Skip to main content
Search

# MoE expert streaming

1 related articles

Subscribe to updates
  1. Hugging Face Blog

    A 180B-parameter model on a 32GB laptop: streaming MoE experts through Transformers lands about 6% behind Strata

    On a MacBook M5 with 32GB of unified memory, streaming Qwen3.8-Flash-Next's 121GB of expert weights from SSD into a fixed-size device cache through Transformers reaches 13.9 tok/s decode, about 6% behind Strata's 14.8 tok/s, and 228 tok/s prefill against Strata's 220 tok/s. The earlier Transformers plus Accelerate path managed only 0.16 tok/s on Qwen3.6-35B-A3B because it copies a whole layer's expert bank per token.

You’re all caught up