A 180B-parameter model on a 32GB laptop: streaming MoE experts through Transformers lands about 6% behind Strata
Synopsis
On a MacBook M5 with 32GB of unified memory, streaming Qwen3.8-Flash-Next's 121GB of expert weights from SSD into a fixed-size device cache through Transformers reaches 13.9 tok/s decode, about 6% behind Strata's 14.8 tok/s, and 228 tok/s prefill against Strata's 220 tok/s. The earlier Transformers plus Accelerate path managed only 0.16 tok/s on Qwen3.6-35B-A3B because it copies a whole layer's expert bank per token.
Interpretation
A MoE model need not fit in VRAM or unified memory at all: streaming only the router-selected experts from disk into a shared cache is enough to run a 180B-class model on a 32GB laptop. Transformers offloading through Accelerate treats a MoE layer as one module and copies its whole expert bank to the GPU before each forward pass, reaching only 0.16 tok/s decode on Qwen3.6-35B-A3B on a single A10G. On the same M5 with the same prompts, the final version decodes at 13.9 tok/s versus Strata's 14.8, prefills at 228 tok/s versus 220, and decodes at 12.4 tok/s versus 13.7 after a 6K document.
The central trick is turning the cache into a slot pool: one preallocated device tensor shared by all layers, plus a per-layer table mapping expert ids to slots. The first version ran a Python loop over one expert at a time and reached only 10.1 tok/s even with every expert cached, against about 32 tok/s for vanilla Transformers, so the loop rather than disk I/O was the bottleneck. On OLMoE-1B-7B, which fits in memory, the slot-pool version reached 34.6 tok/s against 38.1 tok/s for vanilla Transformers.
Once router expert ids are remapped to slots, the pool looks like an ordinary expert bank, so grouped_mm runs on it unchanged and no custom expert matmul is needed. The implementation enters as an experts_offload option of from_pretrained in huggingface/transformers#50717; experts mapped to disk skip tensor loading and switch to the streaming path. Misses are read by 8 I/O threads, one contiguous read per expert, with the next two layers' routers run early to prefetch; experts can be quantized offline to 3 bits, 49GB on disk with 12GiB cached on the device.
Perspective
The demonstrated setting is one MacBook M5 with 32GB of unified memory running Qwen3.8-Flash-Next with experts quantized to 3 bits, compared against Strata 0.1.60, both greedy with thinking, MTP and speculative drafts off and 1,024-token prefill chunks. After loading it uses about 18GiB, and a 6K prompt peaks at 25.4GiB, close to the limit, so the browser has to be closed to keep macOS from swapping.
The author states that about 14 tok/s is still too slow for how he uses coding agents, and that most of the remaining gap is roughly 10ms of host round trips per token, because PyTorch does not expose the native per-layer sync this would need; a spin-kernel workaround saw the GPU react anywhere from a second to never, so it was not built. The API is still under discussion with Transformers maintainers, and quantized experts will probably move to the per-layer quantization plan of #50245.
