A 180B-parameter model on a 32GB laptop: streaming MoE experts through Transformers lands about 6% behind Strata
On a MacBook M5 with 32GB of unified memory, streaming Qwen3.8-Flash-Next's 121GB of expert weights from SSD into a fixed-size device cache through Transformers reaches 13.9 tok/s decode, about 6% behind Strata's 14.8 tok/s, and 228 tok/s prefill against Strata's 220 tok/s. The earlier Transformers plus Accelerate path managed only 0.16 tok/s on Qwen3.6-35B-A3B because it copies a whole layer's expert bank per token.