A 30B-parameter MoE model fits a 16 GB GPU: CECAR folds expert-cache state into routing for 12.5 tokens/s
Synopsis
CECAR folds expert cache residency and predicted eviction priority into MoE routing so that fewer non-resident experts are requested. Prior cache-aware routing relied only on whether an expert was cached, which could destabilize routing; on Qwen3-30B-A3B, with 16 GB of VRAM, about 1 GB of host DRAM and non-resident experts fetched from SSD, CECAR reaches 12.5 tokens/s, a 4.5x speedup over conventional MoE inference and faster than a baseline keeping all experts resident in system DRAM.
Interpretation
CECAR lets memory-constrained devices run a 30B-scale MoE model at usable speed, fetching non-resident experts from SSD instead of requiring all expert weights to stay in memory. Conventional MoE inference follows router logits strictly, so cache-missing experts must be loaded from slower storage and expert loading dominates per-token latency; prior cache-aware routing used only a binary cached-or-not signal. On Qwen3-30B-A3B with 16 GB of VRAM, about 1 GB of host DRAM and a cache of 24 experts per layer, CECAR reaches 12.5 tokens/s, a 4.5x speedup over conventional MoE inference and above the 7 tokens/s baseline that keeps all roughly 56.7 GB of expert parameters resident in system DRAM.
Routing considers cache residency and expert reuse tendency together, biasing selection toward experts already in VRAM while preserving the original router's semantic ranking. Prior pruning methods drop experts by routing weight, and the experts they keep may still be non-resident and require SSD reads; CECAR instead skips non-resident low-priority experts during expert selection. On Qwen3-30B-A3B, DES and ODP activate fewer experts on average (4.67 and 4.71) than CECAR (7.13) yet decode at only 4.7 to 5.0 tokens/s with hit rates around 70% to 73%, against CECAR's 96.6%.
A lightweight learned cache policy predicts an eviction priority for each expert, and its score enters routing as a bonus so cache replacement and expert selection are optimized together. Heuristics such as LRU and LFU fail to capture nonlinear expert reuse patterns across tokens; FlashMoE's learned policy falls below LRU on out-of-distribution workloads and small caches. On TriviaQA at cache size 24, CECAR's learned policy reaches a 73.63% hit rate versus LRU's 69.49%; on HumanEval, MBPP, GPQA-Diamond and MATH-500 it reaches 68.37%, 69.31%, 73.73% and 69.63%, all above LRU's 65.50%, 65.26%, 70.86% and 66.04%.
Perspective
Results come from three representative MoE models and four standard benchmarks, on a memory-constrained desktop GPU with SSD offloading and host DRAM pinned to about 1 GB rather than a phone- or tablet-class SoC; accuracy and throughput are measured on two different paths.
Performance may vary under substantially different model architectures, workloads or hardware platforms; whether the gain holds or grows on lower-bandwidth mobile storage is not verified; multi-seed variance for hit rates, sensitivity to thresholds and a training-data-scale ablation remain to be reported.
