Skip to main content
Search

# On-Device MoE Inference

1 related articles

Subscribe to updates
  1. arXiv

    A 30B-parameter MoE model fits a 16 GB GPU: CECAR folds expert-cache state into routing for 12.5 tokens/s

    CECAR folds expert cache residency and predicted eviction priority into MoE routing so that fewer non-resident experts are requested. Prior cache-aware routing relied only on whether an expert was cached, which could destabilize routing; on Qwen3-30B-A3B, with 16 GB of VRAM, about 1 GB of host DRAM and non-resident experts fetched from SSD, CECAR reaches 12.5 tokens/s, a 4.5x speedup over conventional MoE inference and faster than a baseline keeping all experts resident in system DRAM.

You’re all caught up