Skip to main content
Back to timeline
arXivSource publication:

Pruned MoE experts don't need retraining: GEAR² reuses already-activated experts per token and beats mean substitution on the nine-task zero-shot average at 50% pruning

Synopsis

Pruning experts from a mixture-of-experts language model saves GPU memory but discards what the removed experts contributed to the tokens routed to them. GEAR² leaves the main model untrained: it approximates each pruned expert's output by scaling an expert already activated for the same token, falls back to the mean output when a reliability gate judges that proxy unreliable, and adds a low-rank residual weighted by the missing routing mass. Across five MoE models from 16B to 106B parameters, 25% and 50% pruning, and nine zero-shot tasks, it gives the best pruned result in all 10 settings, and at 50% pruning it beats same-mask mean substitution by 0.96 to 3.44 points.

Interpretation

A pruned expert need not be treated as vanished parameters; its contribution to a token can be reconstructed from the retained experts already activated for that same token. Earlier training-free substitutes either gave each pruned expert a fixed mean output or an affine transform of a fixed proxy expert, or merged functionally similar experts into one; the former does not adapt to the experts activated for the current token, and the latter rewrites the retained experts' own functions. On five MoE models, two pruning ratios of 25% and 50%, and nine zero-shot tasks, GEAR² attains the best pruned average in all 10 settings, 9 of them by itself; at 50% pruning it exceeds the same-mask mean substitute MoNE by 0.96 to 3.44 points, and the substitution method MoRA and the merging method HC-SMoE by 2.75 to 6.41 and 3.31 to 25.34 points respectively.

Compensation runs in two steps: reuse the expert outputs already computed, then add an ultra-light residual for what reuse cannot express. The reuse step, GEAR, estimates a pruned expert's output as a directed scaling of an activated expert's output, with the scale fitted by weighted least squares over the calibration tokens on which both are activated; a reliability gate falls back to that expert's mean output when the proxy error exceeds a threshold. The residual, GEAR², is weighted by the missing mass, the total routing weight the original router assigns to pruned experts; each layer adds one pair of low-rank matrices, distilled for 64 steps on 64 sequences of 512 tokens from C4, and at rank 32 adds about 0.02% or less of each model's parameters.

Perspective

The evaluation covers nine zero-shot understanding and reasoning tasks on five open-source MoE models (Moonlight-16B-A3B, Qwen3-30B-A3B-Instruct-2507, DeepSeek-V2-Lite, Gemma-4-26B-A4B and GLM-4.5-Air), with 25% or 50% of routed experts pruned per layer and calibration and distillation data drawn from C4. The authors list generative tasks, long-context tasks and larger models as future work, and name calibration and distillation data matched to the target domain as a natural extension.

The authors state that the reference implementation is not yet fused into the MoE layer's output aggregation, so the reported inference latency comes from the unfused version; generative and long-context tasks and larger models are not yet evaluated. The gate threshold is chosen per model and pruning ratio from a small candidate set by next-token negative log-likelihood on a development set, and the search involves only forward passes.