Skip to main content
Search

Tianxiao Cao

1 related articles

Subscribe to updates
  1. arXiv

    Shared Low-rank Basis Factorization (SLBF) compresses MoE expert weights data-free and beats pruning, merging, and reconstruction baselines across five MoE architectures from 16B to 122B

    The authors analyze three MoE compression families (expert pruning, expert merging, weight reconstruction) and derive structural error bounds showing that pruning and merging incur non-vanishing errors tied to routing and expert heterogeneity, whereas weight reconstruction avoids these structural costs by preserving expert structure and routing; motivated by this, they propose Shared Low-rank Basis Factorization (SLBF), a data-free weight reconstruction method that replaces full-rank shared bases with rank-r bases shared among experts and applies a bilinear gauge fixing to remove redundant parameters, consistently outperforming methods from all three families across five MoE architectures spanning 16B to 122B parameters.

You’re all caught up