RoShift recenters SiLU activations for sparser LLM decoding: 1.60 points above TEAL at 50% sparsity across five models
RoShift is a training-free activation-sparsification method that recenters SiLU activations near zero and builds rotations that account for both activation statistics and weight influence, yielding representations friendlier to sparsification. It fuses the corrections required by shifting and rotation into GPU kernels; across five models and 11 benchmarks it improves average accuracy over TEAL by 1.60 and 3.29 percentage points at 50% and 60% sparsity, and reaches up to 1.72x decoding speedup on an RTX 5090 at 50% sparsity.