RoShift recenters SiLU activations for sparser LLM decoding: 1.60 points above TEAL at 50% sparsity across five models
Synopsis
RoShift is a training-free activation-sparsification method that recenters SiLU activations near zero and builds rotations that account for both activation statistics and weight influence, yielding representations friendlier to sparsification. It fuses the corrections required by shifting and rotation into GPU kernels; across five models and 11 benchmarks it improves average accuracy over TEAL by 1.60 and 3.29 percentage points at 50% and 60% sparsity, and reaches up to 1.72x decoding speedup on an RTX 5090 at 50% sparsity.
Interpretation
RoShift reshapes SiLU-model activations into a form friendlier to sparsification, so accuracy loss at high sparsity is clearly smaller than with earlier activation-sparsity methods. TEAL, LaRoSA and WINA select components to zero by magnitude or weight information directly on SiLU models; the higher the sparsity, the more output contribution is discarded and the larger the accuracy drop. Across Llama-3.1-8B, Llama-2-7B, Mistral-7B, Qwen2.5-7B and Qwen3-8B on 11 benchmarks, average accuracy at 50% and 60% non-uniform sparsity exceeds TEAL by 1.60 and 3.29 percentage points, LaRoSA by 3.76 and 8.76 points, and WINA by 2.22 and 5.66 points.
The method has two parts: each channel's SiLU activations are shifted near zero by a calibrated channel median with the restoring correction deferred to the end of the FFN, and a rotation concentrates activation energy and weight energy in the same dimensions. Lazy shift lets both the Up and Down projections exploit the extra sparsity from recentering without restoring the original activations midway; the weight-aware rotation targets sparsification error itself rather than only compressing activation variance. On Llama-3.1-8B, compared with an activation-only PCA rotation, the weight-aware rotation raises the share of output-contribution variance in the leading 50% of dimensions from 90.77% to 94.85% and lowers output NMSE under B128 sparsification from 1.33% to 0.70%.
Shifting and rotation each need a matrix correction; RoShift fuses them into matrices precomputed per RMS representative and runs the sparse FFN and correction path in parallel on separate CUDA streams. Without fusion, correction and inter-layer basis conversion run serially, and the extra overhead cancels the savings from sparsity. On an RTX 5090 with FP16, batch size 1 and 50% uniform sparsity, Llama-3.1-8B end-to-end decoding drops from 9.94 to 6.75 ms/token; separate shift correction adds 0.49–0.76 ms/token, fusion removes 0.41–0.71 ms/token, and parallel execution removes a further 0.19–0.24 ms/token.
Perspective
The demonstrated scope is five SiLU base models, 11 benchmarks, 40%/50%/60% target MAC sparsity, and FP16 batch-size-1 decoding measurements on an RTX 5090 and a Jetson AGX Orin; the accuracy comparison counts Transformer-block sparsity without LM-head sparsity, and the speed and Pareto analyses use uniform sparsity.
The authors note that TEAL's fixed-threshold implementation achieves MAC sparsity 0.83–1.68 percentage points below target on average, which may partly explain its relatively strong accuracy against WINA; the speed comparison with LaRoSA reimplements LaRoSA on RoShift's GPU kernel backend and may differ from LaRoSA's original implementation. Under 4-bit quantization, quantizing weights and transition matrices together degrades severely, so transition matrices must stay at 8 or 16 bits; kernels jointly optimized for sparse and quantized computation are left to future work.
