2-bit KV caches get more than a rotation: RoLaQ calibrates quantization range too, topping six benchmarks on three MLA models
Synopsis
Multi-head latent attention (MLA) compresses each token's key-value cache into one shared latent, and at two bits an error in that latent perturbs both attention matching and value readout. RoLaQ refines a shared rotation within subspaces defined by the frozen key and value projections, and jointly learns quantization thresholds whose reconstruction gains are coupled to control the shared signal's response. Across DeepSeek-V2-Lite, Moonlight-16B-A3B and MiniCPM3-4B on six benchmarks, it reaches the highest average accuracy among the compared 2-bit methods both without and with token protection; without protection it beats the strongest baseline by 1.1 to 3.8 points.
Interpretation
Calibration for 2-bit MLA caches moves from tuning a rotation alone to tuning rotation and quantization range together, and three MLA models all score higher six-task averages than the compared 2-bit methods. Earlier rotation-based methods mainly target multi-head or grouped-query attention with separate key and value caches; OSCAR supplies covariance-aware rotations for the shared cache, but its quantization range is not learned alongside the rotation. Without token protection, RoLaQ's six-task averages are 46.7, 60.1 and 59.4 on DeepSeek-V2-Lite, Moonlight-16B-A3B and MiniCPM3-4B, 3.8, 2.8 and 1.1 points above the strongest baseline; with 4 sink and 16 recent tokens kept in BF16 it still has the highest average on all three models.
The method partitions the rotation search space along directions given by the frozen key and value projections, and lets each layer's quantization threshold set its reconstruction gain so the quantized shared signal keeps unit linear response. A rotation changes the input distribution the quantizer sees, so a fixed threshold no longer fits; coupling gain to threshold anchors the component of the shared signal aligned with the input while the decision boundary adapts. Calibration uses 1.43M Pile tokens with model weights frozen, taking about 22 minutes for the 16B models and 34 minutes for MiniCPM3 on one A100; in the ablation, learning rotation and range together lifts DeepSeek-V2-Lite MBPP from 39.4 at initialization to 44.0, versus 41.6 for rotation only.
Perspective
The demonstrated scope is 2-bit content caches on three MLA models, DeepSeek-V2-Lite, Moonlight-16B-A3B and MiniCPM3-4B, with RoPE keys kept in BF16; accuracy is evaluated with simulated quantization in Hugging Face Transformers, and serving is measured on DeepSeek-V2-Lite on one A100 80GB.
Positional keys and group metadata remain a substantial share of the remaining cache, so further savings call for coordinated content and positional precision; serving benefits of low-bit caches depend on hardware and workload, and at shorter contexts quantization and unpacking overheads can offset the bandwidth savings.
