Skip to main content
Back to timeline
arXivSource publication:

LoopVQ squeezes looped-transformer KV caches to 0.5 bit: Ouro-1.4B holds 75.82 on GSM8K where CQ drops to 46.78

Synopsis

Looped transformers reuse the same layers and save parameters, but each loop step still stores its own keys and values, so the KV cache grows linearly with loop steps and becomes the memory bottleneck. LoopVQ packs channels from several consecutive loop steps into one sub-vector that shares a single codebook index, then fine-tunes the codebooks to match the full-precision output. On Ouro-1.4B at 0.5 bits per element it scores 75.82 on GSM8K, against 46.78 for CQ and 57.32 for TaSQ, two vector-quantization methods built for standard LLMs.

Source-provided article image: LoopVQ: Sub 1-bit KV Cache Quantization for Looped Transformers
arXiv

Interpretation

A looped transformer's KV cache can be compressed along the loop axis: channels from several loop steps share one codebook index while each element still costs 0.5 or even 0.25 bits. Earlier KV cache vector quantization grouped channels and learned codebooks per loop step, treating the loop steps as unrelated caches, and lost much of its accuracy below one bit per element. On Ouro-1.4B general benchmarks at 0.5 bit, LoopVQ scores 75.82 on GSM8K, 69.40 on MATH500 and 63.41 on HumanEval, versus 46.78, 17.80 and 14.63 for CQ and 57.32, 38.80 and 24.39 for TaSQ; BF16 scores 78.77, 77.20 and 73.17.

The method concatenates the same channels across several consecutive loop steps into one coupled vector, encodes it with a single lookup, and trains the codebooks to reproduce the full-precision model's output. A coupled vector spans coordinates from multiple loop steps, so one index serves all of them at unchanged bitwidth; codebooks are initialized by Fisher-weighted k-means and then refined by minimizing the KL divergence between quantized and full-precision outputs with weights frozen. On Ouro-1.4B, codebook fine-tuning roughly halves the KL divergence from the full-precision output; an ablation shows coupling supplies most of the gain, fine-tuning adds accuracy on HumanEval, and the two together are best or tied-best in every setting.

Perspective

The demonstrated scope is three looped models, Ouro-1.4B, Ouro-2.6B and Huginn-3.5B; accuracy is measured on general, long chain-of-thought reasoning and long-context retrieval benchmarks, and efficiency on a single RTX 6000 Ada GPU, with A100 results in the appendix. The efficiency comparison is against BF16 serving in SGLang only and does not include the baseline quantization methods.

The efficiency evaluation compares only against BF16 serving, not the baseline quantization methods, because each method's serving efficiency depends on kernels optimized for its own cache format. Accuracy is also not compared with FlashLoop and ResidualQuant, scalar KV cache quantization methods for looped transformers, which are designed for 2 to 4 bits and specify no lower-bitwidth configuration. Accuracy keeps improving as the calibration set grows from 8 to 32 sequences, so results still depend on calibration size.