Skip to main content
Back to timeline
arXivSource publication:

KVPath links each token's key-value states across layers into one path, cutting GQA cache by up to 68% without raising perplexity

Synopsis

KVPath stops caching a full key-value representation at every Transformer layer and instead links a token's key-value states across depth into one path: the first layer stores a full-dimensional base key-value, and each later layer generates only a low-dimensional delta that updates the preceding state. On parameter-matched 12-layer models, the residual and rolling variants cut KV-cache storage by up to 68.65% under GQA and 54.90% under MLA, with average perplexity equal to or lower than standard GQA and MLA.

Source-provided article image: KVPath: Connecting Key-Value Representations Across Transformer Layers
arXiv

Interpretation

In long-context inference the key-value cache grows linearly with sequence length and network depth and is a major memory bottleneck; KVPath has each layer store only the first layer's base key-value plus a low-dimensional delta per later layer instead of a full independent key-value. Earlier cross-layer approaches either let several layers share one key-value or merge or jointly factorize neighboring layers' key-values into a shared low-rank subspace, still treating each layer's representation as an independent static copy or approximation. On 12-layer GQA models with 98,710,016 parameters each, the residual variant ResGQA and the rolling variant RollGQA cut the cache to 31.35% of GQA at 32K context, with average perplexity 3.48% and 0.52% lower than GQA.

Each layer's key-value state is the preceding state plus a low-dimensional delta projected from the current hidden state, so the cache keeps one full-dimensional base key-value and a string of compact deltas. The residual variant up-projects the low-dimensional delta back to full width and adds it to the preceding state; the rolling variant splits the state into narrow slots, drops the oldest slot each layer and appends a new one, so a full-dimensional state is always retained. On MLA, ResMLA and RollMLA cut the cache by 53.44%, 54.61% and 54.90% at 2K, 8K and 32K context, with average perplexity reductions of 1.69%, 1.38% and 1.20% versus 0.96%, 0.98% and 0.82%, and both beat the cross-layer latent-sharing baseline CLLA under the same parameter budget.

The path structure still holds at larger scale: the 0.6B-parameter RollGQA has lower perplexity than parameter-matched GQA on all four benchmarks at all three context lengths. The small-model gains had been shown only at roughly 100M parameters, and at 0.6B only the rolling variant was tested. On 606,405,632-parameter 32-layer models, RollGQA lowers average perplexity by 1.83%, 2.32% and 2.45% at 2K, 8K and 32K, uses 72.27% less cache than GQA at 32K, scores higher on six of eight zero-shot downstream tasks, and lowers LAMBADA target-word perplexity from 28.97 to 25.79, while GQA does better on PIQA and WinoGrande.

Perspective

Experiments cover 12-layer GQA and MLA models of about 100M parameters and one 0.6B-parameter 32-layer GQA model, trained on FineWeb-Edu and evaluated by GPT-2 token-level perplexity on C4, FineWeb-Edu, Wikipedia and PG-19 plus eight zero-shot tasks. The reported figures are compact cache requirements, not throughput or end-to-end serving performance.

The authors state that compute limits kept experiments to about 0.6B parameters, so whether the quality and cache benefits persist at larger scales remains to be evaluated; no specialized attention kernels for KVPath have been developed, so cache requirements are reported instead of throughput. In the ablation, raising the delta dimension from 8 to 32 moves cache share from 20.48% to 54.85%, but perplexity does not vary monotonically with that dimension.