Kwai's KSA V2 reuses summary tokens twice: 40.4% smaller resident cache at 128K, with prefill throughput above the original hybrid
Synopsis
Kwai retrofitted pretrained hybrid KSA models into KSA V2, letting each layer's existing summary tokens serve both as re-compressible distant memory and as a block-level sparse index. On the 1.9B (MHA) backbone at 128K, resident cache falls from 10.59 GiB to 6.31 GiB, a 40.4% reduction, and prefill reaches 44,971 tokens/s versus 29,837 for the original hybrid KSA. The six-length RULER mean is 72.39% against 72.61%, and the 128K score is 65.01% against 65.35%.
Interpretation
Storage and compute costs of long-context inference drop together: on the 1.9B backbone at 128K, KSA V2 cuts resident cache from 10.59 GiB for the original hybrid KSA to 6.31 GiB, and by 73.7% relative to the 24.00 GiB of a full-attention model. In the earlier hybrid KSA, every layer had to use the same summary compression chunk, so per-layer KV cache compression was inflexible; full-attention layers carried the added summary tokens yet still paid quadratic prefill without using them. Resident cache, prefill and decode throughput were measured from 16K to 128K on a single NVIDIA H800 with BF16 and batch size one; at 128K prefill reaches 44,971 tokens/s versus 29,837 for hybrid KSA and 20,830 for full attention, and decode is 1.23–1.32 times faster than hybrid KSA.
The mechanism gives summary tokens two roles without adding tokens or changing the summary layout, so a pretrained hybrid KSA model can be retrofitted directly. Compressed KSA (C-KSA) pools consecutive summary KV entries in each layer into layer-local Meta entries, raising distant-history compression from 8 to 1024 while the summary sequence passed between layers keeps its granularity; Sparse KSA (S-KSA) builds a routing key per text block from the summaries, and a Lightning Indexer selects a few original-text blocks whose original KV the main attention reads. The 1.9B model goes through module-only then full-parameter adaptation, and the S-KSA stage trains only the new aggregation modules and Indexer; the same retrofit on a 4B GQA backbone gives a six-length RULER mean of 84.15% versus 84.45% for hybrid KSA, and 74.41% versus 71.67% at 128K.
Perspective
Evaluation covers 1.9B MHA and 4B GQA backbones with contexts up to 128K; general-capability evaluation and efficiency measurements are limited to 1.9B, and the 4B validation covers RULER retrieval only. Adaptation reuses KSA's training corpus and data mixture at a fixed 128K context length.
At 128K the remaining cache is mostly original-text KV held by the sparse layers; block selection degrades on aggregation-heavy tasks, with the frequent-word-extraction subtask dropping from 45.00 to 26.67. The 4B Indexer needs RMSNorm on projected queries and keys to converge, and whether that change also reduces 1.9B adaptation data remains untested. Longer contexts, larger models, and how to allocate compression factors across layers are left open.
