Sparse attention moves long-context cost into the indexer: it takes 47% of a DeepSeek V3.2 prefill layer
Synopsis
In long-context inference, sparse attention lets each query attend to only a few keys, but the indexer that picks those keys still scores all N of them, so the cost moves rather than disappears. In a DeepSeek V3.2 prefill layer, the indexer and Top-k together take 47% of the time. A sweep over a production FP8 kernel yields eight design guidelines, chiefly keeping the product of query heads per group and query rows per step inside one Tensor Core tile and folding Top-k into the indexer budget.
Interpretation
Sparse attention caps the cost of core attention at long context, at the price of an indexer that still touches the whole context, so the dominant long-context cost now sits in the indexer and Top-k. Dense attention scores every query against every key, so per-query cost grows with context length N and at 128K tokens can take most of a layer's prefill time. In the DeepSeek V3.2 prefill layer, the indexer and Top-k together account for 47% of the time.
The indexer is a small attention-like block: query heads score the current query against a cache of indexer keys, and a ReLU plus weighted sum turns those scores into one logit per key; Top-k then keeps the k highest, and core attention runs softmax over only those k keys and values. Indexer and Top-k cost grows with N, while Top-k and the rest of core attention grow with k, which is typically only a few thousand. Measurements use an FP8 indexer kernel with 64K indexer keys, compression ratio 4, head dimension 128, one indexer key head, prefill batch size 1 and decode batch size 128.
Perspective
The scope is sparse attention trained into the model, not drop-in post-training methods such as BLASST, MInference or Quest; measurements default to an FP8 indexer kernel with 64K indexer keys, compression ratio 4, head dimension 128, one indexer key head, prefill batch size 1 and decode batch size 128.
Decode runtime moves only 2% while group size grows 16x, a result that depends on the single key-cache read setting the runtime; pushing group size times query rows per step past about 200 rows makes the kernel launch more thread blocks and runtime steps up. FP16 accumulation is usable only if the indexer's QK scaling keeps the matmul outputs within FP16 range.
