Skip to main content
Back to timeline
arXivSource publication:

Sparse attention's real bottleneck is finding the keys: LSA clusters them in a learned latent space, and one attention layer prefills 10M tokens 22.7x faster

Synopsis

At ultra-long context, most of a sparse attention layer's prefill time goes to the indexer scoring every key, not to the attention itself. Latent Sparse Attention (LSA) clusters keys hierarchically in a learned latent space so each query scores cluster centroids and then expands the keys of the selected clusters. On GLM-5.3-Flash at 10M tokens, the distilled variant D-LSA prefills one attention layer 22.7x and decodes 4.6x faster than that model's original sparse attention, staying within 4 points on RULER; the natively trained N-LSA matches full-attention validation loss at 128k and trains 1.6x faster per layer.

Source-provided article image: Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention
arXiv

Interpretation

At ultra-long context the cost of sparse attention shifts from the attention computation to key selection: at 10M tokens, 98% of a sparse attention layer's prefill time is spent in the indexer, which still scores every preceding key or block of keys for each query. Earlier improvements, such as compressing every four keys into one before scoring, scoring blocks first and then keys inside selected blocks, or reusing one layer's selection in the next, each divide the indexer cost by a constant, so work per query still grows linearly with context. The comparison comes from breaking down prefill time of a sparse attention layer: the indexer takes 98% at 10M tokens, and a free indexer would make sparse attention faster still in prefill.

LSA groups keys by content, has each query score cluster centroids, and then expands the keys of the selected clusters, so no per-key scan is needed. Clustering happens in a learned latent space: the distilled D-LSA learns a separate pair of reduced-dimension key and query projections per layer, while native N-LSA clusters the attention's own keys. On GLM-5.3-Flash, single-layer prefill speedup rises from 3.5x at 1M tokens to 22.7x at 10M tokens over its original sparse attention, and is 2.1x at 512k tokens; on Qwen-3.8-27B it reaches 72.4x over full attention at 10M tokens.

On accuracy, distilled D-LSA stays close to each model's baseline on knowledge, reasoning and long-context tasks and remains usable at 10M tokens. On Qwen-3.8-27B, D-LSA (Flat) stays within 5 points of full attention up to 1M tokens and reaches 63.9 against 59.9 on RULER and 29.1 against 19.8 on PaperTrail at 10M tokens. On GLM-5.3-Flash, D-LSA (Flat) stays within 3 points of the original sparse attention up to 1M tokens, trails by up to 8 points from 2M to 5M tokens, and is within 4 points on RULER at 10M tokens.

Natively trained N-LSA uses only the next-token prediction loss, adds no parameters and needs no teacher, and matches full-attention validation loss at 128k context. In the same architecture and data, PISA, NSA and sliding-window attention reach validation losses of 1.100, 1.117 and 1.154, while N-LSA and full attention both reach 1.085. In a hybrid model of about 0.5B parameters, N-LSA scores 32.2 on RULER at 128k against 34.7 for full attention, while NSA and PISA fall to 19.2 and 14.6; one layer trains 1.6x faster than full attention at 128k.

Perspective

The demonstrated scope is accuracy on Qwen-3.8-27B and GLM-5.3-Flash extended to 10M tokens on RULER and the newly built PaperTrail, plus single-layer and SGLang end-to-end timings on H200 and B300; native training is validated only on a hybrid model of about 0.5B parameters at 128k context with flat search.

On GLM-5.3-Flash, tree search loses accuracy faster than flat search past 1M tokens, and with a single request it overtakes flat search in decode only from about 5M tokens; whether sparse training would close that gap remains open. Scaling native training to larger models, longer contexts and tree search is also left open.