Skip to main content
Search

# Sparse Attention

16 related articles

Subscribe to updates
  1. Qwen

    Alibaba releases Qwen4-Max: a 2.4-trillion-parameter model with a million-token context, scoring 52.69 on Work360 versus 47.74

    Alibaba's Qwen team released Qwen4-Max, built on the new Qwen4 architecture with a 2.4-trillion-parameter backbone, 71B parameters activated per token, native vision and a one-million-token context window. It scores 52.69 on the team's Work360 industry-task benchmark, up from 47.74 for Qwen3.8-Max, uses fewer tokens at matched effort levels on Work360 and Terminal-Bench 4.0, and carries lower API prices. The release also documents pre-training, long-running agent and safety evaluations.

  2. arXiv

    MMFlash reweights visually dependent tokens in diffusion drafting, lifting VLM decoding to 3.00–3.05× on image benchmarks

    MMFlash extends diffusion-style speculative decoding to vision-language models by measuring how much each token depends on the image and upweighting those tokens during training. The measurement shows DFlash drafter acceptance falling from 46% in the lowest visual-dependence quartile to 22% in the highest, a gap that persists after multimodal training. On the Qwen3.5 series, MMFlash reaches average greedy-decoding speedups of 3.00–3.05× across seven image benchmarks, 10.3–10.9% above DFlash, and 2.33–2.71× on long-video benchmarks with sparse attention and light fine-tuning.

  3. arXiv

    VerKV keeps just 3.13% of the KV cache on GPU: it bounds attention error first, then fetches only what it needs

    Long-context inference makes the KV cache grow linearly with sequence length and often exceed GPU memory. VerKV keeps the full cache in CPU memory and only a Poisson sample of tokens, drawn from importance scores, on the GPU; at each decode step a certificate estimates the attention error and extra entries are fetched from CPU only when the bound exceeds tolerance. Across Qwen3-4B, Qwen3.5-9B and Ministral-3-8B on RULER and LOFT at 32K and 128K context, with 3.13% of the cache resident on GPU, the three retain 94.6%, 84.6% and 79.6% of dense quality, where Fast KVzip and KVzip at the same 3.13% retain only 1.9% to 11.9%.

  4. arXiv

    JSSA makes token-level sparse decoding gather-free, reaching up to full-KV throughput at 128K context

    Long-context decoding is bottlenecked by repeatedly reading a growing KV cache, and sparse attention reads only part of it, yet token-level selection has not reliably delivered both accuracy and lower latency. JSSA scores candidate tokens in a shared low-dimensional query–key subspace and pairs it with a gather-free indexed-attention kernel that keeps a distinct selected token set for each KV head. Across GQA, hybrid sliding-window/global attention and MLA models it stays closest to full-KV accuracy, and on a B200 at 128K context its decoding throughput reaches up to that of full-KV attention.

  5. LMSYS Org

    XM-Sparse moves sparse-attention KV cache off HBM: 49.25% lower time per token at 16K context and matched batch

    Netpreme attaches its X-Mem™ memory tier to SGLang HiSparse so decode GPUs fetch missed KV entries from high-bandwidth external memory instead of host DRAM over PCIe. At 16K context, batch 192 per GPU and a 20% top-k miss rate, average time per output token falls from 167.48 ms to 84.99 ms, a 49.25% reduction; at a 40% miss rate the reduction is 63.58%. At 20% misses, peak decode throughput is 2.09–3.04× that of host DRAM across 16K to 1M contexts.

  6. KV selection stops waiting for the target-layer query: EarlySelectKV precomputes indices from proxy queries, lifting long-output throughput by up to 12.4%

    Query-conditioned sparse KV selection must wait for the target-layer query before scoring candidates and generating indices, so routing sits on the immediate pre-attention path. EarlySelectKV replaces the routing query with a proxy built from an earlier hidden state, precomputing the KV indices the next layer will access while final attention still uses the real query. On RocketKV it differs by +0.018 points on LongBench, about −0.01 points on RULER and +0.25 percentage points on representative NIAH/Passkey settings; in a TensorRT-LLM deployment with CPU-offloaded routing, end-to-end output throughput rises by up to 12.4% and GPU-resident memory falls by 11.1% of the combined KV/K-summary pool.

  7. Mistral opens a 1T-parameter multimodal model: 82% on reproducing and patching real vulnerabilities

    Mistral released Mistral Large 4 (Le Chonk) on Hugging Face, a multimodal Mixture-of-Experts model with about 1T total and 49B active parameters and a 1M-token context window, shipped as open weights in FP8 and NVFP4 variants. It pairs a large expert pool with sparse attention and a Pixtral vision encoder, so the language model selects a few experts per token while an attention indexer selects which earlier positions to attend to. On the Artificial Analysis Cyber Index test that asks a model to reproduce a real vulnerability in open-source software and then patch it, the model scores 82%, which Mistral describes as the highest reported score on that test.

  8. vLLM Blog

    vLLM runs Mistral Large 4 on day one: NVFP4 decodes 1.08–1.19x faster than FP8 with about 1.5x the KV cache

    vLLM added same-day inference support for Mistral Large 4, a 1-trillion-parameter sparse mixture-of-experts multimodal model with 49 billion parameters active per token. The work spans breakable CUDA graphs, hand-written fusions, decode and prefill context parallelism, DSpark speculative decoding, NIXL disaggregated serving and host-memory KV offload. On TP8, NVFP4 decodes 1.08–1.19x faster per token than FP8 and holds about 1.5x the KV cache, while staying within 2.1 points of FP8 on multimodal and long-context evaluations.

  9. arXiv

    Long-context RL training stops waiting on autoregressive rollouts: sparse self-drafting plus full-attention verification lifts throughput 1.39x

    Long-context RLVR rollout generation now drafts tokens with sparse attention and verifies them with full attention, raising end-to-end training throughput to 1.39x and 1.35x of autoregressive on Qwen3-1.7B and Qwen3-8B while keeping comparable task performance.

  10. arXiv

    3D reconstruction drops dense attention layer by layer: up to 5.6x end-to-end speedup at 1,024 views

    Zebra3D swaps dense global attention in pretrained 3D reconstruction models for a per-layer mix of linear and tile-sparse attention, holding pose accuracy close to dense models at roughly 94% mean attention sparsity and reaching up to 5.6x end-to-end speedup at 1,024 views.