Skip to main content
Back to timeline
arXivSource publication:

NVFP4-MLA pushes both MLA KV-cache storage and attention math to FP4, matching BF16 accuracy while shrinking the cache 3.20-fold

Synopsis

Under long-context and agentic workloads, MLA's KV-cache capacity and attention computation dominate serving cost. NVFP4-MLA pushes both MLA KV-cache storage and both attention matrix multiplications to FP4, using selective residual compensation that adds an NVFP4 residual only to quantization-sensitive channels and two-dimensional quantization blocks that let the shared latent cache serve both reduction directions. On nine benchmarks for DeepSeek-R1 and Kimi-K3, average accuracy matches BF16 MLA while the cache shrinks 3.20 and 3.05 times relative to BF16.

Source-provided article image: NVFP4-MLA: Unified FP4 KV-Cache Storage and Attention Computation for Multi-Head Latent Attention
arXiv

Interpretation

MLA's KV cache and attention computation can now be stored and computed in FP4 together, rather than shrinking the cache and dequantizing it back to higher precision for the matrix multiplications. Earlier KV-cache quantization methods target separate key and value caches and still run the attention matrix multiplications at higher precision, so cached values need an extra dequantization step; in MLA all heads share one latent cache, and the two matrix multiplications reduce that same data along different directions. On nine benchmarks for DeepSeek-R1 and Kimi-K3, NVFP4-MLA averages 59.82 versus 59.80 and 80.28 versus 80.27, matching BF16 MLA, while the version quantized directly to NVFP4 without residual compensation drops by 5.71 and 16.48 percentage points.

Accuracy is preserved by adding one NVFP4 residual only to quantization-sensitive channels, and a two-dimensional block layout lets the cache serve both matrix multiplications. Outliers in the MLA cache concentrate in the RoPE channels, which hold 64 of 576 channels; a conventional one-dimensional block layout grouped along channels matches the channel reduction of the first matrix multiplication but not the token reduction of the second, and converting it requires dequantizing, transposing and requantizing. Compensating only the RoPE channels lowers mean whole-cache normalized mean squared error by 60.3% on DeepSeek-R1 and 66.1% on Kimi-K3; the NVFP4 residual reduces mean RoPE error by factors of 115.4 and 116.1 relative to plain NVFP4, whereas FP8 RoPE leaves 9.5 and 9.1 times as much error on average.

With both matrix multiplications running on native FP4 Tensor Cores, the attention kernel pulls further ahead of FP8 MLA as KV length grows. Multi-token prediction reuses each cache read across several query positions, raising FP8 MLA decoding to 1,934 FLOPs per cache byte, beyond the 911 FLOP/B ridge point of a Rubin GPU, so decoding becomes compute-bound. On NVIDIA Rubin GPUs, across 39 measured shapes NVFP4-MLA is faster than FP8 MLA at every shape, by 1.31 to 2.82 times with a geometric mean of 1.87 on DeepSeek-R1 and 1.35 to 2.71 times with a geometric mean of 1.99 on Kimi-K3, with speedup growing from 1.3 to 1.5 times at 2K KV length to 2.2 to 2.8 times at 128K.

Perspective

The demonstrated scope is two frontier open MLA models, DeepSeek-R1 and Kimi-K3, nine benchmarks spanning reasoning, coding, instruction following, long context and agentic tool use, and kernel and serving measurements on NVIDIA Rubin GPUs; Kimi-K3 uses NVFP4 in 22 of its 24 MLA layers with FP8 in the final two, and the cache count excludes the recurrent state of its 69 KDA layers.

Direct FP4 quantization loses accuracy on outliers in the RoPE channels, which is why residual compensation is added; Kimi-K3's MLA cache shares the KV budget with the recurrent state of its KDA layers, and under an equal budget the extra cached tokens range from 47% to 49% depending on the KDA state share; kernel speedup at 2K length is limited by fixed per-call cost to 1.3 to 1.5 times.