AscMLA lets FP4 caches feed MLA decoding directly: one shared payload for K and V, 1.35x faster than TRT-LLM's FP8 kernel
Synopsis
Long-context decoding repeatedly rereads a growing KV cache, and FP4 can shrink that storage, but block scaling clashes with MLA's shared latent: existing routes either dequantize first or encode K and V separately. AscMLA keeps one FP4 payload for both K and V and gives them different scales, one per 16-channel block for K and one per token for V, folding the V scale into the softmax probability. Across forty configurations it reaches geometric-mean decode speedups of 1.35 over TRT-LLM's FP8 kernel and 1.26 over CAKE's NVFP4 MLA kernel, with 38.2% less KV cache memory than FP8.
Interpretation
An MLA decode cache can hold a single FP4 payload that K and V read with scales of different granularity, without dequantizing V separately or storing a second encoding. Earlier FP4 MLA paths either dequantized the latent to FP8 before attention or, as in the MLA adaptation of SageAttention3, encoded K and V separately and duplicated the cache. Across forty workload configurations, AscMLA reaches a 1.35 geometric-mean speedup over TRT-LLM's FP8 MLA kernel and 1.26 over CAKE's NVFP4 MLA kernel, while using 38.2% less KV cache memory than FP8.
The design splits the two scaling needs: V keeps one scale per token and K keeps one scale per 16 channels, both pointing at the same FP4 codes. Block scaling applies scales along the reduction dimension, but MLA's QK reduces over latent channels while PV reduces over tokens, so the same channel-block scales land on an output dimension in PV. The cache first quantizes shared codes under the V constraint, then fits K scales to those codes by least squares; QK runs native NVFP4 matrix multiply-accumulate, and PV uses MXFP8–MXFP4 matrix multiply-accumulate to multiply scaled probabilities by the same FP4 codes.
Scales are delivered according to how long each must stay live, so the asynchronous pipeline does not stall waiting for them. K scales must remain valid until their matrix multiply-accumulate completes, whereas V scales are consumed during probability conversion, so the two cannot share one buffering policy. K scales use four disjoint slots for multibuffering, and removing it drops the speedup to 0.69 at 16K context; V scales use a four-stage shared-memory prefetch queue, and removing it leaves speedups between 0.87 and 0.89.
Perspective
The evaluation covers forty kernel configurations on a single GPU, ninety attention-output error cases, and downstream accuracy of Kimi-K2.6 and Kimi-K3 on GPQA-Diamond; the end-to-end serving experiment uses AscMLA as the decode backend in SGLang 0.5.21 with ModelOpt-NVFP4 Kimi-K2.6 on four prefill and four decode GPUs, with TRT-MLA on the prefill side.
Mean attention-output error is 13.37%, above Sage-MLA's 12.66%, with mean cosine similarities of 0.9908 and 0.9920; on sparse-outlier inputs AscMLA's error is higher than Sage-MLA's because a few large entries set the amax scale for an entire token. Kimi-K2.6 and Kimi-K3 score 89.4% and 92.8% on GPQA-Diamond, against officially reported 90.5% and 93.5%.
