Skip to main content
Back to timeline
arXivSource publication:

QK-MAP drops query-key normalization: lower training and validation loss than QK-Norm from 190M to 7B parameters

Synopsis

Modern large language models commonly use QK-Norm to keep attention logits from growing, but normalization also erases the token-specific information carried by query and key magnitudes. QK-MAP instead encodes input magnitude as fixed-norm phase patterns using multiple learnable frequencies, so magnitude changes show up in the direction and phase of the representation rather than in vector length. In pretraining comparisons from 190M to 7B parameters with shared backbones and data budgets, QK-MAP had lower training and validation cross-entropy than both standard attention and QK-Norm at every evaluated scale, and the best eight-task average accuracy at 7B.

Source-provided article image: Large Language Models without QK-Normalization
arXiv

Interpretation

QK-MAP keeps query and key magnitude information from being erased by normalization while still holding attention logits in a controlled range, letting it replace QK-Norm in language-model pretraining. Standard attention preserves magnitude information but query and key norms grow during training, while QK-Norm controls that growth yet makes different magnitudes along the same direction nearly indistinguishable. At 190M, 370M, 760M, 1B and 7B parameters, variants share the backbone, pretraining data, token budget, batch size and seeds; QK-MAP has the lowest training and validation cross-entropy at every scale, with the validation gain holding across C4, Pile, M2D2-S2ORC and the Dolma domains.

In implementation, QK-MAP maps the representation entering the attention block into multi-frequency sine and cosine features, then projects queries and keys separately from those features. The mapping borrows RoPE's idea of writing position as a bounded combination of phases at several frequencies, but the frequencies are learned and act on the token representation rather than a position index. At the 7B checkpoint, QK-MAP has the lowest first-key attention in all 55 domain-length comparisons across eleven domains and context lengths from 32 to 512 tokens, with a mean 9.1% to 12.7% below standard attention; in synthetic long-context tests its evidence-to-distractor attention SNR gain over QK-Norm grows from 24.9% at 32 tokens to 85.3% at 600 tokens.

Perspective

The conclusions rest on controlled comparisons from 190M to 7B parameters with sequence length 2048, a shared Dolma-based OLMo pretraining mixture and equal token budgets; downstream numbers are the unweighted mean accuracy over eight zero-shot tasks, the norm-shuffling and word-replacement analyses use 190M checkpoints, and the attention diagnostics use 7B checkpoints.

Global injectivity of the multi-frequency map holds only when the frequency ratio is irrational, and learning changes the frequencies, so this is a sufficient condition rather than an unconditional guarantee; the word-replacement experiment scores fixed blocks, unlike QUEST's sliding-window protocol, so its numbers are a matched comparison among these checkpoints rather than a reproduction of the original paper's absolute perplexities.