Skip to main content
Back to timeline
arXivSource publication:

Most of an attention head's self-alignment is background geometry: an earlier-position null explains 48% to 63% across 5,408 heads in nine models

Synopsis

In multi-head attention, the cosine similarity between a head's output and its own value vector is often read as a redundancy signal, but against a null drawn from earlier positions in the same sequence, 48% to 63% of that alignment across nine multi-head pretrained models and 5,408 heads comes from background geometry, with no monotone scale trend. In a pre-registered three-arm, eight-seed paired training factorial at 50.9M parameters, removing the own-value direction lowered validation loss below baseline, a fixed arbitrary rank-one direction did not differ from baseline, and own-direction removal directly outperformed that control.

AI-generated editorial illustration: Sanity Checks for Attention Surgery: Null Models and Matched Interventions for Representation-Motivated Design

Interpretation

A high cosine similarity between a head's output and its own value vector cannot be read directly as redundancy of that direction; much of it comes from the background geometry of a causal model, where a position can attend to earlier positions. The common sequence measures a large statistic on internal representations, removes that direction architecturally, and reports a training gain, without first asking whether the statistic exceeds its noise floor or a geometric null. Across nine multi-head pretrained models, 5,408 heads and 124M to 6.9B parameters, a null value drawn uniformly from earlier positions in the same sequence accounts for 48% to 63% of the head-value cosine alignment, with no monotone scale trend.

Whether a training gain requires the removed direction has to be settled by a matched control with the same parameterisation and initialisation, not by whether loss falls after removal. A fixed arbitrary rank-one direction and the own-value direction share the same gate parameterisation, each is compared against baseline, and the two are compared directly across eight random seeds. In a pre-registered three-arm, eight-seed paired factorial at 50.9M parameters, the fixed arbitrary direction does not differ from baseline, own-direction removal lowers validation loss (significant after Holm correction), and it directly outperforms the fixed arbitrary control.

Under grouped-query attention the alignment depends on which group's value is used: it is positive against the head's own key-value group and reverses sign against a neighbouring group. Multi-head models have no groups sharing a value projection, so this contrast is undefined for them and can only be measured on grouped-query models. On Qwen2.5-0.5B the own-group excess is +0.24 and the neighbouring-group excess is negative, and Qwen2.5-1.5B and TinyLlama-1.1B show the same signs.

Perspective

The demonstrated scope is this: geometry measurements on frozen models cover twelve pretrained models, including nine multi-head and three grouped-query models; the resolvability analysis covers GPT-2, Pythia-160M and Pythia-410M, extended to Pythia-1.4B and Pythia-2.8B under a half-budget protocol; and the training conclusion comes from one 50.9M-parameter, eight-layer, eight-head model trained from scratch. A reader hoping to judge whether a direction can be safely removed from an already trained large model will not find that case covered here.

The training experiment does not determine the sign or magnitude of the effect at larger scale, and the frozen and trained settings differ in gate constraints and first-token handling, so the two are related but not identical interventions. The fixed arbitrary-direction control and own-direction removal differ in projected norm and converged gate strength, and the control direction is fixed across positions while own-direction removal takes out an input-dependent direction at every token, so the contrast confounds own-value specificity with per-token versus static projection. The GPT-2 geometry measurement did not reproduce the project's reference triple within its stated tolerance; a length sweep accounts for most of the gap, but the residual in the self-cosine component has not been traced to corpus, truncation or head-level aggregation.