Skip to main content
Search

Basavaraj R M

1 related articles

Subscribe to updates
  1. arXiv

    Most of an attention head's self-alignment is background geometry: an earlier-position null explains 48% to 63% across 5,408 heads in nine models

    In multi-head attention, the cosine similarity between a head's output and its own value vector is often read as a redundancy signal, but against a null drawn from earlier positions in the same sequence, 48% to 63% of that alignment across nine multi-head pretrained models and 5,408 heads comes from background geometry, with no monotone scale trend. In a pre-registered three-arm, eight-seed paired training factorial at 50.9M parameters, removing the own-value direction lowered validation loss below baseline, a fixed arbitrary rank-one direction did not differ from baseline, and own-direction removal directly outperformed that control.

You’re all caught up