Skip to main content
Back to timeline
arXivSource publication:

Bilateral log-linear attention adds an origin-anchored hierarchy: MQAR mean rises from 76.37% to 96.12%, and the 6144-token retrieval mean nearly doubles

Synopsis

Log-linear attention compresses older context more coarsely and keeps fine-grained summaries only near the query. Bilateral log-linear attention adds a second hierarchy anchored at the sequence origin alongside the existing Fenwick hierarchy, so both early and recent context get finer reads. On MQAR with matched depth and hidden width, mean overall test accuracy rises from 76.37% to 96.12% across ten training seeds; under approximate parameter matching, the six-task RULER-NIAH mean at 6144 tokens rises from 26.97% to 50.03%.

AI-generated editorial illustration: Bilateral Log-Linear Attention

Interpretation

Bilateral log-linear attention adds an origin-anchored causal hierarchy alongside the existing one, so the same causal prefix can be read at finer resolution at both the early and the recent end. Log-linear attention kept only the Fenwick hierarchy, placing recent tokens in small buckets and merging older tokens into progressively larger buckets, so early associations shared coarser summaries. On MQAR with matched depth and hidden width, mean overall test accuracy across ten training seeds rises from 76.37% for forward log-linear attention to 96.12%.

The added branch groups source positions into geometrically growing buckets at fixed levels, stores one fixed-dimension matrix state per bucket, and multiplies each write by an input-dependent scalar gate. The reverse branch has no carry, merge, or temporal decay, so decoding updates only one level's state while reading visible levels at logarithmic cost. Each branch's read is RMS-normalized without affine parameters and mixed per head by a softmax router that depends on the layer input; under approximate parameter matching at 6144 tokens, the six-task RULER-NIAH mean rises from 26.97% to 50.03%.

The two branches share query, key, and base value features, and the combined mechanism keeps log-linear attention's asymptotic cost at fixed feature dimensions and chunk size. The added hierarchy increases computation and persistent state storage only by constant factors, leaving training and prefill work log-linear and decoding work per token logarithmic. Under approximate parameter matching, bilateral attention beats forward log-linear attention on seven of eight downstream retrieval tasks, with a retrieval average of 25.16% versus 21.63%.

Perspective

Evaluation covers 512-token synthetic multi-query associative recall, language models pretrained on roughly 10B tokens with a 6144-token block, and six RULER-NIAH tasks at 2048, 4096, and 6144 tokens plus eight downstream retrieval tasks; all lengths fall within the training block length.

On the six LM-Eval reasoning tasks, bilateral attention averages 41.51%, versus 42.23% for layer-matched forward log-linear attention and 41.26% under approximate parameter matching, so reasoning-task scores change in mixed directions rather than uniformly improving; Transformer still scores higher overall on RULER-NIAH and downstream retrieval.