Skip to main content
Search

# Self-Attention

11 related articles

Subscribe to updates
  1. arXiv

    Pixel diffusion drops attention mixing: Volterra layers beat JiT's FID at every ImageNet scale

    Pixel-space diffusion models had inherited softmax attention from sequence modeling as their token mixer, although the optimal denoiser under Gaussian noise admits a polynomial functional expansion in the observation. PoD compresses each image patch through a low-dimensional bottleneck and stacks cascaded low-rank Volterra layers that mix tokens and channels, with no activation function on the residual path. At matched parameters and FLOPs on ImageNet, PoD beats JiT in FID at every scale, with the largest gap at the G scale.

  2. bioRxiv

    A frozen EEG foundation model needs no fine-tuning: one first-layer attention head's connectivity matrix reaches 42.13 balanced accuracy in cross-subject emotion recognition

    The EEG foundation model REVE no longer needs full fine-tuning or all-layer pooled features to recognize emotion. Such models had not consistently beaten task-specific architectures on cross-subject tasks, and their large representations are costly to compute and store. This approach aggregates the first-layer attention weights of a single head over each trial into a weighted channel-to-channel matrix, then uses a label-free symmetry score to pick the most informative head; on SEED-IV, classical classifiers reach 42.13 balanced accuracy with about 176 times fewer calculations than all-layer attention pooling.

  3. aclanthology.org

    DoLa swaps logits for attention when picking layers: LLaMA-7B MC2 reaches 62.8%, while larger models keep MC1 with DoLa

    Contrastive decoding's DoLa picks its premature layer from differences in output vocabulary distributions; Sakai and colleagues instead use self-attention distributions as the selection signal, introducing Attention-JSD, Attention-Entropy-Max and Attention-Entropy-Min while leaving the DoLa framework intact. On TruthfulQA, LLaMA-7B with Attn-Ent-Min reaches MC2 62.8% and MC3 36.5%, above DoLa's 56.5% and 30.5%; on LLaMA-33B and 65B, however, DoLa keeps the highest MC1, so the gains concentrate on multi-answer metrics.

  4. arXiv

    More Communication Can Widen the Generalization Gap in Streaming Federated Learning, Bounds Grow With √R

    Streaming federated learning trains a shared model from continuously arriving client data, whose streams are often temporally dependent. For clients whose data follow Markov processes, an analysis uses information-theoretic and rate-distortion tools to bound the expected generalization gap of the aggregated model, specializing to linear and self-attention next-token prediction. The bounds show that more communication rounds loosen the guarantee at roughly √(R/N), while temporal dependence enters through the square root of the mixing time; GPT-style pretraining on TinyStories and ETTh2 forecasting both show train and test losses falling with rounds while the gap widens.

  5. EEG attention decoding drops the fixed electrode graph: dynamic topology plus self-attention reaches 91.67% on three classes

    EEG attention recognition has moved away from a fixed electrode-connectivity graph. Multi-band features were previously concatenated and channel relations were handled with a preset adjacency matrix, which struggles to capture task-dependent coupling. DGCSA treats electrodes as graph nodes, updates channel topology through a learnable adjacency matrix, and adds self-attention for global dependencies, while Bayesian-optimized band weights fuse power spectral density and differential entropy. On 873 three-second EEG segments from 20 participants, three-class accuracy reached 91.67%, above SVM, CNN and LSTM baselines; mean attention scores were also higher in VR than in 2D.

  6. arXiv

    Each Transformer layer read as one EM iteration: attention does belief propagation, the layer-to-layer jump updates parameters

    Transformer attention is interpreted as max-product belief propagation on a Markov random field that varies from instance to instance, while the parameter change between layers acts as the parameter-update step of EM, a view that groups self-attention, mixture-of-experts, gated units and selective state-space models under one meta-model principle in which the input generates the parameters.

  7. arXiv

    Universal Attention composes decay gates for over 10x KV-cache compression and beats unpruned Llama3 on RULER

    The work introduces Universal Attention, which combines query-aware, key-aware, and similarity-based decay mechanisms via a geometric mean into an additive bias on the attention matrix, letting the model adaptively prune its KV-cache while keeping RoPE and Softmax; on a 1B Llama3-style model trained on 50B tokens it achieves over 10x compression, with RULER averages of 50.3/49.3 versus 40.8 for unpruned Llama3 at only 21.4/12.9 MB of cache.

  8. arXiv

    Keeping only the top-attention tokens without retraining, nine checkpoints need effective attention sets of 7 to 283 tokens, and longer context raises the required set size

    Without retraining, this work retains only the tokens with the highest attention weights at each head, layer, and query while keeping their original weights unchanged, and estimates the effective attention set size needed to stay within a chosen loss tolerance by measuring the increase in negative log-likelihood (NLL); it finds that relatively small selected sets keep NLL close to the full-attention baseline, that attention-based selection substantially outperforms random selection, that the required set size grows with context while its fraction of context decreases, that in BABILong experiments with a fixed annotated supporting fact additional background pushes support tokens down the attention ranking and reduces their attention mass, and that renormalizing the retained weights can subs

  9. arXiv

    Why Video Diffusion Models Break Physics: Researchers Trace It to RoPE Spatial Anchoring and Fix It by Rescaling RoPE Frequency

    The work presents an interpretability study of the "motion planning" process in text-to-video diffusion models, finding that excessive spatial attention decay induced by RoPE in self-attention makes early candidate regions lock prematurely onto physically implausible positions, and proposes a lightweight architectural change that merely rescales RoPE frequency across denoising steps, improving physical commonsense in both training-free and training-based experiments.

  10. arXiv

    CalcSeg combines confidence-aware curriculum learning with slice-wise self-attention to raise myocardial scar segmentation Dice to 0.677 and low-confidence Dice to 0.644 on single-stack LGE-CMR

    The work presents CalcSeg, a confidence-aware latent 3D context curriculum learning framework that scores each sample using Dice, percentage scar-burden error, and epistemic uncertainty from Monte Carlo Dropout, expands training from easy to hard cases across stages, and uses slice-wise self-attention to infer subject-level 3D anatomical context from single-stack 2D LGE-CMR; on LGE-CMR data from four sites and two segmentation challenges (MICCAI 2012 LV Infarct, EMIDEC 2020) comprising 976 patients, it reaches myocardial scar Dice of 0.677±0.24, low-confidence Dice of 0.644±0.22, scar error of 38.88%, and low-confidence scar error of 35.03%, outperforming TransUNet, AttentionUNet, UNETR, ScarNet, and ScarNet with supervised curriculum learning using expert difficulty labels.