Skip to main content
Search

# Attention Mechanism

13 related articles

Subscribe to updates
  1. medRxiv

    Alzheimer's cohort stops sorting patients by diagnosis alone: TRACE lifts 24-month phenotype purity 3.1 points over the strongest baseline

    TRACE maps longitudinal visit records to interpretable progression phenotypes and lets the number of phenotypes exceed the number of diagnostic classes. In the label-matched setting on the ADNI cohort it achieves the highest purity, NMI, ARI, silhouette index, accuracy and macro-F1 at the current, 12-month and 24-month horizons; at 24 months it exceeds the strongest baseline by 3.1 and 5.8 percentage points in purity and silhouette index. With the phenotype count freed, it splits a new phenotype out of the cognitively-normal- and MCI-dominant groups, one whose cognitive-scale trajectories sit in between while its hippocampal-volume trajectory coincides with the MCI group.

  2. arXiv

    TPU attention kernels stop idling: FlyWheel lifts softmax attention by up to 3.49x and reaches 81% of peak BF16 throughput

    FlyWheel is a set of TPU-native attention kernels covering both softmax attention and Gated DeltaNet linear attention. It targets matrix units left idle by repeated operand loads, vector-heavy work and dependency stalls, by coordinating data layouts, recasting triangular inversion and convolution as matrix products, and explicitly interleaving independent computation. On TPU v6e, softmax attention runs up to 3.49x faster than RPA-v3 and up to 1.56x faster than Splash Attention, reaching 81% of peak BF16 throughput.

  3. Sleep staging stops picking electrodes: any EEG-EOG mix scores a night, and resolution drops to seconds

    Sleep staging has long depended on fixed electrode sets and 30-second expert-scored epochs, which makes multi-center studies hard to harmonize. AnySleep fuses any number of EEG and EOG channels through channel attention, trained and validated on 20,589 overnight recordings from 28 datasets, matching or slightly exceeding U-Sleep at 30-second epochs and staying strong with no EOG or a single derivation. At sub-30-second resolution, the share of expert-annotated arousal time overlapping predicted Wake rises from 7.7% at 30 seconds to 57.7% near 2 seconds.

  4. Arctic sea ice forecasts drop recurrent nets: causal time convolution plus state-space encoding beats MT-IceNet at every lead from one to six months

    Arctic sea ice concentration forecasting has been limited by the difficulty of fusing multi-timescale information, with statistical models missing nonlinear relationships and numerical models depending on initial conditions. STICEnet replaces the U-Net encoder-decoder path with strictly unidirectional temporal causal and transposed convolutions and embeds channel and spatial attention in the skip connections, trained on NSIDC sea ice concentration and ERA5 atmospheric and oceanic variables. Across one- to six-month leads its RMSE, MAE and R2 all beat CNN, ConvLSTM, IceNet and MT-IceNet, reaching 7.51% RMSE and 0.925 R2 at month six.

  5. No More Blackouts for Meter Checks: Kirchhoff and Cable-Heat Constraints Let Voltage Transformers Be Scored Online

    Distribution-network voltage transformers have been checked mainly by de-energizing and comparing them against standard units, which is slow and interrupts supply. A State Grid Wuhan team embedded switching-station Kirchhoff voltage constraints and a cable thermal-balance equation as soft loss terms in a neural network, added radial-basis multi-scale time features and residual attention weighting, and estimated transformer error online. On a laboratory physical simulation platform the method cut RMSE by 35.2% and MAE by 36.4% and raised R² by 2.63 points over the runner-up MRSA-DGCN; in a field pilot in a 10 kV Wuhan area it flagged one out-of-tolerance transformer, matching the de-energized check.

  6. A Transformer reads raw microscopy video and calls cell death up to 8 hours early

    On long-term live-cell videos of cisplatin-treated cancer cells, a Transformer that takes raw image sequences as input separates apoptosis from mitosis at 0.94 balanced accuracy and stays reliable up to 8 hours before the event.

  7. arXiv

    WakeKV replaces frozen KV compression with recoverable CPU residency, cutting miss rate at matched memory and raising throughput

    Measuring attention-head behavior across three models (1.5B–8B) and three regimes (needle retrieval, long chain-of-thought, multi-turn recall), the authors find most heads change their reading behavior at least once during generation and introduce WakeKV, a reactive residency policy that moves cooling heads to a recoverable CPU reservoir instead of freezing or permanently evicting their state; at matched memory or budget WakeKV consistently improves miss rate over frozen classification and destructive eviction, and a FlexiCache/vLLM implementation on Mistral-7B improves throughput while retaining LongBench quality.

  8. arXiv

    MAGEFormer embeds voxel spacing into positional encoding and attention, reaching the lowest HD95 on anisotropic abdominal CT segmentation across BTCV and FLARE 22

    The authors propose MAGEFormer, which embeds physical spacing constraints into representation learning: MASE calibrates positional frequencies by voxel spacing, GCA suppresses physically implausible cross-scale feature correlations, and GVV performs deterministic sub-voxel view aggregation at inference; under a unified 5-fold protocol on BTCV and FLARE 22 it attains the strongest boundary accuracy among compared methods (10.58 mm and 3.40 mm HD95) with consistent Dice gains, and its advantage grows with anisotropy severity.

  9. arXiv

    S3N pairs finite-element LL–HP mapping with quad-spiral state-space scanning to cut 10-day forecast nRMSE to 0.5177

    The work proposes S3N, a spherical spiral scanning network that uses the surface finite-element L2Proj method to bidirectionally map atmospheric fields between the latitude–longitude grid and the equal-area HEALPix grid, and an Attention-Guided Quad-Spiral State-Space Scanning block that propagates information across HEALPix base-face boundaries along four global pole-to-pole spiral paths guided by cross-latitude attention; on ERA5/WeatherBench 2, S3N attains lower nRMSE at 4-, 7-, and 10-day lead times (0.5177 at 10 days versus 0.6398 for OneForecast) with slower error growth, while being less competitive at 6-hour and 1-day lead times.

  10. arXiv

    A survey recasts attention evolution as contextual-memory organization, using 59 release records and 11 open-weight endpoints

    Treating model-internal contextual memory as the shared unit of analysis, this survey introduces five dimensions—Memory Representation, Memory Update, Access, Readout, and Integration—and reviews five research lines (Softmax Attention, Sparse Attention, Linear Attention, State Space Models, and Hybrid Architectures), using 59 release-level records across 14 major model lineages and a frozen comparison of 11 high-performing open-weight endpoints through September 22, 2026 to argue that attention design is diversifying rather than converging, with hybridization and cross-layer reuse making network depth a dimension along which contextual memory is organized.