Skip to main content
Search

# Transformer Encoder

5 related articles

Subscribe to updates
  1. arXiv

    Membrane protein diffusion without more molecular dynamics: a neural stochastic model stretches trajectories to 40,000 frames while holding near-Brownian motion

    Coarse-grained molecular dynamics gives short membrane-protein trajectories, but long-time diffusion statistics are costly to obtain. A neural stochastic model combines a transformer encoding of protein configurations, recurrent memory at multiple time scales and a stochastic latent process with learned memory to generate correlated displacements autoregressively. In the seven-PIEZO system it yields a mean anomalous diffusion exponent of 1.048±0.012 against 1.049±0.022 for MD, where the earlier VAR-SDE model gives 1.563±0.016, and its trajectories extend to 40,000 frames.

  2. Visible and infrared stop being fused in one shot: staged modality control reaches 88.6% mAP50 on M3FD

    Visible and infrared images follow different imaging mechanisms, so direct concatenation or one-shot attention fusion can mistake background texture and thermal interference for complementary evidence. FCI-DETR splits detection into three linked stages: preserving modality-specific structure before fusion, calibrating cross-modal responses during fusion, and injecting infrared context by scale after fusion. It reaches 88.6%, 84.3%, 86.7% and 81.5% mAP50 on M3FD, DroneVehicle, VEDAI and FLIR, and beats a same-backbone RGB+IR RT-DETR baseline by 6.0 points on M3FD under a fixed protocol.

  3. arXiv

    OncoNoteBERT trains oncology-specific encoders on 290,000 UK outpatient notes: continued pretraining cuts perplexity to 2.10, while a from-scratch tokenizer shortens sequences by about 23.6%

    Using 290,026 outpatient oncology notes from 21,564 patients treated for lung and head-and-neck cancer at a UK specialist oncology centre, the study compared RadBERT and PathologyBERT with two local strategies: OncoNote-RadBERT, produced by continued masked language model pretraining, reached a validation perplexity of 2.10, while OncoNoteBERT, trained from scratch with an oncology-specific WordPiece tokenizer, reached 2.83 but the most efficient tokenisation (subword fertility 1.224, normalised sequence length 0.7636, about 23.6% fewer tokens) and returned clinically acceptable predictions for 12 of 13 masked-token probes versus 7 of 13 for OncoNote-RadBERT; both local models represented the institutional placeholder [REMOVED] as a single learnable token.

  4. arXiv

    MAGEFormer embeds voxel spacing into positional encoding and attention, reaching the lowest HD95 on anisotropic abdominal CT segmentation across BTCV and FLARE 22

    The authors propose MAGEFormer, which embeds physical spacing constraints into representation learning: MASE calibrates positional frequencies by voxel spacing, GCA suppresses physically implausible cross-scale feature correlations, and GVV performs deterministic sub-voxel view aggregation at inference; under a unified 5-fold protocol on BTCV and FLARE 22 it attains the strongest boundary accuracy among compared methods (10.58 mm and 3.40 mm HD95) with consistent Dice gains, and its advantage grows with anisotropy severity.

  5. CEEMDAN-decomposed time series forecasting of reported hepatitis B cases using KOA-optimised deep learning: a nationwide study in mainland China (2004-2027)

    This study compiled a national monthly series of hepatitis B notifications in mainland China from January 2004 to December 2025, applied CEEMDAN to isolate multiscale temporal components, and trained four models (GRU, CNN, SVM, and a Transformer encoder) with KOA-optimised hyperparameters on a sliding 12-month window recursively extended to 24-month horizons; the Transformer delivered the best out-of-sample fit on the held-out test split (MAE = 3105.508, MAPE = 0.024, RMSE = 4071.901, R² = 0.928), SVM ranked second, CNN outperformed GRU but not SVM, and forecasts for 2026-2027 remain elevated, signalling little improvement and even possible resurgence.

You’re all caught up