Skip to main content
Search

# Looped Transformer

13 related articles

Subscribe to updates
  1. arXiv

    Decision models stop answering in one pass: looping the same weights lifts JevBench hard questions, 63.2% to 76.2%

    Decision models that answer fixed-option questions usually score every option in one forward pass, with a depth fixed at training time, so easy and hard questions cost the same. LEV swaps in a language-model backbone pretrained as a loop, reads the options after every step and trains all steps jointly, so one checkpoint answers at any depth; on JevBench the 2.6B model rises from 63.2% at depth one to 76.2% at depth four, mostly on hard questions. A small stopper sets depth per question with a threshold fitted on unlabelled requests, saving 15% to 24% of the computation at under one accuracy point.

  2. arXiv

    Looped reasoning stops keeping only the latest state: attention residuals hold solutions at nearly 25,000 layers

    Looped Transformers deepen reasoning by re-running a shared block at test time, but the usual carry-last rule lets later iterations overwrite correct intermediate results. InfiLoop replaces that rule with learned attention over recurrent outputs: a shared pseudo-query scores each candidate once, a learned exponential decay discounts older ones, and the aggregation has an exact streaming recurrence needing only a weighted-state vector and a normalization scalar per token. On Sudoku-Extreme a 7M-parameter model reaches 97.9% exact accuracy versus 94.2% for FPRM, and hits 90.9% at 1,872 layers where FPRM needs 3,744 layers for 90.1%.

  3. arXiv

    LLoCoT refines a compact latent workspace with a looped transformer, matching explicit-CoT accuracy on HumanEval and MBPP while cutting time to first answer token by about 36x

    The authors introduce LLoCoT, which replaces left-to-right latent generation with a few iterative refinement passes of a shared transformer over a compact latent workspace, then samples latent tokens in parallel from a probabilistic head to condition an autoregressive decoder; on HumanEval and MBPP it matches the accuracy of the explicit-CoT baseline Reasoning SFT and outperforms the base model, answer-only SFT and NF-CoT, while reducing time to the first answer token by about 36x, reasoning-phase latency by about 42x, and raising end-to-end throughput by 9.2%.

  4. arXiv

    ResidualQuant anchors on the final-loop KV and stores 2-bit residuals, cutting looped-Transformer KV storage by 80.7% at near-BF16 accuracy under INT2/4

    To address the KV cache memory bottleneck that grows with recurrent depth in looped Transformers, the authors propose ResidualQuant, which uses the final-loop KV states as an anchor and represents the remaining loops as low-precision residuals, combined with least-squares scaling, rotations applied to the residuals, and loop-wise INT4/INT2 mixed precision; across Ouro-1.4B and Huginn-3.5B on mathematical reasoning and code generation benchmarks it improves the accuracy-memory tradeoff over rotation-based KV quantization, reducing theoretical KV storage by 80.7% relative to BF16 while retaining accuracy close to BF16 under mixed precision.

  5. arXiv

    LiFT lets looped DiTs keep improving beyond training depth: L/2 and XL/2 beat larger dense DiTs with fewer parameters and less inference compute

    LiFT supervises each loop of a recurrent DiT toward a point on a straight path from the model's initial velocity estimate to the flow-matching target, so a trained checkpoint can run far more inference loops than its training depth; on class-conditional ImageNet, L/2 and XL/2 checkpoints that reallocate inference compute toward recurrence reach lower FID than larger dense DiTs with fewer parameters and less inference compute, while at B/2 the dense model remains stronger.

  6. arXiv

    LoopCD treats a looped Transformer's earlier iteration as a weak reference, lifting Ouro-2.6B's AIME 2024 pass@1 from 61.88% to 73.33% and cutting 22.5%–48.2% of forward FLOPs at half the loops

    The work introduces LoopCD, a training-free contrastive decoding framework that guides token selection by contrasting a looped Transformer's final prediction with an earlier recurrent pass, in either logit space with one extra output pass (LoopCD-Logits) or hidden-state space with zero output overhead (LoopCD-Hidden); across four looped Transformer families it delivers consistent gains at full recurrent depth, for example raising Ouro-2.6B-Thinking's AIME 2024 pass@1 from 61.88% to 73.33% and Huginn's HumanEval pass@1 from 22.56% to 31.71%, and these gains let it match or exceed full-depth unguided baselines at half the recurrent loops while reducing forward FLOPs by 22.5% to 48.2%.

  7. arXiv

    LoopVL reuses shared parameters across 128 layer calls, lifting a 1B backbone from 55.33 to 63.47 on MMStar and revealing a cross-loop Visual Aha Moment

    The authors introduce LoopVL, extending looped Transformers to vision-language models: the language backbone is pretrained from scratch on HRM-Text's open-source framework and data, using nested Module-Loop and Model-Loop computation over shared L/H parameter stacks, with the default H2L3 configuration executing 128 Transformer-layer calls per forward pass; under the same 0.14T-token training budget, LoopVL outperforms a same-depth Transformer-VL 1B baseline on MMStar, RealWorldQA, and ChartQA and approaches or surpasses larger 4B dense baselines, while the authors report cross-cycle visual attention reallocation they call a Visual Aha Moment.

  8. arXiv

    First joint scaling law for looped mixture of experts: sparsity raises and delays the saturation of recurrence's effective-parameter gain

    The work introduces Loop Scaling Laws, the first scaling law that jointly models recurrence and MoE sparsity alongside model size and training data, using a bounded, sparsity-conditional recurrence mapping to characterize the effective-parameter gain from looping and its saturation; it predicts held-out loss more accurately than linear and power-law mappings, guides the choice of recurrence and expert count under compute and memory budgets, and at trillion-token scale lets a 0.3B-active/1.3B-total looped MoE match a roughly 2x larger non-looped MoE on reasoning benchmarks.

  9. arXiv

    TAPS adapts recurrent step size to the trajectory, lifting Sudoku accuracy to 91.39% and Maze to 79.90% while cutting the loops needed for matched quality

    The work introduces the Trajectory Adaptive Progress–Fluctuation Scheduler (TAPS), which uses exponential moving averages to estimate persistent progress and centered fluctuation in recurrent updates and adapts the per-step update scale online; it proves that under stated conditions TAPS reduces expected terminal loss and reaches a target quality in fewer loops, and empirically improves terminal accuracy with up to roughly 1.5x wall-clock speedup across Sudoku, Maze, language-model recurrence, and intermediate-layer recurrence.

  10. arXiv

    WaveFront Decoding fuses drafting and verification into the same recurrent call, speeding up decoding 2.42x on Ouro-2.6B and 3.54x on Huginn-3.5B

    The work introduces WaveFront Decoding (WFD), a training-free self-speculative decoding framework for looped language models that exploits two properties of these architectures, namely that intermediate recurrence outputs provide effective draft predictions and that weight sharing lets token states at different positions and recurrence depths be processed in one batched recurrent-block call, organizing these mixed-depth states into a diagonal wavefront so that drafting and verification proceed concurrently within the same recurrent calls while rejected drafts are corrected using full-depth predictions; across six Spec-Bench task categories it achieves 2.42x speedup on Ouro-2.6B and 3.54x on Huginn-3.5B over autoregressive decoding, rising to 4.81x on Huginn-3.