Skip to main content
Search

# ARLoop

1 related articles

Subscribe to updates
  1. arXiv

    Looped transformers can skip the full residual history: ARLoop keeps only the last three blocks and its 1.2B model beats Huginn-3.5B on math and code

    ARLoop brings attention residuals to looped transformers, letting each layer select, through attention weights, which representations produced at earlier recurrent depths to reuse instead of accumulating every residual update into one stream with fixed coefficients as PreNorm does. In controlled experiments at 140M and 370M parameters it reaches lower validation perplexity than PreNormLoop and than the earlier Sandwich-Norm, DeepLoop and Parcae residual designs, and at larger recurrence depths a sliding window over only the three most recent residual blocks matches or beats full-history attention. Trained under a realistic pretraining recipe, ARLoop-1.2B uses only 0.5T tokens, outperforms Huginn-3.5B on every evaluated task and is competitive with Qwen3.5-2B.

You’re all caught up