Looped transformers can skip the full residual history: ARLoop keeps only the last three blocks and its 1.2B model beats Huginn-3.5B on math and code
Synopsis
ARLoop brings attention residuals to looped transformers, letting each layer select, through attention weights, which representations produced at earlier recurrent depths to reuse instead of accumulating every residual update into one stream with fixed coefficients as PreNorm does. In controlled experiments at 140M and 370M parameters it reaches lower validation perplexity than PreNormLoop and than the earlier Sandwich-Norm, DeepLoop and Parcae residual designs, and at larger recurrence depths a sliding window over only the three most recent residual blocks matches or beats full-history attention. Trained under a realistic pretraining recipe, ARLoop-1.2B uses only 0.5T tokens, outperforms Huginn-3.5B on every evaluated task and is competitive with Qwen3.5-2B.
Interpretation
A looped transformer can now select, by attention weight, which representations left by earlier recurrent depths to reuse, rather than forcing every residual update into one stream with fixed coefficients. Under PreNorm, residual updates from earlier depths accumulate into a single state with fixed unit coefficients, so deeper recurrence leaves later computation with no explicit way to choose which early-depth representations are useful. Across controlled settings at 140M and 370M parameters and several recurrence depths, ARLoop reaches lower validation perplexity than PreNormLoop and than the earlier residual designs Sandwich-Norm, DeepLoop and Parcae.
A looped transformer does not need access to the full residual history; the most recent residual blocks are enough. Attention residuals on standard transformers normally let each depth-attention operation see all preceding residual blocks, which is full-history depth attention. In controlled experiments at 140M and 370M parameters, sliding-window depth attention matches or lowers perplexity relative to full-history attention at larger recurrence depths, and visualizing the full-history variant's attention weights shows the three most recent residual blocks plus the embedding block receive 96% of the total attention weight at sublayer 47.
The gains from this residual design carry over to realistic large-language-model pretraining and are especially visible on math and code tasks. Earlier residual designs were mostly evaluated under relatively small-scale pretraining, leaving their effect on complex generative tasks such as mathematical reasoning and code generation untested. At the same 0.6B parameters, 180B-token budget and three recurrences, ARLoop-0.6B beats PreNormLoop-0.6B on 7 of 8 evaluated tasks, including +5.2 points on GSM8K and +3.1 points on MBPP.
Perspective
The demonstrated scope is controlled pretraining at 140M and 370M parameters with 6 and 12 layers, plus realistic pretraining recipes at 0.6B and 1.2B parameters with three recurrences and 180B and 500B tokens; evaluation covers general knowledge, mathematical reasoning and code generation, and training code, data-processing code, intermediate checkpoints and final models are released.
The sweep over sliding-window and residual-block sizes was mainly run at 140M, and a more systematic window design is left open; pushing inference-time recurrence beyond the maximum training depth does not improve performance, even when query parameters at unseen depths are trained separately; the comparison with residual methods such as Parcae and DeepLoop was not run at realistic pretraining scale.
