Skip to main content
Back to timeline
arXivSource publication:

Tiny recursive models drop their two-level state: STRM beats TRM on Sudoku-Extreme at equal compute

Synopsis

The tiny recursive model TRM relies on a high-level and a low-level state plus nested loops to avoid memorization, which makes its recursion complicated. STRM replaces both states with a single state updated as a bank of fixed-rate, log-spaced exponential moving averages, and reads the answer only from the slowest-updating channels. With the same data, parameters and compute, STRM improves on TRM on Sudoku-Extreme and PPBench, matches it on Maze-Hard, and no longer shows a late drop in held-out accuracy.

AI-generated editorial illustration: Simpler Tiny Recursive Models

Interpretation

A recursive small model can reason with one state and one uniform update rule, without TRM's high-level and low-level states and nested loops. TRM previously alternated between refining a latent state and an answer state, and used that hierarchy to avoid memorization and overfitting; a plain single-state update loses accuracy on Sudoku-Extreme. On Sudoku-Extreme, STRM's final weights reach 91.2, against 88.3 at TRM's best checkpoint and 77.3 at TRM's final weights, with the same data, parameter count and compute per segment.

The single state trains stably because each channel averages the network's outputs at its own fixed rate, so fast channels act as working memory and slow channels accumulate results over many calls. Earlier single-state designs either overfit or needed an extra skip connection or learned memory tokens to stabilize training. In the ablation, removing averaging (all rates set to 1) drops test accuracy to 55.0, a single shared rate gives 81.9, and the full multi-timescale setting gives 91.2.

Reading the answer only from slow channels moves the information bottleneck that TRM's answer state provided inside the single state. If the output head reads all channels, the loss can be reduced through the fast path that copies the latest output, bypassing slow accumulation. On Sudoku-Extreme, the variant whose heads read all channels reaches 86.9 test accuracy, against 91.2 for the reference setting that reads slow channels only.

Perspective

The demonstrated scope is puzzle-like and abstract-reasoning benchmarks: Sudoku-Extreme, six PPBench puzzle types, Maze-Hard, and ARC-AGI-1 and ARC-AGI-2. The ARC-AGI comparison uses published TRM results; the other comparisons run at equal data, parameter count and compute per segment. The authors state that code will be released upon acceptance of the paper.

The authors position STRM as a drop-in replacement for TRM and expect it to be used with winner-takes-all training, parallel sampling and other follow-up methods, but those combinations are not tested. Rate jitter adds little to final accuracy; its main effect is reducing the late decline in held-out accuracy. Higher reported ARC-AGI scores come from methods such as URM that use four blocks instead of two and twice the training epochs, a different setting from STRM's.