StateBench splits state tracking into four levels: recurrent and hybrid models extrapolate, while frontier models drop to 13% or lower on the longest general finite problems
Synopsis
The Allen Institute for AI released StateBench, which sorts state tracking in long documents into four levels of assignment operation and derives four difficulty-matched problems from the same assignment sequence. On open pretrained models, almost all handled immutable assignment well, lost accuracy on reassignment by value as inputs grew, and stayed at chance on general finite problems; four frontier models dropped to 13% or lower on the longest general finite problems. Among 275-million-parameter models trained from scratch, Gated DeltaNet and hybrid models extrapolated to tens of thousands of assignments near perfectly on the three easier levels, while transformers never learned the hardest level.
Interpretation
Long-document state tracking now has a graded test: one string of variable assignments is rewritten into four difficulty levels, so the level at which a model fails can be located separately. Earlier state-tracking tests concentrated on the hardest cases and did not separate the simpler, potentially more common kinds. The four levels share the same number of variables and distinct values, with assignments at the same points in the sequence; only the assignment type changes, so the difficulty gap does not come from length or vocabulary.
The approach writes state tracking as a sequence of variable assignments and asks the model to output each variable's final value after reading it. The four levels correspond to immutable assignment, reassignment by value, self-referential update, and assignment by reference, the last including operations such as swapping two variables. Pretrained testing used open models with documented training data and context lengths, including the 7-billion-parameter Olmo 3 and Olmo Hybrid, which adds Gated DeltaNet layers, plus smaller 275-million and 2.7-billion-parameter models, none trained on contexts longer than 64,000 tokens.
Training from scratch separates architecture differences from pretraining-data differences: trained directly on these tasks, recurrent and hybrid models handled levels their pretrained counterparts could not. Transformers could never learn the hardest general finite level even when trained directly, so that level is not simply a gap in pretraining data. Six 275-million-parameter models trained on sequences of up to 256 assignments and were tested on sequences as long as 64,000 assignments; the Gated DeltaNet variant with more flexible memory update stayed near perfect out to roughly 10 times its longest training sequence.
Perspective
The tests cover immutable assignment, reassignment by value, self-referential update, and assignment by reference; pretrained testing is limited to open models trained on contexts no longer than 64,000 tokens, and the from-scratch experiments are limited to the 275-million-parameter scale.
A new proof shows that Gated DeltaNet with the more limited memory update of non-negative eigenvalues can solve all aperiodic tasks and certain periodic tasks under the paper's fixed-precision arithmetic with rounding, but its exact placement in the hierarchy remains open, including whether it can solve all soluble tasks and how numerical precision or combining it with attention layers affects its capabilities. StateBench also assumes a fixed number of variables, while real tasks often introduce new entities over time, a setting the current framework does not yet cover.
