Skip to main content
Search

# StateBench

1 related articles

Subscribe to updates
  1. StateBench splits state tracking into four levels: recurrent and hybrid models extrapolate, while frontier models drop to 13% or lower on the longest general finite problems

    The Allen Institute for AI released StateBench, which sorts state tracking in long documents into four levels of assignment operation and derives four difficulty-matched problems from the same assignment sequence. On open pretrained models, almost all handled immutable assignment well, lost accuracy on reassignment by value as inputs grew, and stayed at chance on general finite problems; four frontier models dropped to 13% or lower on the longest general finite problems. Among 275-million-parameter models trained from scratch, Gated DeltaNet and hybrid models extrapolated to tens of thousands of assignments near perfectly on the three easier levels, while transformers never learned the hardest level.

You’re all caught up