StateBench: Testing how language models track state changes over long documents

StateBench is a new benchmark designed to test how language models follow changes to entities in long documents, such as characters in a story or variables in code. The benchmark includes four difficulty levels, ranging from simple updates to complex chains where missing a step can alter the final outcome. For example, tracking whether a knife changes hands multiple times requires following a sequence of assignments, testing the model’s ability to maintain an accurate state over long inputs.

The benchmark evaluates models on problems of increasing complexity, with each level representing a different type of state update operation. These levels are grounded in Krohn–Rhodes theory, which describes how state-tracking problems can be broken down into atomic components. StateBench ensures fairness by using the same number of variables and assignment points across all levels, varying only the type of assignment to isolate differences in performance.

Open pretrained models, including transformer and hybrid variants, were tested on StateBench. Most models performed well on simpler problems but struggled with harder ones as input length increased. Gated DeltaNet and hybrid models, however, generalized better on longer sequences, while transformers degraded or failed entirely on the most complex tasks. The results highlight the importance of architecture and training in handling state tracking.

Frontier models such as Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna were also evaluated. These models performed strongly on easier problems but saw sharp accuracy drops on harder levels as inputs grew longer, falling to 13% or lower on the longest general finite problems. The findings suggest that current state-of-the-art models still face challenges with robust state tracking in long contexts.