Benchmarks show AI agents fail structurally on long-horizon tasks

Multiple research papers and benchmarks document a consistent pattern: AI agents that perform well on short tasks fail substantially as task duration and step count grow, and the failure is structural rather than a simple performance decrement.

OpenAI data, cited by Rohan Paul, shows agent success without human intervention dropping from 86% on tasks under 15 minutes to roughly 16% on tasks lasting 64 to 128 hours. A Microsoft paper testing models on the ToolQA benchmark finds agents appearing near-perfect on short runs collapse to 0 to 33% success by 16 steps, with error compounding across steps; shortening the context made the decline worse. Scale AI's SWE-Bench Pro shows models scoring above 70% on standard SWE-bench dropping to approximately 23% on long-horizon tasks involving multi-file refactors and cross-repository changes. METR's analysis places the 50% success threshold for frontier models at approximately 50 minutes on human expert tasks. Some companies report agent failures on production workloads even when benchmark scores improve.

The HORIZON benchmark, built from 3,100+ trajectories across multiple model families, finds that long-horizon failure is a structural shift in failure composition rather than a simple performance drop, and that performance gaps between frontier and smaller models collapse in the long-horizon failure regime, suggesting diminishing returns from scaling model capacity alone.

Two additional benchmarks further characterize the measurement problem. FM-Bench tests 15 frontier models managing a simulated football club across 20 years; year-5 rankings correlated only 0.19 with final rankings, and no single model dominated the competitive Arena setting. LoopArena isolates the controller model from the coding worker; even the top controller, GPT-5.5, achieved only 24.69% strict success on 27 tasks, and restating the original goal each round scored the same as running with no controller at all.

A factorial experiment found that the choice of agent scaffold explained 7.8x more variance in SWE-bench scores than the choice of model, with GLM-5.1 improving 13 percentage points solely by switching harnesses.

Two proposed fixes address different parts of the problem. ContextPilot uses reinforcement learning to teach agents to actively manage their working context, achieving 69.40 versus 45.93 for the base model across four long-context benchmarks. SPACE, from a joint Amazon-Microsoft paper, trains agents to group safe sequential actions rather than querying the LLM after every step, raising success on ScienceWorld from 35.9% to 67.2% while cutting LLM rounds from 10.2 to 5.2.