Microsoft's ToolQA study, testing nine models, found agent success rates near-perfect on short runs collapse to 0-33% by 16 steps; shortening context worsened performance, ruling out context trimming as a fix. Scale AI's SWE-Bench Pro showed models scoring above 70% on standard SWE-bench drop to roughly 23% on long-horizon tasks involving multi-file refactors and cross-repository changes. METR's July 2025 time-horizon analysis placed the 50% success threshold for frontier models at approximately 50 minutes on human expert tasks. Some companies report agent failures on production workloads even when benchmark scores improve.
A Forethought analysis fit an exponential decay model to research-engineering task data, finding that success rates decline at a constant per-minute failure rate, allowing each agent to be characterized by a half-life -- the task length at which it succeeds 50% of the time, which is also the median remaining lifespan from any mid-task point. The HORIZON benchmark analyzed over 3,100 trajectories and found long-horizon failure is a structural shift in failure composition, not just a performance drop; performance gaps between frontier and smaller models collapse once agents enter this failure regime. Two distinct failure types require different interventions: environment disturbances need monitoring and recovery mechanisms, while catastrophic forgetting requires improved memory and constraint tracking. Controlling for task length, models perform worse on tasks with higher messiness scores, where messiness captures factors including resource limits, novelty, and dynamic or not-easily-resettable environments.
Anthropic reported autonomous AI agents that propose ideas, run experiments, and iterate on research problems -- specifically how to train a strong model using only a weaker model's supervision -- and found they outperform human researchers, suggesting this kind of research automation is practical.