SWE-CoWork released; Opus 5.5 tops leaderboard completing 8 of 101 tasks
Opus 5.5 Beats Human Baselines on Singularity-Bench; End-to-End Gaps Remain
These benchmarks reveal a consistent gap between partial task progress and full end-to-end delivery across ML engineering and scientific workflows. Singularity-Bench's finding that most agents scored near zero, combined with the authors' acknowledgment that the benchmark does not test the full recursive self-improvement loop, sets a more cautious frame around headline scores.
When stories like this move, I write about them in an email each morning.
One free email each morning, around 8 AM US Eastern.
Check your email.
I have sent you a confirmation link. One click and you are in.
Not in your inbox? Look in spam or promotions. If it's there, mark it "Not spam" so future editions land in your inbox.
The full picture
Four benchmarks now measure AI agents on complex ML engineering and scientific workflow tasks, with results that differ sharply by task type. On Singularity-Bench, a six-task Carnegie Mellon benchmark that scores agents relative to production systems, Claude Opus 5.5 paired with Claude Code achieved a Singularity-Index of 1.25, above the human baseline of 1.0, while the other ten evaluated agents averaged 0.58. Its serving engine earned $11.52 per GPU-hour versus $8.18 for SGLang, though six of eleven engines across all agents scored zero. The benchmark's authors acknowledge it is a proxy for the component skills recursive self-improvement would require, and that it does not test the full autonomous loop of an agent building and then improving a successor end to end. The benchmark includes a safety framework called Breakwater to guard against reward-hacking attacks such as prompt-based data exfiltration. FrontierML tests agents on 21 long-horizon ML tasks given a 12-hour CPU sandbox and spending budgets of $10, $50, or $200; Opus 5.5 reached a 45.4-point boost at $10 and 58.8 at $200 against human baselines of 56 and 66.1. In 28% of trajectories, agents added newly trained components to pretrained models or trained a model from scratch. SWE-CoWork, from TokenWave AI, places agents in a simulated workplace where 82% of requirements must be gathered from 26 simulated teammates rather than a written spec. Across 101 tasks and 101 open-source repositories, top models passed about 63% of individual checks but fully completed fewer than 8% of tasks; Opus 5.5 tops the leaderboard with 8 of 101 fully completed. More than half of Opus 5.5's failed checks occurred after the agent already had the requirements, pointing to code translation as the main bottleneck rather than information gathering. Giving GPT-6 Astra the full spec upfront raised its passed checks by 16 points yet still left it completing only 8 of 101 tasks. A fourth benchmark, FrontierChallenge, evaluated 12 frontier models on 97 cross-domain scientific workflow tasks; the best-performing configurations completed 20 of 97 tasks, a 20.6% pass rate, with models frequently claiming completion without satisfying the full-completion criterion.
How it developed
FrontierML results reported: Opus 5.5 approaches human-researcher returns at $200 budget
Singularity-Bench coverage notes Opus 5.5 serving engine outperforms SGLang by ~41%
Singularity-Bench paper published; Opus 5.5 scores 1.25 against human baseline of 1.0
FrontierChallenge published: best agents complete 20 of 97 scientific workflow tasks
Sources
2 more sources
- Agent framework choice shifts scores 27 points; long-horizon failures persist
- Multi-agent AI performance tradeoffs
- Intology's Locus agent on PostTrainBench+
- Princeton shadow evaluation of AI research judgment
BackgroundFrontier models, open models, and how they are compared
Get the daily email
When stories like this move, I write about them in an email each morning.
One free email each morning, around 8 AM US Eastern.
Check your email.
I have sent you a confirmation link. One click and you are in.
Not in your inbox? Look in spam or promotions. If it's there, mark it "Not spam" so future editions land in your inbox.