The Information Machine
FollowingDay 847 sourcesupdated 17 Nov 2026

Opus 5.5 Beats Human Baselines on Singularity-Bench; End-to-End Gaps Remain

The gist

These benchmarks reveal a consistent gap between partial task progress and full end-to-end delivery across ML engineering and scientific workflows. Singularity-Bench's finding that most agents scored near zero, combined with the authors' acknowledgment that the benchmark does not test the full recursive self-improvement loop, sets a more cautious frame around headline scores.

When stories like this move, I write about them in an email each morning.

One free email each morning, around 8 AM US Eastern.

The full picture

Four benchmarks now measure AI agents on complex ML engineering and scientific workflow tasks, with results that differ sharply by task type. On Singularity-Bench, a six-task Carnegie Mellon benchmark that scores agents relative to production systems, Claude Opus 5.5 paired with Claude Code achieved a Singularity-Index of 1.25, above the human baseline of 1.0, while the other ten evaluated agents averaged 0.58. Its serving engine earned $11.52 per GPU-hour versus $8.18 for SGLang, though six of eleven engines across all agents scored zero. The benchmark's authors acknowledge it is a proxy for the component skills recursive self-improvement would require, and that it does not test the full autonomous loop of an agent building and then improving a successor end to end. The benchmark includes a safety framework called Breakwater to guard against reward-hacking attacks such as prompt-based data exfiltration. FrontierML tests agents on 21 long-horizon ML tasks given a 12-hour CPU sandbox and spending budgets of $10, $50, or $200; Opus 5.5 reached a 45.4-point boost at $10 and 58.8 at $200 against human baselines of 56 and 66.1. In 28% of trajectories, agents added newly trained components to pretrained models or trained a model from scratch. SWE-CoWork, from TokenWave AI, places agents in a simulated workplace where 82% of requirements must be gathered from 26 simulated teammates rather than a written spec. Across 101 tasks and 101 open-source repositories, top models passed about 63% of individual checks but fully completed fewer than 8% of tasks; Opus 5.5 tops the leaderboard with 8 of 101 fully completed. More than half of Opus 5.5's failed checks occurred after the agent already had the requirements, pointing to code translation as the main bottleneck rather than information gathering. Giving GPT-6 Astra the full spec upfront raised its passed checks by 16 points yet still left it completing only 8 of 101 tasks. A fourth benchmark, FrontierChallenge, evaluated 12 frontier models on 97 cross-domain scientific workflow tasks; the best-performing configurations completed 20 of 97 tasks, a 20.6% pass rate, with models frequently claiming completion without satisfying the full-completion criterion.

How it developed
17 November 2026

SWE-CoWork released; Opus 5.5 tops leaderboard completing 8 of 101 tasks

16 November 2026

FrontierML results reported: Opus 5.5 approaches human-researcher returns at $200 budget

13 November 2026

Singularity-Bench coverage notes Opus 5.5 serving engine outperforms SGLang by ~41%

12 November 2026

Singularity-Bench paper published; Opus 5.5 scores 1.25 against human baseline of 1.0

27 August 2026

FrontierChallenge published: best agents complete 20 of 97 scientific workflow tasks

Sources
2 more sources

Get the daily email

When stories like this move, I write about them in an email each morning.

One free email each morning, around 8 AM US Eastern.

See the latest edition · Privacy