Agent harnesses and skill distillation drive large AI benchmark gains
Benchmark results that were previously attributed to model capability can be substantially explained by harness and skill-library design, which changes how progress in AI agents should be measured and interpreted. The DisCo and Prime Agent results show that reusable skill storage and state management can produce large performance jumps without changing the underlying model.
Several systems show that the infrastructure around a language model, rather than the model itself, is a primary driver of benchmark performance. NVIDIA's AVO agent system completed all 183 public ARC-AGI-3 levels, with the result framed as showing how much the tooling harness can matter on long-running tasks. Prime Agent, a self-improving harness built around Claude Opus 5, reached 95.5% on ARC-AGI-3 and published a technical report on arXiv describing its architecture: a memory hierarchy spanning model weights, active context, a persistent IPython session, and a disk-backed store of histories, skills, and prompts. Rather than allowing state to be compacted away by the context window, the agent moves state between memory levels with code, and the report frames long-context work as 'an information-management problem rather than a reading problem.' A separate system called DisCo takes a complementary approach: it distills skills from GitHub repositories and papers into a reusable library that tells agents what tool to use, when to use it, and how to handle failures. Using DisCo with GPT-5.5 and Codex, MLE-bench scores rose from 31.11% to 72.89% with no change in task-running budget; DisCo's skill library contains 5,353 skills drawn from 1,000 ML repositories. Harness-Bench, a benchmark introduced by researchers and available on alphaXiv, makes harness design a first-class variable in agent evaluation. It finds that stronger models show lower variance across harness configurations, a pattern the authors call 'harness dependence,' while weaker models show high variance depending on how tools and prompts are structured. Models including GPT-5.4 and Claude-Sonnet-4.6 are described as 'more resilient to sub-optimal harness designs.'
ARC Prize retested GPT-6 Astra on September 8 using a stripped-down harness, dropping the model from 99.9% to 62.7% on ARC-AGI-3.
Three papers on September 9 extended the finding: Zhang et al. measured harness-induced variance at 7.8x model-induced variance; ByteDance's HarnessDev found Opus-generated harnesses dropped from 69.3 to 33.0 on SWE-Pro when Gemini executed them; and Amazon-Microsoft's SPACE raised ScienceWorld from 35.9% to 67.2% by training agents to batch safe sequential actions.
DisCo skill-distillation system reported; MLE-bench scores rose from 31.11% to 72.89% using a 5,353-skill library.
GPT-6 Astra scored 99% on ARC-AGI-3 under OpenAI's native harness on September 3, saturating the benchmark and surpassing human performance on 96% of its levels.
The same model scored 62.7% under ARC Prize's provider-neutral harness, with ARC Prize attributing the gap to how the evaluation wrapper manages memory and long-context state rather than any change in model weights. ARC Prize characterized the two harnesses as answering different questions: cross-provider comparability versus best-case provider-native performance.
JIT-Agent paper shows smaller models can outperform larger ones with task-specific harnesses
A controlled SWE-bench study published September 1 found harness-induced variance averaged 7.80x model-induced variance, with model rankings reversing in 6 of 9 pairings when the harness changed, and called for mandatory disclosure.
The Yuj harness study found context management alone raised F2PF from 28% to 49% across models, while ACES found skill document quality near-zero correlated with actual runtime benefit. Apodex 1.1 reported gains from externalizing task state, and AutoSaddler showed unvalidated harness patching can underperform the original.
WeaveBench, a 114-task benchmark of hybrid GUI-CLI long-horizon work, emerged August 30 showing a best pass rate of 41.2%, against the near-complete ARC-AGI-3 scores NVIDIA AVO and Prime Agent posted days earlier.
LongHorizon-Harness, described alongside it, frames the gap as a task-state problem, where task history becomes unreliable about what is finished, failed, or remaining, and argues that reliability over long task spans depends as much on the harness as on the model.
PRAXIST Beta released; 49 gold medals on MLE-bench vs Claude Opus 4.8's 34
Databricks shows frontier vision agent fails chart count that structured extraction solves
NVIDIA's AVO completed all 183 public ARC-AGI-3 levels, and Prime Agent, paired with Claude Opus 5, scored 95.5% and published an arXiv report on its architecture, both dated August 23-25.
Prime Agent's report describes a memory hierarchy where 'the model moves state between those levels with code instead of having it compacted away,' so long-context work becomes an information-management problem. Both results are framed as showing that agent scaffolding, not raw model capability, is the decisive factor.
DataSpace benchmark published, harness shift from 30.98% to 46.34% accuracy reported
Task-CoEvolve benchmarking cost-reduction paper published
HCL framework and 'harness-level forgetting' concept published
I send one email each morning with the stories that moved. If you would rather just read here, that works too.
Subscribe free