The Information Machine

This story was merged into AI agent long-horizon performance collapse →

Merged·New·in the edition since 2 Sep 2026·2 sources·updated 2 Sep 2026

Agent harness variance in LLM benchmarks

The gist

Agent harness drives 7.8x more SWE-bench variance than model choice

Published LLM agent benchmark comparisons may be misleading if different models were tested with different harnesses, making it unclear whether performance differences reflect model capability or harness design. The Harness Card proposal, if adopted, would require researchers and labs to disclose the full execution environment alongside any benchmark scores.

The full picture

A paper titled 'Stop Comparing LLM Agents Without Disclosing the Harness' reports that in a 3-model by 3-harness study on 100 SWE-bench Verified tasks, the execution harness drove on average 7.80 times more performance variance than the model itself. Switching harnesses while holding the model fixed shifted pass@1 by 8.5 to 13.0 percentage points; switching models while holding the harness fixed changed pass@1 by only 2.5 to 5.0 percentage points. Model rankings were unstable: 6 of 9 model-pair/harness-pair comparisons reversed when the harness changed. The paper proposes a 'Harness Card,' a structured disclosure format organized around a seven-layer ETCSOVG taxonomy covering Execution, Tool, Context, Scheduling, Observation, Verbosity, and Grading, and states that valid comparisons require either a locked-harness protocol or a factorial protocol with reported variance components. The disclosure requirement covers infrastructure responsible for context construction and tool-call mediation.

How it developed
2 September 2026

Two papers posted to arXiv on September 1 and 2, 2026, find that agent execution harnesses dominate model choice in LLM benchmark results.

The first, studying 3 models across 3 harnesses on 100 SWE-bench Verified tasks, found harness-induced variance 7.80 times model-induced variance, with 6 of 9 model rankings reversing on harness swaps, and proposes a 'Harness Card' disclosure format. The second introduces JIT-Agent, which generates harnesses per task; every JIT harness improved performance across 18 backbone-benchmark pairs, with DeepSeek-V4-Flash scoring 85.1 on DeepSearchQA against GPT-5's 76.0 at 36% lower cost.

First citedRohan Paul Twitterarxiv.org
First in the edition · 2 September 2026
31 August 2026

Yuj coding-agent context management study published, showing harness effects raise complete solutions from 43 to 72 at constrained context windows

28 August 2026

PRAXIST Beta released, achieving 49 gold medals on MLE-bench vs. Claude Opus 4.8's 34

27 August 2026

PwC analysis published recommending fleet-wide harness standardization for auditability

26 August 2026

Cambridge and Hong Kong University paper on autonomous agent research direction choice published, arguing fixed scaffolding costs capability

Sources
Rohan Paul Twitter
arxiv.org

Related

The daily email

Want this in your inbox?

I send one email each morning with the stories that moved. If you would rather just read here, that works too.

Subscribe free