Agent harness variance in LLM benchmarks

A paper on arXiv argues that LLM agent benchmark scores cannot validly compare models unless the execution harness is fully disclosed. Testing 3 models against 3 harnesses on 100 SWE-bench Verified tasks, the study found harness-induced variance averaged 7.80 times model-induced variance. Switching harnesses while holding the model fixed shifted pass@1 by 8.5 to 13.0 percentage points; switching models while holding the harness fixed changed pass@1 by only 2.5 to 5.0 percentage points. Model rankings were unstable: 6 of 9 model-pair/harness-pair comparisons reversed when the harness changed. The paper proposes a structured 'Harness Card' disclosure format organized around a seven-layer ETCSOVG taxonomy, and argues valid cross-model comparisons require either a locked-harness or factorial protocol. A separate paper introduces JIT-Agent, which generates agent harnesses dynamically per task; across 18 matched backbone-benchmark pairs, every JIT-generated harness improved the underlying model's performance, with DeepSeek-V4-Flash scoring 85.1 on DeepSearchQA against GPT-5's 76.0, at 36% lower average cost than the cheapest fixed harness alternative.