ConcludedRan 1 dayin the edition since 14 Sep 20263 sourcesupdated 14 Sep 2026

GPT-6 Astra's ARC-AGI-3 harness divergence

The gist

Latest: ARC Prize to Report Both Harness Results for GPT-6 Astra on ARC-AGI-3

The same model produces scores ranging from 54.8% to 99.9% on the same benchmark depending on evaluation scaffolding, which raises questions about what published benchmark numbers actually measure. ARC Prize's decision to label and publish both harness results gives readers a clearer basis for interpreting future scores.

When stories like this move, I write about them in an email each morning.

One free email each morning, around 8 AM US Eastern.

The full picture

ARC Prize evaluated GPT-6 Astra on ARC-AGI-3 under two harnesses and published the results on its blog, including cost figures and a policy change. Under the Standard harness at max reasoning effort, the model scored 62.7% at a cost of $26,098. Under the Provider Adapter harness at max reasoning effort, it scored 98.6% at $17,332. At high reasoning effort, the gap is wider: 99.9% under the Provider Adapter harness versus 54.8% under the Standard harness. The Standard harness provides a minimal, provider-neutral interface where the model manages its own notes; the Provider Adapter harness preserves opaque reasoning state between requests and uses compaction. ARC Prize stated it believes a future AGI should be able to solve ARC-AGI-3 under Standard harness conditions, and announced it will report both harness results on the leaderboard going forward, with each evaluation condition clearly labeled.

How it developed

First in the edition · 14 September 2026

8 September 2026in the 14 Sep edition

ARC Prize published a blog post on September 14 showing GPT-6 Astra scored 62.7% on ARC-AGI-3 under the Standard harness and 98.6% under the Provider Adapter harness at max reasoning effort, at costs of $26,098 and $17,332 respectively.

ARC Prize confirmed the gap stems from scaffolding rather than model weights, stated that a future AGI should solve ARC-AGI-3 under Standard harness conditions, and announced it will report both harness results on the leaderboard with each evaluation condition clearly labeled.

First citedRohan Paul on X The Neuron arcprize.org

3 September 2026

Initial reporting on GPT-6 Astra's divergent ARC-AGI-3 scores under different harnesses published

Sources

Rohan Paul on X