$ cat wiki/concepts/eval-harness-configuration.md
Eval Harness Configuration
Definition
The harness is the scaffolding around a model during a benchmark run: how context is carried between steps, whether reasoning traces survive a turn boundary, how tools are exposed, how many attempts are allowed, how results are parsed. It is not part of the model and not part of the benchmark's task set — and on agentic benchmarks it can move the reported score by more than the gap between two model generations.
A benchmark number is therefore a claim about a (model, harness) pair, never about a model alone.
Why It Matters
On multi-step agentic tasks the harness decides what the model remembers. Discard the chain of thought after each action and the model rebuilds its state every turn; preserve it and the same weights behave like a different system.
The 2026-07-29 ARC-AGI-3 episode is the clean demonstration (source):
| Configuration | GPT-5.6 Sol |
|---|---|
| ARC Prize official harness (ARC Prize verified) | 7.8% |
| Official harness, as re-run and reported by OpenAI | 13.3% |
| Responses API + retained reasoning + compaction | 38.3% |
| Same model, same task set, a 4.9× spread from harness settings alone. For scale, the | |
| verified SOTA that Claude Opus 5 set on 2026-07-27 was 30.2%, and the average | |
| human tester scores 48%. |
- Retained reasoning preserves the chain of thought between steps.
- Compaction summarizes older context instead of truncating it.
Neither was built for ARC-AGI-3; both are general Responses API settings available to any API user, and OpenAI reports they also cut output tokens 6×.
The comparability problem
Two things are true at once and the tension between them is the whole subject:
- A setting available to all users, not built for the benchmark, is a legitimate way to run a model. ARC Prize's reported position is that such settings are fair game if properly reported, and co-founder François Chollet is reported as agreeing (source).
- Official leaderboard scores use one standardized harness without provider-specific settings, precisely so that numbers from different labs remain like-for-like.
So OpenAI's 38.3% and Anthropic's 30.2% cannot be compared. Nothing published states whether Claude Opus 5 was measured with any equivalent state-preserving configuration, and without that the higher number is not evidence of a better model (source).
Open Problems
- No convention for reporting harness configuration. A score is published as a scalar; the configuration that produced it usually is not.
- Provider-specific settings have no neutral equivalent. Retained reasoning is a Responses API feature. There is no vendor-independent way to grant a competing model the same state-preservation, so "run everyone with their best settings" and "run everyone the same way" are both defensible and give different rankings.
- The gap grows with task length. Harness effects are small on single-turn benchmarks and large on long-horizon agentic ones — which is the direction evals are moving.
- Self-reported runs are unverified by default. ARC Prize verification is a separate step; most published agentic numbers never get one.
Practical consequence for this wiki
Benchmark rows on model pages record the figure with the harness or source that produced it
where that is known, and a disputed figure goes to ## Conflicting Reports rather than being
silently reconciled. The two "official harness" numbers for GPT-5.6 Sol — 7.8% verified by ARC
Prize, 13.3% as re-run by OpenAI — are recorded as disagreeing because nothing read reconciles
them.
The same discipline applies to the leaderboard snapshots this repo scrapes: LMArena is quoted as a percentage with its confidence interval rather than as Elo, and Artificial Analysis figures are quoted with the column heading above them (source).
Key Sources
- OpenAI, "How enabling two settings tripled our scores on the ARC-AGI-3 benchmark" (2026-07-29) (source)
- ARC Prize results — Claude Opus 5
- ARC Prize, analyzing GPT-5.5 & Opus 4.7 with ARC-AGI-3
- LMArena snapshot (source)
- Artificial Analysis snapshot (source)