Back to AI Research

AI Research

Terminal-Bench maintenance study shows why benchmark scores need version context

Key Takeaways

  • An audit reports task repairs and resource-budget effects that change agent scores without replacing the underlying models.
  • ## Benchmarks accumulate maintenance debt
  • [Benchmarks as Software](https://arxiv.org/abs/2611.08417) argues that agent evaluations need ongoing validation after release.
  • A model may discover a valid solution an old verifier rejects, while changes in dependencies or infrastructure can break a previously sound task.
  • The researchers apply a continuous-validation workflow to Terminal-Bench 2.0, combining community reports, task artifacts, agent traces, oracle checks and human-gated repairs.

Benchmarks accumulate maintenance debt

Benchmarks as Software argues that agent evaluations need ongoing validation after release. A model may discover a valid solution an old verifier rejects, while changes in dependencies or infrastructure can break a previously sound task.
The researchers apply a continuous-validation workflow to Terminal-Bench 2.0, combining community reports, task artifacts, agent traces, oracle checks and human-gated repairs. They report issues in roughly 30% of the benchmark and release repairs as Terminal-Bench 2.1.

Repairs change the measured result

The paper reports accuracy increases of more than 10 percentage points and leaderboard movements of up to three ranks after the repairs. The overall ranking remains strongly correlated, with a reported Spearman correlation of 0.958.
These observations can coexist: a benchmark may preserve much of the broad ordering while materially changing an individual score or close comparison. The version of the tasks and verifiers is therefore part of what a benchmark number means.

Time and token budgets belong in the specification

The authors also examine resource allocation. Extending time limits closes a reported 20-percentage-point harness gap for GPT 5.4 and reduces another gap for GPT 5.4 mini from 30 to five percentage points.
Franklin’s analysis is that a harness comparison cannot be separated from its resource budget. A timeout may reflect a measurement constraint as well as an agent’s ability to solve the task.
When reading a coding-agent leaderboard, record the benchmark version, harness, time allowance and output-token policy together. The preprint does not imply that every improved score is a flawed test; it provides evidence that maintenance and evaluation settings can alter conclusions and should be documented alongside model changes.

Comments