Back to AI Research

AI Research

HarnessOpt-Bench: Evaluating LLMs at Harness Optimi... | AI Research

Key Takeaways

  • HarnessOpt-Bench is a new benchmark designed to measure how well Large Language Models (LLMs) can perform "harness optimization"—the process of iteratively i...
  • Yet the community lacks a common protocol for measuring how well frontier LLMs perform at this task.
  • We introduce HarnessOpt-Bench, a benchmark for end-to-end harness optimization under expensive and stochastic evaluation.
  • An optimizer, an LLM paired with a coding harness, receives a target agent's seed harness, graded evaluation feedback, and a fixed target-evaluation budget.
  • It edits the harness and nominates a final candidate, which is scored by its normalized gain over the seed on a held-out test partition that remains inaccessible throughout search.
Paper AbstractExpand

As LLMs are increasingly deployed within agentic systems, their capabilities depend not only on the model weights but also on the harness: the prompts, tools, control flow, memory, and orchestration code surrounding them. This makes automated harness optimization -- the iterative and evaluation-guided improvement of a harness by an AI system -- both an important route to improving AI systems and a demanding capability for AI systems themselves. Yet the community lacks a common protocol for measuring how well frontier LLMs perform at this task. We introduce HarnessOpt-Bench, a benchmark for end-to-end harness optimization under expensive and stochastic evaluation. An optimizer, an LLM paired with a coding harness, receives a target agent's seed harness, graded evaluation feedback, and a fixed target-evaluation budget. It edits the harness and nominates a final candidate, which is scored by its normalized gain over the seed on a held-out test partition that remains inaccessible throughout search. A trusted execution environment enforces the evaluation boundary, meters target-agent resource use, and preserves candidate versions for audit. We evaluate 5 frontier LLMs as optimizers both under a shared coding harness and under their native harnesses across 4 downstream tasks, over 111 scored runs. Experiment results show that optimizer models separate more than the coding harnesses they act through, native harnesses are not consistently superior, and gains vary substantially across tasks and seed regimes. These results establish harness optimization as a measurable and discriminative capability with large space for improvement.

HarnessOpt-Bench is a new benchmark designed to measure how well Large Language Models (LLMs) can perform "harness optimization"—the process of iteratively improving the prompts, tools, and control flow surrounding an AI agent. Because an agent’s performance depends heavily on this surrounding "harness," the researchers at Scale AI argue that automating its improvement is a critical, yet currently under-measured, capability for frontier AI systems.

The Challenge of Harness Optimization

Harness optimization is difficult because it requires more than just coding ability. It involves diagnosing failures in complex, stochastic systems where evaluation is expensive and noisy. Unlike standard code benchmarks where a test suite provides immediate, cheap feedback, evaluating a harness change requires running an agent over many cases. The researchers define this as a "long-horizon" task that requires reasoning-driven interpretation of a system to reach a specific performance metric.

How the Benchmark Works

HarnessOpt-Bench provides a controlled, reproducible environment to test optimizer models. The protocol follows these steps:

  • The Setup: An optimizer (an LLM paired with a coding harness) is given a target agent’s seed harness, a fixed budget for evaluation, and access to development and validation feedback.

  • The Boundary: A trusted execution environment enforces the evaluation boundary. It prevents the optimizer from accessing held-out test data, meters resource usage, and tracks every candidate version for audit.

  • The Goal: The optimizer must edit the harness and nominate a final candidate. This candidate is then scored on a held-out test partition that was inaccessible during the search process.

  • The Metric: Performance is measured by "normalized gain," which calculates the fraction of potential improvement captured by the optimizer over the original, untuned seed harness.

Key Findings

The researchers evaluated five frontier LLMs across four downstream tasks, resulting in 111 scored runs. Their findings include:

  • Model vs. Harness: The choice of the optimizer model has a greater impact on performance than the specific coding harness it uses. When comparing models using a shared coding harness, the differences between the models were larger than the differences observed when switching between a shared harness and a model’s "native" harness.

  • Native Harnesses: Using a model's native harness does not provide a consistent performance advantage over a shared, standardized coding harness.

  • Search Behavior: Models that explored a broader range of "harness levers"—such as modifying different components of the agent's structure—tended to achieve higher gains. Interestingly, the researchers found that detailed inspection of failure traces was rarely used by the models and did not correlate with better outcomes.

Limitations and Scope

The researchers note that while HarnessOpt-Bench is currently implemented using coding agents as end-to-end optimizers, the benchmark is agnostic to the specific design of the optimizer. Any system that can operate within the defined budget and edit the harness is eligible for evaluation. The benchmark is specifically designed for tasks where evaluation is costly and noisy, rather than tasks where scoring is cheap, as the researchers believe this regime is more representative of the challenges involved in building complex, real-world AI systems.

Comments (0)

No comments yet

Be the first to share your thoughts!