Back to AI Research

AI Research

Verify Smarter, Evolve Further: Efficient Harness E... | AI Research

Key Takeaways

  • HarnessLens is a framework designed to automate the evolution of language-model agent harnesses—the instructions, tools, and runtime components that dictate...
  • Agent harnesses shape how language-model agents use instructions, tools, and runtime components, but adapting these harnesses requires costly verification.
  • Existing propose-and-verify methods typically score every candidate on a fixed task set, wasting rollouts on unrelated behaviors and allowing aggregate scores to obscure specific regressions.
  • We introduce HarnessLens, a budget-aware framework for automated harness evolution.
  • Across three agent harnesses and four benchmarks, HarnessLens improves average held-out performance by 7.6-13.6% while consuming substantially less evaluation budget than competing baselines.
Paper AbstractExpand

Agent harnesses shape how language-model agents use instructions, tools, and runtime components, but adapting these harnesses requires costly verification. Existing propose-and-verify methods typically score every candidate on a fixed task set, wasting rollouts on unrelated behaviors and allowing aggregate scores to obscure specific regressions. We introduce HarnessLens, a budget-aware framework for automated harness evolution. HarnessLens jointly explores the task space and user-configurable components, derives candidate modifications from execution trajectories, and selectively verifies each candidate on behavior-relevant tasks using an attributable-evidence gate. Across three agent harnesses and four benchmarks, HarnessLens improves average held-out performance by 7.6-13.6% while consuming substantially less evaluation budget than competing baselines. These results demonstrate that behavior-aware verification with explicit attribution enables more reliable and sample-efficient harness evolution under constrained interaction budgets. Our code is available at this https URL .

HarnessLens is a framework designed to automate the evolution of language-model agent harnesses—the instructions, tools, and runtime components that dictate how an agent performs tasks. By replacing traditional, resource-heavy verification methods with a behavior-aware approach, HarnessLens improves agent performance while significantly reducing the number of task rollouts required during the evolution process.

The Problem with Current Verification

Existing methods for improving agent harnesses typically use a "propose-and-verify" approach, where candidate modifications are tested against a fixed set of tasks. The authors, Jinghan Xu et al., note that this is inefficient because many tasks in a fixed set are irrelevant to the specific behavior a modification is intended to change. This wastes computational budget on unrelated rollouts and can mask whether a modification actually achieved its goal or introduced unintended regressions.

How HarnessLens Works

HarnessLens operates through three distinct stages to ensure that verification is targeted and evidence-based:

  • Context Exploration: The framework characterizes the available tasks and identifies which components of the harness are user-configurable. This creates a map of the agent's environment without executing any tasks.

  • Trajectory Diagnosis: Instead of relying solely on binary success or failure rewards, the system analyzes execution trajectories to extract reusable experiences and identify recurring deficiencies. This evidence is linked to specific agent behaviors.

  • Harness Evolution: Using the diagnosed evidence, the framework proposes modifications and selects a specific, relevant subset of tasks for verification. It then reviews the results to confirm that the change improved the intended behavior without causing regressions in other areas.

Performance and Efficiency

In experiments across three agent harnesses (OpenCode, Codex, and Pi) and four benchmarks (including Terminal-Bench 2.0 and BIRD Mini-Dev), HarnessLens outperformed existing methods. It achieved average performance improvements of 7.6% to 13.6% on held-out tasks. Notably, the framework achieved these results while operating under a strict budget of 200 total units (rollouts and sessions), which is substantially lower than the budgets used by baseline methods like Self-Harness or Meta-Harness.

Key Considerations

The effectiveness of HarnessLens relies on its ability to isolate and attribute behavioral changes to specific modifications. Because the framework is designed to be harness-independent, it can be applied to various agent configurations. However, the authors note that the process is constrained by a fixed interaction budget, meaning the system must prioritize the most promising proposals and rely on the quality of the evidence extracted during the diagnosis phase to make informed decisions about which modifications to adopt.

Comments (0)

No comments yet

Be the first to share your thoughts!