Towards Direct Evaluation of Harness Optimizers via...

Towards Direct Evaluation of Harness Optimizers via... | AI Research

Key Takeaways

The Challenge of Evaluating Agent Optimizers Automated agent creation relies on "harness optimization," where an optimizer agent iteratively updates the soft...
Harness optimization enables automated agent creation by having an optimizer agent iteratively update the harness of target agents.
Despite its success, current studies evaluate optimizers solely by observing target agents' performance gains.
This indirect end-improvement evaluation neglects optimizers' actions at intermediate steps, which are often erroneous and hinder agent performance.
Therefore, it is unclear whether harness optimization is driven by optimizers' informed update actions or simply trial-and-error.

Paper AbstractExpand

Harness optimization enables automated agent creation by having an optimizer agent iteratively update the harness of target agents. Despite its success, current studies evaluate optimizers solely by observing target agents' performance gains. This indirect end-improvement evaluation neglects optimizers' actions at intermediate steps, which are often erroneous and hinder agent performance. Therefore, it is unclear whether harness optimization is driven by optimizers' informed update actions or simply trial-and-error. This necessitates direct evaluation of harness optimizers. However, evaluating harness optimizers directly is non-trivial and costly due to the lack of oracle harnesses. To address this, we present a simple, low-cost design to directly evaluate them, namely priority ranking. By asking harness optimizers to rank components (e.g., tools) in a given harness by their potential to improve/hinder agent performance when updated, our design quantifies optimizer ability at the step level without expensive rollouts or manual examination. More importantly, optimizers' ranking performance correlates with their ability to improve agents in actual multi-step harness optimization, establishing priority ranking as a reliable predictor of optimization ability. Priority ranking is enabled by Shor, a collection of 182 human-verified optimization scenarios spanning across domains, designs, and time stages. Codes and data can be found at this https URL .

The Challenge of Evaluating Agent Optimizers

Automated agent creation relies on "harness optimization," where an optimizer agent iteratively updates the software layer—the harness—surrounding a target agent to improve its performance. Currently, these optimizers are evaluated like black boxes: researchers only look at whether the target agent performs better at the end of the entire process. This approach is problematic because it ignores the individual steps taken by the optimizer. Research shows that nearly half of these intermediate steps are actually detrimental to the agent, yet these errors often persist in the final design. Because current evaluation methods only measure end results, it remains unclear whether an optimizer is making smart, informed decisions or simply relying on trial-and-error.

Introducing Priority Ranking

To move beyond simple end-result evaluation, this paper introduces "priority ranking." Instead of waiting for a full, costly, and time-consuming optimization cycle to finish, this method asks the optimizer to rank the components of a harness (such as prompts, tools, memory, and workflow) based on their potential to improve or hinder agent performance. By focusing on these relative priorities, the evaluation can assess the optimizer's decision-making process at each individual step. This approach is significantly more efficient, as it avoids the need for expensive rollouts or manual human examination of every version of a harness.

The Shor Dataset

To make this evaluation possible, the authors created Shor, a collection of 182 human-verified optimization scenarios. Each scenario provides a snapshot of a harness at a specific point in its development, along with a "consensus ranking" of which components should be prioritized for updates. These rankings were determined by running various agent configurations and measuring the actual performance impact of updating each component, ensuring that the labels are grounded in real-world performance data.

Key Findings and Implications

The study demonstrates that an optimizer's performance in priority ranking is a reliable predictor of its actual ability to improve agents in multi-step optimization. Specifically, the ranking performance correlates with real-world success, with a correlation coefficient of 0.602. Furthermore, the priority ranking method is at least 8 times cheaper and 17 times faster than traditional evaluation techniques. Beyond serving as an evaluation tool, the researchers found that explicitly training optimizers to be aware of these priorities helps them better correct flawed harnesses, suggesting that this framework can also serve as a guide for building more effective, intelligent optimization agents.