How Do Agent Harnesses Create Value? Planning Infor... | AI Research

How Do Agent Harnesses Create Value? Planning Information and Release Control in Stateful LLM Agents
This paper investigates how the "harnesses"—the surrounding systems that organize an AI agent’s work—actually contribute to its success. While much research focuses on the models themselves, this study breaks down the specific impact of planning guidance and terminal verification on agent performance. By testing these components across retail and airline tasks, the authors aim to determine which parts of an agent's support system provide the most value and how they affect the balance between successful task completion and the risk of accepting incorrect work. The ai search story also surfaces in EU Regulators Demand Apple and Google..., adding another angle.

Isolating the Value of Planning

To understand if providing a plan actually helps an agent, the researchers compared "Fixed" plans (prewritten, task-specific instructions) against "Sham" plans (shuffled text that matches the word count and structure of the real plans). This method isolates the effect of the actual planning content from the mere presence of extra text. The study found that Fixed plans improved success rates by 7.17 percentage points, with the most significant gains appearing in more complex tasks. Interestingly, the shuffled "Sham" text did not harm performance, suggesting that the specific guidance provided in the Fixed plans is a key driver of success.

The Role of Terminal Verification

The researchers also examined "release control," or the use of a terminal verifier to check an agent's work before it is finalized. In the retail experiments, this verifier rejected 61% of incorrect episodes, though it also mistakenly withheld 17% of correct ones. Despite these occasional false rejections, the verifier significantly reduced the overall rate of "false passes" (incorrect work accepted as complete). Notably, a standalone verifier was found to capture nearly all the benefits of a full, complex support stack at only a fraction of the cost. The ai search story also surfaces in OpenAI Says AI Found Possible Navier–Stokes..., adding another angle.

Balancing Risk and Cost

The value of these harness components depends heavily on the goals of the deployment. The authors propose a framework to weigh the benefits of success against the "liability" of accepting an incorrect result. When the cost of an error is low, the performance boost from planning guidance is the most valuable factor. However, when the cost of an error is high, the verifier becomes the most important component. By analyzing these trade-offs, the study suggests that organizations can optimize their agent systems by choosing the right support mechanisms based on their specific tolerance for risk and their budget for execution costs. The ai search story also surfaces in House Intelligence Committee Warns AI Safeguards..., adding another angle. as detailed in the full paper on Arxiv