StarHarness: Evolving Harnesses with Stratified Sea... | AI Research

StarHarness is a framework designed to improve the performance of Large Language Model (LLM) agents in enterprise environments by evolving the "harness"—the executable code and configuration surrounding the model—without changing the model's internal weights. By optimizing how an agent interacts with tools, interprets state, and manages task workflows, the framework addresses the persistent mismatch between general-purpose models and the specific, stateful requirements of enterprise software.

Evolving the Agent Harness

The StarHarness approach treats the agent’s harness as an editable software component. The system uses a "proposer" (an LLM) to generate candidate patches to the harness, which may include changes to prompt framing, tool interfaces, subagent structures, and execution policies.
To ensure these changes are effective and generalizable, the framework employs a stratified search protocol:

• Task Stratification: Tasks are categorized based on baseline failure modes (such as context loss or incorrect tool usage), scores, and verifier pass rates.

• Separation of Concerns: The system splits tasks into a "search" set (visible to the proposer), a "selection" set (used to validate improvements), and a "holdout" set (reserved for final evaluation to test generalization).

• Evolution Loop: The system iteratively proposes, validates, and tests changes. Only modifications that improve performance on the selection set are committed to the harness, ensuring that the agent learns environment-specific conventions rather than memorizing individual task solutions.

Performance and Transferability

In experiments across three enterprise benchmarks—ITBench SRE, EnterpriseOps-Gym ITSM, and AutomationBench Finance—StarHarness improved full-benchmark performance by 20–35 percentage points compared to default harnesses.
The research indicates that these improvements are not limited to the model used during the evolution process. When the evolved harnesses were applied to different model families, including GPT and Qwen, they consistently improved performance without requiring further re-evolution. Additionally, trace analysis showed that the evolved harnesses reduced inference costs and shortened agent trajectories by automating repetitive reasoning steps and repairing interface-level friction.

Learned Specialization

The study identifies three primary ways the evolved harnesses improve agent behavior:

• Interface Repair: The framework optimizes how agents handle tool arguments and schemas, stripping away unnecessary or misleading data that often causes errors in strict enterprise environments.

• Environment Conventions: The harness encodes implicit operational rules—such as specific database update requirements or triage procedures—directly into the agent’s execution logic.

• Search Compression: By delegating deterministic tasks (like date calculations or forensics overviews) to the harness, the agent can bypass open-ended reasoning for routine operations, leading to fewer false-positive diagnoses and more efficient task completion.

Considerations for Deployment

StarHarness relies on guardrails to prevent the agent from overfitting to specific tasks. The system explicitly forbids the proposer from accessing ground-truth labels, hard-coding answers, or including verifier content in the agent's prompts. While the framework demonstrates significant gains in performance and efficiency, the authors note that the effectiveness of the system depends on the quality of the baseline benchmarks and the ability of the proposer to identify meaningful, reusable environment patterns. The resulting edits are implemented as standard code changes, allowing them to be tested, audited, or reverted as needed.