An Empirical Study of Harness Design for Coding Agents | AI Research

This paper investigates how the individual components of a coding harness—the software layer that allows AI agents to perform software engineering tasks—impact performance. While many existing studies evaluate these harnesses as "all-in-one" systems, it remains unclear which specific parts are responsible for an agent's success or failure. By building a modular harness with a fixed execution loop, the researchers systematically varied three core components—planning, action space, and context management—to determine how they influence agent behavior across different models, task types, and resource budgets.

Dissecting the Coding Harness

To isolate the effects of specific design choices, the researchers created a flexible framework that allows for component-level testing. The harness follows a "ReAct" loop, where the agent reasons, takes an action, and receives an observation. The study focused on three areas:

• Planning: An explicit, persistent task plan that the model can update.

• Action Space: A choice between a set of predefined tools (such as file editors and search functions) or a "bash-only" interface.

• Context Management: Five different strategies for handling long interaction histories, ranging from no management to combinations of elision (replacing old data with stubs), external storage (making elided data recoverable), and LLM-based summarization. The same large language models question is explored in MeClear, which adds a research perspective.

The Role of Context Management

The study found that context management is most critical when the available "context window" (the amount of information the model can "see" at once) is tight. Its primary benefit is preventing the agent from crashing due to context overflow, which allows the agent to continue working toward code verification. Interestingly, the researchers discovered that making elided content recoverable through external storage is rarely used by models and provides no meaningful accuracy gain. Instead, a staged approach—where rule-based elision is performed before LLM summarization—offers the best balance of efficiency and performance.

Planning and Tooling Trade-offs

The impact of planning and action interfaces depends heavily on the capability of the model being used:

• Planning: For weaker models, planning acts as a necessary scaffold that keeps the agent on track, increasing success rates. For stronger models, however, planning serves primarily as a cost-saving measure by reducing redundant verification steps, with little impact on overall accuracy.

• Action Space: Predefined tools are highly beneficial for models that struggle with bash commands. Conversely, models that are already proficient with bash can operate effectively using only a shell interface, which significantly lowers the cost of completing tasks. The same ai evaluation question is explored in Compositional Policy Violations, which adds a research perspective.

Key Takeaways for Future Design

The researchers conclude that there is no "one-size-fits-all" design for coding agents. Instead, the effectiveness of a harness is deeply tied to the model's inherent capabilities and the specific constraints of the task. By analyzing trajectories, the team observed that context management extends the duration of a task without changing the agent's core behavior, while planning changes where an agent decides to stop, and the action space dictates the granularity of the code being written. These findings provide a modular framework for developers to design more efficient, model-aware coding agents. The same ai evaluation question is explored in Procedural Graphs, which adds a research perspective. as detailed in the full paper on Arxiv