Harness-of-Harness: Multi-Day Autonomous Software D... | AI Research

Harness-of-Harness (HoH) is a framework designed to enable AI coding agents to build complex software systems from scratch without human intervention. While many existing coding agents require constant human guidance to manage tasks and fix errors, HoH allows agents to work autonomously over long periods. By organizing development into iterative loops of planning, coding, and testing, the framework ensures that the software consistently improves, remains functional, and avoids the common pitfalls of long-term autonomous projects, such as losing track of original requirements or introducing conflicting code.

How the Framework Works

HoH operates by breaking down the development process into a repeating three-step cycle. First, a Project Planner reviews the high-level requirements and the current state of the project to define a small, manageable goal for the next iteration. Second, a Developer implements that specific goal, building upon the existing codebase. Finally, a QA Tester independently evaluates the new changes to ensure they work as intended and do not break previously finished parts of the system.
Crucially, HoH maintains two types of records between these loops: the "artifact state" (the actual code and files) and the "evidence state" (a history of what has been tested, what failed, and what was successfully validated). This ensures that the agents always know what has already been achieved, preventing them from repeating work or forgetting previous design decisions.

Maintaining Continuity

A major challenge in autonomous development is that AI agents often struggle to maintain focus over long trajectories. To solve this, HoH uses "progressive disclosure." Instead of trying to keep the entire project history in the AI’s active memory, the framework stores project documents, histories, and reports in a file system. These are indexed and only retrieved when the agent specifically needs them. This allows the system to scale to complex projects—such as building a full-featured game—without overwhelming the AI’s context window or losing track of the project's evolution.

Performance and Results

The researchers tested HoH across three established coding benchmarks and a real-world, multi-day project. In the benchmarks, HoH consistently outperformed standalone coding agents, achieving an average relative performance gain of over 52%. In a more practical test, the framework was tasked with building a first-person shooter game from scratch. Over the course of more than 70 iterations, the system autonomously developed a playable game that included a coherent storyline, combat mechanics, a heads-up display, and polished audio and visual elements, demonstrating the framework's ability to handle complex, long-horizon software engineering tasks.

Key Design Principles

HoH succeeds by enforcing a strict separation of roles while allowing the agents autonomy in how they achieve their goals. By constraining the "what" (the required output and verifiable goals) rather than the "how" (the specific workflow or tools), the framework remains flexible. It encourages the reuse of existing tools and skills rather than forcing the agent to recreate standard engineering solutions from scratch. This approach ensures that the development process remains structured and verifiable, even as the software grows in complexity.