ExecCritic: Learn to Test, Test to Improve for Codi... | AI Research

ExecCritic: Learn to Test, Test to Improve for Coding Agents
Coding agents often struggle to fix bugs in large software repositories because they lack a reliable way to verify their work. When an agent is responsible for both writing a patch and creating the test to validate it, it may inadvertently create a "false confidence" loop: the patch and the test might share the same misunderstanding of the bug, causing the code to pass a flawed test while failing to actually resolve the issue. ExecCritic addresses this by decoupling the testing process from the repair process, using specialized agents trained to generate high-quality, independent tests and perform feedback-guided code revisions. To see openai in practice, Gemini's now Generates Files! walks through a concrete example.

A Two-Stage Scaffold for Repair

The core of ExecCritic is a "test–verify–revise" framework that separates responsibilities. First, a dedicated Test agent explores the repository to generate a "test bundle"—a set of repository-native tests, execution commands, and behavioral contracts. A fail-closed harness then qualifies these tests, ensuring they fail on the buggy code and pass on the correct version. Once qualified, the test is "frozen," meaning the Repair agent cannot modify it. The Repair agent then attempts to fix the source code, receiving execution feedback from the fixed test. This structure prevents the Repair agent from "cheating" by weakening the test to make its own flawed code pass.

Role-Specific Training

ExecCritic uses a reinforcement learning recipe to train the Test and Repair agents separately, ensuring each develops the specific skills required for its role. The Test agent is trained to produce tests that are behaviorally valid and capable of distinguishing between correct and incorrect patches. Meanwhile, the Repair agent is trained to handle both the initial task of writing a patch and the iterative process of revising that patch based on the feedback provided by the fixed test. By training these roles independently, the system ensures that the feedback provided to the Repair agent is both reliable and diagnostic. The ai agents story also surfaces in OpenAI Unveils GPT-Red an Automated Model..., adding another angle.

Performance Gains

Experiments on the SWE-bench Verified benchmark demonstrate that the quality of generated tests is critical to an agent's success. When using lower-quality tests, an agent’s performance can actually decrease compared to a baseline with no tests at all. However, by using the trained ExecCritic agents, the system achieved a 72.6% resolution rate. This represents an 11.4-point improvement over the original no-test baseline. These results highlight that simply adding execution feedback is not enough; the system must be specifically trained to construct meaningful behavioral targets that accurately reflect the requirements of the issue.

Key Considerations

While ExecCritic significantly improves performance, it is important to note that the system's effectiveness relies on the harness's ability to qualify tests. The harness ensures that tests are executable and fail on the buggy repository, but it does not guarantee that a test is perfectly aligned with the underlying issue. Furthermore, the system is designed to operate without relying on stronger models or external "Oracle" feedback during the evaluation phase, proving that agents can learn to self-correct through structured, independent verification. The ai agents story also surfaces in Alibaba Releases Page Agent to Control..., adding another angle. as detailed in the full paper on Arxiv