Back to AI Research

AI Research

CodeRescue: Budget-Calibrated Recovery Routing for... | AI Research

Key Takeaways

  • CodeRescue addresses the challenge of managing costs when coding agents encounter errors.
  • Coding agents increasingly operate in executable environments where a failed attempt produces actionable feedback rather than merely an incorrect answer.
  • Existing cost-aware systems typically treat such failures as cascade decisions: try a cheap model first, then escalate hard cases to a stronger and more expensive model.
  • In coding, however, execution feedback can also make further cheap-model recovery worthwhile, raising a budgeted deployment question: when should an agent spend more cheap compute, and when should it escalate?
  • We formulate this post-failure decision as recovery routing over heterogeneous actions and train a supervised router from execution rollouts.
Paper AbstractExpand

Coding agents increasingly operate in executable environments where a failed attempt produces actionable feedback rather than merely an incorrect answer. Existing cost-aware systems typically treat such failures as cascade decisions: try a cheap model first, then escalate hard cases to a stronger and more expensive model. In coding, however, execution feedback can also make further cheap-model recovery worthwhile, raising a budgeted deployment question: when should an agent spend more cheap compute, and when should it escalate? We formulate this post-failure decision as recovery routing over heterogeneous actions and train a supervised router from execution rollouts. To make the same router usable under changing budgets, we add a Conformal Risk Control (CRC) layer that selects a deployment-time cost penalty without retraining and provides marginal expected-cost control under exchangeability. Across held-out failures from five coding benchmarks, cheap recovery and escalation exhibit complementary success patterns. The calibrated frontier improves over fixed actions, prompt-only routers, and a binary cascade baseline; in the main GPT-5.4-nano/GPT-5.4 setting, one CRC-calibrated frontier point exceeds always-escalate solve rate while using 35% of its mean recovery cost. Code is available at this https URL .

CodeRescue addresses the challenge of managing costs when coding agents encounter errors. In modern software development, agents often work in environments where they can run code and receive feedback, such as compiler errors or failed tests. When an initial attempt fails, the agent must decide how to proceed: should it try to fix the code itself, start over with a new plan, or pay for a more powerful model to solve the problem? CodeRescue provides a systematic way to make these "recovery" decisions while staying within a specific budget.

Routing Recovery Actions

Rather than simply choosing between a cheap model and an expensive one, CodeRescue treats failure as a decision point for three distinct recovery actions: "reflect," "replan," and "escalate." Reflecting involves using the error feedback to patch the existing code, while replanning involves having the cheap model try a completely different approach. Escalating, the most expensive option, involves passing the problem to a stronger, more capable model. The system uses a supervised router trained on execution rollouts to determine which of these three paths is most likely to succeed for a given error.

Budget-Calibrated Control

A key innovation in this research is the use of Conformal Risk Control (CRC) to manage costs. Instead of retraining the model every time a user wants to change their budget, the researchers added a calibration layer on top of the trained router. This layer uses a cost-penalty "knob" that can be adjusted at deployment time. By shifting this penalty, the system can automatically favor cheaper recovery actions when the budget is tight or allow for more frequent escalation when the budget is more flexible. This provides a way to maintain control over average costs without needing to rebuild the underlying AI model.

Performance and Efficiency

The researchers evaluated CodeRescue across five different coding benchmarks. They found that the different recovery actions—reflecting, replanning, and escalating—often complement one another, as some problems are better suited for cheap, iterative fixes while others require the advanced reasoning of a larger model. In tests using the GPT-5.4-nano and GPT-5.4 models, the calibrated system was able to exceed the success rate of a strategy that always escalates to the stronger model, while using only 35% of the average cost.

Practical Considerations

The system is designed to be flexible, allowing it to adapt to different users and workloads without complex re-optimization. Because the calibration is based on statistical guarantees, it provides a reliable way to ensure that the average cost of the agent's recovery attempts stays within the user's defined limits. This approach acknowledges that not all failures are the same; by intelligently routing based on the specific type of error and the available budget, the system maximizes the utility of cheaper models before resorting to expensive, high-compute alternatives.

Comments (0)

No comments yet

Be the first to share your thoughts!