Back to AI Research

AI Research

Training agent confidence against the final outcome improves failure prediction

Key Takeaways

  • An experimental calibration method asks whether the whole agent attempt will succeed, rather than whether one step looks correct.
  • ## Confidence should refer to a defined outcome
  • The [outcome-grounded calibration paper](https://arxiv.org/abs/2610.18614) trains an agent's per-step confidence to predict the eventual result of a multi-step attempt.
  • That differs from rewarding confidence for the correctness of the current action alone.
  • Under group-relative policy optimization, multiple rollouts for a task provide success labels.

Confidence should refer to a defined outcome

The outcome-grounded calibration paper trains an agent's per-step confidence to predict the eventual result of a multi-step attempt. That differs from rewarding confidence for the correctness of the current action alone.
Under group-relative policy optimization, multiple rollouts for a task provide success labels. A proper scoring rule compares each step's stated confidence with the episode's realized outcome. The intended use is a signal that helps decide whether to continue, abstain or request assistance during a longer task.

Reported gains have a specific scope

In offline experiments with recorded trajectories from a Qwen2.5-7B agent, the authors evaluate ALFWorld, TextWorld and WebShop. They report that the ability of step confidence to distinguish eventual success from failure, measured by AUROC, rose from near chance to 0.68–0.85.
Online reinforcement learning on ALFWorld also improved early-step discrimination. Expected calibration error fell from 0.42 to below 0.12, while selective execution improved over an untrained agent and a format-only training control without degrading task success in those experiments.
The evaluation separates episode depth, task difficulty and foresight about an individual attempt. The strongest finding is that the trained agent can recognize early which tasks it is likely to fail. Distinguishing a successful attempt from an unsuccessful attempt on the same task remains harder.

Test the decision rule, not confidence language

Franklin analysis: a deployment trial should compare the actual consequences of its escalation threshold. Count successful completions, unnecessary handoffs and failures that the confidence signal allowed through.
Keep harder tasks separate from repeated attempts at the same task. Otherwise a system may appear to anticipate failure mainly because it recognizes difficult assignments. The paper offers evidence that training can make confidence more useful, while leaving a clear measurement question for any application that wants to act on it.

Comments