Back to AI Research

AI Research

RADAR separates confidence recalibration from adopting and auditing the repair

Key Takeaways

  • The framework checks what a scalar confidence score can express, tests whether a repair improves risk and examines errors left behind.
  • ## A better score does not settle every decision
  • RADAR studies confidence scores used to accept, review or regenerate language-model answers.
  • The authors argue that recalibrating a scalar score can repair only the errors visible through that score.
  • Information lost when a richer predictor is compressed to one number may remain unavailable to the recalibration map.

A better score does not settle every decision

RADAR studies confidence scores used to accept, review or regenerate language-model answers. The authors argue that recalibrating a scalar score can repair only the errors visible through that score. Information lost when a richer predictor is compressed to one number may remain unavailable to the recalibration map.
The framework separates three jobs: repair confidence, decide whether the repair is supported by evidence and audit the residual errors. That separation prevents a lower prediction-risk estimate from being treated automatically as a better human-review policy.

Repair under a stated constraint

Probe-Anchored Isotonic Calibration learns an order-preserving map using correctness-probe predictions on unlabelled target prompts. The paper relates its excess risk to score-conditional probe bias and fitting error, rather than treating the probe's overall accuracy as the decisive property.
For adoption, RADAR uses a paired procedure with finite-sample control of false improvements. Its audit uses a prompt-clustered kernel test to look for feature-aligned residual errors while accounting for answers that depend on the same prompt.

Keep the reported percentages attached to the experiment

Across 47,400 replayed raw-score allocations, the authors report that paired betting adopts candidates with lower held-out test risk in 42.95% of allocations, versus 15.09% for one-sided empirical Bernstein. Those percentages concern an adoption experiment; they are not the language model's answer accuracy or a universal probability that a repair is useful.
The paper also reports that richer confidence predictors can lose gains when projected back to a scalar, and that lower risk need not produce better review decisions. Code and data are described as planned for release upon acceptance.
Franklin's reading is to test a proposed repair against the decision it will support. Check the evidence for adopting it, then inspect which prompt groups still produce systematic errors. A better-looking confidence curve alone does not answer those operational questions.
Read the RADAR paper.

Comments