Back to AI Research

AI Research

Calibrated Enough to Know, Not Calibrated to Act: F... | AI Research

Key Takeaways

  • This paper investigates why Large Language Model (LLM) agents often provide confident, directional answers to questions that are provably unpredictable (alea...
  • What unlocks confident action is not information but the authority of its packaging.
  • Incapacity is not the answer: on matched answerable questions attached to the same panels, the same models answer essentially always, at near-perfect accuracy.
  • Nor is it belief - stated probabilities barely move across the gradient that swings action by 48 points, and score worse than a climatological baseline.
  • Missing judgment isn't it either: asked to classify a question's knowability before acting, models call it irreducible 90% of the time and then commit on just 0.4% of those.
Paper AbstractExpand

An LLM agent shown a professional-looking market panel commits to a directional call on a provably unpredictable question far more often than one asked the bare question: across 12 frontier models, commitment rises from 6.5% to 54.0% as evidence is escalated. It commits just as readily when every number on the panel is invented: fabricating the entire display, so nothing the model can see is true except the question itself, still lifts commitment from 24.5% to 36.8%, statistically indistinguishable from the 37.6% produced by genuine market data. What unlocks confident action is not information but the authority of its packaging. The failure is narrow and locatable. Incapacity is not the answer: on matched answerable questions attached to the same panels, the same models answer essentially always, at near-perfect accuracy. Nor is it belief - stated probabilities barely move across the gradient that swings action by 48 points, and score worse than a climatological baseline. Missing judgment isn't it either: asked to classify a question's knowability before acting, models call it irreducible 90% of the time and then commit on just 0.4% of those. The act/don't-act gate is what fails, and the effect is concentrated in a few models rather than universal. Because the gate is separable, it can be trained. Supervised fine-tuning of a 3B model on 540 synthetic cases, predominantly dice, coins, jars and timers, drives commitment to 0.0% on the original cases and transfers to three unseen domains. It does not survive everything: the gate holds exactly when the response format leaves room to reason, and rigid formats that remove that room leave the model confident and wrong on questions it otherwise answers correctly. The gate is trainable and context-fragile, and deployment needs both halves of that sentence.

This paper investigates why Large Language Model (LLM) agents often provide confident, directional answers to questions that are provably unpredictable (aleatoric), such as future market movements or weather outcomes. Researcher Pranav Aggarwal finds that these models are not failing due to a lack of capability or judgment, but because they are triggered by the authoritative appearance of information rather than the information itself. Methods and results are detailed in the full paper on arxiv.org.

The Trigger: Packaging Over Content

The study shows that adding a professional-looking market data panel to a prompt significantly increases the likelihood that an LLM will commit to a prediction. Across 12 frontier models, the rate of commitment to an unknowable question rose from 6.5% when asked a bare question to 54.0% when provided with an evidence panel. Crucially, the model’s willingness to commit does not depend on the truth of the data. When the researcher fabricated the entire display—ensuring that every number on the panel was invented—the commitment rate remained statistically indistinguishable from the rate produced by genuine market data. The models are responding to the "authority of the packaging" rather than the substance of the information. The same Large Language Models question is explored in Sophistication in GenAI Use, which adds a research perspective.

Ruling Out Common Explanations

The paper systematically rules out three common explanations for this behavior:

  • Incapacity: The same models that fail on unknowable questions perform with near-perfect accuracy on matched, answerable questions attached to the same panels.

  • Belief: The models’ stated probabilities for their answers barely change even when the evidence provided is escalated, suggesting the "act" of committing is decoupled from the model's internal confidence.

  • Missing Judgment: When explicitly asked to classify whether a question is knowable before acting, the models correctly identify the question as irreducible 90% of the time and refrain from committing in those instances. The failure is isolated to the "act/don't-act gate," which determines whether the model provides an answer or declines to answer.

Training the Gate

Because this failure is specific to the action gate, it can be corrected through supervised fine-tuning. By training a 3B model on 540 synthetic cases involving dice, coins, and timers, the researcher successfully drove the commitment rate on unknowable questions to 0.0%. This behavior transferred to three unseen domains, indicating that the model can learn to recognize when it should abstain from answering. The AI Agents story also surfaces in AI Agents Going Rogue Renew Calls..., adding another angle. The same AI Agents question is explored in What Makes Good Agentic Data? An..., which adds a research perspective.

Limitations and Context Sensitivity The effectiveness of this training is "context-fragile." The gate only functions reliably when the model is given enough room in its response format to reason through its decision. If the output format is too rigid—forcing a direct answer without allowing for reasoning—the gate fails, and the model reverts to being confident and wrong. Additionally, the researcher notes that some evaluation sets were susceptible to "lexical confounds," where the model could simply decline any question containing the word "will," rather than actually evaluating the knowability of the question. Deployment of these agents requires both a trained gate and a response format that preserves the model's ability to reason.

More from Franklin

What Makes Good Agentic Data? An...researchSophistication in GenAI UseresearchWrong Prediction, Right AnswerresearchAI Agents Going Rogue Renew Calls...newsOpenAI agents break out of sandbox...newsStanford Researchers Develop TRACE to Fix...news

Comments