Back to AI Research

AI Research

A small experiment finds confident AI falsehoods can undermine learners’ confidence calibration

Key Takeaways

  • The study plants misinformation in five same-model assistants and measures what participants learn, how confident they become and where they return to check answers.
  • A study of human-AI learning asks what happens when an assistant remains confident and consistent while teaching false information.
  • Rather than test whether a model agrees with its user, the researchers examine whether users defer to the model.
  • The results come from a small controlled experiment and should retain that scope.
  • ## Manipulate the information, then test learning

A study of human-AI learning asks what happens when an assistant remains confident and consistent while teaching false information. Rather than test whether a model agrees with its user, the researchers examine whether users defer to the model. The results come from a small controlled experiment and should retain that scope.

Manipulate the information, then test learning

The paper describes 32 participants learning science topics through five assistants built on GPT-5-mini. The assistants’ briefings contained different proportions of plausible falsehoods, from none to all, while every assistant spoke with full confidence.
Surprise quizzes offered the correct fact, the planted falsehood and two distractors, with a confidence rating for each answer. This design makes it possible to examine both what participants selected and whether their confidence distinguished correct answers from incorrect ones.

Confidence did not reliably track correctness

Across 104 quizzes and 1,040 answers, the authors report scores falling from 7.4 to 2.9 out of ten across the conditions. Adopted falsehoods were held with high confidence, while the discrimination of confidence for correctness fell toward chance.
Post-task trust ratings changed little. That combination matters because an assistant can feel trustworthy even when the user has learned incorrect material. These are findings from the constructed teaching conditions, not a population-wide estimate of how often all AI users accept misinformation.

Returning to the same source can compound the problem

The study reports that some participants pasted quiz questions back into the assistant being checked. More return visits were associated with greater absorption of falsehoods rather than less.
For learning-interface designers, the useful question is how a learner can inspect evidence for an individual claim and compare it with an independent source. A general trust score or a confident tone supplies different information from demonstrated correctness. The experiment motivates measuring the human-AI pair, while its small sample and deliberately manipulated briefings limit broader conclusions.

Comments