Ahana Biswas and Yu-Ru Lin studied how people respond to evidence and uncertainty in AI answers. Their November 5 preprint analyzes 489,283 assistant–user response pairs from ShareChat and WildChat. Its most striking descriptive result is how often an answer invokes outside authority without including an explicit URL.
What the study counts as evidence
The authors distinguish broad attribution, such as invoking experts or research, from inspectable evidence. The latter can include a written URL, a Markdown link or a source object preserved in the conversation record. Among qualifying responses, broad attribution appeared in 31.9% and explicit URLs in 1.33%.
The measures describe features preserved in the data. They help separate a model’s expression of support from a piece of evidence a reader can follow. The study also codes uncertainty cues, confidence, refusals and reasoning text among the features it compares with the user’s next response.
How people check and challenge answers
The authors look at accepting, checking and challenging in the next user turn. In pooled results, uncertainty shows a small statistically distinguishable association favoring checking relative to challenging. The associations vary by task. For technical help, uncertainty favors checking. For information seeking, the study finds associations between refusals or reasoning traces and challenging.
The analysis also finds user scrutiny concentrated in some conversations. Apparent persistence across adjacent turns becomes much smaller when the authors examine differences between conversations and the ordering of responses separately.
What answer designers can measure
These are observational associations in conversation records. A product team could use the study’s distinctions to measure whether its interface preserves actual source links and whether those links help users resolve a specific question. Counting attribution phrases alone would miss the difference between a reference to authority and evidence that someone can open.
Comments