Back to AI Research

AI Research

A smoking-cessation chatbot audit separates persona variation from appropriate personalization

Key Takeaways

  • Six models answered the same questions for twelve personas; a factual-floor prompt reduced differences but did not improve every model in the same way.
  • ### Ask the same question through different scenarios
  • A [research audit of smoking-cessation information](https://arxiv.org/abs/2610.42039) examines how chatbot answers change with the apparent user.
  • The researchers tested six models from four providers using four questions and twelve personas, collecting 2,880 responses.
  • The personas differed in life stage, stated expertise and smoking history.

Ask the same question through different scenarios

A research audit of smoking-cessation information examines how chatbot answers change with the apparent user. The researchers tested six models from four providers using four questions and twelve personas, collecting 2,880 responses.
The personas differed in life stage, stated expertise and smoking history. Because several attributes changed together, the study reports differences between scenarios rather than isolating a causal effect of age or professional expertise.

More uniform answers are not automatically better

Every evaluated model mentioned prescription cessation medications less often for teen personas than for adults, with reported gaps of 32 to 67 percentage points. In five of the six models, a persona claiming healthcare expertise received fewer facts than a persona claiming none.
The paper cautions against interpreting every difference as a failure. It notes that clinical appropriateness can depend on age, so a useful baseline should be age-aware. Equal mention rates alone would not establish the quality of an answer.
The researchers also tested a short system prompt describing a factual floor: relevant core information to preserve while allowing tone and vocabulary to adapt. It narrowed persona differences in all six models, but the effect on overall fact coverage varied in both magnitude and direction. A prompt that works well on one model therefore needs evaluation on another.

Use the audit design without turning it into treatment advice

The released responses and analysis code provide material for examining consistency, omissions and the effects of system instructions. The study evaluates model outputs; it does not establish patient outcomes or provide an individualized treatment plan.
For developers, the practical lesson is to measure factual coverage and appropriate personalization together. A useful test set should include multiple user scenarios, an explicit relevance rubric and separate checks for whether an intervention reduces gaps while preserving the information each scenario actually needs.

Comments