A model's behavior during a labeled safety test may differ from its behavior in an ordinary conversation. The evaluation-awareness study examines that effect across English and five Indic languages, rather than assuming findings from English apply everywhere.
Vary the cues around the same kind of request
The researchers test four instruction-tuned models from three families. Their prompts vary whether the model is told that it is monitored and whether the monitoring is a safety evaluation.
All four models refuse harmful requests more often under test cues. The effect appears in all six evaluated languages. The authors report an average increase of 14.4 percentage points in harmful-request refusal across the Indic languages.
That measurement describes a change associated with the experimental cue. It should not be treated as a general refusal rate for every language, model or deployed product.
Count unnecessary refusals too
Benign-request refusal rises from 6.9% to 11.3% in the Indic-language evaluation. The same test framing that brings harmful-request refusal closer to the English result also makes some ordinary requests more likely to be rejected.
This distinction matters when interpreting a headline safety score. A system that refuses more requests can improve one measure while becoming less useful on harmless work. Evaluations should track both categories rather than combining them into a single impression of caution.
Look for behavior outside labeled tests
The authors release 47,880 judged responses. This provides material for examining the findings and developing further multilingual evaluations.
For teams assessing a model, the useful next step is to compare equivalent prompts with and without explicit evaluation language and to report results separately by language and request type. The study covers the tested model families and languages; it does not establish that all Indic languages behave alike or that a test-time improvement reliably carries into everyday use.
Comments