AI Glossary

Safety classifier

Also called content moderation model

What does “Safety classifier” mean?

Safety classifier: Separate model that screens prompts or responses and flags or blocks content in categories like violence, self-harm, hate or weapons instructions. It acts as a guardrail running alongside the main model, so a harmful output can be caught even if the main model was tricked. Classifiers must balance missed harms against blocking harmless requests.

Safety classifier visual guide: Separate model that screens prompts or responses and flags or blocks content in categories like violence, self-harm, hate or weapons instructions. It acts as a guardrail running alongside the main model, so a harmful output can be caught even if the main model was tricked. Classifiers must balance missed harms against blocking harmless requests.
Visual guide Open full size (new tab)

What is an example of “Safety classifier”?

A chat app might run every reply through a safety classifier and replace flagged responses with a refusal message before the user sees them.

Keep learning

See also

GuardrailsTermJailbreakTermResponsible AITermRed teamingTerm

Same topic

SycophancyTermReward hackingTermWatermarkingTermModel cardTermMMLUTerm