Safety classifier
Checks prompts and responses for harmful content, so an AI app can block or flag them.
What it is
Safety classifiers screen what goes into and comes out of an AI system. They flag requests or answers involving violence, self-harm, illegal activity, jailbreak attempts or personal data, letting an app refuse, rewrite or escalate.
How it works
A model, often a small fine-tuned language model, reads the conversation and outputs whether it is safe and which policy category it breaks. Running one before the main model checks the prompt; running one after checks the response.
What it is good for
- Guardrails for chatbots and agents
- Moderating user-generated content
- Detecting prompt injection and jailbreaks
Limits to know
- False positives can block legitimate requests
- Determined attackers find gaps
- Policies differ by product and region
