Online Safety Monitoring for LLMs | AI Research

Online Safety Monitoring for LLMs
Large Language Models (LLMs) are increasingly used for critical tasks, yet they remain prone to generating harmful, incorrect, or hallucinated content. While developers use alignment training and pre-deployment testing to mitigate these risks, these measures cannot account for every possible scenario. This paper introduces a real-time monitoring framework designed to detect unsafe outputs as they are being generated, allowing systems to intervene—such as by halting generation or triggering a human review—before the full response is delivered.

A Simple Approach to Safety

The researchers propose a straightforward statistical method to monitor LLM outputs. Instead of using complex, computationally heavy systems, they use a "verifier signal"—a score from an external model that estimates the safety of the text generated so far. The monitor applies a single, time-invariant threshold to this signal. If the verifier’s score drops below this threshold, the monitor raises an alarm. The key to the system's effectiveness is how this threshold is chosen: it is calibrated using "risk control" techniques, which provide mathematical guarantees that the monitor will not exceed a user-specified rate of false alarms.

Calibration for Reliability

To ensure the monitor is trustworthy, the authors use two calibration procedures based on a held-out dataset:

• Control in Expectation: This method ensures that the average risk remains below a chosen limit. It is efficient and requires less data to set up.

• High-Probability Control: This provides a stronger guarantee, ensuring that the risk stays below the limit in almost all cases. This is more conservative and typically requires more calibration data, making it ideal for high-stakes environments where safety violations are particularly costly.

Performance and Efficiency

In experiments involving mathematical reasoning and toxic content detection, the researchers found that their simple threshold-based monitor is highly competitive with more advanced methods, such as sequential hypothesis testing. Notably, the risk-controlling monitor often detects failures earlier in the generation process, which limits the user's exposure to harmful content and saves on computational costs. The study also explored using the LLM’s own internal token probabilities as a "free" signal. While this is computationally cheaper than using an external verifier, the results showed that it is significantly less effective at detecting errors, highlighting a clear trade-off between cost and safety performance.

Limitations and Future Directions

While the proposed monitor is effective and easy to deploy, it has notable limitations. The system’s performance is entirely dependent on the quality of the verifier signal; if the verifier is inaccurate or lacks robustness, the monitor will struggle. Additionally, the current model uses a single threshold for the entire generation process, which ignores the fact that safety signals might naturally fluctuate over time. Future research could explore combining multiple signals for better accuracy, optimizing the balance between computational cost and safety, or implementing dynamic, step-specific thresholds to further improve detection capabilities.