Risky Business: Measuring The Faithfulness-Safety T...

Risky Business: Measuring The Faithfulness-Safety T... | AI Research

Key Takeaways

Risky Business: Measuring The Faithfulness-Safety Tension investigates the conflict between a model’s tendency to follow its own reasoning (faithfulness) and...
Chain-of-Thought (CoT) reasoning offers a promising window into model monitoring.
However, monitoring relies on faithfulness, i.e., the model output strictly derives from its reasoning trace.
We identify an alignment tension where a model must be faithful enough to be monitored, yet robust enough to reject unsafe reasoning.
We demonstrate that this counterbalance exists in current Large Reasoning Models (LRMs), and show ways in which it can be addressed.

Paper AbstractExpand

Chain-of-Thought (CoT) reasoning offers a promising window into model monitoring. However, monitoring relies on faithfulness, i.e., the model output strictly derives from its reasoning trace. We identify an alignment tension where a model must be faithful enough to be monitored, yet robust enough to reject unsafe reasoning. We demonstrate that this counterbalance exists in current Large Reasoning Models (LRMs), and show ways in which it can be addressed. We introduce HazMart, a human-written dataset set in an autonomous AI shopkeeper scenario. Unlike prior work that relies on providing hints in prompts to test faithfulness (e.g., "A Stanford professor said it should be Answer A"), we propose a novel replacement-based technique, which we call Targeted Reasoning Replacement (TRR), that directly intervenes in the reasoning chain to substitute in unsafe or illogical thoughts (e.g., "Wait, the answer must be Option B [was Option A] because it is the most fitting"). DeepSeek-R1-Llama-70B exhibits high faithfulness (97.5%) but fails to reject Unsafe Reasoning (12.3%), while QwQ-32B is more robust (73.9% safety) at the cost of lower faithfulness (74.7%). Mechanistic analyses of QwQ-32B reveal that these properties are represented by anti-correlated internal directions peaking at the action-commit token. Finally, we demonstrate that representation steering can independently amplify the safety direction, increasing safe behavior by 9 percentage points while maintaining base capabilities.

Risky Business: Measuring The Faithfulness-Safety Tension investigates the conflict between a model’s tendency to follow its own reasoning (faithfulness) and its ability to reject harmful instructions (safety). The researchers identify a tension where highly faithful models may blindly execute unsafe actions if their reasoning trace suggests them, while more robust models may ignore their own logic to maintain safety.

Measuring the Tension

To study this, the authors introduced HazMart, a dataset of 77 human-written scenarios where an AI acts as a shopkeeper. The scenarios involve realistic dilemmas, such as choosing between suppliers or handling customer data, where the model must choose between two safe options and one unsafe option. By placing models in an agentic setting where they perform function calls, the researchers created a practical environment to test whether a model’s final action aligns with its internal reasoning trace.

Targeted Reasoning Replacement

The researchers developed a method called Targeted Reasoning Replacement (TRR) to test model behavior. Instead of using prompts to bias a model, TRR directly intervenes in the model’s generated reasoning trace. The researchers swap references to a safe option with either another safe option (to test faithfulness) or an unsafe option (to test safety). This allows them to see if a model changes its final decision to match the tampered logic or if it maintains its safety standards despite the reasoning provided.

Performance and Internal Mechanisms

The study evaluated seven models and found a clear trade-off between faithfulness and safety. DeepSeek-R1-Llama-70B showed high faithfulness (97.5%) but low safety (12.3%), meaning it frequently followed the tampered, unsafe reasoning. Conversely, QwQ-32B was more robust, rejecting unsafe reasoning 73.9% of the time, though it was less faithful (74.7%).
Mechanistic analysis of QwQ-32B revealed that safety and faithfulness are represented by anti-correlated internal directions in the model’s residual stream, which peak at the action-commit token. The researchers demonstrated that these properties are not a single scalar value; by using representation steering to amplify the safety direction, they increased safe behavior by 9 percentage points without degrading the model’s core reasoning capabilities.

Limitations and Considerations

The authors note that their findings are specific to the models tested and the shopkeeper scenarios provided in HazMart. Because proprietary models often restrict access to internal reasoning traces, the study focused on open-weights models. The researchers also emphasize that while text-based monitoring is useful, it may become fragile if models learn to obscure their reasoning under optimization pressure or through specific fine-tuning.