Auditing the Risk Claims of Distributional Reinforc... | AI Research

Auditing the Risk Claims of Distributional Reinforcement Learning
This paper investigates whether the "risk" reported by modern distributional reinforcement learning (RL) agents is a genuine reflection of the environment or merely a byproduct of how these models are trained. While these agents are increasingly used to make safety-critical decisions—such as monitoring autonomous systems or calculating risk-sensitive policies—their internal "risk" assessments have rarely been measured against ground truth. By conducting a rigorous, decision-level audit, the author finds that the risk claims made by these agents are largely unfounded, acting as structural artifacts rather than accurate representations of environmental uncertainty.

A New Way to Measure Risk

To determine if an agent’s risk claims are true, the author developed a "decision-relevant" metric called the excess Wasserstein gap. This metric identifies states where an agent claims a risk trade-off exists between two actions—meaning one action is not simply better than the other, but carries a different risk profile. If this gap is greater than zero, it signals that a risk-sensitive policy would choose differently than a standard "mean-greedy" policy. The audit then compares these claims against ground truth by using "snapshot-restart" Monte Carlo simulations, where the environment is reset thousands of times to observe the actual outcomes of the agent's chosen actions.

The Findings: Artifacts Over Reality

Across several popular RL architectures (QR-DQN, C51, and IQN), the audit reveals that the agents' most confident risk claims are frequently wrong. In the top 2% of states where these agents claim the most significant risk trade-offs, between 40% and 95% of those claims were refuted at 95% confidence. The research shows that these "risk" signals are structural artifacts that appear early in training, remain uncorrelated with the agent's final performance, and are unique to each training seed. Essentially, the agents report risk even in "doomed" states where every possible future outcome is identical, proving that the reported risk is a failure of the model's internal representation rather than a feature of the game environment.

Why the Audit Matters

To ensure the results were not caused by a flawed testing process, the author used "positive controls"—environments with known, built-in risk trade-offs. In these controlled tests, the audit successfully confirmed 96–100% of the true claims, demonstrating that the audit tool itself is accurate and sensitive. When the agents were tasked with making decisions based on their own risk assessments, their performance ranged from beneficial to significantly worse than chance. Ultimately, the paper concludes that the "risk" heads in these agents are uninformative. Neither training specifically for risk nor using ensembles of models successfully removed these artifacts, suggesting that current distributional RL methods may not be as reliable for safety-critical applications as their internal outputs might suggest.