AI Research

ResidencyRL: Reinforcement Learning in Simulated Cl... | AI Research

Key Takeaways

ResidencyRL is a reinforcement learning method designed to train clinical AI agents by simulating the full sequence of a patient encounter.
Much of clinical reasoning relies on the patient encounter, a dialogue in which a clinician elicits history, refines diagnostic hypotheses, and decides management under uncertainty.
While large language models (LLMs) excel on static medical benchmarks, methods to optimize the full sequence of clinical decisions remain underdeveloped.
On held-out evaluations, the ResidencyRL agent improves diagnostic accuracy by 7.0% under adversarial conditions (88.0% vs.
81.0%) and reduces missed red flag rates by 31%, demonstrating rigorous mitigation of premature closure.

Paper AbstractExpand

In medical education, physicians convert academic knowledge into clinical expertise through residency: years of training across thousands of encounters, with diverse sources of feedback and progressively greater autonomy. Much of clinical reasoning relies on the patient encounter, a dialogue in which a clinician elicits history, refines diagnostic hypotheses, and decides management under uncertainty. While large language models (LLMs) excel on static medical benchmarks, methods to optimize the full sequence of clinical decisions remain underdeveloped. We present ResidencyRL, a reinforcement learning (RL) method for training clinical artificial intelligence (AI) agents through simulated multi-turn clinical encounters (up to 60 dialogue turns and 8 tool calls per trajectory). ResidencyRL pairs the policy agent with LLM simulators capable of complex, adversarial behaviors, training against a structured reward aligned to diagnostic accuracy, management quality, communication, documentation, and safety. On held-out evaluations, the ResidencyRL agent improves diagnostic accuracy by 7.0% under adversarial conditions (88.0% vs. 81.0%) and reduces missed red flag rates by 31%, demonstrating rigorous mitigation of premature closure. Blinded expert clinicians validated these gains, preferring the trained agent in 87.6% of side-by-side comparisons. The procedural competencies transfer to unseen benchmarks: the agent outperforms the base model across all six clinical axes of the AMIE multi-visit benchmark, and shows consistent directional improvements on AgentClinic and CRAFT-MD. Our findings demonstrate that sequential clinical decision-making can be effectively learned through multi-turn RL in simulation, yielding robust, generalizable capabilities, paving the way towards clinical mastery. Prospective validation with real-world workflows remains necessary to establish clinical utility.

ResidencyRL is a reinforcement learning method designed to train clinical AI agents by simulating the full sequence of a patient encounter. Rather than relying on static medical benchmarks, the researchers developed a system that allows an AI to practice multi-turn dialogues, diagnostic reasoning, and management planning in a simulated environment, mirroring the way human physicians develop expertise during residency.

Training Through Simulated Practice

The ResidencyRL agent, initialized from a Gemini 3.5 Flash model, learns by interacting with LLM-driven patient simulators. These simulators are programmed to exhibit diverse behaviors, including complex, adversarial, or uncooperative traits, which force the agent to navigate uncertainty. The training process covers three types of environments: routine telehealth consultations, targeted history-taking to uncover hidden information, and safety-focused scenarios. Throughout these encounters, the agent performs up to 60 dialogue turns and 8 tool calls, such as documenting diagnoses or management plans. An automated rater evaluates these interactions based on diagnostic accuracy, management quality, communication, documentation, and safety, providing a reward signal that guides the agent’s improvement via Group Relative Policy Optimization.

Performance and Generalization

In held-out evaluations, the ResidencyRL-trained agent showed significant improvements over the base model. Under adversarial conditions, diagnostic accuracy increased from 81.0% to 88.0%, and the rate of missed safety-critical red flags dropped by 31%. These gains were validated by 97 board-certified clinicians in blinded side-by-side comparisons, where the trained agent was preferred in 87.6% of cases. The agent’s competencies also transferred to unseen benchmarks, including the AMIE multi-visit benchmark, AgentClinic, and CRAFT-MD, as well as specialist oncology cases not encountered during training.

Clinical Reasoning and Safety

The research indicates that the ResidencyRL approach effectively mitigates "premature closure"—a common diagnostic error where a clinician stops gathering information too early. By training the agent to handle long-horizon, multi-turn interactions, the system encourages more thorough information gathering and more appropriate management planning. The evidence suggests that sequential decision-making in clinical settings can be optimized through simulation, providing a structured way to improve AI performance across the entire arc of a patient visit.

Limitations and Future Outlook

While the results show consistent improvements across multiple clinical dimensions, the authors note that prospective validation with real-world clinical workflows is necessary. The current findings are based on simulated environments, and further research is required to establish whether these competencies translate directly to patient-level efficacy and safety in actual clinical practice.

Comments (0)

No comments yet

Be the first to share your thoughts!