Back to AI Research

AI Research

MentalHospital: A Virtual Environment for Evaluatin... | AI Research

Key Takeaways

  • MentalHospital: A Virtual Environment for Evaluating Psychiatric Clinical Encounters Large language models (LLMs) have shown promise in specific psychiatric...
  • We introduce $\textbf{MentalHospital}$, a virtual evaluation environment for LLM-based psychiatric clinical encounters.
  • Each encounter is assessed through a dual-track protocol that combines objective comparison against EHR-derived references with subjective assessment of clinical process quality.
  • Survey responses from 22 clinicians support MentalHospital's clinical fidelity (3.88/5), while MentalEval achieves strong expert alignment with an average QWK of 0.944.
  • Benchmarking shows that even the strongest LLM trails clinicians by 37.28 percentage points in objective psychiatric competence, with mental status assessment as a key bottleneck.
Paper AbstractExpand

Large language models (LLMs) have shown strong performance on isolated psychiatric tasks, including dialogue, diagnosis, and treatment planning, yet existing benchmarks rarely simulate complete psychiatric clinical encounters. We introduce $\textbf{MentalHospital}$, a virtual evaluation environment for LLM-based psychiatric clinical encounters. MentalHospital instantiates the Subjective Interviewing, Objective Examination, Diagnostic Assessment, and Treatment Planning (S.O.A.P.) workflow, using skill-augmented standardized patients constructed from 1,193 de-identified psychiatric electronic health record (EHR) cases spanning all major ICD-11 categories and 76 disorders. Each encounter is assessed through a dual-track protocol that combines objective comparison against EHR-derived references with subjective assessment of clinical process quality. To scale specialist judgment, we develop $\textbf{MentalEval}$, five domain-specific evaluators covering communication empathy, interviewing professionalism, clinical-note quality, diagnostic rigor, and treatment appropriateness, trained with rubric-grounded SFT and expert-guided DPO. Survey responses from 22 clinicians support MentalHospital's clinical fidelity (3.88/5), while MentalEval achieves strong expert alignment with an average QWK of 0.944. Benchmarking shows that even the strongest LLM trails clinicians by 37.28 percentage points in objective psychiatric competence, with mental status assessment as a key bottleneck.

MentalHospital: A Virtual Environment for Evaluating Psychiatric Clinical Encounters
Large language models (LLMs) have shown promise in specific psychiatric tasks, but they often lack the ability to handle the full, integrated workflow of a real-world clinical encounter. To address this, researchers have introduced MentalHospital, a virtual environment designed to evaluate how AI models perform across the entire psychiatric process. By simulating the standard Subjective, Objective, Assessment, and Planning (S.O.A.P.) workflow, the environment allows for a more comprehensive assessment of an AI’s ability to interview patients, request examinations, diagnose disorders, and plan treatments.

A Realistic Simulation Environment

MentalHospital is built upon 1,193 de-identified psychiatric electronic health record (EHR) cases, covering 76 different disorders and all major ICD-11 categories. Unlike previous benchmarks that rely on simple, reactive chatbots, this environment uses "skill-augmented" standardized patients. These virtual patients possess individualized symptom presentations and a dynamic memory, allowing them to interact in a way that mimics real psychiatric patients. The environment also includes a hospital-side module that provides clinical evidence only when the AI agent correctly requests the appropriate examinations.

Dual-Track Evaluation

To measure performance, the researchers implemented a dual-track protocol. First, they use objective comparisons to see how well the AI’s findings and decisions match the actual clinical records. Second, they assess the quality of the clinical process—such as empathy and professionalism—which is difficult to measure with automated metrics alone. Because general-purpose AI judges often struggle to provide expert-level feedback, the team developed MentalEval. This is a specialized suite of five evaluators trained to assess communication, note-taking, diagnostic rigor, and treatment planning, achieving high alignment with human clinical experts.

Performance Gaps and Key Findings

Experiments conducted with various LLMs, medical trainees, and human experts revealed a significant performance gap. Even the most advanced models currently trail human experts by an average of 37.28 percentage points in objective psychiatric competence. A specific bottleneck identified in the research is the "mental status assessment," where models struggle to effectively gather and synthesize the necessary clinical information. While models show potential in areas like empathic communication, they are not yet at the level of human clinicians in managing the full complexity of a psychiatric encounter.

Clinical Fidelity and Future Use

The environment has been validated by 22 clinicians, who gave it a high score for clinical fidelity (3.88/5), indicating that the simulations feel authentic to real-world practice. By providing a standardized, scalable way to test AI in a controlled setting, MentalHospital serves as a valuable tool for training and evaluating medical AI. It highlights that while AI can handle isolated tasks, the integrated, multi-step nature of psychiatry remains a significant challenge that requires further development and specialized evaluation.

Comments (0)

No comments yet

Be the first to share your thoughts!