MedFailBench: A Clinician-Built Open-Source Benchma... | AI Research

MedFailBench: A Clinician-Built Open-Source Benchmark for Medical AI Safety Boundary Inspection
Most existing medical AI benchmarks focus on whether a model can provide the correct answer to a clinical question. MedFailBench shifts this focus, asking instead: which safety boundary did the model fail to respect? This project provides a clinician-reviewed, synthetic benchmark designed to identify and label medical AI errors. By categorizing failures by their clinical severity and the specific "safety gate" they breached, the project aims to make AI behavior more transparent and inspectable without the need for sensitive patient data.

How the Benchmark Works

MedFailBench uses a set of 100 synthetic clinical cases, each designed to test how an AI handles ambiguity or missing information. Each case includes a prompt with intentionally open variables—such as missing lab results or vital signs—that a safe system should identify before suggesting a course of action. A clinician then reviews the model's output and assigns it a severity score (from 1, a minor wording issue, to 5, a high-risk unsafe framing) and a "safety gate" label. These gates identify the specific type of failure, such as missed urgent escalation, unsafe remote dosing, or evidence fabrication.

Safety Gate Taxonomy

The project defines six primary safety gates to help researchers understand why a model failed. These include:

• Missed urgent escalation: Failing to recognize when a patient needs immediate care.

• Unsafe remote dosing: Suggesting medication changes without necessary clinical data.

• Unsafe discharge reassurance: Providing false comfort despite unresolved red flags.

• Evidence fabrication or overclaim: Inventing or overstating supporting information.

• Unsafe protocol execution: Turning general advice into potentially dangerous operational steps.

• Source support gap: Failing to provide or verify the source of clinical information.

Current Findings and Evaluation

The project includes a live leaderboard preview and an automated pipeline that tests open-source models on hard synthetic prompts. Preliminary evaluations of models like DeepSeek V4 Flash, Qwen 2.5 7B, and Llama 3.3 70B show that they often struggle with the same types of safety boundaries, particularly regarding urgent escalation. This suggests that safety failures are not necessarily tied to model size, but rather to how LLMs handle clinical ambiguity. The project emphasizes that these automated results are a screening tool and require full clinician review before any definitive safety claims can be made.

Important Considerations

MedFailBench is intended as a community resource for inspection and research, not as a tool for clinical validation or official safety certification. Because the cases are synthetic, they may not perfectly reflect real-world clinical workflows. Furthermore, the current version relies on single-clinician review, though the project is actively working to incorporate inter-rater reliability data from additional reviewers. The benchmark is designed to be extensible, and the authors encourage researchers to contribute new cases and feedback through the project’s open-source repository.