Domain-Adapted Small Language Models for Reliable C... | AI Research

Domain-Adapted Small Language Models for Reliable Clinical Triage
Emergency departments often struggle with the Emergency Severity Index (ESI), a system used to classify how quickly a patient needs care. Because triage notes are written in free-text and can be highly variable, assigning an ESI level is often subjective, leading to inconsistencies and inefficiencies. This research explores whether small, open-source language models (SLMs) can act as reliable, privacy-preserving tools to assist clinicians in making these assignments more accurately and consistently.

A Smarter Way to Process Clinical Notes

The researchers found that feeding raw, messy triage notes directly into a model is ineffective. Instead, the most accurate results came from first converting these notes into "clinical vignettes"—concise, structured summaries of the patient’s condition. By testing various models, the team identified that the Qwen2.5-7B model provided the best balance of accuracy, stability, and speed. Unlike larger, proprietary models that require external cloud services, this smaller model can be deployed locally, keeping sensitive patient data secure within the hospital’s own infrastructure.

Training for Clinical Accuracy

To make the model effective for a specific hospital environment, the researchers used a process called fine-tuning. They combined expert-curated triage examples with a large set of "silver-standard" data—thousands of real-world pediatric encounters where the model helped generate summaries that were then matched against the triage nurse's original ESI assignment. By using a technique called QLoRA, the team was able to train the model efficiently on standard hospital hardware. This targeted training allowed the model to learn the specific nuances of pediatric triage, significantly reducing errors compared to baseline models.

Key Findings and Performance

The fine-tuned Qwen2.5-7B model outperformed both basic open-source models and larger, more complex proprietary systems like GPT-4o. A major advantage of this approach is its computational efficiency; the model can process a patient encounter in less than one second, making it fast enough for real-time use in a busy emergency department. The results showed that this method not only reduced overall discordance—the difference between the model's prediction and the nurse's assignment—but also minimized "significant" errors, such as misclassifying a high-acuity patient as low-acuity.

Considerations for Real-World Use

While the model shows great promise, the researchers emphasize that its goal is to support human decision-making rather than replace it. The study highlights that the quality of the input data is critical; when information is missing or poorly documented, the model’s performance can be affected. Furthermore, because the model is designed to reproduce standard clinical triage patterns, it is limited by the inherent variability of human-assigned ESI levels. Future work will focus on how these tools can handle incomplete data in real-world settings, ensuring they remain reliable even when clinical information is evolving during a patient's visit.