AI Research

A Multi-Agent System for Autonomous, Fine-Tuning-Fr... | AI Research

Key Takeaways

Clinical notes are a goldmine of patient information, but much of this data remains trapped in unstructured text rather than being recorded in standardized f...
Clinical notes contain many of the signs and symptoms that bring patients to care, yet this information rarely reaches structured fields.
Existing extraction approaches either rely on context-insensitive rules that generate false positives or on supervised models that require substantial fine-tuning.
We present Pythia, a multi-agent system that autonomously writes and optimizes extraction prompts for clinical concepts without manual prompt engineering or fine-tuning.
Running on a locally hosted open-weights model, Pythia keeps clinical notes on local infrastructure and selects prompts using development-set sensitivity and specificity.

Paper AbstractExpand

Clinical notes contain many of the signs and symptoms that bring patients to care, yet this information rarely reaches structured fields. Existing extraction approaches either rely on context-insensitive rules that generate false positives or on supervised models that require substantial fine-tuning. We present Pythia, a multi-agent system that autonomously writes and optimizes extraction prompts for clinical concepts without manual prompt engineering or fine-tuning. Running on a locally hosted open-weights model, Pythia keeps clinical notes on local infrastructure and selects prompts using development-set sensitivity and specificity. We compared Pythia with a curated lexicon across 72 signs and symptoms from 400 clinical notes representing 387 patients. Development (n=300) and validation (n=100) sets were partitioned independently for each concept. Pythia achieved mean sensitivity of 0.76 and specificity of 0.95, compared with 0.82 and 0.76 for the lexicon, and matched or exceeded the lexicon on both metrics for 20 of 62 directly comparable concepts. For 14 concepts where the lexicon labeled every note positive, Pythia recovered mean specificity of 0.97 by requiring a present-tense, patient-attributed finding rather than any textual mention of a term. Specificity transferred from development to validation with minimal degradation across prevalences, whereas sensitivity transfer weakened below 5% prevalence, reaching a mean gap of 0.25 below 2% prevalence. A BERT classifier fine-tuned per concept on the same development set achieved mean sensitivity of 0.23 and collapsed to zero sensitivity for concepts below roughly 5% prevalence. These findings suggest that autonomous, fine-tuning-free prompt optimization can produce symptom extraction prompts that generalize effectively from development to validation while remaining deployable on local infrastructure.

Clinical notes are a goldmine of patient information, but much of this data remains trapped in unstructured text rather than being recorded in standardized fields. While researchers need this information to track symptoms and improve care, current methods for extracting it are flawed: rule-based systems often produce too many false positives, while advanced machine learning models require expensive, time-consuming fine-tuning and large amounts of labeled data. This paper introduces Pythia, a multi-agent system that autonomously writes and optimizes its own extraction prompts to identify clinical symptoms without requiring manual engineering or model retraining.

How Pythia Works

Pythia functions as a "loop-engineered" system, meaning it operates through a cycle of goal-setting, testing, and refinement. It uses a locally hosted, open-weights language model to ensure that sensitive clinical notes remain within the institution's private infrastructure. The system consists of three specialized agents: a "Specialist" that applies a prompt to clinical notes, an "Improver" that analyzes errors (such as false positives or negatives), and a "Summarizer" that updates the prompt based on those findings. A separate, deterministic controller monitors the process, deciding when to continue, backtrack, or stop based on the system's performance on a development set.

Comparing Performance

The researchers tested Pythia against two traditional methods: a curated rule-based lexicon and a fine-tuned BERT classifier. Across 72 different signs and symptoms, Pythia achieved a mean sensitivity of 0.76 and a mean specificity of 0.95. In contrast, the lexicon was more sensitive (0.82) but significantly less specific (0.76), often flagging every note as positive for certain symptoms. The fine-tuned BERT model struggled significantly, achieving a mean sensitivity of only 0.23 and failing to detect symptoms entirely when they appeared in less than 5% of the notes. Pythia’s ability to distinguish between a symptom being present and a symptom being mentioned in a different context—such as a family history or a negated statement—allowed it to maintain high specificity where other methods failed.

Key Findings and Limitations

The study demonstrates that autonomous prompt optimization can effectively bridge the gap between simple rule-based matching and complex, resource-heavy supervised learning. Pythia’s performance remained stable when moving from development data to validation data, showing that its optimized prompts generalize well. However, the researchers noted that the system’s sensitivity weakens for very rare symptoms, specifically those appearing in less than 2% of the notes. These results suggest that while Pythia is a powerful tool for clinical extraction, its effectiveness is tied to the prevalence of the condition being tracked and the specific operating points set by the user.

Comments (0)

No comments yet

Be the first to share your thoughts!