Self-Evolving Human-Centered Framework for Explaina...

Self-Evolving Human-Centered Framework for Explaina... | AI Research

Key Takeaways

Self-Evolving Human-Centered Framework for Explainable Depression Symptom Annotation This paper introduces a new framework designed to improve how we label a...
Annotation quality is a major bottleneck in building reliable and explainable artificial intelligence (XAI) systems for mental health research.
We propose a self-evolving, expert-in-the-loop annotation framework for Major Depressive Disorder (MDD) that combines large language model (LLM)-assisted labeling with expert verification.
The framework is intended to support the construction of explainable, DSM-5-TR-aligned datasets rather than to perform clinical diagnosis.
It operates in three stages: candidate evidence selection from textual records, criterion-level DSM-5-TR analysis, and case-level synthesis that produces label-level diagnostic and severity annotations.

Paper AbstractExpand

Annotation quality is a major bottleneck in building reliable and explainable artificial intelligence (XAI) systems for mental health research. In depression-related datasets, labels are often assigned without structured evidence, symptom-level justification, or traceable alignment with the criteria of the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition, Text Revision (DSM-5-TR), limiting both transparency and downstream model interpretability. We propose a self-evolving, expert-in-the-loop annotation framework for Major Depressive Disorder (MDD) that combines large language model (LLM)-assisted labeling with expert verification. The framework is intended to support the construction of explainable, DSM-5-TR-aligned datasets rather than to perform clinical diagnosis. It operates in three stages: candidate evidence selection from textual records, criterion-level DSM-5-TR analysis, and case-level synthesis that produces label-level diagnostic and severity annotations. A dual-memory architecture, composed of Example Memory and Reflection Memory, is designed to internalize expert feedback and iteratively improve future annotations without retraining. We describe this mechanism and leave its evaluation across multiple feedback cycles to future work. In addition to final labels, the framework exports clinical evidence, reasoning traces, and edit histories, enabling comprehensive auditability. In a pilot study using expert-reviewed samples, the proposed approach improves annotation consistency and explainability while reducing manual revision effort.

Self-Evolving Human-Centered Framework for Explainable Depression Symptom Annotation
This paper introduces a new framework designed to improve how we label and analyze data related to Major Depressive Disorder (MDD). Currently, many AI systems for mental health lack transparency, often providing labels without explaining the clinical evidence behind them. This framework addresses that gap by creating a collaborative, "expert-in-the-loop" system that aligns AI-generated annotations with the official clinical standards of the DSM-5-TR. Rather than attempting to perform automated clinical diagnoses, the system acts as a tool to help experts build high-quality, explainable datasets more efficiently.

How the Framework Works

The system operates through a three-stage pipeline that mimics clinical reasoning. First, it screens raw text to identify relevant evidence, reducing the amount of reading an expert must do. Second, it performs a detailed analysis of this evidence against the nine specific criteria for MDD defined in the DSM-5-TR, providing a clinical rationale and highlighting key phrases for each. Finally, it synthesizes these findings into a structured case profile. Throughout this process, the system provides a "conflict warning" if it detects contradictory information, ensuring the expert can intervene where necessary.

A Self-Evolving Design

A core innovation of this framework is its "dual-memory" architecture, which allows the system to improve over time without needing to be retrained. It uses an "Example Memory" to store gold-standard cases verified by experts and a "Reflection Memory" to store distilled clinical insights and lessons learned from previous corrections. When an expert reviews and edits the AI’s work, the system records these changes as feedback. This creates a closed-loop process where the AI’s future suggestions become more accurate and better aligned with expert intuition as it "learns" from the audit trail of human edits.

Results and Efficiency

In a pilot study involving 10 complex clinical cases, the framework demonstrated strong agreement with expert human annotators. The system proved capable of identifying relevant evidence and correctly mapping it to DSM-5-TR criteria across several different AI models. Most notably, the framework significantly reduced the burden on human experts. By shifting the workload from manual annotation to a review-and-verify process, the system saved experts between 63% and 75% of the time typically required for manual labeling, all while maintaining high levels of accuracy and providing a clear, auditable history of how every decision was reached.

Important Considerations

The authors emphasize that this framework is intended to support researchers and clinicians, not to replace professional judgment. While the system is model-agnostic—meaning it can work with various underlying AI technologies—the quality of the output remains dependent on expert oversight. The study focused on the initial implementation and performance of the framework; the researchers noted that a long-term evaluation of how the system evolves across many feedback cycles remains a task for future work.