Back to AI Research

AI Research

Towards Expert-level Medical AI for Real-time Video... | AI Research

Key Takeaways

  • Towards Expert-level Medical AI for Real-time Video Consultations presents AMIE (Articulate Medical Intelligence Explorer), a multi-agent AI system designed...
  • Audio-visual interaction is the standard for patient-physician consultations, enabling natural communication and effective assessment of illness through non-verbal cues.
  • While text-based AI has shown promise, it discards essential perceptual dimensions and limits patients who cannot articulate symptoms in writing.
  • Early efforts to extend medical AI to audio-visual interaction have demonstrated feasibility but not reached clinician-level performance.
  • Here, we provide the first demonstration of expert-level AI in real-time clinical video consultations using AMIE (Articulate Medical Intelligence Explorer) in a video configuration.
Paper AbstractExpand

Audio-visual interaction is the standard for patient-physician consultations, enabling natural communication and effective assessment of illness through non-verbal cues. While text-based AI has shown promise, it discards essential perceptual dimensions and limits patients who cannot articulate symptoms in writing. Early efforts to extend medical AI to audio-visual interaction have demonstrated feasibility but not reached clinician-level performance. Here, we provide the first demonstration of expert-level AI in real-time clinical video consultations using AMIE (Articulate Medical Intelligence Explorer) in a video configuration. AMIE (Video) is a Gemini-based multi-agent system integrating low-latency dialogue, clinical reasoning, and real-time audio-visual perception. To guide development, we established a taxonomy and automated evaluations for clinical audio-visual cues in telehealth settings. In a randomized Objective Structured Clinical Examination (OSCE) study with 30 primary care physicians (PCPs), 15 patient actors and 100 clinical scenarios, we compared AMIE (Video), its text-only counterpart AMIE (Text), and PCPs consulting via video. Clinical evaluators rated AMIE (Video) on par or better than PCPs in history-taking, diagnosis, management, and physical observation and examination. Patient actors preferred AMIE's approach to assessing and explaining conditions, while PCPs were preferred for rapport and partnership building. In modality ablation, patient actors preferred AMIE (Video)'s interface over text chat for communicative effectiveness, convenience, and feeling understood. Limitations remain in fine anatomical precision, subtle affective nuances, and high-frequency movements. While further research is needed before real-world translation, these results mark an important milestone toward AI systems capable of augmenting care across the sensory complexity of clinical practice.

Towards Expert-level Medical AI for Real-time Video Consultations presents AMIE (Articulate Medical Intelligence Explorer), a multi-agent AI system designed to conduct clinical consultations via real-time video. The research aims to bridge the gap between text-based medical AI and the sensory-rich environment of live patient-physician interactions, where non-verbal cues like facial expressions, posture, and vocal tone are essential for accurate assessment.

System Architecture

AMIE (Video) uses a three-agent architecture built on Gemini models to manage the complexities of live video consultations:

  • Talker Agent: Manages the conversational interface, aiming for low-latency responses by processing the most recent five seconds of video and audio. It coordinates with the other agents to ensure responses are informed by clinical goals.

  • Planner Agent: Maintains a persistent memory of the clinical state, including patient symptoms, differential diagnoses, and management plans. It tracks "milestones," such as specific information to collect or physical actions for the patient to perform.

  • Perception Agent: Analyzes continuous audio and video streams to identify and log clinically relevant cues, such as a patient’s affect or physical signs, ensuring this information remains accessible for the system’s reasoning process throughout the encounter.

Evaluation Methodology

The researchers conducted a randomized Objective Structured Clinical Examination (OSCE) study to test the system. The study involved 30 primary care physicians (PCPs), 15 professional patient actors, and 100 clinical scenarios. The performance of AMIE (Video) was compared against its text-only counterpart, AMIE (Text), and PCPs conducting video consultations. Clinical evaluators, consisting of 20 board-certified PCPs, assessed the consultations using standard OSCE criteria and case-specific rubrics.

Key Findings

Clinical evaluators rated AMIE (Video) as performing on par with or better than human PCPs in areas including history-taking, diagnosis, management, and physical observation. Patient actors reported a preference for AMIE’s approach to assessing and explaining conditions, while they preferred human PCPs for building rapport and partnership. In a comparison of modalities, patient actors preferred the video interface over text chat, citing improved communicative effectiveness, convenience, and a greater sense of being understood.

Limitations and Considerations

The researchers identified several areas where the system requires further development. Current limitations include a lack of fine anatomical precision, difficulty in capturing subtle affective nuances, and challenges in processing high-frequency movements. The authors note that while these results represent a milestone in AI-augmented care, additional research is necessary before the system can be translated into real-world clinical practice.

Comments (0)

No comments yet

Be the first to share your thoughts!