NeuroVFM: University of Michigan's New Neuroimaging Foundation Model

A research team from the University of Michigan has introduced NeuroVFM, a generalist visual foundation model designed to advance neuroimaging diagnosis. Published in Nature Medicine, the model leverages a self-supervised learning approach known as Vol-JEPA to analyze brain anatomy and pathology directly from uncurated clinical MRI and CT volumes. By training on 5.24 million volumes from the UM-NeuroImages dataset, the model bypasses the need for paired radiology reports or disease-specific labels, addressing a significant gap in clinical AI where patient privacy concerns often limit the use of neuroimaging data in general-purpose models.

The Vol-JEPA Architecture

NeuroVFM utilizes Vol-JEPA, a vision-only algorithm that extends I-JEPA and V-JEPA methods to volumetric medical data. Instead of performing traditional pixel reconstruction, the model learns by predicting representations within a latent space. During the training process, 3D volumes are tokenized into patches, with the model tasked with predicting masked-region latents based on visible context. This approach incorporates foreground-focused masking and precomputed head masks to ensure the encoder prioritizes neuroanatomy over background noise.
The efficiency of this training method is notable, as a full Vol-JEPA run requires fewer than 1,000 GPU hours. This represents a significant performance improvement, training over seven times faster than 3DINO baselines while supporting larger batch sizes. The resulting base encoder contains 85.8 million parameters, providing a robust foundation for various diagnostic tasks.

Clinical Performance and Diagnostic Capabilities

In evaluations across 156 diagnostic tasks, NeuroVFM achieved a macro-averaged AUROC of 92.68 on CT scans and 92.49 on MRI scans. These results outperformed several baselines, including models trained with report supervision and voxel reconstruction. Beyond basic classification, the model supports downstream applications such as structured report generation, triage, and grounded predictions. When paired with a Qwen3-14B model in a system dubbed NeuroVFM-LLaVA, the framework demonstrated the ability to generate clinical findings and determine acuity levels with high accuracy.
A one-week prospective study involving 1,155 cases confirmed the model's potential in clinical settings, where it reached 92.6% balanced triage accuracy. Despite this performance, the researchers emphasize that NeuroVFM is intended as a decision-support tool rather than a system for autonomous screening. The model maintains an 86.5% sensitivity rate for critical findings, meaning some urgent cases may still be missed.

Implementation and Considerations

The NeuroVFM stack is available for research use, with code released under an MIT license and model weights provided under a CC-BY-NC-SA-4.0 license. Users can access the encoder, diagnostic heads, and vision-language model components through the project's repository. While the model shows versatility across different manufacturers and demographic subgroups, its current application is limited by its status as a non-FDA-approved tool. Furthermore, the researchers note that the model remains susceptible to biases inherent in its architecture and the single-site academic data used for training. Future deployment will require careful integration into clinical workflows to account for its current sensitivity limitations.