Multimodal Pretraining for Generalizable EEG Repres...

Multimodal Pretraining for Generalizable EEG Repres... | AI Research

Key Takeaways

Multimodal Pretraining for Generalizable EEG Representation Learning This research introduces a new foundation model designed to improve how we detect and an...
Electroencephalography (EEG) models used for epilepsy are often limited to specific datasets and tasks.
This limited approach can make it challenging to apply these models across different datasets or in various situations.
However, recent studies in foundation models and self-supervised learning suggest that an adaptable EEG backbone could support a range of EEG related tasks.
The pretraining process relies on several innovative techniques, such as masked modeling, cross-view contrastive alignment, and temporal consistency losses.

Paper AbstractExpand

Electroencephalography (EEG) models used for epilepsy are often limited to specific datasets and tasks. This limited approach can make it challenging to apply these models across different datasets or in various situations. However, recent studies in foundation models and self-supervised learning suggest that an adaptable EEG backbone could support a range of EEG related tasks. In this study, we have developed a multimodal EEG foundation model that combines a raw signal encoder based on the Mamba architecture, a Vision Transformer (ViT)-style encoder for time-frequency data, and a lightweight encoder for text, all within a shared embedding space. The pretraining process relies on several innovative techniques, such as masked modeling, cross-view contrastive alignment, and temporal consistency losses. These methods are designed to create rich, seizure-relevant representations without requiring labeled data. To assess the efficacy and generalization of our pretrained model, we fine-tuned it on the canonical CHB-MIT seizure detection benchmark and additional seizure detection datasets, and conducted extensive experiments comparing different model variants. On the standard CHB-MIT split, our best single model achieved an AUROC of 0.874, and an ensemble variant reached 0.878 AUROC, representing state-of-the-art performance on this benchmark. In addition to standard train-test splits, we evaluated performance under a leave-one-subject-out (LOSO) protocol, which is rarely reported in prior EEG seizure modeling work and highlights the difficulty of patient-independent seizure detection, with a mean LOSO balanced accuracy of 0.558 across 19 subjects. Across datasets and evaluation settings, our multimodal foundation model enabled robust seizure detection and straightforward adaptation to new seizure detection scenarios, while also supporting interpretable seizure localization.

Multimodal Pretraining for Generalizable EEG Representation Learning
This research introduces a new foundation model designed to improve how we detect and analyze seizures in EEG data. Traditionally, EEG models are built for specific tasks or datasets, making them difficult to use in new clinical environments or with different patients. By using a multimodal approach—combining raw brain signals, time-frequency patterns, and clinical text—this model creates a more adaptable "backbone" that can learn rich, seizure-relevant features without needing large amounts of labeled data.

A Multimodal Approach to Brain Signals

The model processes EEG data through three distinct but connected pathways. First, a Mamba-based encoder analyzes the raw time-domain signals. Second, a Vision Transformer (ViT)-style encoder processes time-frequency "scalograms," which visualize how brain wave frequencies change over time. Finally, a lightweight text encoder incorporates clinical context. These three streams are aligned within a shared space, allowing the model to leverage complementary information—such as the raw shape of a wave and its specific frequency signature—to identify seizures more accurately.

Training for Robustness

To ensure the model learns effectively without relying on human-labeled data, the researchers employed several self-supervised techniques. These include masked modeling (where the model learns to reconstruct missing parts of the data), cross-view contrastive alignment (ensuring the raw signal and frequency views match), and temporal consistency checks. This pretraining process allows the model to build a deep understanding of EEG patterns that can then be fine-tuned for specific clinical tasks, such as detecting seizures in new patients.

Performance and Generalization

The researchers tested the model on the CHB-MIT seizure detection benchmark, where it achieved state-of-the-art performance. A key strength of this study is its use of a "leave-one-subject-out" (LOSO) protocol. This is a rigorous testing method that evaluates the model on patients it has never seen before, providing a more realistic estimate of how the tool would perform in a real-world clinical setting. While the results highlight the inherent difficulty of patient-independent seizure detection, the model demonstrated strong adaptability and superior performance compared to existing methods.

Interpretability in Clinical Settings

Beyond just detecting seizures, the model provides insights into why a specific decision was made. Using techniques like GradCAM, the researchers showed that the model focuses on physiologically meaningful patterns, such as spike-and-wave discharges and specific frequency bands known to be associated with epilepsy. This focus on interpretable features suggests that the model is not just finding statistical shortcuts, but is instead identifying the actual clinical signatures of seizures, which is essential for building trust in medical AI tools.