Contrastive Mixed Prompt Learning for Incomplete Mu... | AI Research

Contrastive Mixed Prompt Learning for Incomplete Multimodal Sentiment Analysis with Unseen Modality Combination addresses the challenge of sentiment analysis models failing when they encounter combinations of data modalities (such as text, audio, and video) during testing that were not present during training. The authors introduce the task of Incomplete Multimodal Sentiment Analysis with Unseen Modality Combinations (IMSAUMC) to improve model generalization in real-world scenarios where sensor failures or privacy constraints lead to inconsistent data availability.

Addressing Unseen Modality Combinations

Existing sentiment analysis models often assume that all possible combinations of modalities are observed during training. However, in practical applications, missing data patterns are frequently structured rather than random. This leads to a performance gap when a model encounters a combination of modalities at inference time that it never saw during the training phase. The researchers propose the Contrastive Mixed Prompt Learning (CMPL) model to bridge this gap, marking the first attempt to specifically address the IMSAUMC task.

Label-Guided Contrastive Feature Learning

To improve how the model understands multimodal data, the authors developed a label-guided contrastive feature learning (LCFL) module. Traditional contrastive learning often ignores the structural relationships between samples, which can lead to suboptimal representations. The LCFL module pulls together representations of samples that share the same sentiment labels while maintaining distances between different samples that are proportional to their label dissimilarity. This ensures that the model preserves semantic structure even when some modalities are missing.

Mixed Prompts and Routing

The model utilizes a mixed prompts learning mechanism to handle different modality combinations. Instead of relying on a single approach for all inputs, the system uses a soft router that dynamically selects and weights specific prompts based on the available input data. By blending these prompts, the model can better represent the relationships between different modality combinations. This is further supported by three prompt contrastive learning strategies—modality information preservation, cross-combination complementarity, and conditional information alignment—which allow the model to learn from the relationships between known and unknown combinations.

Performance and Results

The researchers evaluated CMPL on three datasets: CMU-MOSI, CMU-MOSEI, and SIMS-V2. According to the paper, the CMPL model achieved an improvement of more than 5% in accuracy compared to existing state-of-the-art approaches. This performance suggests that incorporating label-guided feature learning and dynamic prompt routing helps the model maintain stability and accuracy when faced with diverse, unseen testing scenarios.