Back to AI Research

AI Research

SHAP-Weighted Cross-Modal Expert Fusion for Emotion... | AI Research

Key Takeaways

  • This research investigates how to effectively combine different types of data—such as text, audio, and facial expressions—for emotion and sentiment recogniti...
  • Multimodal emotion and sentiment recognition is commonly addressed by early fusion, which concatenates modalities before classification, or late fusion, which combines independently trained unimodal predictors.
  • Early fusion can be accurate but monolithic, while late fusion is modular but may lose cross-modal interactions.
  • This paper revisits XAI-guided adaptive fusion (\xgaf), a tree-based mixture of unimodal and cross-modal experts whose sample-level weights are derived from TreeSHAP attribution magnitudes.
  • We focus on the effect of SHAP attribution reduction when experts have unequal feature dimensionalities.
Paper AbstractExpand

Multimodal emotion and sentiment recognition is commonly addressed by early fusion, which concatenates modalities before classification, or late fusion, which combines independently trained unimodal predictors. Early fusion can be accurate but monolithic, while late fusion is modular but may lose cross-modal interactions. This paper revisits XAI-guided adaptive fusion (\xgaf), a tree-based mixture of unimodal and cross-modal experts whose sample-level weights are derived from TreeSHAP attribution magnitudes. We focus on the effect of SHAP attribution reduction when experts have unequal feature dimensionalities. In this setting, mean-abs and median-abs reductions can suppress high-dimensional cross-modal experts, whereas sum-abs reduction preserves total attribution mass. On MELD 7-class emotion recognition, sum-abs \xgaf{} nearly matches early fusion across three face-sequence aggregators; the Transformer variant reaches 0.5983 \wf{}, compared with 0.6018 for early fusion and 0.4598 for probability-average late fusion. McNemar testing shows no significant difference between sum-abs \xgaf{} and early fusion on MELD ($p=1.000$), while \xgaf{} remains significantly better than late fusion ($p<0.0001$). On CMU-MOSEI 3-class sentiment recognition, sum-abs \xgaf{} reaches 0.6519 \wf{}, slightly exceeding early fusion (0.6485) and late fusion (0.5696). Ablation studies show that the main gain comes from adding cross-modal experts, especially the trimodal expert, rather than from complex per-sample routing. Diagnostics further show that mean-abs and median-abs weights are nearly uniform, while sum-abs weights concentrate on the trimodal expert. Thus, the main contribution is a transparent empirical analysis of how SHAP reduction, expert dimensionality, and cross-modal expert design affect modular multimodal fusion.

This research investigates how to effectively combine different types of data—such as text, audio, and facial expressions—for emotion and sentiment recognition. While standard methods like "early fusion" (combining all data at once) are accurate, they are often rigid and difficult to interpret. Conversely, "late fusion" (combining independent predictions) is modular but often fails to capture the complex interactions between different modalities. This paper explores an alternative approach called XGAF (XAI-guided adaptive fusion), which uses a pool of specialized experts and a decision-making gate based on SHAP attribution values to determine which expert should be trusted for a given sample.

The Challenge of Expert Dimensionality

A core focus of the study is the technical hurdle of combining experts that have different input sizes. When using SHAP values to weigh the importance of each expert, the researchers found that simply taking the "mean" or "median" of these values can be misleading. Because these methods calculate importance on a per-feature basis, they inadvertently suppress high-dimensional experts—those that process more complex or larger sets of data. The paper demonstrates that using the "sum" of absolute SHAP values is a more effective strategy, as it preserves the total "attribution mass" of each expert, allowing the model to properly value the contribution of cross-modal experts.

How the Fusion Works

The proposed framework organizes experts into a pool that includes unimodal (e.g., text-only), bimodal, and trimodal (text, audio, and face combined) classifiers. For every input, the system calculates SHAP values to explain which features are driving the prediction. These values are converted into scores, which are then passed through a "temperature-scaled" softmax function to assign weights to each expert. This creates a modular architecture where the system can dynamically favor the most relevant expert for a specific piece of content, rather than relying on a single, monolithic model.

Key Findings and Performance

Experiments on the MELD and CMU-MOSEI datasets revealed that the sum-based weighting method significantly outperforms standard late fusion and matches or slightly exceeds the performance of early fusion. Notably, the researchers discovered that the performance gains were not driven by complex, per-sample routing, but rather by the simple inclusion of cross-modal experts—specifically the trimodal expert. Diagnostic analysis showed that while mean and median-based gates resulted in nearly uniform, uninformative weights, the sum-based gate successfully concentrated influence on the most capable experts.

Important Limitations

The authors emphasize that this study is an empirical analysis of fusion design rather than an attempt to create a new state-of-the-art model for dialogue emotion recognition. Because the model processes each utterance independently, it does not account for conversation context or speaker history. Furthermore, while the SHAP-based gate provides a level of transparency, the current implementation does not address real-world challenges like missing or noisy data. The researchers present these findings as a transparent lesson on how attribution-based gating behaves, serving as a guide for future work in modular multimodal systems.

Comments (0)

No comments yet

Be the first to share your thoughts!