Tracing and Coordinating Cross-Layer Influence for... | AI Research

Tracing and Coordinating Cross-Layer Influence for Multimodal Model Merging
Multimodal model merging aims to combine several specialized AI experts—such as those trained for visual reasoning or chart understanding—into a single, efficient model. While previous methods typically combine these experts layer by layer, this approach often ignores how an update in one layer changes the information passed to the next. This paper introduces TAC-Merge, a framework designed to trace how expert updates influence a model across its entire depth and coordinate these updates to ensure visual and textual information interact correctly in the final, merged model. The same large language models question is explored in DRACO, which adds a research perspective.

Understanding Cross-Layer Influence

In deep multimodal models, an expert update does not just affect the specific layer where it is applied; it alters the representations that all subsequent layers receive. When visual and language updates are combined, their effects become coupled, making it difficult to isolate the contribution of each expert. TAC-Merge addresses this by using Multimodal Influence Mapping (MIM), which constructs "influence graphs." These graphs track how specific expert updates at different depths and token positions affect the model's output, allowing the system to distinguish between changes in visual processing and changes in how the model follows textual instructions.

Coordinating Expert Updates

Once the influence of each expert is mapped, the challenge is to decide how much of each expert to include in the final model. TAC-Merge uses a module called Coupled Merge Control (CMC) to manage this. Instead of adjusting every parameter independently, which is computationally expensive, CMC identifies a compact "control subspace" based on the measured responses of the model. It then fits a local quadratic model to predict how different combinations of weights will perform. By jointly optimizing these regional weights, the framework ensures that the merged model maintains the complementary strengths of the original experts. The same large language models question is explored in Beyond Aggregate Scores, which adds a research perspective.

Performance and Generalization

The researchers evaluated TAC-Merge across two different multimodal backbones, testing it on eight seen tasks and four unseen tasks. The results demonstrate that TAC-Merge outperforms existing merging methods in average performance. Notably, the model’s success on unseen tasks suggests that the framework effectively consolidates capabilities in a way that allows the merged model to generalize beyond the specific datasets used to train the individual experts.

Key Considerations

The effectiveness of TAC-Merge relies on its ability to balance two competing objectives: matching the "Ricci curvature" of task-expert graphs—which helps preserve the internal structure of how information flows—and ensuring the merged model’s predictions align with the original experts. By using a calibration set of inputs to guide this process, the framework provides a structured way to synthesize a single, static model that retains the specialized knowledge of its components without the overhead of deploying multiple separate models. The same large language models question is explored in Kernel-Managed Shared Memory for System-Wide Personalization, which adds a research perspective. as detailed in the full paper on Arxiv