Rethinking Heterogeneous LLM Merging: A Weighted Mo... | AI Research

Rethinking Heterogeneous LLM Merging: A Weighted Model Averaging Perspective
This paper explores whether large language models (LLMs) with different parameter sizes can be combined—or "merged"—using simple mathematical averaging. Traditionally, merging models requires complex techniques like fine-tuning, adapters, or aligning internal representations to ensure the models are compatible. This research challenges that assumption by testing whether a straightforward, training-free approach can successfully blend the capabilities of models with different architectures and scales.

A Simple Recipe for Merging

The researchers propose two primary methods to make mismatched models compatible for direct weighted averaging. The first, "union-style" merging, takes a smaller model and expands its parameter space to match that of a larger model. The second, "intersection-style" merging, takes a larger model and truncates it to fit the dimensions of a smaller one. Both methods avoid any additional training, data distillation, or complex alignment steps. By placing the models into a shared parameter space, the researchers can then use simple interpolation—calculating a weighted average of the two models' weights—to create a new, merged checkpoint.

Performance and Capability Transfer

The study tested these methods across various Qwen-family models, ranging from 3 billion to 32 billion parameters. The results demonstrate that this simple approach is surprisingly effective. In many cases, the merged models outperformed the individual source models by transferring complementary strengths. For example, when merging a 14B model with a 32B model, the resulting model showed improvements across benchmarks covering mathematical reasoning, code generation, and instruction following. The researchers found that dimensional adaptation acts as a functional bridge, allowing the models to share knowledge without needing to be explicitly trained to work together.

The Limits of Direct Fusion

Despite these successes, the research highlights clear boundaries for this technique. The effectiveness of the merge is highly sensitive to the "mixing ratio"—the weight assigned to each model. While small-ratio interpolation often leads to performance gains, attempting to balance the models equally (near-balanced interpolation) frequently causes the model's performance to collapse. Furthermore, the researchers observed a "seesaw effect," where improvements in one capability (such as coding) often come at the cost of regressions in another (such as general knowledge).

Implications for Model Merging

These findings suggest that simple parameter averaging is a powerful baseline for heterogeneous model merging. The fact that such a basic, training-free method can achieve competitive results implies that the primary challenges in merging models may not always require complex, resource-heavy solutions. Instead, the limits of this approach likely point to fundamental compatibility bottlenecks between models that even more sophisticated methods may struggle to overcome. This work provides a clear, interpretable framework for understanding how and when different LLMs can be combined to create more capable systems.