Back to AI Research

AI Research

COMPASS estimates model-pair compatibility before expensive training

Key Takeaways

  • A two-part diagnostic ranks model pairs for cross-modal alignment and checkpoint merging.
  • ## Choose compatible models before committing compute
  • [COMPASS](https://arxiv.org/abs/2610.18638) addresses a practical cost in combining pretrained models: teams often learn whether a pair works only after running alignment or merging experiments.
  • The proposed diagnostic estimates compatibility before that expensive step.
  • The authors focus on local neighborhoods in model representations.

Choose compatible models before committing compute

COMPASS addresses a practical cost in combining pretrained models: teams often learn whether a pair works only after running alignment or merging experiments. The proposed diagnostic estimates compatibility before that expensive step.
The authors focus on local neighborhoods in model representations. Their pair-specific ease measure uses overlap between the encoders' nearest neighbors. A second measure, the paradigm-level ceiling, represents how much agreement the training paradigms can support. The distinction separates a pair that currently disagrees from a pairing whose training setup may limit agreement.

Two scores suggest different experiments

The framework prioritizes pairs with both strong initial agreement and compatible paradigms. Low agreement with a compatible paradigm can justify additional compute. A low ceiling instead suggests reconsidering the pretraining paradigm; pairs weak on both dimensions receive lower priority.
The paper evaluates 72 cross-modal alignment experiments and 392 task-arithmetic model merges. It reports average Pearson correlations of 0.91 for cross-modal alignment and 0.70 for model merging between its estimates and downstream success.
Those are correlations within the evaluated settings. A high score is not a guarantee that a merged model will satisfy a particular task, preserve desired behavior or outperform every alternative. The diagnostic ranks experiments; downstream evaluation still determines whether the resulting system is useful.

Use the score to narrow a real search

Franklin analysis: begin with candidate pairs that would otherwise compete for the same training budget. Record the predicted ranking before running the downstream experiments, then compare that ranking with held-out task results and total compute spent.
Keep the two COMPASS dimensions separate in that record. If an initially weak pair improves with more training, that is a different result from a pair whose paradigm prevents useful alignment. A single compatibility label would hide the choice the diagnostic is intended to support.

Comments