Back to AI Research

AI Research

CART coordinates vision and language adaptation during continual instruction tuning

Key Takeaways

  • Compatible Routing maintains experts in both modules and combines them according to relevance and cross-module compatibility.
  • ## Forgetting can begin at the visual interface
  • Compatible Routing, or CART, examines multimodal models that learn sequential tasks through continual instruction tuning.
  • The authors focus on the projector that aligns visual representations with the language model.
  • Protecting the language component alone can leave the projector changing in ways that increase mismatch between the two.

Forgetting can begin at the visual interface

Compatible Routing, or CART, examines multimodal models that learn sequential tasks through continual instruction tuning. The authors focus on the projector that aligns visual representations with the language model. Protecting the language component alone can leave the projector changing in ways that increase mismatch between the two.
Their argument is that this asymmetric adaptation can worsen catastrophic forgetting. A model may retain language parameters while the representation entering those parameters drifts away from what earlier tasks relied on.

Route both modules together

CART keeps task-specific experts for the projector and the language model. It combines them through weighted mixtures, jointly inferring weights using both expert relevance and compatibility across modules.
This makes coordination part of the routing decision. Choosing a relevant visual expert and a relevant language expert separately does not necessarily establish that the pair works well together. The paper's design explicitly represents that relationship instead of treating the projector as an always-updated connection outside the forgetting problem.

Consolidate experts as the task sequence grows

The framework also introduces relevance-guided expert consolidation. Within each module, it merges experts whose relevance profiles are similar across samples, aiming to limit model growth as new tasks arrive.
The authors report strong experimental performance, but the captured abstract does not supply a universal numerical reduction in forgetting, memory use or training cost. Those outcomes should be checked in the paper's particular task sequence and comparison setup rather than inferred from the method's name.
Franklin's reading is to evaluate the boundary between vision and language as well as each module separately. A reproduction should measure performance on earlier tasks after each update and inspect the effect of consolidation on both capacity and retained capability. CART offers a coordinated adaptation strategy; it does not establish that all expert mixtures remain compatible under every new task or distribution change.
Read the CART paper.

Comments