CHARM: A Multimodal Graph Foundation Model with Hie...

CHARM: A Multimodal Graph Foundation Model with Hie... | AI Research

Key Takeaways

CHARM is a multimodal graph foundation model designed to enable zero-shot transfer across different graph domains without requiring target-domain labels or m...
Graph foundation models (GFMs) have emerged as a promising paradigm for transferring knowledge across graph domains and tasks.
Real-world graphs associate nodes with text, images, and other modalities, making multimodal graphs essential for representing complex entities and relations.
Moreover, collecting labels and adapting models for every new graph domain is costly and often infeasible, motivating zero-shot transfer.
Unfortunately, zero-shot transfer on multimodal graphs remains underexplored.

Paper AbstractExpand

Graph foundation models (GFMs) have emerged as a promising paradigm for transferring knowledge across graph domains and tasks. Real-world graphs associate nodes with text, images, and other modalities, making multimodal graphs essential for representing complex entities and relations. Moreover, collecting labels and adapting models for every new graph domain is costly and often infeasible, motivating zero-shot transfer. Unfortunately, zero-shot transfer on multimodal graphs remains underexplored. Existing GNN-based graph foundation models typically require downstream adaptation, whereas LLM-based graph methods mainly address unimodal graphs or tasks within a single domain. This setting presents two key challenges. First, models must generalize knowledge from individual modalities while capturing transferable cross-modal relations. Second, without target-domain fine-tuning, node representations remain entangled with domain-specific structures and modality-specific characteristics, obscuring shared concepts in unseen domains. To address these challenges, we propose CHARM, a multimodal graph foundation model with hierarchical context modeling for zero-shot transfer. CHARM replaces isolated raw nodes with hierarchical graph contexts that capture multimodal semantics and cross-modal relations. These contexts map domain-specific node patterns to shared high-level concepts, reducing reliance on target-domain supervision or adaptation. A modality-aware graph context encoder integrates multimodal information with graph structure and converts the resulting representations into graph tokens for a large language model . Experiments show consistent improvements on zero-shot multimodal graph tasks.

CHARM is a multimodal graph foundation model designed to enable zero-shot transfer across different graph domains without requiring target-domain labels or model fine-tuning. By organizing nodes into a hierarchical structure and incorporating cross-modal relationships, the model allows a large language model (LLM) to reason about graph data in new, unseen environments.

Addressing the Zero-Shot Challenge

Existing graph foundation models often struggle with zero-shot transfer because node representations are typically tied to domain-specific features or local structures. When a model encounters a new domain, these specific details—such as unique vocabulary or visual styles—can obscure the shared concepts necessary for accurate prediction. The authors, including Ankang Yang and colleagues from Tianjin University and The George Washington University, identify that raw node features are often dominated by domain bias, making it difficult for models to identify semantically related nodes across different graphs.

Hierarchical Context Modeling

To overcome domain bias, CHARM replaces isolated node inputs with a "synthetic hierarchical graph." This process organizes raw nodes into three levels of abstraction:

Layer-1 and Layer-2 Anchors: These act as semantic summaries that group nodes based on textual and visual features, moving from fine-grained instances to broader concepts.
Global Anchors: These provide a shared reference space across different domains, allowing the model to associate related concepts even when the underlying data distributions differ.
The model also constructs "modality-complementary bridges." These connections link nodes that are structurally related but exhibit strong similarity in only one modality (e.g., text or image). This ensures that the model does not discard useful evidence just because a node’s visual appearance or textual description is inconsistent with its neighbors.

Graph Context Encoding

For any given target node, CHARM retrieves a compact "graph context" that includes its local structural neighbors, hierarchical semantic anchors, and bridge neighbors. This context is converted into continuous graph tokens, which are then processed by a frozen LLM. By providing the LLM with both local topology and high-level semantic references, the model can perform reasoning tasks—such as node classification and link prediction—without needing to be retrained on the target domain.

Performance and Scope

The researchers report that CHARM shows consistent improvements on zero-shot multimodal graph tasks. The approach is specifically intended for scenarios where collecting labels for every new graph domain is costly or infeasible. By using hierarchical abstraction, the model reduces the reliance on target-domain supervision, allowing it to leverage pretrained multimodal reasoning capabilities to interpret unseen graph structures and attributes.