Back to AI Research

AI Research

Does Forgetting Transfer Across Modalities? A Real-... | AI Research

Key Takeaways

  • Vision-Language Models (VLMs) can memorize sensitive or harmful information during pretraining.
  • Vision-Language Models (VLMs), like Large Language Models (LLMs), may memorize sensitive, copyrighted, or harmful knowledge from their pretraining corpora.
  • Removing such knowledge is essential for building trustworthy AI systems.
  • However, existing studies primarily focus on forgetting within individual modalities.
  • Although recent work has begun to explore cross-modal consistency in unlearning, the cross-modal transfer of real-world knowledge unlearning remains insufficiently studied.
Paper AbstractExpand

Vision-Language Models (VLMs), like Large Language Models (LLMs), may memorize sensitive, copyrighted, or harmful knowledge from their pretraining corpora. Removing such knowledge is essential for building trustworthy AI systems. However, existing studies primarily focus on forgetting within individual modalities. Although recent work has begun to explore cross-modal consistency in unlearning, the cross-modal transfer of real-world knowledge unlearning remains insufficiently studied. To address this gap, we introduce UNLINK-VL, a real-world benchmark for cross-modal knowledge unlearning in VLMs. Under a post-hoc unlearning setting in which the original forget and retain corpora are unavailable, UNLINK-VL selects visually identifiable real-world entities as unlearning targets and associates them with corresponding images and one-hop and multi-hop facts derived from Wikidata. The benchmark comprises four complementary subsets that evaluate direct forgetting of target knowledge, the propagation of forgetting through relational knowledge, the preservation of related non-target knowledge, and robustness to semantically equivalent queries. We train models under text-only and multimodal unlearning settings and evaluate forgetting effectiveness and retained utility across textual, visual, and cross-modal scenarios. Extensive experiments reveal a pronounced asymmetry in cross-modal transfer: multimodal unlearning remains effective under textual evaluation, whereas text-only unlearning transfers poorly to visual and cross-modal scenarios. Meanwhile, the evaluated methods largely preserve the models' general capabilities. These findings demonstrate that relying solely on intra-modal evaluation, particularly text-only evaluation, may substantially overestimate the effectiveness of knowledge unlearning in VLMs, underscoring the need for cross-modal unlearning and evaluation.

Vision-Language Models (VLMs) can memorize sensitive or harmful information during pretraining. While machine unlearning aims to remove this data, most existing research focuses on forgetting within a single modality (text-only or image-only). The paper "Does Forgetting Transfer Across Modalities? A Real-World Benchmark for Cross-Modal Knowledge Unlearning Evaluation" investigates whether unlearning a fact in one modality—such as text—successfully removes that knowledge when the model is queried through another modality, such as an image.

The UNLINK-VL Benchmark

To evaluate cross-modal knowledge transfer, the authors introduced UNLINK-VL, a benchmark consisting of visually identifiable real-world entities. The researchers linked these entities to factual knowledge from Wikidata, creating both text-only and text-image versions of the same information. The benchmark evaluates four key areas:

  • Direct forgetting: The ability to remove specific target facts.

  • Relational propagation: Whether forgetting extends to related two-hop facts.

  • Knowledge preservation: Ensuring unrelated, non-target information remains intact.

  • Robustness: Whether the model remains "forgetful" when faced with semantically equivalent queries or paraphrases.

Testing Cross-Modal Transfer

The researchers tested four unlearning methods—GA, DPO, NPO, and RT—across four experimental settings: text-to-text (T→T), image-to-image (M→M), text-to-image (T→M), and image-to-text (M→T). By keeping the target knowledge constant while varying the input modality, the authors sought to determine if unlearning is consistent across different ways of accessing information.

Key Findings on Asymmetry

The experiments revealed a significant asymmetry in how unlearning transfers across modalities. Multimodal unlearning (using text-image inputs) proved effective even when the model was evaluated using text-only queries. Conversely, text-only unlearning transferred poorly to visual and cross-modal scenarios. This suggests that a model might appear to have forgotten a fact when tested with text, but could still retrieve that same information when presented with a corresponding image. The study also found that these unlearning methods generally preserved the models' broader capabilities, such as mathematical reasoning and OCR performance.

Implications for AI Safety

The findings indicate that relying solely on intra-modal evaluation—specifically text-only testing—can lead to an overestimation of a VLM's unlearning effectiveness. Because VLMs process information across multiple modalities, the authors argue that effective knowledge removal requires both cross-modal unlearning and comprehensive evaluation that accounts for how information is accessed.

Franklin Analysis

The evidence provided by the authors suggests that the modality used for unlearning is a critical variable in VLM safety. The observed asymmetry—where text-only unlearning fails to cover visual retrieval paths—indicates that "forgetting" in a VLM is not a monolithic process. Because the study utilized a controlled pipeline to ensure that text and image queries were semantically identical, the results provide a strong basis for the conclusion that current text-centric evaluation methods are insufficient for modern multimodal models.

Comments (0)

No comments yet

Be the first to share your thoughts!