Back to AI Research

AI Research

Failure-Informed Image Self-Augmentation for Multim... | AI Research

Key Takeaways

  • Failure-Informed Image Self-Augmentation (FISA) is a framework designed to improve Multimodal Large Language Models (MLLMs) by generating synthetic training...
  • Multimodal large language models (MLLMs) have achieved remarkable performance across vision-language tasks, but their progress depends heavily on large-scale, high-quality multimodal data that are costly to annotate.
  • Self-augmentation offers a promising alternative by enabling models to expand their own training data without external supervision.
  • We propose Failure-informed Image Self-Augmentation (\textbf{FISA}), a framework for MLLM self-improvement that constructs augmented images from the model's own failure cases.
  • Our method generates visually challenging yet answer-preserving image complications, verifies their utility through self-examination, and applies dual fidelity filtering to avoid semantic distortion.
Paper AbstractExpand

Multimodal large language models (MLLMs) have achieved remarkable performance across vision-language tasks, but their progress depends heavily on large-scale, high-quality multimodal data that are costly to annotate. Self-augmentation offers a promising alternative by enabling models to expand their own training data without external supervision. However, existing MLLM self-augmentation methods are largely text-centric, while image augmentation remains underexplored and typically relies on generic or handcrafted transformations that are weakly aligned with the model's actual incapability. We propose Failure-informed Image Self-Augmentation (\textbf{FISA}), a framework for MLLM self-improvement that constructs augmented images from the model's own failure cases. Our method generates visually challenging yet answer-preserving image complications, verifies their utility through self-examination, and applies dual fidelity filtering to avoid semantic distortion. Experiments on visual question answering benchmarks show that the proposed method consistently improves performance across both in-distribution and out-of-distribution settings. Further experiments validate the compatibility of FISA with existing textual self-augmentation approaches, the superior data efficiency of the synthesized samples over generic image augmentation baselines, and the practical effectiveness of the proposed filtering strategy.

Failure-Informed Image Self-Augmentation (FISA) is a framework designed to improve Multimodal Large Language Models (MLLMs) by generating synthetic training data that specifically targets the model's own weaknesses. Instead of relying on generic image transformations, FISA uses the model’s failure cases to create challenging, answer-preserving images that help the model learn more effectively without requiring additional human annotations.

Addressing the Modality Imbalance

Current self-improvement methods for MLLMs often focus on text-based augmentation, while image-based methods remain limited. Existing image augmentation techniques typically rely on simple, generic transformations like rotation or cropping, or handcrafted rules that do not account for the model's specific capabilities. The authors argue that these methods are often "capability-agnostic," meaning they do not address the actual reasons why a model fails a task. FISA aims to bridge this gap by creating images that are intentionally designed to expose the model's visual understanding limitations.

How the FISA Framework Works

The FISA process follows a three-stage pipeline to ensure the generated data is both challenging and accurate:

  1. Failure-Oriented Complication: The model uses a three-stage instruction template to generate an original caption, a "thinking" process to identify how to make the image more difficult while preserving the answer, and a modified caption. This modified caption is then used by a text-to-image model to synthesize a new, more challenging image. 2. Utility and Fidelity Filtering: To ensure quality, the framework filters out images that do not meet specific criteria. It checks for utility (does the image actually cause the model to fail?) and fidelity (does the image still contain the information needed to answer the original question correctly?). This involves both text-level checks on the modified caption and image-level checks using a rollout-based self-response criterion. 3. Augmented Training: Once filtered, the successful images are combined with the original seed dataset. The authors further enrich this training data by prompting the model to generate "self-explanations" for the image-question-answer triples, which helps mitigate distribution gaps.

Performance and Efficiency

Experiments conducted on models including Qwen3-VL-2B, Gemma3-4B, and LLaVA1.5-7B show that FISA consistently improves performance across both in-distribution and out-of-distribution benchmarks, such as A-OKVQA, SEEDBench, and MME.
The authors compared FISA against a baseline called Cap2Aug, which synthesizes images from captions without the failure-inducing step. The results indicate that FISA is more data-efficient, providing better performance gains under the same data volume. Additionally, the researchers found that FISA is compatible with existing textual self-augmentation methods, suggesting that image-side and text-side improvements can be combined without interference.

Considerations

The authors note that the image-level fidelity filtering relies on a rollout-based criterion, which they acknowledge is imperfect, though they found it more effective than other unsupervised alternatives like CLIPScore. The framework is designed to function without external supervision, relying entirely on the model's own internal capabilities and self-examination to generate and verify training data.

Comments (0)

No comments yet

Be the first to share your thoughts!