Back to AI Research

AI Research

SciMIF: Understanding Multimodal Instruction Follow... | AI Research

Key Takeaways

  • SciMIF is a benchmark designed to evaluate how well Multimodal Large Language Models (MLLMs) follow complex, domain-specific instructions in scientific field...
  • Understanding instruction-following capabilities in scientific domains is essential for effectively leveraging Multimodal Large Language Models (MLLMs) to advance the development of scientific fields.
  • In this work, we introduce SciMIF, a novel benchmark designed to evaluate the capability of MLLMs in following complex scientific instructions.
  • Guided by this taxonomy, we develop a high-fidelity instruction injection pipeline to systematically augment existing scientific datasets.
  • We conduct comprehensive experiments on multiple state-of-the-art closed-source and open-source MLLMs.
Paper AbstractExpand

Understanding instruction-following capabilities in scientific domains is essential for effectively leveraging Multimodal Large Language Models (MLLMs) to advance the development of scientific fields. In this work, we introduce SciMIF, a novel benchmark designed to evaluate the capability of MLLMs in following complex scientific instructions. Specifically, based on an extensive analysis of 22 distinct tasks across 5 representative scientific disciplines, we propose a comprehensive taxonomy comprising 10 constraint groups that captures both general functional requirements and discipline-specific characteristics. Guided by this taxonomy, we develop a high-fidelity instruction injection pipeline to systematically augment existing scientific datasets. We conduct comprehensive experiments on multiple state-of-the-art closed-source and open-source MLLMs. Our findings reveal significant performance disparities across different scientific disciplines, with chemistry posing greater challenges for current MLLMs. Furthermore, we observe that increasing the model scale does not yield corresponding improvements in constraint adherence, and current models still struggle severely with fine-grained constraints and instructions requiring the deep application of disciplinary knowledge. SciMIF fills the current void in evaluating multimodal instruction adherence within scientific domains, laying a crucial foundation for future enhancements of MLLMs in rigorous scientific applications. Data and code will be released at this https URL .

SciMIF is a benchmark designed to evaluate how well Multimodal Large Language Models (MLLMs) follow complex, domain-specific instructions in scientific fields. While general-purpose benchmarks exist for instruction following, they often fail to account for the specialized knowledge and rigorous operational requirements—such as specific units, chemical nomenclature, or experimental procedures—necessary for scientific applications.

A New Taxonomy for Scientific Constraints

To address the lack of specialized evaluation, the authors developed a taxonomy of 10 functional constraint groups across five scientific disciplines: chemistry, geography, biology, materials science, and physics. These groups include requirements like "Procedure" (following a specific sequence of steps), "Number" (specifying quantities like atoms or bonds), "Unit" (maintaining dimensional consistency), and "Terminology" (using standardized scientific nomenclature). By analyzing 22 distinct scientific tasks, the researchers created a framework that captures both general functional requirements and the specific, often multimodal, demands of scientific inquiry.

The Instruction Injection Pipeline

The researchers built the SciMIF benchmark by transforming existing scientific datasets into instruction-following evaluations. Their pipeline involves:

  • Seed Preparation: Identifying original scientific questions and their reference answers.

  • Constraint Recognition: Determining which requirements are already implicit in the task.

  • Constraint Injection: Systematically adding compatible scientific and general constraints to the queries without altering the underlying scientific correctness of the answer.

  • Verification: Using a double automatic validation process and human review to ensure the injected constraints are logically coherent and that the ground-truth answers remain valid.
    The final dataset contains 2,527 samples, 27.5% of which include multimodal inputs such as molecular structures, biological diagrams, or physical plots.

Performance Disparities in MLLMs

Experiments conducted on state-of-the-art closed-source and open-source MLLMs revealed significant performance gaps. Key findings include:

  • Disciplinary Variation: Models struggle more with chemistry and geography than with other scientific domains.

  • Scaling Limitations: Increasing the scale of a model does not consistently lead to better constraint adherence, suggesting that parameter size alone is not the solution to these challenges.

  • The Correctness Gap: There is a weak coupling between scientific correctness (getting the right answer) and instruction adherence (following the requested format or procedure). A model may provide a scientifically valid answer while failing to meet the specific operational constraints requested by the user.

Limitations and Future Directions

The authors note that current models struggle significantly with fine-grained constraints and instructions that require the deep application of disciplinary knowledge. The study suggests that future research should focus on domain-aware instruction alignment and the development of structure-aware or tool-assisted methods. By separating scientific reasoning from instruction adherence, SciMIF provides a foundation for improving how MLLMs function in rigorous, real-world scientific environments.

Comments (0)

No comments yet

Be the first to share your thoughts!