Back to AI Research

AI Research

Benchmarking Multimodal Large Language Models for S... | AI Research

Key Takeaways

  • Benchmarking Multimodal Large Language Models for Scientific Visualization Literacy Multimodal large language models (MLLMs) are increasingly used to interpr...
  • We evaluate three closed-source and three open-source models under a closed-world protocol and compare their performance using data from 485 human participants.
  • Results show that current MLLMs do not exhibit uniform SciVis literacy.
  • Gemini is the strongest model overall, exceeding the human mean across the evaluated subsets, whereas the open-source models remain below the human baseline.
  • Error analysis reveals recurring failures in fine-grained quantitative estimation, flow-direction interpretation, and grounded encoding interpretation.
Paper AbstractExpand

Multimodal large language models (MLLMs) are increasingly used to interpret visualizations, yet current evaluations remain largely chart-centric and provide limited evidence of understanding of scientific visualization (SciVis). We benchmark six MLLMs on the scientific visualization literacy assessment test, a standardized SciVis literacy assessment comprising 49 items based on 18 scientific visualizations and illustrations, spanning 8 techniques and 11 task types. We evaluate three closed-source and three open-source models under a closed-world protocol and compare their performance using data from 485 human participants. Results show that current MLLMs do not exhibit uniform SciVis literacy. Gemini is the strongest model overall, exceeding the human mean across the evaluated subsets, whereas the open-source models remain below the human baseline. Performance is highly uneven across techniques and tasks: models perform best on scientific illustration, search, and spatial understanding, but struggle on texture-based and integration-based visualizations and on quantitative estimation. Error analysis reveals recurring failures in fine-grained quantitative estimation, flow-direction interpretation, and grounded encoding interpretation. These findings position SciVis literacy as a necessary benchmark dimension for evaluating multimodal AI systems. Our code and model outputs are publicly available at this https URL .

Benchmarking Multimodal Large Language Models for Scientific Visualization Literacy
Multimodal large language models (MLLMs) are increasingly used to interpret images, but most research has focused on standard charts and graphs. This paper investigates whether these models can accurately interpret scientific visualizations (SciVis)—complex images like medical scans, climate simulations, and spatial data that require understanding physical structure and motion. The authors use the Scientific Visualization Literacy Assessment Test (SVLAT) to determine if current AI systems possess the specialized literacy required to analyze these scientific representations.

Evaluating AI Literacy

The researchers tested six different MLLMs—three closed-source (Gemini, GPT, and Claude) and three open-source (Qwen, InternVL, and LLaVA-OneVision)—against a baseline of 485 human participants. The test consisted of 49 items covering 8 distinct visualization techniques and 11 types of interpretation tasks. To ensure a fair comparison, the models were evaluated under a "closed-world" protocol, meaning they were instructed to rely only on the provided images and captions rather than their own external knowledge.

Performance Strengths and Weaknesses

The study found that MLLMs do not have a uniform level of scientific literacy. Gemini emerged as the strongest model, outperforming the human average across the board. While all models performed well on tasks involving scientific illustrations, basic search, and spatial understanding, they struggled significantly with more complex requirements. Specifically, the models had difficulty with quantitative estimation—such as measuring distances or values—and interpreting motion in dense, texture-based visualizations like animated wind fields.

Recurring Failure Patterns

By analyzing the models' reasoning, the researchers identified three primary reasons for failure. First, models often struggle with fine-grained quantitative tasks, such as misreading contour lines on a map or miscalculating the radius of a structure. Second, they frequently misinterpret local motion in animated data, sometimes confusing the orientation of visual patterns with the actual direction of flow. Finally, models occasionally make high-level interpretations that are not grounded in the actual data, instead relying on unsupported assumptions or "hallucinated" mappings that ignore the specific visual evidence provided.

Future Directions

The authors conclude that while some MLLMs show promise, they are not yet broadly literate in scientific visualization. The study highlights that SciVis literacy is a distinct and necessary benchmark for evaluating AI, as it demands a level of spatial and temporal reasoning that general-purpose image benchmarks often miss. Future research is needed to test larger models and more diverse scientific datasets to improve the reliability and accuracy of AI-assisted scientific communication.

Comments (0)

No comments yet

Be the first to share your thoughts!