Back to AI Research

AI Research

Lost in Perception: Isolating Perceptual and Reason... | AI Research

Key Takeaways

  • Lost in Perception: Isolating Perceptual and Reasoning Failures in Multimodal Physics and Geometry Reasoning Multimodal Large Language Models (MLLMs) are inc...
  • Multimodal LLMs report strong performance on scientific reasoning benchmarks, yet most treat perception and reasoning as a single measurable process.
  • We introduce a five-task diagnostic experiment across physics and geometry benchmarks that isolates failures to perception, reasoning, or both.
  • Incorrect diagram interpretation degrades performance even on problems models solve correctly from text alone, and accuracy generally rises from raw images to human-authored captions.
  • Recovery under corrected captions is high for some models, separating perception-blocked failures from genuine reasoning bottlenecks.
Paper AbstractExpand

Multimodal LLMs report strong performance on scientific reasoning benchmarks, yet most treat perception and reasoning as a single measurable process. We introduce a five-task diagnostic experiment across physics and geometry benchmarks that isolates failures to perception, reasoning, or both. Incorrect diagram interpretation degrades performance even on problems models solve correctly from text alone, and accuracy generally rises from raw images to human-authored captions. Recovery under corrected captions is high for some models, separating perception-blocked failures from genuine reasoning bottlenecks. Which reasoning error follows a perception failure depends on domain: physics failures resolve into calculation errors, geometry into conceptual misapplication. As a discussion beyond our core experiments, InternS1-mini, despite heavy scientific pretraining and thinking capabilities, falls below the weakest model from experiments on every task, with reasoning traces frequently truncating before completion.

Lost in Perception: Isolating Perceptual and Reasoning Failures in Multimodal Physics and Geometry Reasoning
Multimodal Large Language Models (MLLMs) are increasingly used for scientific tasks, but they often struggle to distinguish between "seeing" a diagram correctly and "reasoning" about the information it contains. This paper introduces a diagnostic framework to separate these two processes. By testing models across five different conditions—ranging from raw images to human-authored text descriptions—the researchers isolate whether a model’s failure is caused by a breakdown in visual perception or a fundamental inability to perform the required scientific logic. The same reasoning question is explored in A Unified Physics-Aware Quantum Machine Learning..., which adds a research perspective.

A New Diagnostic Framework

To understand where models fail, the authors created a five-task experiment. They compared performance when models were given raw images (IQ), text-only prompts (Q), model-generated captions (CQ), question-aware captions (CQ*), and perfect human-authored captions (GQ). By systematically changing the visual input while keeping the reasoning prompt constant, the researchers could pinpoint exactly where the "reasoning chain" breaks. They also categorized errors into a taxonomy: perception errors (misidentifying elements or values) and reasoning errors (misapplying concepts or making calculation mistakes).

How Perception Errors Propagate

The study reveals that visual input is not always helpful; in many cases, adding a diagram actually causes models to fail on problems they could have solved using text alone—a phenomenon the authors call the "Vision Hurts Rate." Furthermore, the researchers found that the domain of the problem dictates how a perception error evolves. In physics, a visual misinterpretation typically leads to a calculation error. In geometry, the same type of visual failure is more likely to result in a conceptual misapplication, where the model uses the wrong mathematical principle entirely. The same large language models question is explored in Beyond Confidence, which adds a research perspective.

The Bottleneck of Visual Grounding

A key finding is that perception acts as a structural bottleneck. While some models show high recovery rates when provided with perfect human-authored captions (proving they have the reasoning skills but were simply "blinded" by the image), others show little to no improvement. This suggests that for many models, the issue isn't just poor vision—it is a genuine reasoning deficit. The study also highlights that "question-aware" captions help in physics because the questions provide enough context to guide the model's focus, but this approach often backfires in geometry, where minimal questions lead the model to hallucinate details that aren't in the diagram.

Performance of Specialized Models

The authors also evaluated InternS1-mini, a model specifically designed with heavy scientific pretraining and extended "thinking" capabilities. Despite these features, the model performed worse than the other systems tested. It frequently failed to reach a conclusion, with its reasoning traces often truncating or "spiraling" before a final answer could be produced. This suggests that simply increasing scientific training or allowing for longer internal deliberation is not a silver bullet for solving the complex integration of visual perception and logical deduction. The same ai evaluation question is explored in ConvMem, which adds a research perspective. as detailed in the full paper on Arxiv

More from Franklin

From Symbolic Perception to Logical DeductionresearchBeyond Surface ImitationresearchA Unified Physics-Aware Quantum Machine Learning...researchBeyond ConfidenceresearchConvMemresearchWhen Agents Disagreeresearch

Comments (0)

No comments yet

Be the first to share your thoughts!