The Illusion of Visual Tool-Use: A Causal Audit of...

The Illusion of Visual Tool-Use: A Causal Audit of... | AI Research

Key Takeaways

The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images investigates why multimodal large language models (MLLMs) often fail to see performan...
The "thinking-with-images" paradigm equips multimodal LLMs with active visual operations such as crop-and-zoom.
However, models using these operations often achieve only marginal or negative gains over direct inference at substantially higher token cost.
They may also repeatedly crop irrelevant regions and fail on questions that direct inference answers correctly.
We ask whether the returned visual evidence causally affects the answer.

Paper AbstractExpand

The "thinking-with-images" paradigm equips multimodal LLMs with active visual operations such as crop-and-zoom. However, models using these operations often achieve only marginal or negative gains over direct inference at substantially higher token cost. They may also repeatedly crop irrelevant regions and fail on questions that direct inference answers correctly. We ask whether the returned visual evidence causally affects the answer. To answer this question, we formulate visual tool-use as a causal graph that separates observation-mediated paths from action-induced shortcuts. We then audit it through interventions at the three levels: policy (comparing tool-use with direct inference), trajectory (corrupting all observations during rollout), and step (counterfactually replacing one individual observation under a fixed prefix). Our step-level estimand, Visual Evidence Gain, isolates the contribution of each returned observation. Across six representative models and five fine-grained perception benchmarks, we uncover policy miscalibration with two failure modes. In Calling Without Looking, returned observations have no causal effect on the answer. In Looking Without Planning, observations are informative but the call schedule is incoherent. A trajectory-level diagnostic decomposes the policy-level accuracy gain and shows that the gain is concentrated in a Calibrated minority. We term this discrepancy the illusion of visual tool-use: despite aggregate accuracy gains, visual tool-use is not causally effective across a broad range of rollouts. The code is available at this https URL .

The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images investigates why multimodal large language models (MLLMs) often fail to see performance gains when using "thinking-with-images" tools, such as crop-and-zoom operations. The authors, Zhiheng Wang, Bo Peng, Lai Wei, and Chaochao Lu, argue that while these tools are intended to provide fine-grained visual evidence, they frequently result in marginal or negative accuracy improvements despite significantly higher token costs.

The Causal Framework

To determine if visual evidence actually influences model answers, the researchers formulated visual tool-use as a causal graph. This graph distinguishes between three paths from the input to the final answer:

Direct path: The model answers based on the original image and query without tools.
Observation-mediated path: The tool returns a crop that provides genuine visual grounding for the answer.
Action-induced shortcut: The model's answer is influenced by the mere act of calling a tool, regardless of the visual content returned.
The authors developed a three-level intervention protocol to audit these paths: policy-level (comparing tool-use to direct inference), trajectory-level (corrupting observations during a full rollout), and step-level (counterfactually replacing individual observations).

Identifying Failure Modes

The audit of six representative models across five perception benchmarks revealed that policy miscalibration is the primary bottleneck. The researchers identified two specific failure modes:

Calling Without Looking (CWL): The visual content returned by the tool has no causal effect on the final answer.
Looking Without Planning (LWP): The visual content is informative, but the schedule of tool calls is incoherent, such as repeatedly cropping irrelevant regions or continuing to call tools after the correct answer is already determined.
The study concludes that aggregate accuracy gains in these models are often driven by a "Calibrated" minority of successful rollouts, while the majority of tool calls are either decoupled from the answer or poorly planned.