Back to AI Research

AI Research

Bridging Physical Reasoning and Task Generalization... | AI Research

Key Takeaways

  • Vision-language models (VLMs) are increasingly used for physical reasoning tasks, but they often struggle to generalize to new environments or unseen tasks.
  • Vision-language models (VLMs) struggle to generalize in interactive physical reasoning, particularly under unseen tasks and environments.
  • Two key failure modes are prominent: hallucinated chain-of-thought (CoT) reasoning that contradicts physical reality, and misalignment between the model's reasoning and actions.
  • We present VAORA (Visual Action Outcome Reasoning Alignment), a novel reward design that directly addresses both issues.
  • Together, these rewards suppress hallucinated CoT and reduce the gap between reasoning and behavior.
Paper AbstractExpand

Vision-language models (VLMs) struggle to generalize in interactive physical reasoning, particularly under unseen tasks and environments. Two key failure modes are prominent: hallucinated chain-of-thought (CoT) reasoning that contradicts physical reality, and misalignment between the model's reasoning and actions. We present VAORA (Visual Action Outcome Reasoning Alignment), a novel reward design that directly addresses both issues. VAORA introduces two complementary rewards: Visual Alignment Reward, which anchors VLM reasoning to the visual context independent of the agent action itself, and Visual-Action Alignment Reward, which grounds reasoning in the visual outcome induced by the model's action. Together, these rewards suppress hallucinated CoT and reduce the gap between reasoning and behavior. To improve training stability, we further employ smooth, dense rewards by estimating success probabilities using a pre-trained in-domain expert agent. Experiments on PHYRE and Virtual Tool support our performances across novel-task and unseen-environment settings, confirming that grounded and generalizable physical intelligence can be induced through VAORA.

Vision-language models (VLMs) are increasingly used for physical reasoning tasks, but they often struggle to generalize to new environments or unseen tasks. They frequently suffer from two major issues: "hallucinated" reasoning, where the model’s internal logic contradicts physical reality, and a misalignment between what the model says it will do and the actual actions it takes. The paper "Bridging Physical Reasoning and Task Generalization via Visual Action Outcome Reasoning Alignment" introduces VAORA, a new reward framework designed to ground VLM reasoning in physical reality, ensuring that the model's thoughts, visual perceptions, and physical actions are all consistent with one another.

The Problem with Current Training

Existing methods for training VLMs generally fall into two categories, both of which have significant drawbacks. Supervised fine-tuning often teaches models to mimic the style of expert reasoning without actually understanding the underlying physics, leading to fluent but incorrect "hallucinations." Conversely, reinforcement learning that only rewards task success often encourages models to take "shortcuts." Instead of learning to reason, these models learn to exploit dataset biases to achieve a goal, which causes them to fail when they encounter new, unfamiliar environments.

How VAORA Works

VAORA addresses these failures by introducing three specific reward signals that force the model to align its reasoning with the physical world:

  • Visual Alignment Reward: This anchors the model’s reasoning to the initial scene. By comparing the model’s perception of objects and their locations to the actual visual input, the system suppresses hallucinations before the model even attempts an action.

  • Visual-Action Alignment Reward: This grounds the model’s reasoning in the consequences of its actions. It rewards the model only when its predicted collision events and object placements match what actually happens in the physics simulator after an intervention.

  • Expert-Guided Stability: Because reinforcement learning can be unstable, the researchers use a pre-trained expert agent to provide dense, smooth estimates of success probability. This helps the model learn more reliably, especially in complex, continuous action spaces where success signals are otherwise sparse.

Results and Generalization

The researchers tested VAORA across several benchmarks, including PHYRE and Virtual Tool. The results demonstrate that VAORA significantly improves the model's ability to generalize. It outperforms existing baselines on unseen tasks and successfully transfers to entirely different physics simulators without additional training. Furthermore, the improvements in physical reasoning were not limited to just choosing the right action; the model also showed a deeper, more accurate understanding of physical dynamics when answering questions about the scenes, confirming that the framework successfully induces more grounded and generalizable intelligence.

Comments (0)

No comments yet

Be the first to share your thoughts!