Back to AI Research

AI Research

Rule-Compliant Visual Spatial Planning for Multimod... | AI Research

Key Takeaways

  • Rule-Compliant Visual Spatial Planning for Multimodal Large Language Models introduces a framework to improve how AI models navigate environments while follo...
  • This setting requires models to jointly understand spatial layouts, interpret natural-language rules, and plan valid actions accordingly.
  • To address this gap, we introduce RuleMaze, a controllable benchmark in which MLLMs must navigate mazes while obeying natural-language rules of varying complexity.
  • RuleMaze isolates rule-compliant spatial planning by requiring accurate perception, rule interpretation, and constrained action planning.
  • To improve rule following and generalization, we introduce Disentangled Multimodal Planning (DMP), which separates perception, execution, and rule verification through interpretable reasoning primitives.
Paper AbstractExpand

Multimodal large language models (MLLMs) combine linguistic reasoning with visual perception, yet their ability to perform visual spatial planning under explicit or previously unseen rule constraints remains underexplored. This setting requires models to jointly understand spatial layouts, interpret natural-language rules, and plan valid actions accordingly. To address this gap, we introduce RuleMaze, a controllable benchmark in which MLLMs must navigate mazes while obeying natural-language rules of varying complexity. RuleMaze isolates rule-compliant spatial planning by requiring accurate perception, rule interpretation, and constrained action planning. To enable scalable and systematic rule construction, we propose Language-Logic-Function Hybridization, which automatically generates natural-language rules and translates them into logical representations and executable validators, eliminating manual rule engineering. To improve rule following and generalization, we introduce Disentangled Multimodal Planning (DMP), which separates perception, execution, and rule verification through interpretable reasoning primitives. By disentangling these components, DMP facilitates systematic generalization to more complex and previously unseen rules, while providing transparent intermediate planning traces. Experiments demonstrate that DMP substantially improves rule compliance and planning success compared to end-to-end textual planning baselines. Overall, RuleMaze establishes a principled benchmark for studying grounded and interpretable rule-based spatial planning in MLLMs. Code is available at this https URL .

Rule-Compliant Visual Spatial Planning for Multimodal Large Language Models introduces a framework to improve how AI models navigate environments while following specific, natural-language rules. The researchers address the challenge of "rule-compliant spatial planning," where an agent must not only understand a visual layout but also adhere to constraints that may change or appear for the first time during operation.

The RuleMaze Benchmark

To evaluate this capability, the authors developed RuleMaze, a benchmark that pairs procedurally generated maze images with natural-language rules. The researchers created two scenarios: "RegularMaze," which focuses on basic spatial navigation with color-coded rules, and "QuestMaze," which adds memory-dependent tasks, such as collecting items or navigating toward changing targets. To ensure the benchmark is rigorous, the team used a "Language–Logic–Function Hybridization" pipeline. This process automatically converts natural-language rules into logical representations and then into executable Python functions, which act as validators to confirm whether an AI’s planned path actually follows the rules.

Disentangled Multimodal Planning (DMP)

The authors propose Disentangled Multimodal Planning (DMP) to move away from "end-to-end" models, which often struggle to coordinate perception, planning, and rule-following simultaneously. Instead of forcing a single model to learn everything at once, DMP breaks the task into three modular, interpretable components:

  • Perception: Tools that identify the agent's starting position and grid contents.

  • Execution: Tools that update the agent's position based on chosen actions.

  • Verification: Tools that check if a planned trajectory complies with the given rules.
    The model acts as a controller that decides when to call these specific tools. Because the rule-verification tools are externalized, the system can adapt to new rules without requiring the entire model to be retrained.

Performance and Results

The researchers report that DMP significantly improves rule compliance and overall planning success compared to standard end-to-end textual planning models. By separating the reasoning process into distinct, reusable tools, the model becomes more reliable when facing complex or previously unseen rules. The authors note that the automatically generated validators achieved 93.3% correctness before undergoing manual verification to ensure data quality.

Franklin Analysis

The evidence suggests that the primary bottleneck in current multimodal models is the "entanglement" of perception and reasoning. By externalizing the rule-checking process into a modular toolset, the authors successfully shift the burden from the model's internal weights to an explicit, verifiable logic layer. This approach is particularly effective for tasks where rules are dynamic, as it allows for "plug-and-play" updates to the rule-verification logic without needing to modify the underlying vision-language model. The reliance on procedurally generated mazes and automated validator synthesis provides a scalable path for testing AI agents in environments that require strict adherence to safety or operational constraints.

Comments (0)

No comments yet

Be the first to share your thoughts!