Back to AI Research

AI Research

SceneActBench: Can Agents Act on the 3D Scenes They... | AI Research

Key Takeaways

  • SceneActBench: Can Agents Act on the 3D Scenes They See?
  • Vision-language model (VLM) agents are increasingly being used to interact with 3D environments rath...
  • Vision-language model (VLM) agents increasingly use tools to act on 3D scenes rather than only describe them.
  • Existing 3D benchmarks score textual responses or single-object operations, leaving agent action on complete multi-object 3D scenes under evaluated.
  • We present SceneActBench, a benchmark for visually conditioned action across five 3D tasks under a unified agent-environment loop.
Paper AbstractExpand

Vision-language model (VLM) agents increasingly use tools to act on 3D scenes rather than only describe them. Existing 3D benchmarks score textual responses or single-object operations, leaving agent action on complete multi-object 3D scenes under evaluated. We present SceneActBench, a benchmark for visually conditioned action across five 3D tasks under a unified agent-environment loop. Given PNG images or sampled video frames and, where applicable, supplied 3D assets, an agent acts on a 3D environment. We evaluate each final output against hidden ground truth with task-specific geometric metrics. SceneActBench comprises five tasks built from 210 source instances, yielding 520 task cases including paired input conditions. Every task runs through one fixed agent loop to keep the comparison fair. Across eleven proprietary VLM configurations, Overall scores span 38.6-50.2, and none performs consistently well across tasks. We further analyse where and how failures manifest.

SceneActBench: Can Agents Act on the 3D Scenes They See?
Vision-language model (VLM) agents are increasingly being used to interact with 3D environments rather than simply describing them. However, current benchmarks often focus on static tasks, such as answering questions about a scene or performing simple, single-object edits. This paper introduces SceneActBench, a new benchmark designed to evaluate how well AI agents can perform complex, multi-object actions within 3D environments. By requiring agents to act on complete scenes based on visual input, the benchmark provides a more rigorous test of an agent's true 3D spatial and kinematic understanding.

A Unified Testing Environment

To ensure a fair comparison across different AI models, the researchers developed a standardized evaluation harness. In this setup, an agent is given visual evidence—such as PNG images or sampled video frames—and must interact with a 3D environment using a shared tool interface. The agent operates in a "headless" mode, meaning it uses code to manipulate 3D assets in Blender without a graphical interface. The agent’s final output is then compared against hidden ground truth data using specific geometric metrics to determine how accurately the agent reconstructed or modified the scene.

Five Core 3D Tasks

SceneActBench evaluates agents across five distinct tasks, each testing a different aspect of 3D reasoning:

  • Layout: Testing spatial grounding by requiring the agent to map 2D images to the correct 3D positions and rotations of furniture.

  • Camera: Testing egocentric spatial reasoning by asking the agent to infer the 3D position and orientation of a camera based on a single view of a scene.

  • Articulated: Testing kinematic reasoning by having the agent identify movable parts and reproduce their motion from video frames.

  • Reconstruction: Testing shape imagination by requiring the agent to build and texture 3D meshes from partial views.

  • Dynamic: Testing dynamic reasoning by asking the agent to reconstruct the motion of multiple objects over time.

Performance and Limitations

The researchers tested eleven proprietary VLM configurations using this benchmark. The results showed that overall performance varied significantly, with scores ranging from 38.6 to 50.2. Notably, no single model performed consistently well across all five tasks, suggesting that different models have distinct strengths and weaknesses in their 3D capabilities. By analyzing where and how these models failed, the study provides a diagnostic look at the current limitations of VLM agents, highlighting that while these models are becoming more capable, they still struggle to maintain accuracy when tasked with complex, multi-object 3D coordination.

Comments (0)

No comments yet

Be the first to share your thoughts!