ChronoGraph gives robot planning an explicit record of what actions change

A robot opening a drawer needs more than an object label. It must find the handle, anticipate how pulling it changes access to the contents, and check the resulting scene before continuing. ChronoGraph tackles that connection between interpreting an interaction and planning the next one. The researchers represent both observed and predicted changes with functional 4D scene graphs.

Recording actions at the part level

Each graph contains task-relevant objects and the parts an agent can manipulate. Nodes carry semantic states and 3D information; edges describe functional and spatial relationships. An action targets an affordance part and updates the graph. A sequence of those transitions records how the scene evolves.
That distinction matters in the drawer example. Identifying the drawer does not specify where to pull, and a plan written before opening it may no longer describe what the robot can reach afterward. ChronoGraph asks models to represent those action-dependent changes explicitly rather than treating every observation as an isolated scene description.
The benchmark tests observed interaction understanding alongside two planning settings. Static-conditioned planning starts with an image and a goal. History-conditioned planning also provides previous interactions, which can reveal states missing from the current camera view, such as whether a refrigerator was left open.

Building and training on graph traces

ChronoGraphBench combines human egocentric recordings from HD-EPIC and HOI! with simulated robot trajectories from RoboCasa. The authors report 1,669 interaction samples, 5,143 graph states and 15,094 question-answer pairs spanning 357 object categories. Their annotation pipeline uses model-generated semantic graphs and visual grounding, with human review triggered by invalid fields, ambiguous evidence or low-confidence outputs.
ChronoGraphVLM learns in two stages. Supervised fine-tuning teaches a graph–evidence–answer output sequence: the model reconstructs or predicts scene transitions, selects supporting graph properties, then answers. Reinforcement learning subsequently rewards graph properties and answer correctness.
The graph reward covers more than a well-formed response. It checks semantic states, relations, action targets and geometric information. The researchers align predicted and reference graph sequences so that a missing state does not shift every later comparison.

What the evaluation establishes

The held-out evaluation contains 202 interaction samples and 1,811 visual questions. The split separates households or simulation setups across supervised training, reinforcement learning and evaluation. Multiple-choice answers use exact-match accuracy; numerical grounding receives a distance-based score relative to the target box.
The authors report improvements over the corresponding pretrained baselines and transfer to VLM4D without task-specific training there. They also describe real-world mobile-manipulation demonstrations using existing robot skills without additional fine-tuning.
Those demonstrations do not establish reliable operation across arbitrary homes, objects or robot hardware. The benchmark measures specific interaction and planning questions, while execution still depends on the skills available to the robot. The captured paper also says code and data will be released, so reproducibility requires checking their actual availability rather than assuming an open implementation already exists.