Back to AI Research

AI Research

From Passive Video to Editable Experience: Physical... | AI Research

Key Takeaways

  • From Passive Video to Editable Experience: Physically Grounded Experience Synthesis for Embodied Intelligence addresses the "embodiment gap" that prevents ro...
  • The key bottleneck in embodied AI is not model architecture but data.
  • Although billions of human manipulation videos exist online, robots cannot directly learn from them due to the embodiment gap between human morphology and robot hardware.
  • We introduce Pegasus, a low-resource framework that bridges this gap by translating human demonstrations into robot-learnable data through structured knowledge transfer.
  • A hierarchical affordance latent space models the relationship between object states, affordances, and tasks, enabling generalization beyond object identities.
Paper AbstractExpand

The key bottleneck in embodied AI is not model architecture but data. Although billions of human manipulation videos exist online, robots cannot directly learn from them due to the embodiment gap between human morphology and robot hardware. We introduce Pegasus, a low-resource framework that bridges this gap by translating human demonstrations into robot-learnable data through structured knowledge transfer. Instead of relying on raw video prompts, Pegasus constructs a graph-based intermediate representation: a Task Graph extracted from human videos is transformed through Affordance and Constraint Graphs into a Robot Planning Graph for robot-conditioned video generation. A hierarchical affordance latent space models the relationship between object states, affordances, and tasks, enabling generalization beyond object identities. A closed-loop physics verifier further filters invalid generations using kinematic feasibility, collision constraints, and joint limits. We evaluate Pegasus across a range of egocentric manipulation benchmarks, including GTEA Gaze+ and EPIC-KITCHENS-100, and diverse robot embodiments, assessing Task Correctness, Executability, State Consistency, and Learnability. Results demonstrate reliable cross-embodiment translation and show that robot data generation can be reframed from a hardware collection problem into a scalable, low-resource knowledge transfer problem.

From Passive Video to Editable Experience: Physically Grounded Experience Synthesis for Embodied Intelligence addresses the "embodiment gap" that prevents robots from learning directly from the billions of human manipulation videos available online. Because human and robot morphologies, kinematics, and action spaces differ, raw video cannot be directly used as training data. Author Jia Luo introduces Pegasus, a framework that translates human demonstrations into robot-learnable data by converting video into structured, physically grounded graphs rather than relying on pixel-level imitation.

Bridging the Embodiment Gap

Pegasus treats robot data generation as a knowledge transfer problem rather than a hardware collection task. The framework extracts a Task Graph from human videos, which captures sub-goals, objects, and spatial relationships. This graph is then transformed through an Affordance Graph and a Constraint Graph into a Robot Planning Graph. By focusing on these intermediate representations, the system separates embodiment-invariant task semantics from embodiment-specific execution requirements, such as joint limits and collision constraints.

Hierarchical Affordance Latent Space

To enable generalization, Pegasus uses a hierarchical Affordance Latent space. Instead of relying on specific object identities—which limits a robot to objects it has seen before—the model organizes representations by object states, affordances (such as graspability or pourability), and task intentions. This allows the system to perform tasks on unseen objects by recognizing their functional properties. For example, the model can determine how to handle a watermelon by identifying it as "graspable, heavy, and fragile," even if it has never encountered that specific object during training.

Physics-Constrained Verification

Visual fidelity does not guarantee that a robot can actually perform a movement. Pegasus includes a closed-loop physics verifier that checks generated trajectories against kinematic feasibility, collision constraints, and joint limits. If a generated plan fails these checks, the system produces error signals that are fed back into the planning module to iteratively refine the trajectory. Experiments show this process reduces inverse-kinematics errors by 93.5% and collision rates by 90.9%, with 94.1% of trajectories becoming executable within five iterations.

Performance and Policy Learning

In experiments using benchmarks like GTEA Gaze+ and EPIC-KITCHENS-100, Pegasus-generated data improved downstream robot policy performance. When training models like OpenVLA and ACT, adding Pegasus-generated data to real-robot demonstrations resulted in an 11.3% improvement in success rates compared to using real-robot data alone. Furthermore, the framework’s graph-conditioned generation outperformed prompt-only video generation methods by 12.3% in task correctness and 15.7% in executability, suggesting that structured task representation is a primary driver of successful cross-embodiment transfer.

Comments (0)

No comments yet

Be the first to share your thoughts!