JEPA-WAM: Connecting Generated Visual Instructions... | AI Research

JEPA-WAM: Connecting Generated Visual Instructions to World Action Models through JEPA Latent Representations
Robotic systems often struggle to follow language instructions accurately because they rely too heavily on visual patterns in their training data rather than the actual meaning of the commands. When a robot is trained on a limited set of language labels paired with complex visual actions, it may learn to "shortcut" the task by recognizing the scene rather than understanding the instruction. This paper introduces JEPA-WAM, a new approach that improves how robots interpret instructions by augmenting text commands with diverse, AI-generated visual examples of what a completed task should look like. The same ai evaluation question is explored in Beyond Confidence, which adds a research perspective.

Enriching Instructions with Visual Hypotheses

The core problem identified by the researchers is a "structural imbalance" in robot learning: language labels are often sparse and repetitive, while visual-action data is dense. To fix this, JEPA-WAM uses an off-the-shelf text-to-image generator to create a "bank" of multiple images for every instruction. These images serve as visual goal references, showing different ways a task might be completed. By providing the robot with these diverse visual cues, the model is forced to ground its actions in the specific semantic goal of the instruction rather than just memorizing the visual context of a specific training scene.

Translating Visuals into Latent Semantics

Because AI-generated images may not perfectly match the robot’s actual workspace in terms of layout or lighting, the researchers needed a way to extract the "meaning" of these images without getting distracted by irrelevant visual details. They use a frozen V-JEPA 2.1 encoder to transform these generated images into compact, spatially structured latent representations. These representations act as a bridge, allowing the robot’s video and action experts to "see" the goal in a way that is robust to appearance variations. By compressing these features into goal tokens, the model can effectively condition its future movements on the intended outcome. The same ai evaluation question is explored in CERA-MoA, which adds a research perspective.

Performance on Real-Robot Benchmarks

To test the effectiveness of this approach, the authors created a real-robot benchmark featuring 11 complex, overlapping manipulation tasks. They evaluated the model in three challenging scenarios: familiar settings (in-distribution), scenes with new distractors (out-of-distribution scene), and scenarios where the instructions were rephrased in ways the robot had never seen before (out-of-distribution instruction). JEPA-WAM consistently outperformed existing models like $\pi_0$ and Fast-WAM, achieving success rates of 87.3%, 74.5%, and 80.9% across these settings. These results demonstrate that providing diverse visual goal references significantly improves a robot's ability to handle scene shifts and novel language formulations.

Key Considerations

JEPA-WAM is designed to be efficient for deployment. The image generation and latent encoding steps are performed offline or once per instruction, meaning the robot does not need to generate new images during every millisecond of movement. Furthermore, because the image generator and the V-JEPA encoder remain frozen—meaning they are not trained or fine-tuned during the robot learning process—the system remains computationally manageable, allowing the entire inference stack to run on a single workstation. The same ai evaluation question is explored in JarvisGUI, which adds a research perspective. as detailed in the full paper on Arxiv