DECOWAM: Decoupled Whole-Body World-Action Model fo... | AI Research

DECOWAM is a whole-body world-action model designed to improve how legged mobile robots coordinate arm manipulation with base locomotion. By explicitly separating camera ego-motion from robot actions, the model enables more accurate visual prediction and control in environments where the robot's viewpoint changes as it moves.

Addressing the Mobile Manipulation Challenge

Existing world-action models are primarily built for fixed-base platforms where the camera remains stationary. Legged mobile manipulators face a more complex problem: the on-board camera moves with the base, meaning image changes are a mixture of scene dynamics, arm movements, and the robot's own travel. Furthermore, arm joints require high-rate control (15–30 Hz), while base velocity commands operate at a lower rate (3–5 Hz). DECOWAM addresses these issues by decoupling these factors rather than treating them as a single, undifferentiated stream of data.

How DECOWAM Works

The model builds on the FastWAM backbone by introducing specialized interfaces for different robot functions:

• Staged Adaptation: The system uses a two-stage training process. First, it aligns the model to the robot's specific data. Second, it freezes the backbone and trains only 25.95M residual parameters, making the system parameter-efficient.

• Decoupled Latents: The model uses adversarial training to separate base-velocity commands from arm-joint commands. This ensures that the navigation-scale decisions do not interfere with manipulation-scale corrections.

• Ego-Motion Conditioning: Instead of forcing the video model to guess camera motion from pixels alone, DECOWAM explicitly inputs the robot's current base velocity as a conditioning signal for future video prediction.

• Future-Information Bottleneck: During training, the model uses a "privileged" teacher that sees future frames to distill information into a "causal" student. This student is the only part used during deployment, ensuring the robot can act based only on current observations.

The ARMDOG Dataset

To support this research, the authors introduced ARMDOG, a real-robot dataset specifically for quadrupedal mobile manipulation. It contains 1,487 synchronized episodes—totaling over 343,000 frames—that align video, whole-body state and action, and language instructions. This dataset allows researchers to train models that understand the relationship between a robot's physical movement and the resulting visual changes in its environment.

Performance and Results

In testing, DECOWAM outperformed the baseline FastWAM model on a fixed replay protocol, reducing action mean squared error (MSE) by 21.7%. In 79 closed-loop trials on a physical robot, the model demonstrated higher whole-body coordination and better robustness against perturbations compared to the baseline systems. While task completion rates remained comparable to the strongest baseline, the authors conclude that this embodiment-aware factorization provides a more efficient way to handle joint visual prediction and control under moving viewpoints.