AcrossVAM1.0: Particle World Modeling for Text-Assi...

AcrossVAM1.0: Particle World Modeling for Text-Assi... | AI Research

Key Takeaways

AcrossVAM1.0 is a lightweight video prediction model designed to help robots anticipate future scenes by separating motion reasoning from visual appearance.
We present AcrossVAM1.0, a lightweight, text-assisted video action model that factorizes future prediction into object-centric motion and dense appearance.
A frozen SAM3-DLP codec decomposes four context frames into semantic particles for the robot, arm, and gripper, together with a background latent.
A 0.28M-parameter spatio-temporal Transformer aligns particle identities, rolls their states forward, and is modulated by a frozen OpenCLIP instruction embedding through FiLM.
On our VRS benchmark constructed from diverse real-robot trajectories, particle dynamics reduce trajectory error by 21.0\% over persistence.

Paper AbstractExpand

Predicting robot videos requires both precise motion reasoning and preservation of high-frequency appearance, yet monolithic pixel models entangle these objectives and often conceal their progress behind a strong last-frame baseline. We present AcrossVAM1.0, a lightweight, text-assisted video action model that factorizes future prediction into object-centric motion and dense appearance. A frozen SAM3-DLP codec decomposes four context frames into semantic particles for the robot, arm, and gripper, together with a background latent. A 0.28M-parameter spatio-temporal Transformer aligns particle identities, rolls their states forward, and is modulated by a frozen OpenCLIP instruction embedding through FiLM. A causal dual-stream decoder combines particle-rendered motion with appearance encoded exclusively from the last observed frame; a residual refiner and learned delivery mask produce five future frames without access to future appearance. On our VRS benchmark constructed from diverse real-robot trajectories, particle dynamics reduce trajectory error by 21.0\% over persistence. Across three delivery-mask seeds, AcrossVAM1.0 improves future-frame PSNR/SSIM from 19.97/0.796 to 20.573/0.8004, while raw particle generation improves motion-region PSNR from 11.89 to 13.23. The delivered model does not yet beat persistence in LPIPS, and correct-versus- shuffled language changes trajectory error by only 2.8--3.1%. We report these limitations alongside oracle, negative-control, multi-seed, and per-robot analyses. The results show that explicit particle dynamics are a promising low-dimensional interface for robot video prediction, while robust language grounding and appearance delivery remain the principal open challenges.

AcrossVAM1.0 is a lightweight video prediction model designed to help robots anticipate future scenes by separating motion reasoning from visual appearance. By decomposing video into object-centric particles and a dense appearance stream, the model aims to solve the common issue where predictive models simply copy the last observed frame to achieve high accuracy scores without actually learning how objects move.

How the Model Works

The system uses a frozen SAM3-DLP codec to break down four context frames into semantic particles representing the robot, its arm, and its gripper, alongside a background latent. A small, 0.28M-parameter Transformer then predicts the future movement of these particles. To ensure the model remains responsive to human input, it uses OpenCLIP instruction embeddings to modulate these dynamics.
The final video is generated by a dual-stream decoder: one stream renders the predicted motion of the particles, while the other uses only the last observed frame to fill in visual details. A learned delivery mask then combines these two streams to produce the final five-frame prediction.

Performance and Results

The researchers evaluated AcrossVAM1.0 on the Video Robot Segmentation (VRS) benchmark, which consists of diverse real-robot trajectories. The model reduced trajectory error by 21.0% compared to a simple persistence baseline (which just repeats the last frame).
Across three different delivery-mask seeds, the model improved future-frame PSNR from 19.97 to 20.573 and SSIM from 0.796 to 0.8004. Additionally, the raw particle generation improved motion-region PSNR from 11.89 to 13.23, suggesting that the particle-based approach effectively captures articulated motion.

Limitations and Challenges

Despite these improvements, the authors report several open challenges:

Visual Fidelity: The model does not yet outperform the persistence baseline in LPIPS, a metric often used to measure perceptual similarity.
Language Grounding: The model’s reliance on language is currently limited; changing the text instructions only altered trajectory error by 2.8–3.1%.
Synthesis Accuracy: The gap between the model and a pixel-selection oracle—which chooses the best possible output per pixel—indicates that the system still struggles to decide exactly where to synthesize new pixels versus where to rely on existing context.

Franklin Analysis

The evidence suggests that explicit particle dynamics provide a viable, low-dimensional interface for robot video prediction. By freezing the perception and language components, the authors successfully isolated the dynamics core, allowing for a clearer assessment of how much the model actually "understands" motion versus simply relying on static image priors. However, the modest impact of language instructions and the persistent gap in perceptual metrics indicate that while the model is effective at structured motion, it has not yet achieved robust, instruction-driven visual synthesis.