ReWorld: An Interactive World Model with Long-Horiz... | AI Research

ReWorld is an interactive world model designed to simulate environments that respond to user actions while maintaining long-term spatial memory. The system aims to solve the structural tension between control, which requires a short-term focus to respond to immediate commands, and memory, which requires an unbounded history to recognize previously visited locations. By separating these tasks during training and using a bounded cache at inference, ReWorld generates consistent, high-fidelity video in real time.

Separating Control and Memory

To balance immediate responsiveness with long-term recall, the researchers developed a training method that splits attention heads by window size. Most heads are designated as "local" and attend only to a short, recent window of frames to ensure precise control. A smaller set of "global" heads attends to the entire causal history to facilitate memory.
To prevent these capabilities from binding to specific heads—which would be problematic at inference when the full history is unavailable—the model uses random head routing. During training, the system switches which heads act as global or local at every step. This ensures that every head learns to function under both conditions, allowing the model to operate effectively at inference using a shared, fixed-size memory cache.

Managing Memory at Scale

At inference, ReWorld maintains a fixed memory budget to keep the system running in real time. It uses a "landmark bank" to store snapshots of the environment. When the camera travels, the system saves a landmark only if the camera has moved a sufficient distance, and it evicts the most spatially redundant landmarks when the bank reaches capacity.
To ensure the model can read this sparse, non-contiguous history, the authors introduced "chunk-drop training." By randomly masking parts of the history during training, the model learns to reconstruct the scene state from incomplete data. This makes the sparse caches used during deployment "in-distribution," preventing the blur and drift often seen when models are forced to work with limited memory.

Performance and Real-Time Capability

The researchers implemented a metric-aligned data engine that standardizes camera movement across eight different sources, including real-world footage, game roaming, and synthetic fly-throughs. This allows a single action command to produce consistent physical movement across different styles of worlds.
To achieve real-time performance, the team used distribution-matching distillation to compress the sampling process into four steps within a LoRA adapter. This allows a single backbone to serve both high-fidelity multi-step generation and real-time interactive streaming. In testing, ReWorld achieved a 11.95° rotation error and maintained consistent camera motion. On minute-long "out-and-back" rollouts, the model successfully regenerated the starting view using its fixed 12-chunk cache, a task where standard sliding-window models typically lose the necessary historical evidence.

Franklin Analysis

The design of ReWorld addresses a common failure point in interactive world models: the trade-off between immediate action-following and long-term consistency. By using pose-indexed attention (MRoPE) to anchor memory to specific camera coordinates rather than just temporal distance, the model creates a robust way to retrieve past scenes. The evidence suggests that the combination of random head routing and chunk-drop training is effective, as these techniques allow the model to maintain performance even when the available memory is significantly smaller than the total rollout duration.