Actionable World Representation

The research community is increasingly focused on building "physical world models"—systems that act as internal simulators for AI agents to understand, predict, and interact with the physical environment. A core challenge in this field is that while large language models have mastered conceptual reasoning, they often lack a grounded understanding of how physical objects behave. The paper introduces WorldString, a neural architecture designed to serve as a foundational building block for these models. By treating objects as "actionable entities" that change states based on their properties, WorldString creates a digital twin of real-world objects, allowing for a unified way to model articulated, skinned, and soft objects directly from point clouds or video data.

A Unified Approach to Physical Objects

Current methods for modeling objects often rely on separate techniques for different types of movement, such as video generation for visual rollouts or physics simulations for mechanical accuracy. WorldString proposes a more principled, unified framework. It categorizes physical reality into three primary types: articulated objects (like robots with joints), skinned objects (like humans or animals with skeletons), and soft objects (like deformable materials). By using a shared neural architecture, the model can represent the state of these diverse entities through a common language of keypoints and canonical embeddings, bridging the gap between rigid kinematics and complex, non-rigid deformation.

How WorldString Works

The architecture functions as a fully differentiable pipeline that translates sparse structural information into a complete 3D representation. It operates in three main stages:

• State Conditioning: The model uses a "State Transformer" to take canonical base embeddings and condition them on sparse keypoints, effectively grounding the object's base geometry in its current pose.

• Structural Coherence: An "Object Transformer" then applies self-attention to these embeddings, ensuring that the localized movements of keypoints propagate correctly to maintain the object's overall structural integrity.

• Voxel Reconstruction: Finally, a "Voxel Transformer" queries the latent space to predict a continuous occupancy field, which allows the model to reconstruct the explicit 3D geometry of the object in Cartesian space.
  This design is fully differentiable, meaning it can be integrated into future policy learning and neural dynamics systems, allowing an AI agent to "think" about how its actions will physically deform or move an object.

Performance and Versatility

To ensure the model is grounded in reality, the researchers developed a data acquisition pipeline that processes raw RGB-D video into tracked 3D point clouds and keypoint sequences. Experiments demonstrate that WorldString is highly effective at reconstructing complex rigid shapes and maintaining structural coherence during movement. When tested against retrieval-based baselines and specialized models for articulated robots, WorldString consistently provided more accurate representations of joint limits and connectivity. By successfully mapping implicit base states to explicit target states, the model proves its potential as a versatile digital twin capable of learning from diverse, real-world interaction data.