MIT's SceneSmith Uses AI Agents to Build Realistic Robot Training Grounds

Researchers at the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) have developed a new system called “SceneSmith” that utilizes collaborative AI agents to generate highly realistic 3D environments. By creating diverse virtual playgrounds—ranging from kitchens and hotels to restaurants—the system allows robots to practice everyday chores and test various action plans in a simulated space before they are deployed in the real world. This approach addresses a major bottleneck in robotics: the labor-intensive and time-consuming nature of physical training.

A Collaborative Agent Framework

SceneSmith operates through the coordination of three distinct AI agents, each powered by a vision-language model (VLM) known as GPT-5.2. The process begins with a “designer” agent that generates the initial elements of a scene, followed by a “critic” agent that evaluates the design for realism. Finally, an “orchestrator” manages the interaction between the two, determining when the scene is complete. This collaborative structure allows the system to construct 3D environments in a manner similar to a human designer, improvising creative and diverse arrangements without needing specific manual prompts.
The system’s ability to handle complex spatial knowledge enables it to populate rooms with up to six times more objects than prior methods. Because these environments are loaded directly into physics simulation software, they include articulated items like cabinets that robots can open and close, providing a more accurate training ground for tasks such as moving soda cans or placing fruit on plates.

Validating Realism and Performance

To ensure these virtual worlds are effective for training, the researchers tested them by placing a pretrained robot policy—an AI controller trained on real-world data—into the generated scenes. The robot successfully performed tasks like moving an apple from a bowl to a cutting board, demonstrating that the virtual environments closely mirror the settings the robot had previously learned from. Furthermore, the team conducted experiments where they teleoperated robots through the virtual spaces to navigate rooms and interact with objects, confirming that the environments remain stable under sustained physical interaction.
The system’s performance was also validated by over 200 users, who found SceneSmith’s visuals to be more realistic than other approaches over 90 percent of the time. By allowing engineers to weed out flawed robotic approaches in simulation, SceneSmith significantly reduces the need for trial and error in physical settings.

Advancing Robotics Research

While the current process can take several hours to produce a single scene due to the intensive scrutiny of each object, the framework represents a significant step forward in simulation technology. According to Jeremy Binagia, an applied scientist at Amazon Robotics, the system pushes the limits of object density and ensures that assets are physically accurate rather than just visually convincing.
The research team, which includes MIT PhD student Nicholas Pfaff and Toyota Research Institute roboticists, presented their findings as a spotlight at the International Conference on Machine Learning. The project was supported by the Toyota Research Institute, the U.S. Office of Naval Research, the U.S. National Science Foundation, and Amazon. Looking ahead, the researchers aim to expand the system’s capabilities to include deformable objects, such as sponges, as 3D libraries continue to evolve.