iARCS: Iterative Agentic RL for Controllable 3D Sce... | AI Research

iARCS is an iterative agentic reinforcement learning framework designed to improve the functional utility of synthetic 3D indoor scenes. While existing generative models excel at creating visually realistic scenes, they often fail to meet specific functional requirements like accessibility, traversability, and spatial rule compliance. This framework adapts pretrained scene generators to satisfy these constraints through a two-stage optimization process, allowing them to serve as more effective data sources for computer vision and embodied AI training.

A Two-Stage Optimization Strategy

The researchers, Saugat Adhikari, Ashok Prasad Neupane, Pramish Paudel, Ajad Chhatkuli, and Danda Pani Paudel, propose a two-stage strategy to refine scene generators. In the first stage, the model undergoes universal-reward pretraining to improve basic physical plausibility, such as reducing object collisions and ensuring objects are placed within room boundaries. In the second stage, the model undergoes task-specific fine-tuning. Here, an LLM agent translates natural-language user prompts into executable reward programs. These programs are then used to guide the model toward satisfying specific functional constraints, such as ensuring a clear walking path or reachable furniture placement.

Agentic Reward Synthesis and Reflection

The framework utilizes an LLM agent to automate the creation of reward functions, moving away from manually engineered objectives. The agent follows a three-step process: reasoning about the user's prompt to identify restrictive constraints, decomposing those constraints into measurable geometric checks, and outputting executable Python code to calculate rewards. To prevent issues like reward hacking, the system includes a reward-reflection module. This module monitors the statistics of generated scenes and allows the LLM to debug or refine the reward code and weights, ensuring the model progressively learns to meet high-level intent without losing scene diversity.

Performance and Data Utility

Experiments conducted on the 3D-FRONT dataset show that iARCS improves constraint fidelity across walkability, reachability, and clearance-focused tasks. The researchers report that training the MiDiffusion model with synthetic data generated by iARCS improves the base generator’s performance in both functional utility and physical plausibility. Notably, the FID (Fréchet Inception Distance) scores remained matched, indicating that the framework preserves the generative quality and diversity of the original model while adding functional control.

Considerations for Implementation

The authors note that their approach relies on post-training adaptation rather than rebuilding generators from scratch. By using Denoising Diffusion Policy Optimization (DDPO), the framework can optimize for non-differentiable functional objectives that standard gradient-based guidance methods cannot handle. However, the effectiveness of the system is tied to the LLM's ability to decompose constraints and the model's capacity to balance these new requirements against the original learned distribution. The researchers emphasize that this framework is intended to function as a practical synthetic-data engine, providing a method to generate task-critical edge cases that are often missing from standard training datasets.