Mind the Sim-to-Real Gap & Think Like a Scientist | AI Research

Mind the Sim-to-Real Gap & Think Like a Scientist explores how to effectively combine pre-trained simulators with real-world experimentation. While simulators are inexpensive to query, they often suffer from "calibration drift"—meaning they reflect the biases of the historical data used to build them rather than the true dynamics of the real world. This paper provides a framework for planners to decide when to trust a simulator and when to step into the field to conduct deliberate, randomized experiments to close the gap between simulated predictions and real-world outcomes.

The Value Gap Decomposition

The authors identify that the error in a simulator’s predictions is not uniform. They decompose the "value gap"—the difference between the simulator’s suggested policy and the optimal real-world policy—into two distinct parts. The first is a "local" component, which involves states that a policy already visits during its normal operation. The second is a "reachability" component, which involves states that the policy never visits. The paper proves that while passive learning (simply observing data as it comes) can eventually fix the local component, it is mathematically incapable of closing the reachability gap. To reach those unexplored states, a planner must move beyond passive observation and perform active, designed experimentation.

Thinking Like a Scientist

The core prescription of the paper is to shift how we use simulators. Instead of using a simulator solely to generate a final policy for deployment, planners should use it as a tool to design experiments. The authors propose "Fisher-SEP," a simulation-aided experimental policy. This approach uses the simulator to calculate where real-world data would be most valuable—specifically, by minimizing the "posterior predictive variance" of a target policy’s value. By using the simulator to identify these high-uncertainty areas, the planner can allocate limited experimental resources to the specific state-action pairs that matter most for improving long-term performance.

Real-World Regimes

The paper illustrates these concepts through two practical scenarios. In a vending-machine supply chain, the problem is largely local; as the planning horizon grows, the benefits of front-loaded experimentation eventually outweigh the costs of initial testing. In contrast, an HIV mobile-testing program highlights the reachability problem. In this case, there is a "corridor" separating well-surveilled regions from poorly-surveilled ones. Because the simulator cannot learn about the poorly-surveilled regions from existing data, the only way to understand those areas is to intentionally divert resources to them, even if it means sacrificing immediate efficiency in well-known areas.

Key Considerations

It is important to note that not all errors can be fixed through experimentation. The authors distinguish between "calibration-deployment shifts," which randomization can identify, and "parametric residuals," which are inherent to the simulator’s design and cannot be reduced by further interaction with the real world. Furthermore, the framework assumes that the latent dynamics of the system have a "forgetting property," where the influence of past hidden states fades over time. If a system does not exhibit this property, the complexity of the problem increases, as the history of the system continues to influence future outcomes in ways that simple Markov models cannot capture.