The Dually Flat Geometry of Planning as Inference | AI Research

The Dually Flat Geometry of Planning as Inference
This paper introduces a new way to understand reinforcement learning (RL) by viewing it through the lens of information geometry. The authors propose a "resetting planning process"—a mathematical model where an agent’s simulated life periodically restarts. By focusing on the "visitation measure" (the stationary probability of being in a certain state and taking a specific action), the researchers show that the entire planning process can be mapped onto a "dually flat statistical manifold." This framework provides a unified geometric language that connects reinforcement learning, variational inference, and theoretical neuroscience.

A New Perspective on Planning

Standard reinforcement learning often struggles to bridge the gap between high-level planning objectives and the specific algorithms used to solve them. The authors address this by creating a latent controlled Markov chain that resets at a state-action-dependent rate. This resetting mechanism allows the agent’s occupancy measure to be treated as a stationary visitation measure. By doing so, the researchers transform the complex task of planning into a problem of conditioning a generative model on desirable outcomes, placing control and perception on the same mathematical footing. The ai agents story also surfaces in EU Regulators Demand Apple and Google..., adding another angle.

The Geometry of Decision Making

The core contribution of the paper is the discovery that these visitation measures form a dually flat statistical manifold. This structure relies on two "affine charts"—different ways of describing the same point on the manifold:

• Mixture coordinates: Represented by the visitation probabilities themselves.

• Exponential coordinates: Represented by the log-policies of the agent.
  These two charts are linked by conditional entropy, acting as Legendre-dual pairs. This geometric structure explains why different popular RL techniques, such as natural policy gradients and policy mirror descent, are actually the same update rule viewed through different coordinate systems. The ai agents story also surfaces in Stanford Researchers Develop TRACE to Fix..., adding another angle.

Generalizing Beyond Linear Rewards

Because the visitation manifold is dually flat, the authors demonstrate that planning-as-inference can move beyond simple linear rewards. Their framework allows for the use of nonlinear functionals of the visitation, such as those found in convex reinforcement learning and active inference. In this model, each iteration of the planning process can be solved using a single natural-gradient step. Furthermore, the temporal-difference error—a staple of RL and neuroscience—is reinterpreted as a marginal-utility estimate, providing a potential bridge between artificial intelligence and how the brain processes rewards.

Key Implications

This geometric approach provides a rigorous foundation for understanding why certain RL algorithms are effective. By showing that the visitation measure is the natural object for information geometry, the authors provide a single, coherent account of how agents learn to act. This framework not only simplifies the theoretical landscape of reinforcement learning but also offers new tools for researchers in theoretical neuroscience to model how biological systems might perform complex planning and decision-making tasks. The ai agents story also surfaces in Alibaba Releases Page Agent to Control..., adding another angle. as detailed in the full paper on Arxiv