PAMD: Structured Adaptive Distances for Bisimulatio... | AI Research

PAMD: Structured Adaptive Distances for Bisimulation Representations in Visual Reinforcement Learning
Visual reinforcement learning (RL) often relies on "bisimulation," a method that groups states together if they result in similar rewards and future transitions. To make this work, algorithms must learn a "latent representation"—a compressed version of an image—that matches these behavioral similarities. Current methods typically use fixed, rigid distance metrics (like standard $\ell_p$ norms) to measure this similarity. However, these fixed metrics are often too restrictive to capture the complex geometry of an environment. PAMD (Pairwise Adaptive Mahalanobis Distance) introduces a more flexible, adaptive approach that allows the model to learn a custom, pair-conditioned distance metric, leading to better representations and improved performance in visual control tasks.

The Problem with Fixed Metrics

In many existing RL algorithms, the encoder is trained to ensure that the distance between two latent states matches a target behavioral distance. Because this target is often fixed, researchers usually rely on simple, pre-defined metrics to measure the distance between latent points. The authors argue that this is a "one-size-fits-all" approach that fails to account for the fact that different regions of the state space may require different ways of measuring similarity. Conversely, making the distance metric completely unconstrained can lead to "degenerate solutions," where the model learns to cheat the math without actually improving the quality of the visual representation.

How PAMD Works

PAMD acts as a plug-in module that replaces fixed distance metrics with a "pairwise-conditioned positive-definite quadratic form." Instead of using a static rule, PAMD uses a lightweight neural network to output a matrix that changes based on the specific pair of states being compared. This allows the model to capture "anisotropy"—the ability to measure distance differently depending on the direction and context of the latent space. By using a Cholesky decomposition and trace normalization, the method ensures that the distance remains mathematically stable, symmetric, and non-negative, providing enough structure to prevent the model from collapsing into useless representations.

Key Findings and Performance

The researchers tested PAMD on visual MuJoCo continuous-control tasks, which are standard benchmarks for visual RL. The results showed that equipping existing bisimulation-based algorithms with PAMD led to substantial improvements in final performance. The authors also conducted ablation studies to confirm that their structured approach effectively prevents "representation under-training." By providing a middle ground between rigid, hand-designed metrics and fully unconstrained learning, PAMD ensures that the encoder is forced to learn meaningful, well-organized features that align with the underlying behavioral dynamics of the environment.