Manifold Drift in Flow Preference Optimization: A R... | AI Research

Manifold Drift in Flow Preference Optimization: A Root Cause of Reward Hacking investigates why continuous-time generative models, such as flow-based models, often experience a decline in sample quality when fine-tuned using preference optimization. The authors, Yansen Han, Shengyi Liao, Yuanxing Zhang, Pengfei Wan, and Tao Lin, identify a failure mode they term "manifold drift," where reward-driven updates push generated samples away from the pretrained data manifold, leading to visual artifacts and semantic distortion.

The Problem: Manifold Drift

In flow matching, models generate data by transporting noise along learned trajectories toward a terminal manifold—a set of points representing valid data. While standard preference optimization (like DPO) is effective for discrete models, applying it to continuous models often modifies these transport trajectories. The authors demonstrate that if a preference update induces a displacement with a nonzero normal component relative to the pretrained manifold, the model will drift off that manifold. This drift is a structural cause of reward hacking, where the model achieves higher reward scores by producing samples that are technically "preferred" but no longer reside within the learned generative support.

The Solution: ThermoDPO

To address this, the authors propose ThermoDPO, a temperature-controlled objective designed to anchor preference optimization to the pretrained manifold. By introducing a time-dependent temperature function, the method balances two competing goals: maximizing the reward signal and preserving the integrity of the terminal manifold.
The objective functions by adding a "winner-side anchor" to the optimization process. Theoretically, this approach bridges the gap between two existing techniques:

• Rejection Sampling Fine-Tuning (RFT): As the temperature approaches zero, ThermoDPO reduces to an RFT-like objective, which prioritizes reconstruction accuracy.

• FlowDPO: At higher temperatures, the objective decomposes into a temperature-scaled FlowDPO component combined with a non-negative anchoring term that penalizes drift.
  To ensure the anchor remains effective near the terminal endpoint, the authors also introduced a reweighted variant, ThermoDPO-weighted.

Performance and Results

The authors evaluated their method on both a toy benchmark and real-world image generation tasks using the SD3.5-M model.

• Toy Benchmark: ThermoDPO-weighted achieved a StrictScore of 0.899, outperforming FlowDPO (0.629) and FlowDPO+RFT (0.857).

• Real-Image Generation: On SD3.5-M at a Classifier-Free Guidance (CFG) scale of 4.5, the method improved Optical Character Recognition (OCR) metrics by 47.5% and the average of four quality metrics by 16.0%.

Franklin Analysis

The research provides a clear geometric explanation for why preference alignment in continuous models often degrades visual fidelity. By formalizing manifold drift, the authors move the discussion of reward hacking from an empirical observation to a structural, mathematical problem. The strength of the approach lies in its ability to provide an upper bound on a reconstruction-based surrogate for manifold distance, offering a theoretical guarantee that the model remains anchored to the pretrained support while still responding to preference signals. However, the authors note that while their method improves the trade-off between alignment and preservation, it remains a pointwise approach rather than a full distribution-level guarantee.