AI Glossary

Direct preference optimization

Also called DPO

What does “Direct preference optimization” mean?

Direct preference optimization: Trains a model directly on pairs of preferred and rejected responses. It skips the separate reward model and the reinforcement learning loop that classic RLHF requires, which generally makes it simpler and more stable to run.

Direct preference optimization visual guide: Trains a model directly on pairs of preferred and rejected responses. It skips the separate reward model and the reinforcement learning loop that classic RLHF requires, which generally makes it simpler and more stable to run.
Visual guide Open full size (new tab)

What is an example of “Direct preference optimization”?

Open-model developers often choose DPO because they can align a model with a dataset of "better versus worse" answer pairs and ordinary training code.

Keep learning

See also

Reinforcement learning from human feedbackTermReward modelTermFine-tuningTermReinforcement learning from AI feedbackTerm

Same topic

EpochTermData labelingTermCurriculum learningTermGradient descentTermComputeTerm