Reward model
Scores how good a response is, used to train and steer other models toward preferred behaviour.
What it is
A reward model predicts which of two responses a person would prefer, or how good one response is. It is not used by end users directly; it is the judge that reinforcement learning uses to make assistants more helpful, honest and safe, and it can also rank candidate answers at run time.
How it works
People compare pairs of model answers and pick the better one. A language model with a scoring head is trained so the preferred answer gets the higher score. That score then becomes the reward signal in reinforcement learning from human feedback (RLHF).
What it is good for
- Training assistants with RLHF
- Picking the best of several generated answers
- Evaluating model outputs at scale
Limits to know
- Models can learn to exploit its blind spots (reward hacking)
- Reflects the preferences and biases of the raters
- Needs fresh data as models improve
