OpenRLHF for distributed preference and RL training

How it works

OpenRLHF separates work such as rollout generation and training among distributed processes. Recipes define how the learner, reference or reward components are placed and connected.

What you need

Configure the worker topology and its model roles before trying the largest example. A small initial run should prove that rollout, scoring and updates connect correctly.

What to try first

Start with a small example using the documented topology. Log generation rate and trainer progress separately to see which stage limits throughput.

Track the distribution of responses and rewards throughout training. A throughput improvement can make it easier to overlook repeated or degenerate rollouts.

Where to look for failures

The placement of models and processes drives memory use. Plan around checkpoint recovery and verify reward behavior with held-out prompts.

Related Franklin pages

Measure rollout generation and training independently. Distributed jobs can stall in data exchange or worker placement even when the model operation itself is fast.

• Run training workflows that coordinate rollout, scoring and policy updates. verl for experimenting with distributed LLM reinforcement learning
• Scale Python ML work across workers while keeping resource needs explicit. Ray for distributed training and serving

For the distinction between training from scratch and adapting a checkpoint, see Franklin’s glossary. Fine-tuning

Project documentation

• Current project README
• README at the reviewed revision