AI GitHub
verl for experimenting with distributed LLM reinforcement learning
Run training workflows that coordinate rollout, scoring and policy updates.
Key Takeaways
- The framework composes distributed workers for different phases of reinforcement learning.
- Set a measurable reward and choose the supported execution backends before scaling.
- Use a small task with a deterministic reward, inspect generated rollouts and check reward calculations before increasing cluster size.
- A rising reward can reflect shortcuts in the reward function.
How it works
The framework composes distributed workers for different phases of reinforcement learning. Data exchange and batching between rollout and trainer are central to a run.
What you need
Set a measurable reward and choose the supported execution backends before scaling. Keep sample rollout records so reward and generation bugs are easier to reproduce.
What to try first
Use a small task with a deterministic reward, inspect generated rollouts and check reward calculations before increasing cluster size.
Save a small batch of rollout inputs and rewards for offline inspection. It provides a stable reference when changing the configuration of distributed workers.
Where to look for failures
A rising reward can reflect shortcuts in the reward function. Compare behavior on held-out tasks and track time spent transferring or waiting on work.
Related Franklin pages
Measure rollout generation and training independently. Distributed jobs can stall in data exchange or worker placement even when the model operation itself is fast.
- Scale Python ML work across workers while keeping resource needs explicit. Ray for distributed training and serving
- Coordinate model training, rollout generation and related workloads across accelerators. OpenRLHF for distributed preference and RL training
For the distinction between training from scratch and adapting a checkpoint, see Franklin’s glossary. Fine-tuning
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗