AI GitHub
TRL for supervised and preference-based training
Train transformer models with documented fine-tuning and preference-learning trainers.
Key Takeaways
- TRL provides trainer classes for different objectives, which expect different data fields and roles.
- Select the trainer whose objective matches the available examples.
- Run a short supervised job and inspect the formatted example batches.
- Changing the objective and dataset at the same time makes results hard to attribute.
How it works
TRL provides trainer classes for different objectives, which expect different data fields and roles. The choice of objective affects what signals each sample supplies.
What you need
Select the trainer whose objective matches the available examples. A labeled completion, a pair of preferences and a reward-driven rollout require different input contracts.
What to try first
Run a short supervised job and inspect the formatted example batches. Test a preference objective only after checking that pairs and evaluation prompts match the intended behavior.
Inspect which tokens contribute to the loss. Different training tasks can need different masks even if they read the same conversation data.
Where to look for failures
Changing the objective and dataset at the same time makes results hard to attribute. Save the base checkpoint, templates and training configuration with each run.
Related Franklin pages
Choose a training objective separately from how you update or store weights. Verify the data format, then reload the resulting artifact against its original base model.
- Adapt a supported model by training a subset of parameters and saving compact artifacts. PEFT for training and loading model adapters
- Study published instruction tuning and alignment recipes with code and data references. Open Instruct for reproducing post-training experiments
For the distinction between training from scratch and adapting a checkpoint, see Franklin’s glossary. Fine-tuning
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗