AI GitHub
LlamaFactory for configuring model fine-tuning
Train supported language models with configurable datasets and tuning methods.
Key Takeaways
- LlamaFactory organizes model, data and training choices in configurations and offers multiple interfaces for running them.
- Choose the base checkpoint and an appropriate conversation template before creating data records.
- Fine-tune a small checkpoint on a short dataset and inspect one formatted sample.
- A valid dataset can use the wrong roles or template.
How it works
LlamaFactory organizes model, data and training choices in configurations and offers multiple interfaces for running them. Chat templates determine how examples are formatted for training.
What you need
Choose the base checkpoint and an appropriate conversation template before creating data records. Validate one formatted sample before committing accelerator time.
What to try first
Fine-tune a small checkpoint on a short dataset and inspect one formatted sample. Compare base and tuned outputs on examples held out before training.
Include examples with different lengths and roles in a preprocessing test. It catches formatting and truncation problems before a long training job starts.
Where to look for failures
A valid dataset can use the wrong roles or template. Pin preprocessing, track memory and verify model and dataset terms before distributing adapters.
Related Franklin pages
Run a tiny supervised job and check the prepared examples before investing in a large run. Save the base checkpoint, data revision and validation outputs with the recipe.
- Describe model fine-tuning runs in configuration files and reuse the recipes across experiments. Axolotl for recipe-based language model training
- Fine-tune supported model architectures using optimized training paths and documented recipes. Unsloth for memory-conscious model tuning
For the distinction between training from scratch and adapting a checkpoint, see Franklin’s glossary. Fine-tuning
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗