AI GitHub
DVC for versioning datasets and ML pipelines
Track large artifacts and dependency-aware stages alongside a Git project.
Key Takeaways
- DVC stores references in Git while data and model files can live in a configured remote.
- Choose a data remote and decide who can access it from a fresh checkout.
- Track one dataset and a training stage, then reproduce the result from a clean checkout.
- Every input that affects the result needs to be declared.
How it works
DVC stores references in Git while data and model files can live in a configured remote. Pipeline stages describe commands, inputs and outputs used to reproduce work.
What you need
Choose a data remote and decide who can access it from a fresh checkout. Keep the Git commit and artifact pointers together when sharing a reproducible run.
What to try first
Track one dataset and a training stage, then reproduce the result from a clean checkout. Change a data file and confirm which stage is invalidated.
Test a fresh checkout with an empty artifact cache. A successful run on the original machine can hide a missing remote or undeclared dependency.
Where to look for failures
Every input that affects the result needs to be declared. Secure remote storage independently and check how large artifact history affects retention costs.
Related Franklin pages
Choose what needs versioning and what needs tracking. A run's curves are easier to reproduce when they link to the exact data, configuration and saved artifact.
- Record parameters, metrics and artifacts alongside model runs. MLflow for tracking experiments and model artifacts
- Log experiment metrics and inspect curves and metadata across runs. Aim for comparing training runs
For the steps that prepare raw data for training or evaluation, see Franklin’s glossary. Data preprocessing
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗