AI GitHub
DataComp for studying vision-language training data
Compare data selection methods under a documented training and evaluation setup.
Key Takeaways
- The benchmark provides data pools and a framework for selecting subsets and training models.
- Choose a data scale and a training budget that allow a fair comparison.
- Pick a tractable scale and inspect the records chosen by a baseline filter.
- Candidate pairs can contain noisy captions or problematic data.
How it works
The benchmark provides data pools and a framework for selecting subsets and training models. Holding the training recipe fixed makes the data selection decision easier to isolate.
What you need
Choose a data scale and a training budget that allow a fair comparison. Check which metadata, images and evaluations are required by that benchmark setup.
What to try first
Pick a tractable scale and inspect the records chosen by a baseline filter. Compare another filter under the same compute and evaluation settings.
Plot basic label and language statistics before and after filtering. A filter can improve a benchmark while changing who or what the dataset represents.
Where to look for failures
Candidate pairs can contain noisy captions or problematic data. Review provenance, retrieval and dataset terms before downloading or sharing subsets.
Related Franklin pages
Inspect what a data rule removes before measuring its effect on a model. Aggregate changes can conceal examples, labels or languages that disappeared from the processed corpus.
- Prioritize examples for review using model predictions and data-quality methods. cleanlab for finding likely dataset problems
- Filter, transform and deduplicate large text collections with composable stages. Datatrove for building repeatable text data pipelines
For the steps that prepare raw data for training or evaluation, see Franklin’s glossary. Data preprocessing
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗