AI GitHub
Datatrove for building repeatable text data pipelines
Filter, transform and deduplicate large text collections with composable stages.
Key Takeaways
- Pipeline blocks read, process and write records.
- Choose readers and writers for the corpus format and record the pipeline configuration.
- Run a small corpus through filtering and deduplication.
- Rules can remove valid minority-language or unusual content.
How it works
Pipeline blocks read, process and write records. Execution can be configured for local testing or supported larger-scale environments.
What you need
Choose readers and writers for the corpus format and record the pipeline configuration. A small representative run should produce both outputs and diagnostic counts before scaling.
What to try first
Run a small corpus through filtering and deduplication. Examine examples removed at each stage and save per-stage counts before increasing data size.
Measure deduplication across partitions on a synthetic set with known copies. Exact and near-duplicate rules remove different examples and change the resulting distribution.
Where to look for failures
Rules can remove valid minority-language or unusual content. Compare kept and removed samples and version the original dataset and filtering code.
Related Franklin pages
Inspect what a data rule removes before measuring its effect on a model. Aggregate changes can conceal examples, labels or languages that disappeared from the processed corpus.
- Compare data selection methods under a documented training and evaluation setup. DataComp for studying vision-language training data
- Prioritize examples for review using model predictions and data-quality methods. cleanlab for finding likely dataset problems
For the steps that prepare raw data for training or evaluation, see Franklin’s glossary. Data preprocessing
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗