cleanlab for finding likely dataset problems

How it works

cleanlab can use predicted probabilities to flag examples whose labels merit inspection. Out-of-sample predictions help distinguish probable label issues from overfitting.

What you need

Prepare predictions from examples the model did not train on when testing likely label issues. Use the review outcomes to judge whether the flagged examples were useful.

What to try first

Generate cross-validated predictions for a small classifier and manually review the highest-ranked examples. Record whether each case is a mislabel, ambiguity or a rare valid case.

Inspect flagged samples across groups rather than only the first few overall. An error category can be concentrated in rare classes with weak training coverage.

Where to look for failures

A high issue score is evidence for review. Removing all flagged examples can erase useful edge cases or reinforce the current model's bias.

Related Franklin pages

Separate annotation from review of likely data problems. Check which record identifiers survive an export so reviewer decisions can be traced back to inputs.

• Define annotation records, fields and questions for people reviewing model outputs. Argilla for collecting feedback on AI datasets
• Filter, transform and deduplicate large text collections with composable stages. Datatrove for building repeatable text data pipelines

For the steps that prepare raw data for training or evaluation, see Franklin’s glossary. Data preprocessing

Project documentation

• Current project README
• README at the reviewed revision