AI GitHub
Argilla for collecting feedback on AI datasets
Define annotation records, fields and questions for people reviewing model outputs.
Key Takeaways
- A dataset contains records and a feedback schema.
- Define what reviewers see, what they can change and how they mark uncertainty.
- Create a dozen records with a clear labeling rubric and compare two reviewers' answers.
- Reviewer agreement depends on examples and instructions.
How it works
A dataset contains records and a feedback schema. Reviewers can label, rank or correct examples using questions that reflect the intended task.
What you need
Define what reviewers see, what they can change and how they mark uncertainty. Choose stable source identifiers before importing records from several datasets.
What to try first
Create a dozen records with a clear labeling rubric and compare two reviewers' answers. Export results and check that the identifiers match the original inputs.
Add an example that deserves two plausible labels and one that should be rejected. Include a review path for uncertainty before collecting a larger dataset.
Where to look for failures
Reviewer agreement depends on examples and instructions. Keep rejected or ambiguous labels visible during dataset QA instead of silently discarding them.
Related Franklin pages
Separate annotation from review of likely data problems. Check which record identifiers survive an export so reviewer decisions can be traced back to inputs.
- Create annotation projects for text, images, audio and other supported inputs. Label Studio for configurable data annotation
- Prioritize examples for review using model predictions and data-quality methods. cleanlab for finding likely dataset problems
For the steps that prepare raw data for training or evaluation, see Franklin’s glossary. Data preprocessing
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗