AI GitHub
OpenAI Evals for reproducible model evaluations
Use a registry and evaluation runner to define test cases for model responses.
Key Takeaways
- The repository contains a framework and examples built around evaluation tasks and completion interfaces.
- Choose an evaluation task and a compatible completion interface.
- Create a small deterministic evaluation with expected answers, then inspect the per-sample records.
- Check model compatibility in the current README and keep the fixtures and scoring logic independently versioned.
How it works
The repository contains a framework and examples built around evaluation tasks and completion interfaces. A task defines samples and how outputs are judged.
What you need
Choose an evaluation task and a compatible completion interface. Keep the sample registry and score implementation alongside the commands used to run them.
What to try first
Create a small deterministic evaluation with expected answers, then inspect the per-sample records. Compare exact match with a criterion that tolerates harmless variations.
Include alternate spellings or equivalent units where the rubric permits them. A metric should measure the task rather than a preferred textual representation.
Where to look for failures
Examples and APIs can age. Check model compatibility in the current README and keep the fixtures and scoring logic independently versioned.
Related Franklin pages
Start with a few failures that a person can label confidently. Keep prompt tests, retrieval metrics and model judges tied to the exact evidence used to score each example.
- Run repeatable model tests against fixtures, assertions and red-team cases. Promptfoo for testing prompts and model behavior
- Write model and retrieval evaluations as tests with explicit metric thresholds. DeepEval for application-level LLM tests
Franklin’s benchmark pages offer examples of evaluating models on specific tasks. Franklin’s AI benchmarks
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗