AI GitHub
DeepEval for application-level LLM tests
Write model and retrieval evaluations as tests with explicit metric thresholds.
Key Takeaways
- DeepEval defines test cases and metrics that score generated outputs, retrieval evidence or agent behavior.
- Use a test case that contains the fields your metric expects.
- Turn one known production failure into a fixture and add cases just above and below the expected threshold.
- Judge models bring cost and their own failure modes.
How it works
DeepEval defines test cases and metrics that score generated outputs, retrieval evidence or agent behavior. Some metrics use a judge model, while others can use deterministic criteria.
What you need
Use a test case that contains the fields your metric expects. For metrics that invoke another model, document the judge and use a limited budget during calibration.
What to try first
Turn one known production failure into a fixture and add cases just above and below the expected threshold. Run the same test repeatedly to measure grader variance.
Preserve the judge's reason and the evaluated response together. If a score changes after a prompt edit, you can inspect whether the reasoning changed as well.
Where to look for failures
Judge models bring cost and their own failure modes. Compare metric output with human labels before using a threshold to block releases.
Related Franklin pages
Start with a few failures that a person can label confidently. Keep prompt tests, retrieval metrics and model judges tied to the exact evidence used to score each example.
- Measure retrieval and answer behavior with metrics designed for RAG workflows. Ragas for evaluating retrieval-augmented generation
- Use a registry and evaluation runner to define test cases for model responses. OpenAI Evals for reproducible model evaluations
Franklin’s benchmark pages offer examples of evaluating models on specific tasks. Franklin’s AI benchmarks
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗