AI GitHub
Promptfoo for testing prompts and model behavior
Run repeatable model tests against fixtures, assertions and red-team cases.
Key Takeaways
- Configuration defines prompts, providers, test inputs and assertions.
- Start with a test configuration, at least one model endpoint and fixtures with expected outcomes.
- Use a few hand-labeled examples and add one deterministic assertion plus one scored criterion.
- A passing score depends on the assertions.
How it works
Configuration defines prompts, providers, test inputs and assertions. The same scenario can be run across models to see which outputs changed and why.
What you need
Start with a test configuration, at least one model endpoint and fixtures with expected outcomes. Keep provider credentials and private prompt material outside shared test configuration.
What to try first
Use a few hand-labeled examples and add one deterministic assertion plus one scored criterion. Run the suite on two prompt versions and review failures side by side.
Start with one assertion that a person can verify independently, such as a required citation matching an allowed set. That gives you a baseline before adding a judge model.
Where to look for failures
A passing score depends on the assertions. Keep generated test cases separate from private inputs and verify that any model-based grader applies the intended rubric.
Related Franklin pages
Start with a few failures that a person can label confidently. Keep prompt tests, retrieval metrics and model judges tied to the exact evidence used to score each example.
- Write model and retrieval evaluations as tests with explicit metric thresholds. DeepEval for application-level LLM tests
- Measure retrieval and answer behavior with metrics designed for RAG workflows. Ragas for evaluating retrieval-augmented generation
Franklin’s benchmark pages offer examples of evaluating models on specific tasks. Franklin’s AI benchmarks
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗