Ragas for evaluating retrieval-augmented generation

How it works

The library can score aspects such as answer faithfulness and the quality of retrieved context. Different metrics need different inputs, including references or a judge model in some cases.

What you need

Capture the retrieved contexts as well as each question and response. Reference-dependent metrics need a human or independently sourced answer tied to the same example.

What to try first

Label a small set of questions and capture their retrieved passages. Score the same answers with original and deliberately corrupted context to understand how the metric reacts.

Separate answers that correctly say evidence is absent from ones that hallucinate. Evaluators can conflate these when the fixture lacks an explicit expected response.

Where to look for failures

Synthetic questions and model judges can share blind spots. Include human-labeled examples and record the model and metric configuration used.

Related Franklin pages

Start with a few failures that a person can label confidently. Keep prompt tests, retrieval metrics and model judges tied to the exact evidence used to score each example.

• Use a registry and evaluation runner to define test cases for model responses. OpenAI Evals for reproducible model evaluations
• Run repeatable model tests against fixtures, assertions and red-team cases. Promptfoo for testing prompts and model behavior

Franklin’s benchmark pages offer examples of evaluating models on specific tasks. Franklin’s AI benchmarks

Project documentation

• Current project README
• README at the reviewed revision