AI GitHub
LightEval for configurable language-model benchmarks
Evaluate checkpoints and inference backends using standardized or custom tasks.
Key Takeaways
- LightEval provides task definitions, metrics and interfaces to supported model backends.
- Choose the backend and task before choosing evaluation scale.
- Choose a task you can inspect by hand and run it on a small subset.
- Output formatting and answer extraction can drive a score change.
How it works
LightEval provides task definitions, metrics and interfaces to supported model backends. A benchmark run can produce per-example information in addition to aggregates.
What you need
Choose the backend and task before choosing evaluation scale. Keep model, tokenizer and dataset revisions together if the result will be compared with another run.
What to try first
Choose a task you can inspect by hand and run it on a small subset. Compare the prompt, parsed prediction and metric output before scaling to a full run.
Inspect a case where parsing fails after a plausible generation. That failure is important if the benchmark reports only aggregate accuracy.
Where to look for failures
Output formatting and answer extraction can drive a score change. Pin the dataset revision, sampling settings and parser as well as the model.
Related Franklin pages
Pin the checkpoint, rendered prompts and task version for a comparison. Inspect raw completions and parsed answers so a scoring error stays distinguishable from a generation error.
- Build model evaluations with datasets, solvers, tools and scorers. Inspect AI for evaluating tool-using models
- Run documented language-model benchmark tasks through a shared evaluation interface. LM Evaluation Harness for benchmark runs
Franklin’s benchmark pages offer examples of evaluating models on specific tasks. Franklin’s AI benchmarks
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗