AI GitHub
LM Evaluation Harness for benchmark runs
Run documented language-model benchmark tasks through a shared evaluation interface.
Key Takeaways
- The harness separates task definitions from model adapters.
- Select the model adapter and define the task configuration before launching a large run.
- Evaluate a small model on one task and inspect a few rendered prompts and answers.
- Reported scores depend on task version, prompt formatting and evaluation settings.
How it works
The harness separates task definitions from model adapters. Prompt templates, few-shot settings and scoring rules are part of a run's configuration.
What you need
Select the model adapter and define the task configuration before launching a large run. Save the exact prompt rendering so readers can interpret the reported result.
What to try first
Evaluate a small model on one task and inspect a few rendered prompts and answers. Then repeat the same setup on a second checkpoint.
Check one task with and without few-shot examples while keeping the checkpoint fixed. This shows how much the benchmark score depends on the prompt regime.
Where to look for failures
Reported scores depend on task version, prompt formatting and evaluation settings. Record them alongside the model revision and avoid comparing incompatible runs.
Related Franklin pages
Pin the checkpoint, rendered prompts and task version for a comparison. Inspect raw completions and parsed answers so a scoring error stays distinguishable from a generation error.
- Evaluate checkpoints and inference backends using standardized or custom tasks. LightEval for configurable language-model benchmarks
- Build model evaluations with datasets, solvers, tools and scorers. Inspect AI for evaluating tool-using models
Franklin’s benchmark pages offer examples of evaluating models on specific tasks. Franklin’s AI benchmarks
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗