AI GitHub
Inspect AI for evaluating tool-using models
Build model evaluations with datasets, solvers, tools and scorers.
Key Takeaways
- Inspect separates the examples a model receives from the solver that runs the task and the scorer that checks it.
- Define a dataset, solver and scorer with observable behavior.
- Write one task with a deterministic outcome and review its log.
- Isolation depends on the configured sandbox and its privileges.
How it works
Inspect separates the examples a model receives from the solver that runs the task and the scorer that checks it. Some tasks can run tools in a controlled environment.
What you need
Define a dataset, solver and scorer with observable behavior. Add environment access only when the task needs it and scope the sandbox separately from the model.
What to try first
Write one task with a deterministic outcome and review its log. Add a sandboxed tool and test an attempted action beyond the expected scope.
Save the scorer's decision next to the evidence it used. Tool-using tasks can appear successful in text even when the resulting environment is wrong.
Where to look for failures
Isolation depends on the configured sandbox and its privileges. Pin dependencies, keep secrets outside the environment and inspect individual failures.
Related Franklin pages
Pin the checkpoint, rendered prompts and task version for a comparison. Inspect raw completions and parsed answers so a scoring error stays distinguishable from a generation error.
- Run documented language-model benchmark tasks through a shared evaluation interface. LM Evaluation Harness for benchmark runs
- Run targeted probes and detectors against supported model endpoints. garak for probing language-model failure modes
For broader questions about the risks of deploying AI systems, see Franklin’s glossary. AI safety
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗