AI Glossary

Agent evals

Also called agent evaluation, evals for agents

What does “Agent evals” mean?

Agent evals: Tests that measure how well an agent completes realistic, multi-step tasks, checking final outcomes and sometimes the path taken, cost and safety. Because agents behave differently from run to run, results are usually averaged over many attempts.

Agent evals visual guide: Tests that measure how well an agent completes realistic, multi-step tasks, checking final outcomes and sometimes the path taken, cost and safety. Because agents behave differently from run to run, results are usually averaged over many attempts.
Visual guide Open full size (new tab)

What is an example of “Agent evals”?

SWE-bench is a well-known example: it checks whether coding agents can resolve real issues taken from open-source projects.

Keep learning

See also

BenchmarkTermEvalsTermLong-horizon taskTermCoding agentTerm

Same topic

Agent guardrailsTermA2ATermAgent loopTermAgent memoryTermAgent permissionsTerm