AI Glossary

MMLU

Also called Massive Multitask Language Understanding, MMLU-style benchmarks

What does “MMLU” mean?

MMLU: Multiple-choice knowledge test covering 57 academic and professional subjects, from law to physics, and the family of similar benchmarks it inspired. It was long a standard headline metric, but top models now score so high that harder successors such as MMLU-Pro are used to tell them apart.

MMLU visual guide: Multiple-choice knowledge test covering 57 academic and professional subjects, from law to physics, and the family of similar benchmarks it inspired. It was long a standard headline metric, but top models now score so high that harder successors such as MMLU-Pro are used to tell them apart.
Visual guide Open full size (new tab)

What is an example of “MMLU”?

When most frontier models score in the high 80s or above, a few points' difference on MMLU says little about which is more useful.

Keep learning

See also

BenchmarkTermBenchmark contaminationTermLeaderboardTermEvalsTerm

Same topic

Model cardTermMechanistic interpretabilityTermRed teamingTermLLM-as-a-judgeTermResponsible AITerm