DataTerminology.comNEW
Also known as: Massive Multitask Language Understanding
A benchmark assessing language models across 57 academic subjects to measure knowledge and reasoning via multiple-choice questions.
DESCRIPTION
MMLU evaluates a model’s breadth of knowledge and reasoning on tasks spanning STEM, humanities, and social sciences. It presents multiple-choice questions at varying difficulty levels. Performance correlates with general language understanding and pretraining quality. MMLU scores are widely reported in leaderboards and model cards. Caveats include potential data contamination and limited coverage of real-world tasks. It remains a standard baseline for academic knowledge assessment.
ALTERNATIVE DEFINITIONS
arXiv
We introduce the Massive Multitask Language Understanding (MMLU) benchmark, a test consisting of multiple-choice questions from 57 tasks including elementary mathematics, US history, computer science, law, and more.
Stresses breadth across 57 subjects and multiple-choice format.
Wikipedia
Massive Multitask Language Understanding (MMLU) is a benchmark designed to measure a language model’s knowledge acquired during pretraining by testing it on questions from diverse academic fields.
Emphasizes pretraining knowledge measurement across diverse subjects.
RELATED TERMS
TruthfulQA AGIEval HELM Benchmarking Few-Shot Learning Knowledge Assessment Multiple Choice QA LM Evaluation
COMPETING TERMS
BIG-bench MMLU-Pro
KNOWLEDGE AREAS
AI Data Science Testing & Validation MLOps
TARGET AUDIENCE
AI Engineer Data Scientist MLOps Engineer AI Product Manager Chief AI Officer
EXTERNAL SOURCES
- Wikipedia - Massive Multitask Language Understanding
- arXiv - Original MMLU paper and description
- Papers with Code - Dataset overview and leaderboards
TOP QUESTIONS
What does MMLU measure in language models?
It measures broad academic knowledge and reasoning using multiple-choice questions across 57 subjects, serving as a baseline indicator of general competence.
Category: Explanation
Why is MMLU still widely used?
Its breadth and longevity enable comparisons across generations of models, providing a historical baseline and common reference.
Category: Relevancy
What are limitations of MMLU?
Potential contamination and a focus on multiple-choice items can limit realism. It doesn’t capture interactive or tool-augmented abilities.
Category: Limitations
How should I interpret an MMLU score?
Use it as one axis of evaluation—pair with domain benchmarks, safety, latency, and cost to form a complete view.
Category: BestPractice
What benchmarks compete with or complement MMLU?
BIG-bench and MMLU-Pro provide alternative or more robust tests. TruthfulQA and AGIEval cover specific dimensions.
Category: Alternatives
Does MMLU predict performance on enterprise tasks?
Not directly. It indicates general knowledge, but task-specific evaluations are necessary for deployment decisions.
Category: Misunderstanding
How can contamination be mitigated for MMLU?
Use controlled evaluation pipelines, stable test sets, and document data provenance and model pretraining sources.
Category: BestPractice
How frequently should I reevaluate models on MMLU?
When upgrading models or changing evaluation harness versions. Track versioning for comparability.
Category: Process
Is MMLU suitable for fine-tuning evaluation?
It can detect knowledge improvements, but domain-held-out tests and safety checks should accompany fine-tuning assessments.
Category: HowTo