For several years, one number appeared in almost every language model announcement: the score on MMLU. Introduced in 2020 by Dan Hendrycks and colleagues at UC Berkeley and collaborators, the benchmark became the default way to say “our model knows a lot”. Understanding how it works — and why it eventually stopped being useful — is a good lesson in how AI evaluation evolves.

The problem it addressed

Earlier language benchmarks focused on narrow skills: reading comprehension, sentence similarity, common-sense completion. By 2020, large models were saturating many of them. Researchers wanted a test of broad knowledge and problem solving that would be hard for years to come.

How MMLU works

MMLU — Massive Multitask Language Understanding — contains about 16,000 multiple-choice questions spanning 57 subjects. They range from elementary mathematics to professional law, from high-school biology to clinical knowledge, philosophy, econometrics and US foreign policy. Questions were collected from freely available practice exams and educational sources.

Each question has four answer options. The model is asked to choose one, and the score is the percentage answered correctly. Random guessing gives 25%.

The authors evaluated models in zero-shot and few-shot settings: with no examples, or with five worked examples of the same subject included in the prompt.

What the original paper found

The results in 2020 were sobering. Most models scored close to random chance. The largest model tested, GPT-3 with 175 billion parameters, reached about 44% in the five-shot setting — better than chance, but far from expert level, which the authors estimated at roughly 90%.

The paper also highlighted that:

  • performance was lopsided: models did relatively well on some subjects and near chance on others, including subjects requiring calculation such as physics and mathematics;
  • models were poorly calibrated: their confidence did not match how often they were right;
  • models struggled with socially important subjects such as law and morality.

Why it became the standard

MMLU had the right properties for its moment: broad coverage, easy automatic scoring, a large gap between models and experts, and a single summary number. Labs adopted it widely, and progress on it became one of the clearest signs of how fast language models were improving. Within a few years, top models climbed from the 40s to scores around and above 85–90%.

Why it stopped being enough

Saturation. Once leading models scored around the estimated expert level, differences between them shrank to a few points — too small to rank models reliably.

Errors in the dataset. Later analyses found a share of MMLU questions with wrong answer keys, ambiguous wording or missing context. Near the top of the scale, those flaws limit how high an honest model can score.

Contamination. MMLU questions are public and have circulated online for years, so some may have entered training data, inflating scores.

Format limits. Four-option multiple choice rewards elimination strategies and does not test whether a model can produce a correct answer on its own.

The successors

Researchers responded with harder variants and new benchmarks:

  • MMLU-Pro (2024) expands options from four to ten and emphasizes reasoning-heavy questions, reducing the benefit of guessing.
  • GPQA (2023) uses graduate-level questions in biology, physics and chemistry written by domain experts and designed to be “Google-proof” — hard to answer even with web search.
  • Further benchmarks target expert-level questions across many fields, as well as tasks that require actions rather than answers.

The lesson

MMLU illustrates a recurring cycle in AI: a benchmark is designed to be hard, becomes the industry’s yardstick, gets saturated within a few years, and is replaced by something harder. When you see a high MMLU score today, read it as a basic competence check rather than a sign of frontier performance — and look at newer, harder benchmarks and real-world evaluations for the full picture.

Frequently asked questions

What score on MMLU is considered good?

Random guessing gives 25%. The original authors estimated expert-level accuracy at about 90%. Leading models now score in the high 80s or above, so MMLU mostly separates weaker models from capable ones rather than ranking frontier systems.

Why do two reports give different MMLU scores for the same model?

Settings differ: zero-shot versus five-shot prompting, whether chain-of-thought reasoning is allowed, how answers are extracted from the output, and which version of the dataset is used. Always compare scores obtained under the same settings.

Is MMLU still worth reporting?

As a sanity check, yes: a low score reveals a gap in general knowledge. For comparing top models, harder and cleaner benchmarks such as MMLU-Pro and GPQA, plus task-based evaluations, are more informative.

Can a model “cheat” on MMLU?

Not deliberately, but contamination can inflate scores. If benchmark questions and answers appeared in the web data used for training, a model may partly recall them. Careful evaluators test for this and use private or regularly refreshed question sets.