Skip to content

Dataset · lighteval

MMLU benchmark with questions across 57 knowledge domains

lighteval/mmlu

A classic multitask multiple-choice test for evaluating the general knowledge of language models.

What's inside

A well-known academic benchmark for testing erudition and language understanding (Massive Multitask Language Understanding). It includes English multiple-choice questions across 57 disciplines spanning humanities, social sciences, STEM, and other fields. The text dataset contains between 1 and 10 million records.

What you can do with it

  • Evaluating the quality and erudition of large language models (LLMs) across 57 subjects
  • Comparing models before and after SFT or domain fine-tuning
  • Validating question-answering retrieval systems
Tasks
Question answering
Languages
English
Size
1M–10M rows
Downloads this month
12.4K
Likes
44

How to load it

from datasets import load_dataset

ds = load_dataset("lighteval/mmlu")