Dataset · tasksource
Lightweight duplicate-free MMLU benchmark
tasksource/mmlu
Cleaned and fast version of the well-known MMLU benchmark in English for evaluating language model knowledge.
What's inside
This is a repackaged version of the classic English MMLU benchmark (hendrycks_test), from which the auxiliary training set (auxiliary train) was removed. Due to the absence of duplicate data, the size was reduced from 162 MB to 7 MB with a volume of 10,000 to 100,000 examples. The data is intended for multiple-choice, classification, and question-answering tasks.
What you can do with it
- Rapid evaluation of erudition and general knowledge of language models across dozens of disciplines
- Integration into LLM automated testing pipelines without excessive disk and memory overhead
- Tasks
- Text classification, Multiple choice, Question answering
- Languages
- English
- Size
- 10K–100K rows
- Downloads this month
- 30.8K
- Likes
- 37
Models trained on it
- sileod/deberta-v3-base-tasksource-nliZero shot classification 8K
- sileod/deberta-v3-large-tasksource-nliZero shot classification 1.8K
- TheBloke/Manticore-13B-GGUFText generation 852
- mradermacher/manticore-13b-v2-i1-GGUF 827
- TheBloke/Manticore-13B-GPTQText generation 242
- sileod/deberta-v3-xsmall-tasksource-nliZero shot classification 240
- openaccess-ai-collective/manticore-13bText generation 225
- mradermacher/manticore-13b-v2-GGUF 198
How to load it
from datasets import load_dataset
ds = load_dataset("tasksource/mmlu")