Skip to content

Dataset · google-research-datasets

MBPP Python Tasks Benchmark

google-research-datasets/mbpp

About 1,000 Python tasks with descriptions, solutions, and tests for code generation evaluation.

What's inside

The Mostly Basic Python Problems benchmark contains about 1,000 crowdsourced tasks for beginners: programming basics, standard library, and more. Each task includes a description, a reference solution, and three automatic tests; part of the data was manually verified by the authors. The language is English.

What you can do with it

  • Evaluating LLMs on Python code generation
  • Testing code models via pass@k
  • Comparing SFT models on the benchmark
Languages
English
Size
1K–10K rows
Downloads this month
288.7K
Likes
279

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("google-research-datasets/mbpp")