Skip to content

Dataset · openai

HumanEval: 164 Programming Tasks

openai/openai_humaneval

A benchmark of 164 Python tasks with signatures, docstrings, and unit tests for code generation evaluation.

What's inside

The dataset contains 164 Python programming tasks: function signature, docstring, body, and several unit tests. The tasks are hand-written to prevent contamination in code generation model training data; natural text in comments is in English. This is the OpenAI HumanEval benchmark for code generation evaluation.

What you can do with it

  • Testing LLMs on Python code generation
  • Evaluating pass@k for code models
  • Comparing fine-tuned models on HumanEval
Languages
English
Size
up to 1K rows
Downloads this month
293.5K
Likes
420

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("openai/openai_humaneval")