Dataset · openai
HumanEval: 164 Programming Tasks
openai/openai_humaneval
A benchmark of 164 Python tasks with signatures, docstrings, and unit tests for code generation evaluation.
What's inside
The dataset contains 164 Python programming tasks: function signature, docstring, body, and several unit tests. The tasks are hand-written to prevent contamination in code generation model training data; natural text in comments is in English. This is the OpenAI HumanEval benchmark for code generation evaluation.
What you can do with it
- Testing LLMs on Python code generation
- Evaluating pass@k for code models
- Comparing fine-tuned models on HumanEval
- Languages
- English
- Size
- up to 1K rows
- Downloads this month
- 293.5K
- Likes
- 420
Models trained on it
- mradermacher/DPO-trained-Qwen2.5-Python-Coder-7B-i1-GGUF 5.4K
- mradermacher/DPO-trained-Qwen2.5-Python-Coder-7B-GGUF 1.5K
- ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7BText generation 547
- mradermacher/Atomight-V2.1-0.5B-Inference-i1-GGUFText generation 492
- rico03/Qwopus-GLM-9B-DualReason-Distilled-GGUFText generation 280
- mradermacher/Atomight-V2.1-0.5B-Inference-GGUFText generation 202
- Robot-Haus/Qwen3.8-Flash-Next-oQ3.5e-fp16-mtpImage text → text 176
- mondk/Msh-Tiny-47M-GGUFText generation 121
How to load it
from datasets import load_dataset
ds = load_dataset("openai/openai_humaneval")