Skip to content

Dataset · bigcode

Code generation and completion benchmark

bigcode/bigcodebench

1,140 tasks to evaluate models on prompt-based code generation and docstring-based completion.

What's inside

BigCodeBench includes two variants: Complete, for code completion based on structured docstrings, and Instruct, for natural language instruction-based code generation. Each variant has 1,140 tasks with an average of 5.6 test cases and about 99% coverage; the language is code and modality is text. Average prompt lengths are also provided in the card, but the file format is not specified.

What you can do with it

  • Evaluating code LLMs on instruction-based generation
  • Testing code auto-completion models
  • Comparing code models on BigCodeBench-Instruct/Complete
Languages
Code
Size
1K–10K rows
Downloads this month
168.9K
Likes
99

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("bigcode/bigcodebench")