Dataset · bigcode
Code generation and completion benchmark
bigcode/bigcodebench
1,140 tasks to evaluate models on prompt-based code generation and docstring-based completion.
What's inside
BigCodeBench includes two variants: Complete, for code completion based on structured docstrings, and Instruct, for natural language instruction-based code generation. Each variant has 1,140 tasks with an average of 5.6 test cases and about 99% coverage; the language is code and modality is text. Average prompt lengths are also provided in the card, but the file format is not specified.
What you can do with it
- Evaluating code LLMs on instruction-based generation
- Testing code auto-completion models
- Comparing code models on BigCodeBench-Instruct/Complete
- Languages
- Code
- Size
- 1K–10K rows
- Downloads this month
- 168.9K
- Likes
- 99
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("bigcode/bigcodebench")