Dataset · SWE-bench
Python GitHub real-world task benchmark
SWE-bench/SWE-bench_Lite
A compact set of 300 real Python repository tasks for evaluating coding agents and language models.
What's inside
This is a lightweight version of the well-known SWE-bench benchmark, containing exactly 300 test pairs of issues and pull requests. All examples are taken from 11 popular open Python repositories on GitHub and presented in text format. Correctness checking is performed automatically by running unit tests.
What you can do with it
- Evaluating the ability of LLMs and AI agents to independently fix bugs in code
- Comparing coding models on a standard reproducible benchmark
- Size
- up to 1K rows
- Downloads this month
- 39.8K
- Likes
- 35
How to load it
from datasets import load_dataset
ds = load_dataset("SWE-bench/SWE-bench_Lite")