Dataset · SWE-bench
SWE-bench Verified: 500 verified GitHub tasks
SWE-bench/SWE-bench_Verified
500 issue-pull request pairs from Python repositories to evaluate GitHub task resolution via unit tests.
What's inside
The dataset contains 500 test issue-pull request pairs from popular Python repositories, with quality verified manually. Solutions are evaluated by running unit tests, and post-PR behavior serves as the benchmark. This is a text selection for automatically resolving GitHub issues.
What you can do with it
- Evaluate agent systems and LLMs on real GitHub issues
- Assess code generation models using unit tests
- Benchmark automated bug-fixing pipelines
- Size
- up to 1K rows
- Downloads this month
- 112.1K
- Likes
- 205
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("SWE-bench/SWE-bench_Verified")