Dataset · princeton-nlp
300 GitHub Tasks for Code Evaluation
princeton-nlp/SWE-bench_Lite
300 Python repository issue-PR pairs for evaluating automated GitHub issue resolution.
What's inside
SWE-bench Lite is a subset of SWE-bench containing 300 test issue and pull request pairs from 11 popular Python repositories. Evaluation is performed by checking unit tests where the benchmark behavior is post-PR. The dataset was released as part of the SWE-bench paper.
What you can do with it
- Evaluating automated GitHub issue resolution systems
- Benchmarking LLM agents for code patches
- Testing code models on real Python tasks
- Size
- up to 1K rows
- Downloads this month
- 54K
- Likes
- 77
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("princeton-nlp/SWE-bench_Lite")