Dataset · princeton-nlp
SWE-bench: GitHub issues and patches
princeton-nlp/SWE-bench
2,294 issue–pull request pairs from Python repositories for evaluating GitHub issue resolution.
What's inside
Collects 2,294 issue–pull request pairs from 12 popular Python repositories. Systems must automatically resolve GitHub issues, with evaluation based on unit tests where post-PR behavior serves as the gold-standard solution. Released alongside the paper 'SWE-bench: Can Language Models Resolve Real-World GitHub Issues?'.
What you can do with it
- Evaluating a coding agent or LLM on real GitHub issues
- Comparing models on passing unit tests after patching
- Fine-tuning patch generation models on issue–PR pairs
- Size
- 10K–100K rows
- Downloads this month
- 37.3K
- Likes
- 157
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("princeton-nlp/SWE-bench")