Dataset · princeton-nlp
SWE-bench Verified: 500 GitHub Issues
princeton-nlp/SWE-bench_Verified
500 human-verified issue-pull request pairs from Python repositories for evaluating automated GitHub issue resolution.
What's inside
SWE-bench Verified is a human-verified subset of 500 examples from the SWE-bench test set. The set contains issue-pull request pairs from popular Python repositories. Systems must automatically resolve GitHub issues, with evaluation based on unit tests and post-PR behavior as the reference solution.
What you can do with it
- Benchmark LLM agents for GitHub issue resolution
- Evaluate code agents on Python repositories
- Test models after fine-tuning on SWE tasks
- Size
- up to 1K rows
- Downloads this month
- 184.3K
- Likes
- 411
Models trained on it
- mradermacher/next-70b-i1-GGUFText generation 1.4K
- mradermacher/Kronumos-Kairos-v2-i1-GGUFReinforcement learning 1.2K
- bartowski/NadevA23_Kronumos-Kairos-v2-GGUFText generation 1.1K
- thelamapi/next-70b-i1-GGUFText generation 542
- Kronumos/Kronumos-Kairos-v2Text generation 497
- mradermacher/Kronumos-Kairos-v2-GGUFReinforcement learning 461
- mradermacher/next-70b-GGUFText generation 409
- Kronumos/Kronumos-14B-KairosText generation 316
How to load it
from datasets import load_dataset
ds = load_dataset("princeton-nlp/SWE-bench_Verified")