Skip to content

Dataset · princeton-nlp

SWE-bench Verified: 500 GitHub Issues

princeton-nlp/SWE-bench_Verified

500 human-verified issue-pull request pairs from Python repositories for evaluating automated GitHub issue resolution.

What's inside

SWE-bench Verified is a human-verified subset of 500 examples from the SWE-bench test set. The set contains issue-pull request pairs from popular Python repositories. Systems must automatically resolve GitHub issues, with evaluation based on unit tests and post-PR behavior as the reference solution.

What you can do with it

  • Benchmark LLM agents for GitHub issue resolution
  • Evaluate code agents on Python repositories
  • Test models after fine-tuning on SWE tasks
Size
up to 1K rows
Downloads this month
184.3K
Likes
411

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("princeton-nlp/SWE-bench_Verified")