Skip to content

Dataset · princeton-nlp

SWE-bench: GitHub issues and patches

princeton-nlp/SWE-bench

2,294 issue–pull request pairs from Python repositories for evaluating GitHub issue resolution.

What's inside

Collects 2,294 issue–pull request pairs from 12 popular Python repositories. Systems must automatically resolve GitHub issues, with evaluation based on unit tests where post-PR behavior serves as the gold-standard solution. Released alongside the paper 'SWE-bench: Can Language Models Resolve Real-World GitHub Issues?'.

What you can do with it

  • Evaluating a coding agent or LLM on real GitHub issues
  • Comparing models on passing unit tests after patching
  • Fine-tuning patch generation models on issue–PR pairs
Size
10K–100K rows
Downloads this month
37.3K
Likes
157

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("princeton-nlp/SWE-bench")