Skip to content

Dataset · SWE-bench

SWE-bench Verified: 500 verified GitHub tasks

SWE-bench/SWE-bench_Verified

500 issue-pull request pairs from Python repositories to evaluate GitHub task resolution via unit tests.

What's inside

The dataset contains 500 test issue-pull request pairs from popular Python repositories, with quality verified manually. Solutions are evaluated by running unit tests, and post-PR behavior serves as the benchmark. This is a text selection for automatically resolving GitHub issues.

What you can do with it

  • Evaluate agent systems and LLMs on real GitHub issues
  • Assess code generation models using unit tests
  • Benchmark automated bug-fixing pipelines
Size
up to 1K rows
Downloads this month
112.1K
Likes
205

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("SWE-bench/SWE-bench_Verified")