Dataset · ScaleAI
SWE-bench Pro: benchmark for code agents
ScaleAI/SWE-bench_Pro
642 real-world PR tasks where an agent edits a repository and a patch is verified by hidden tests.
What's inside
A benchmark of long-horizon software engineering tasks from real pull requests across 11 open-source repositories in Go, Python, JavaScript, and TypeScript. For each task, the agent is given a repository at a base commit and a PR description with requirements and interfaces. Patches are evaluated using hidden fail-to-pass and pass-to-pass tests in a clean container; the V2 config contains 642 tasks.
What you can do with it
- Evaluating code agents on real-world PRs
- Testing patch generation models
- Comparing agent frameworks
- Size
- 1K–10K rows
- Downloads this month
- 39.7K
- Likes
- 209
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("ScaleAI/SWE-bench_Pro")