Skip to content

Dataset · ScaleAI

SWE-bench Pro: benchmark for code agents

ScaleAI/SWE-bench_Pro

642 real-world PR tasks where an agent edits a repository and a patch is verified by hidden tests.

What's inside

A benchmark of long-horizon software engineering tasks from real pull requests across 11 open-source repositories in Go, Python, JavaScript, and TypeScript. For each task, the agent is given a repository at a base commit and a PR description with requirements and interfaces. Patches are evaluated using hidden fail-to-pass and pass-to-pass tests in a clean container; the V2 config contains 642 tasks.

What you can do with it

  • Evaluating code agents on real-world PRs
  • Testing patch generation models
  • Comparing agent frameworks
Size
1K–10K rows
Downloads this month
39.7K
Likes
209

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("ScaleAI/SWE-bench_Pro")