Dataset · SWE-bench-Live
Updating benchmark for real GitHub software tasks
SWE-bench-Live/SWE-bench-Live
A continuously growing software engineering task set for evaluating autonomous AI agents.
What's inside
It contains real software issues from English repositories, with a scale ranging from 1 to 10 thousand entries. The data is continuously updated using an automated collection pipeline. The dataset card provides links to the research paper, leaderboard, and evaluation script from Microsoft.
What you can do with it
- Evaluating and comparing AI agents for code writing and debugging
- Testing LLM ability to autonomously resolve bugs in real projects
- Size
- 1K–10K rows
- Downloads this month
- 33.2K
- Likes
- 10
How to load it
from datasets import load_dataset
ds = load_dataset("SWE-bench-Live/SWE-bench-Live")