Skip to content

Dataset · SWE-bench-Live

Updating benchmark for real GitHub software tasks

SWE-bench-Live/SWE-bench-Live

A continuously growing software engineering task set for evaluating autonomous AI agents.

What's inside

It contains real software issues from English repositories, with a scale ranging from 1 to 10 thousand entries. The data is continuously updated using an automated collection pipeline. The dataset card provides links to the research paper, leaderboard, and evaluation script from Microsoft.

What you can do with it

  • Evaluating and comparing AI agents for code writing and debugging
  • Testing LLM ability to autonomously resolve bugs in real projects
Size
1K–10K rows
Downloads this month
33.2K
Likes
10

How to load it

from datasets import load_dataset

ds = load_dataset("SWE-bench-Live/SWE-bench-Live")