Skip to content

Dataset · SWE-bench

Python GitHub real-world task benchmark

SWE-bench/SWE-bench_Lite

A compact set of 300 real Python repository tasks for evaluating coding agents and language models.

What's inside

This is a lightweight version of the well-known SWE-bench benchmark, containing exactly 300 test pairs of issues and pull requests. All examples are taken from 11 popular open Python repositories on GitHub and presented in text format. Correctness checking is performed automatically by running unit tests.

What you can do with it

  • Evaluating the ability of LLMs and AI agents to independently fix bugs in code
  • Comparing coding models on a standard reproducible benchmark
Size
up to 1K rows
Downloads this month
39.8K
Likes
35

How to load it

from datasets import load_dataset

ds = load_dataset("SWE-bench/SWE-bench_Lite")