Skip to content

Dataset · SWE-bench

Multilingual GitHub task benchmark

SWE-bench/SWE-bench_Multilingual

300 real-world tasks (issues and pull requests) across 9 programming languages verified with unit tests.

What's inside

An extension of the well-known SWE-bench benchmark to 9 programming languages, collected from 41 popular repositories. Includes 300 test issue and pull request pairs in English. Solution quality is verified by automatic execution of unit tests, where the code state after accepting the PR serves as the reference.

What you can do with it

  • Evaluating autonomous AI agents for code generation and bug fixing across multiple languages
  • Comparing LLMs in software engineering tasks using objective unit tests
Languages
English
Size
up to 1K rows
Downloads this month
40.5K
Likes
32

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("SWE-bench/SWE-bench_Multilingual")