Dataset · SWE-bench
Multilingual GitHub task benchmark
SWE-bench/SWE-bench_Multilingual
300 real-world tasks (issues and pull requests) across 9 programming languages verified with unit tests.
What's inside
An extension of the well-known SWE-bench benchmark to 9 programming languages, collected from 41 popular repositories. Includes 300 test issue and pull request pairs in English. Solution quality is verified by automatic execution of unit tests, where the code state after accepting the PR serves as the reference.
What you can do with it
- Evaluating autonomous AI agents for code generation and bug fixing across multiple languages
- Comparing LLMs in software engineering tasks using objective unit tests
- Languages
- English
- Size
- up to 1K rows
- Downloads this month
- 40.5K
- Likes
- 32
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("SWE-bench/SWE-bench_Multilingual")