Skip to content

Dataset · benchmark-anon-2026

Complex multi-step benchmark for coding agents

benchmark-anon-2026/RoadmapBench

115 real-world version upgrade tasks for open-source projects across 5 programming languages.

What's inside

An English-language benchmark for evaluating AI programming agents on long planning horizons. It contains 115 tasks from 17 open-source repositories in Python, TypeScript, Go, Rust, and C++. Each task is based on a real project version upgrade and requires implementing several interrelated features.

What you can do with it

  • Testing autonomous coding agents on complex refactoring and codebase evolution tasks
  • Comparing LLM context retention when working with an entire repository
Tasks
Text generation
Languages
English
Size
up to 1K rows
Downloads this month
28K
Likes
1

How to load it

from datasets import load_dataset

ds = load_dataset("benchmark-anon-2026/RoadmapBench")