Dataset · McGill-NLP
Web agent auto-evaluation benchmark
McGill-NLP/agent-reward-bench
A set of 1–10k English web agent trajectories for evaluating automated evaluator quality.
What's inside
The AgentRewardBench dataset from McGill NLP researchers is designed to evaluate systems that automatically verify web agent actions. The corpus contains from 1,000 to 10,000 English records, including text data and images (web trajectory screenshots). The project is accompanied by a research paper, code, and a leaderboard.
What you can do with it
- Evaluation and calibration of LLM-as-a-Judge and reward models for web agents
- Development of more reliable metrics for task success in browsers
- Error analysis of automatic verifiers working with GUI and web interfaces
- Tasks
- Robotics
- Languages
- English
- Size
- 1K–10K rows
- Downloads this month
- 22.3K
- Likes
- 7
How to load it
from datasets import load_dataset
ds = load_dataset("McGill-NLP/agent-reward-bench")