Skip to content

Dataset · McGill-NLP

Web agent auto-evaluation benchmark

McGill-NLP/agent-reward-bench

A set of 1–10k English web agent trajectories for evaluating automated evaluator quality.

What's inside

The AgentRewardBench dataset from McGill NLP researchers is designed to evaluate systems that automatically verify web agent actions. The corpus contains from 1,000 to 10,000 English records, including text data and images (web trajectory screenshots). The project is accompanied by a research paper, code, and a leaderboard.

What you can do with it

  • Evaluation and calibration of LLM-as-a-Judge and reward models for web agents
  • Development of more reliable metrics for task success in browsers
  • Error analysis of automatic verifiers working with GUI and web interfaces
Tasks
Robotics
Languages
English
Size
1K–10K rows
Downloads this month
22.3K
Likes
7

How to load it

from datasets import load_dataset

ds = load_dataset("McGill-NLP/agent-reward-bench")