Skip to content

Dataset · lmsys

MT-bench Human Judgment Dataset

lmsys/mt_bench_human_judgments

3.3k expert pairwise preference annotations for six model responses to 80 MT-bench questions.

What's inside

The dataset contains 3.3k expert pairwise human preferences between responses from six models to 80 MT-bench questions. Models include GPT-4, GPT-3.5, Claude-v1, Vicuna-13B, Alpaca-13B, and LLaMA-13B. Annotators are primarily graduate students knowledgeable in the question domains; language is English.

What you can do with it

  • Train a reward model on pairwise preferences
  • Evaluate an LLM-as-a-judge against human evaluations
  • Benchmark models on MT-bench questions
Tasks
Question answering
Languages
English
Size
1K–10K rows
Downloads this month
15.8K
Likes
150

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("lmsys/mt_bench_human_judgments")