Skip to content

Dataset · allenai

LLM benchmark for questions with false premises

allenai/XYBench-dataset

An English dataset of questions with erroneous assumptions to evaluate LLM pragmatic responses.

What's inside

This benchmark from AllenAI (Best Resource Award at EMNLP 2026) tests whether language models can correctly respond to user queries containing misconceptions. It compiles thousands of English questions from Stack Overflow (1,557), Stack Exchange (184), WikiHow (6,268), and 102 manually curated examples, supplemented with human reference answers.

What you can do with it

  • Evaluating LLM ability to gently correct user misconceptions
  • SFT fine-tuning of assistants for more pragmatic and accurate dialogue
  • Testing models for sycophancy resistance under incorrect premises
Tasks
Question answering
Languages
English
Size
10K–100K rows
Downloads this month
998
Likes
3
Access
on request on Hugging Face

How to load it

from datasets import load_dataset

ds = load_dataset("allenai/XYBench-dataset")