Dataset · allenai
LLM benchmark for questions with false premises
allenai/XYBench-dataset
An English dataset of questions with erroneous assumptions to evaluate LLM pragmatic responses.
What's inside
This benchmark from AllenAI (Best Resource Award at EMNLP 2026) tests whether language models can correctly respond to user queries containing misconceptions. It compiles thousands of English questions from Stack Overflow (1,557), Stack Exchange (184), WikiHow (6,268), and 102 manually curated examples, supplemented with human reference answers.
What you can do with it
- Evaluating LLM ability to gently correct user misconceptions
- SFT fine-tuning of assistants for more pragmatic and accurate dialogue
- Testing models for sycophancy resistance under incorrect premises
- Tasks
- Question answering
- Languages
- English
- Size
- 10K–100K rows
- Downloads this month
- 998
- Likes
- 3
- Access
- on request on Hugging Face
How to load it
from datasets import load_dataset
ds = load_dataset("allenai/XYBench-dataset")