Dataset · McGill-NLP
Stereotype and bias evaluation benchmark in language models
McGill-NLP/stereoset
17,000 English sentences to measure gender, racial, religious, and occupational stereotypes in LLMs.
What's inside
It contains 17,000 English sentences organized in a multiple-choice format to check language model bias. The dataset from McGill-NLP researchers evaluates model bias tendencies across four dimensions: gender, race, religion, and profession.
What you can do with it
- Testing and auditing language models for social stereotypes and bias
- Comparing safety and ethics of different LLMs before production deployment
- Tasks
- Text classification
- Languages
- English
- Size
- 1K–10K rows
- Downloads this month
- 8.1K
- Likes
- 31
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("McGill-NLP/stereoset")