Dataset · mrlbenchmarks
Common sense benchmark for 136 language variants
mrlbenchmarks/global-piqa-nonparallel
Multiple-choice questions to test understanding of everyday logic and cultural features in rare languages.
What's inside
A benchmark for checking logic and common sense, covering 136 language varieties from 18 language families and 24 writing systems (including rare ones such as Amharic and Arabic dialects). It includes 10K to 100K examples in text and tabular formats with images. Over 350 researchers from 65+ countries created these tasks manually, with over half of the questions touching on local food, traditions, and culture.
What you can do with it
- Evaluate multilingual LLMs on understanding local cultural context
- Test models on rare and low-resource languages
- Compare multimodal and text models in multiple-choice tasks
- Tasks
- Question answering, Multiple choice
- Languages
- acm, acq, Tunisian Arabic, afb, Albanian, Amharic, Amharic, apc and 4 more
- Size
- 10K–100K rows
- Downloads this month
- 17.6K
- Likes
- 43
How to load it
from datasets import load_dataset
ds = load_dataset("mrlbenchmarks/global-piqa-nonparallel")