Skip to content

Dataset · mrlbenchmarks

Common sense benchmark for 136 language variants

mrlbenchmarks/global-piqa-nonparallel

Multiple-choice questions to test understanding of everyday logic and cultural features in rare languages.

What's inside

A benchmark for checking logic and common sense, covering 136 language varieties from 18 language families and 24 writing systems (including rare ones such as Amharic and Arabic dialects). It includes 10K to 100K examples in text and tabular formats with images. Over 350 researchers from 65+ countries created these tasks manually, with over half of the questions touching on local food, traditions, and culture.

What you can do with it

  • Evaluate multilingual LLMs on understanding local cultural context
  • Test models on rare and low-resource languages
  • Compare multimodal and text models in multiple-choice tasks
Tasks
Question answering, Multiple choice
Languages
acm, acq, Tunisian Arabic, afb, Albanian, Amharic, Amharic, apc and 4 more
Size
10K–100K rows
Downloads this month
17.6K
Likes
43

How to load it

from datasets import load_dataset

ds = load_dataset("mrlbenchmarks/global-piqa-nonparallel")