Dataset · google-research-datasets
TyDi QA multilingual question answering dataset
google-research-datasets/tydiqa
204 thousand question-answer pairs from Google Research in 11 typologically diverse languages.
What's inside
This is a text dataset for the question-answering task, containing 204 thousand question-answer pairs. Metadata specifies Arabic, Bengali, English, Finnish, Indonesian, and Japanese (11 languages total stated in description). Collected by Google Research authors with an emphasis on linguistic diversity to test model generalization.
What you can do with it
- Training and testing multilingual text question-answering systems
- Checking LLM robustness to rare linguistic phenomena
- Tasks
- Question answering
- Languages
- Arabic, Bangla, English, Finnish, Indonesian, Japanese, Korean, Russian and 3 more
- Size
- 100K–1M rows
- Downloads this month
- 12.4K
- Likes
- 39
Models trained on it
- madebyaris/rerank-indonesiaText ranking 117
- mzahran1/arabic-english-e5-small-retrievalSentence similarity 107
- Trentz/IndoLM-M-v0.2-InstructText generation 83
- mzahran1/arabic-english-reranker-mminilmText ranking 77
- mmetamong/ko-decision-roberta-largeText classification 76
- Waqf-AI/arabic-modernbert-unifiedFill mask 68
- MostafaAhmed98/MARBERTv2-finetuned-ar-tydiqaQuestion answering 64
- mmetamong/ko-decision-bge-m3Text classification 60
How to load it
from datasets import load_dataset
ds = load_dataset("google-research-datasets/tydiqa")