Dataset · clips
Massive multilingual web QA corpus
clips/mqa
Hundreds of millions to a billion question-answer pairs in 6 languages, including Russian, collected from Common Crawl.
What's inside
The collection contains text pairs of questions and answers extracted from the Common Crawl web archive. The data is split into two categories: FAQ pages and CQA forums and question-answering services. The corpus covers six languages (English, Russian, German, French, Spanish, and Catalan) and contains between 100 million and 1 billion entries.
What you can do with it
- Pre-training and fine-tuning multilingual models for question answering (QA) and RAG systems
- Training embeddings and rerankers for semantic search in Russian and European languages
- Filtering and generating synthetic dialogue data
- Tasks
- Question answering
- Languages
- Catalan, English, German, Spanish, French, Russian, Japanese, Italian and 4 more
- Size
- 100M–1B rows
- Downloads this month
- 4.5K
- Likes
- 58
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("clips/mqa")