Skip to content

Dataset · clips

Massive multilingual web QA corpus

clips/mqa

Hundreds of millions to a billion question-answer pairs in 6 languages, including Russian, collected from Common Crawl.

What's inside

The collection contains text pairs of questions and answers extracted from the Common Crawl web archive. The data is split into two categories: FAQ pages and CQA forums and question-answering services. The corpus covers six languages (English, Russian, German, French, Spanish, and Catalan) and contains between 100 million and 1 billion entries.

What you can do with it

  • Pre-training and fine-tuning multilingual models for question answering (QA) and RAG systems
  • Training embeddings and rerankers for semantic search in Russian and European languages
  • Filtering and generating synthetic dialogue data
Tasks
Question answering
Languages
Catalan, English, German, Spanish, French, Russian, Japanese, Italian and 4 more
Size
100M–1B rows
Downloads this month
4.5K
Likes
58

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("clips/mqa")