Skip to content

Dataset · Shitao

Multilingual Long-Document Retrieval

Shitao/MLDR

A long-document retrieval dataset in 13 languages with GPT-3.5-generated questions.

What's inside

MLDR is built on Wikipedia, Wudao, and mC4, covering 13 typologically diverse languages; metadata lists ar, de, en, es, fr, hi. Paragraphs are randomly selected from long articles, and GPT-3.5 generates questions based on them. The question-article pairs are designed for text retrieval.

What you can do with it

  • Train a retrieval model for long documents
  • Evaluate multilingual retrieval and rerankers
  • Test RAG systems on long texts
Tasks
Text retrieval
Languages
Arabic, German, English, Spanish, French, Hindi, Italian, Japanese and 4 more
Downloads this month
2.1K
Likes
83

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("Shitao/MLDR")