Dataset · Shitao
Multilingual Long-Document Retrieval
Shitao/MLDR
A long-document retrieval dataset in 13 languages with GPT-3.5-generated questions.
What's inside
MLDR is built on Wikipedia, Wudao, and mC4, covering 13 typologically diverse languages; metadata lists ar, de, en, es, fr, hi. Paragraphs are randomly selected from long articles, and GPT-3.5 generates questions based on them. The question-article pairs are designed for text retrieval.
What you can do with it
- Train a retrieval model for long documents
- Evaluate multilingual retrieval and rerankers
- Test RAG systems on long texts
- Tasks
- Text retrieval
- Languages
- Arabic, German, English, Spanish, French, Hindi, Italian, Japanese and 4 more
- Downloads this month
- 2.1K
- Likes
- 83
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("Shitao/MLDR")