Dataset · wikimedia
Wikipedia in all languages
wikimedia/wikipedia
Cleaned Wikipedia articles in multiple languages, with one train split per language.
What's inside
The dataset is gathered from Wikipedia dumps and contains cleaned full articles across all languages, featuring a separate subset with one train split for each language. The texts are cleaned of markdown and unwanted sections like links, processed using recent dumps. Ranging from 10M to 100M in size, the metadata lists numerous languages, including ab, ace, and ady.
What you can do with it
- Pretraining multilingual LLMs
- Language modeling training
- Fill-mask task
- Tasks
- Text generation, Fill mask
- Languages
- Abkhazian, Acehnese, Adyghe, Afrikaans, Southern Altai, Amharic, ami, Aragonese and 4 more
- Size
- 10M–100M rows
- Downloads this month
- 333.5K
- Likes
- 1.8K
Models trained on it
- deepvk/USER2-baseSentence similarity 1.1M
- neuralcrew/neutrino-instructText generation 14.4K
- deepvk/USER2-smallSentence similarity 12.1K
- mradermacher/zorka-vieniera-12b-i1-GGUF 4.8K
- MatterWaveAI/grt-30mText generation 2.7K
- mradermacher/Qwen3.5-0.8B-Japanese-SFT-v2-i1-GGUF 2.6K
- nomic-ai/nomic-bert-2048Fill mask 2.4K
- JustANormalTinkerer/hayai-ocr-v2.5-novaImage → text 1.8K
How to load it
from datasets import load_dataset
ds = load_dataset("wikimedia/wikipedia")