Skip to content

Dataset · wikimedia

Wikipedia in all languages

wikimedia/wikipedia

Cleaned Wikipedia articles in multiple languages, with one train split per language.

What's inside

The dataset is gathered from Wikipedia dumps and contains cleaned full articles across all languages, featuring a separate subset with one train split for each language. The texts are cleaned of markdown and unwanted sections like links, processed using recent dumps. Ranging from 10M to 100M in size, the metadata lists numerous languages, including ab, ace, and ady.

What you can do with it

  • Pretraining multilingual LLMs
  • Language modeling training
  • Fill-mask task
Tasks
Text generation, Fill mask
Languages
Abkhazian, Acehnese, Adyghe, Afrikaans, Southern Altai, Amharic, ami, Aragonese and 4 more
Size
10M–100M rows
Downloads this month
333.5K
Likes
1.8K

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("wikimedia/wikipedia")