Skip to content

Dataset · graelo

Clean Wikipedia articles across all languages

graelo/wikipedia

Cleaned full Wikipedia articles in multiple languages, with one split per language.

What's inside

The dataset is compiled from Wikipedia dumps and contains cleaned full articles in all languages. Markdown and unwanted sections such as links and footnotes are removed from the text. Each example is a single article; volume ranges from 100M to 1B examples across many languages.

What you can do with it

  • Pre-training and fine-tuning multilingual language models
  • Fill-mask and language modeling for low-resource languages
  • Evaluating the quality of cleaning and tokenization on Wikipedia
Tasks
Text generation, Fill mask
Languages
Abkhazian, Acehnese, Adyghe, Afrikaans, Akan, Albanian, Southern Altai, Amharic and 4 more
Size
100M–1B rows
Downloads this month
1.2K
Likes
72

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("graelo/wikipedia")