Dataset · graelo
Clean Wikipedia articles across all languages
graelo/wikipedia
Cleaned full Wikipedia articles in multiple languages, with one split per language.
What's inside
The dataset is compiled from Wikipedia dumps and contains cleaned full articles in all languages. Markdown and unwanted sections such as links and footnotes are removed from the text. Each example is a single article; volume ranges from 100M to 1B examples across many languages.
What you can do with it
- Pre-training and fine-tuning multilingual language models
- Fill-mask and language modeling for low-resource languages
- Evaluating the quality of cleaning and tokenization on Wikipedia
- Tasks
- Text generation, Fill mask
- Languages
- Abkhazian, Acehnese, Adyghe, Afrikaans, Akan, Albanian, Southern Altai, Amharic and 4 more
- Size
- 100M–1B rows
- Downloads this month
- 1.2K
- Likes
- 72
Models trained on it
- mradermacher/Starstreak-7b-alpha-i1-GGUF 1.1K
- mradermacher/Starstreak-7b-alpha-GGUF 453
- azale-ai/Starstreak-7b-alphaText generation 48
- tensorblock/lightblue_karasu-7B-GGUF 48
- KennethTM/bert-base-uncased-danishFill mask 39
- lightblue/karasu-7BText generation 33
- azale-ai/Starstreak-7b-betaText generation 25
- Amal17/wikipedia-20230601.aceText generation 18
How to load it
from datasets import load_dataset
ds = load_dataset("graelo/wikipedia")