Dataset · open-index
Cleaned Wikipedia across all languages in Markdown format
open-index/open-wikipedia-markdown
Articles from all Wikipedia language editions, cleaned of MediaWiki markup and converted to pure Markdown.
What's inside
The dataset combines articles from various Wikipedia language editions (including rare languages) converted from original Wikimedia dumps into Markdown format. The texts preserve headings, font formatting, code blocks, and internal links, while utility templates, infoboxes, tables, and categories are removed. The corpus volume is between 10 million and 100 million documents.
What you can do with it
- Pre-training and fine-tuning language models on clean encyclopedic texts
- Creating knowledge bases for RAG systems and training models on rare languages
- Tasks
- Text generation, Embeddings, Text classification, Question answering, Summarization, Translation
- Languages
- Bhojpuri, Hiri Motu, kge, Mirandese, Tulu, Zhuang, Zeelandic, Persian and 4 more
- Size
- 10M–100M rows
- Downloads this month
- 2.2K
- Likes
- 23
Models trained on it
- hudsongouge/AOMTS-Base-100M-3k-1MTP-Cosine-v1Text generation 21
- hudsongouge/AOMTS-TST-s6-100M-3k-2MTP-v1Text generation 20
- hudsongouge/Chinchilla-300M-FFN-SwigluText generation 19
- hudsongouge/AOMTS-Base-100M-3k-0MTP-v1-run2Text generation 17
- hudsongouge/AOMTS-Base-100M-3k-2MTP-v1Text generation 17
- hudsongouge/Chinchilla-300M-FFN-ConvSwiGLUText generation 17
- hudsongouge/rrt-40m-wiki-v1Text generation 16
- hudsongouge/rrt-40m-wiki-v1-alt 16
How to load it
from datasets import load_dataset
ds = load_dataset("open-index/open-wikipedia-markdown")