Skip to content

Dataset · open-index

Cleaned Wikipedia across all languages in Markdown format

open-index/open-wikipedia-markdown

Articles from all Wikipedia language editions, cleaned of MediaWiki markup and converted to pure Markdown.

What's inside

The dataset combines articles from various Wikipedia language editions (including rare languages) converted from original Wikimedia dumps into Markdown format. The texts preserve headings, font formatting, code blocks, and internal links, while utility templates, infoboxes, tables, and categories are removed. The corpus volume is between 10 million and 100 million documents.

What you can do with it

  • Pre-training and fine-tuning language models on clean encyclopedic texts
  • Creating knowledge bases for RAG systems and training models on rare languages
Tasks
Text generation, Embeddings, Text classification, Question answering, Summarization, Translation
Languages
Bhojpuri, Hiri Motu, kge, Mirandese, Tulu, Zhuang, Zeelandic, Persian and 4 more
Size
10M–100M rows
Downloads this month
2.2K
Likes
23

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("open-index/open-wikipedia-markdown")