Skip to content

Dataset · uonlp

Multilingual corpus in 167 languages

uonlp/CulturaX

Cleaned multilingual corpus in 167 languages with 6.3 trillion tokens for training LLMs.

What's inside

CulturaX is a public multilingual text corpus for large language model development. It contains 6.3 trillion tokens across 167 languages and has undergone multi-step cleaning and deduplication. Languages include af, als, am, an, ar, arz, and others.

What you can do with it

  • Pre-training a multilingual LLM
  • Fine-tuning on low-resource languages
  • Evaluating language model quality
Tasks
Text generation, Fill mask
Languages
Afrikaans, Albanian, Amharic, Aragonese, Arabic, Egyptian Arabic, Assamese, Asturian and 4 more
Size
1B–10B rows
Downloads this month
18.9K
Likes
712
Access
on request on Hugging Face

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("uonlp/CulturaX")