Dataset · uonlp
Multilingual corpus in 167 languages
uonlp/CulturaX
Cleaned multilingual corpus in 167 languages with 6.3 trillion tokens for training LLMs.
What's inside
CulturaX is a public multilingual text corpus for large language model development. It contains 6.3 trillion tokens across 167 languages and has undergone multi-step cleaning and deduplication. Languages include af, als, am, an, ar, arz, and others.
What you can do with it
- Pre-training a multilingual LLM
- Fine-tuning on low-resource languages
- Evaluating language model quality
- Tasks
- Text generation, Fill mask
- Languages
- Afrikaans, Albanian, Amharic, Aragonese, Arabic, Egyptian Arabic, Assamese, Asturian and 4 more
- Size
- 1B–10B rows
- Downloads this month
- 18.9K
- Likes
- 712
- Access
- on request on Hugging Face
Models trained on it
- Tochka-AI/ruRoPEBert-e5-base-2kEmbeddings 18.6K
- stabilityai/stablelm-2-1_6bText generation 7.5K
- sapienzanlp/Minerva-3B-base-v1.0Text generation 5.8K
- sapienzanlp/Minerva-350M-base-v1.0Text generation 5.7K
- ltg/normistral-7b-warmText generation 4.8K
- mradermacher/TildeOpen-30B-Instruct-LV-i1-GGUF 4.8K
- TildeAI/TildeOpen-30bText generation 2.3K
- LazarusNLP/NusaBERT-baseFill mask 2K
How to load it
from datasets import load_dataset
ds = load_dataset("uonlp/CulturaX")