Dataset · Viet-Mistral
Cleaned multilingual text corpus in 75 languages (15 TB)
Viet-Mistral/CulturaY
Massive web text corpus in 75 languages based on HPLT v1.1 for language model pre-training.
What's inside
A large-scale cleaned text dataset from the creators of CulturaX, containing 1 billion to 10 billion documents across 75 languages (including Belarusian, Bulgarian, Arabic, and others). The data was obtained by cleaning the HPLT v1.1 corpus and takes up 15 TB uncompressed or 3 TB compressed with zstd. The set was used to train the authors' own language models.
What you can do with it
- Pre-training and continued pre-training of large language models (LLMs) from scratch
- Train multilingual fill-mask models and embeddings
- Collect high-purity monolingual corpora for 75 different languages
- Tasks
- Text generation, Fill mask
- Languages
- Afrikaans, Arabic, Azerbaijani, Belarusian, Bulgarian, Bangla, Catalan, Czech and 4 more
- Size
- 1B–10B rows
- Downloads this month
- 9.3K
- Likes
- 41
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("Viet-Mistral/CulturaY")