Skip to content

Dataset · Viet-Mistral

Cleaned multilingual text corpus in 75 languages (15 TB)

Viet-Mistral/CulturaY

Massive web text corpus in 75 languages based on HPLT v1.1 for language model pre-training.

What's inside

A large-scale cleaned text dataset from the creators of CulturaX, containing 1 billion to 10 billion documents across 75 languages (including Belarusian, Bulgarian, Arabic, and others). The data was obtained by cleaning the HPLT v1.1 corpus and takes up 15 TB uncompressed or 3 TB compressed with zstd. The set was used to train the authors' own language models.

What you can do with it

  • Pre-training and continued pre-training of large language models (LLMs) from scratch
  • Train multilingual fill-mask models and embeddings
  • Collect high-purity monolingual corpora for 75 different languages
Tasks
Text generation, Fill mask
Languages
Afrikaans, Arabic, Azerbaijani, Belarusian, Bulgarian, Bangla, Catalan, Czech and 4 more
Size
1B–10B rows
Downloads this month
9.3K
Likes
41

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("Viet-Mistral/CulturaY")