Dataset · domyn
Domyn Edge pre-training mix of 8.44 trillion tokens
domyn/Domyn-Edge-Pretraining-Mix-8.4T
Huge multilingual corpus of web texts, code, scientific papers, and mathematics for the first LLM pre-training stage.
What's inside
This is the first pre-training mix stage for the Domyn Edge Base model, containing 8.44 trillion tokens (1 to 10 billion records). It includes web texts, source code, scientific papers, mathematics, and multilingual data in English, Arabic, German, Spanish, French, and Indonesian.
What you can do with it
- Pre-training large and compact multilingual language models from scratch
- Continued pre-training to enhance skills in code, science, and mathematics
- Exploring the composition and filtering of ultra-large pre-training mixes
- Tasks
- Text generation
- Languages
- English, Arabic, German, Spanish, French, Indonesian, Italian, Japanese and 4 more
- Size
- 1B–10B rows
- Downloads this month
- 1.1K
- Likes
- 6
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("domyn/Domyn-Edge-Pretraining-Mix-8.4T")