Skip to content

Dataset · domyn

Domyn Edge pre-training mix of 8.44 trillion tokens

domyn/Domyn-Edge-Pretraining-Mix-8.4T

Huge multilingual corpus of web texts, code, scientific papers, and mathematics for the first LLM pre-training stage.

What's inside

This is the first pre-training mix stage for the Domyn Edge Base model, containing 8.44 trillion tokens (1 to 10 billion records). It includes web texts, source code, scientific papers, mathematics, and multilingual data in English, Arabic, German, Spanish, French, and Indonesian.

What you can do with it

  • Pre-training large and compact multilingual language models from scratch
  • Continued pre-training to enhance skills in code, science, and mathematics
  • Exploring the composition and filtering of ultra-large pre-training mixes
Tasks
Text generation
Languages
English, Arabic, German, Spanish, French, Indonesian, Italian, Japanese and 4 more
Size
1B–10B rows
Downloads this month
1.1K
Likes
6

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("domyn/Domyn-Edge-Pretraining-Mix-8.4T")