Skip to content

Dataset · OptimalScale

400-billion token NVIDIA corpus for LLM pre-training

OptimalScale/ClimbMix

Cleaned English corpus of 400 billion tokens for efficient language model pre-training.

What's inside

The dataset is a massive English text corpus containing 400 billion tokens (metadata indicates 100 million to 1 billion rows), released by NVIDIA. The data is divided into 1,000 thematic groups and filtered through two classifiers to remove ads and junk.

What you can do with it

  • Pre-training language models from scratch with a limited token budget
  • Researching filtering and balancing algorithms for training mixtures
Tasks
Text generation
Languages
English
Size
100M–1B rows
Downloads this month
16.2K
Likes
39

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("OptimalScale/ClimbMix")