Dataset · OptimalScale
400-billion token NVIDIA corpus for LLM pre-training
OptimalScale/ClimbMix
Cleaned English corpus of 400 billion tokens for efficient language model pre-training.
What's inside
The dataset is a massive English text corpus containing 400 billion tokens (metadata indicates 100 million to 1 billion rows), released by NVIDIA. The data is divided into 1,000 thematic groups and filtered through two classifiers to remove ads and junk.
What you can do with it
- Pre-training language models from scratch with a limited token budget
- Researching filtering and balancing algorithms for training mixtures
- Tasks
- Text generation
- Languages
- English
- Size
- 100M–1B rows
- Downloads this month
- 16.2K
- Likes
- 39
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("OptimalScale/ClimbMix")