Dataset · nvidia
High-quality corpus for LLM pretraining
nvidia/Nemotron-ClimbMix
An English dataset for LLM pretraining focused on high quality and efficient scaling.
What's inside
ClimbMix is a selection of high-quality data for pretraining language models. The card notes that continuous training of a 1B model on it yields a 2.0 percent gain relative to Llama-3.2-1B and better scaling compared to other data. Language is English, volume is 100M to 1B records, format is tabular.
What you can do with it
- Pretraining LLMs around 1B size
- Studying scaling laws on an English corpus
- Comparing data selection strategies for pretraining
- Tasks
- Text generation
- Languages
- English
- Size
- 100M–1B rows
- Downloads this month
- 7.1K
- Likes
- 152
Models trained on it
- MinimaLabs/KLM-1.6Text generation 3.2K
- AxiomicLabs/GPT-X3-150MText generation 2.1K
- MinimaLabs/KLM-1.5Text generation 1.5K
- MinimaLabs/min-spark-1.7Text generation 723
- AxiomicLabs/GPT-S3-3MText generation 720
- MinimaLabs/min-spark-1.6Text generation 278
- qikp/kite-8.1-11m-baseText generation 114
- qikp/kite-9-16m-baseText generation 36
How to load it
from datasets import load_dataset
ds = load_dataset("nvidia/Nemotron-ClimbMix")