Skip to content

Dataset · nvidia

High-quality corpus for LLM pretraining

nvidia/Nemotron-ClimbMix

An English dataset for LLM pretraining focused on high quality and efficient scaling.

What's inside

ClimbMix is a selection of high-quality data for pretraining language models. The card notes that continuous training of a 1B model on it yields a 2.0 percent gain relative to Llama-3.2-1B and better scaling compared to other data. Language is English, volume is 100M to 1B records, format is tabular.

What you can do with it

  • Pretraining LLMs around 1B size
  • Studying scaling laws on an English corpus
  • Comparing data selection strategies for pretraining
Tasks
Text generation
Languages
English
Size
100M–1B rows
Downloads this month
7.1K
Likes
152

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("nvidia/Nemotron-ClimbMix")