Skip to content

Dataset · nvidia

Specialized pretraining data for Nemotron

nvidia/Nemotron-Pretraining-Specialized-v1

Updated corpus for Nemotron 3: math, code, English web, and synthetic data.

What's inside

Expansion of previously released Nemotron pretraining datasets: fresher, higher quality, and more diverse data across math, code, English Common Crawl, and large synthetic corpora. Created for the NVIDIA Nemotron 3 LLM family. The card mentions newly extracted code from Common Crawl and 2.5 trillion new English web tokens; metadata indicates a size of 10–100 million items.

What you can do with it

  • Pretraining or fine-tuning LLMs in the Nemotron 3 style
  • Adapting models on math and code
  • Experiments with web and synthetic data mixtures
Tasks
Text generation
Size
10M–100M rows
Downloads this month
13.9K
Likes
91

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("nvidia/Nemotron-Pretraining-Specialized-v1")