Skip to content

Dataset · nvidia

Nemotron code and math pre-training corpus

nvidia/Nemotron-Pretraining-Code-v2

NVIDIA's updated pre-training corpus with code, math, English web text, and synthetics.

What's inside

Nemotron-Pre-Training-Dataset-v2.1 expands NVIDIA's previous pre-training sets with higher quality and more diverse data on mathematics, code, English Common Crawl, and large synthetic corpora. The card mentions new code extraction from Common Crawl and 2.5T new English web data. The dataset was created for the NVIDIA Nemotron 3 model family; volume ranges from 100 million to 1 billion records.

What you can do with it

  • Pre-train or continue pre-training code-LLMs
  • Fine-tune the model on math and code
  • Evaluate code generation quality on the NVIDIA corpus
Tasks
Text generation
Size
100M–1B rows
Downloads this month
5.5K
Likes
139
Access
on request on Hugging Face

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("nvidia/Nemotron-Pretraining-Code-v2")