Skip to content

Dataset · nvidia

Nemotron-Pretraining-Indic Corpus for Indian Languages

nvidia/Nemotron-Pretraining-Indic-Specialized-v1

A massive specialized text corpus from NVIDIA containing 100M to 1B records in Indian languages for language model pre-training.

What's inside

A specialized portion of NVIDIA's Nemotron-Pretraining-Indic corpus, including 100 million to 1 billion text and tabular records. Data is presented in multiple Indian languages, including Hindi (hi), Bengali (bn), Gujarati (gu), Assamese (as), Bodo (brx), and Dogri (doi). The set filters out only records from the Nemotron-Pretraining-Specialized-v1 and v1.1 repositories without overlapping with CC web crawls.

What you can do with it

  • Pre-training and language adaptation of LLMs for Indian languages
  • Improving text generation quality in specialized domains for Hindi and Bengali
  • Creating multilingual tokenizers and base models from NVIDIA
Tasks
Text generation
Languages
Assamese, Bangla, Bodo, Dogri, Gujarati, Hindi, Kannada, Konkani and 4 more
Size
100M–1B rows
Downloads this month
21.3K
Likes
7

How to load it

from datasets import load_dataset

ds = load_dataset("nvidia/Nemotron-Pretraining-Indic-Specialized-v1")