Skip to content

Dataset · nvidia

Synthetic corpus in 22 Indian languages from NVIDIA

nvidia/Nemotron-Pretraining-Indic-CC-v1

Huge text corpus of 1 to 10 billion records in 22 Indian languages for language model pre-training.

What's inside

A large-scale synthetic corpus from NVIDIA with a volume of 1 to 10 billion text records for pre-training language models in 22 Indian languages, including Assamese, Bengali, Bodo, Dogri, Gujarati, and Hindi. For twelve priority languages, the data is presented in both the original script and romanized Latin script, and for ten lower-resource languages, in the original script. The dataset is one of two parts of the Nemotron-Pretraining-Indic corpus.

What you can do with it

  • Pre-training and continued pre-training of LLMs to support Indian languages
  • Improving the understanding of transliterated Latin texts for Indian languages
Tasks
Text generation
Languages
Assamese, Bangla, Bodo, Dogri, Gujarati, Hindi, Kannada, Konkani and 4 more
Size
1B–10B rows
Downloads this month
5.5K
Likes
6
Access
on request on Hugging Face

How to load it

from datasets import load_dataset

ds = load_dataset("nvidia/Nemotron-Pretraining-Indic-CC-v1")