Dataset · nvidia
Nemotron-Pretraining-Indic Corpus for Indian Languages
nvidia/Nemotron-Pretraining-Indic-Specialized-v1
A massive specialized text corpus from NVIDIA containing 100M to 1B records in Indian languages for language model pre-training.
What's inside
A specialized portion of NVIDIA's Nemotron-Pretraining-Indic corpus, including 100 million to 1 billion text and tabular records. Data is presented in multiple Indian languages, including Hindi (hi), Bengali (bn), Gujarati (gu), Assamese (as), Bodo (brx), and Dogri (doi). The set filters out only records from the Nemotron-Pretraining-Specialized-v1 and v1.1 repositories without overlapping with CC web crawls.
What you can do with it
- Pre-training and language adaptation of LLMs for Indian languages
- Improving text generation quality in specialized domains for Hindi and Bengali
- Creating multilingual tokenizers and base models from NVIDIA
- Tasks
- Text generation
- Languages
- Assamese, Bangla, Bodo, Dogri, Gujarati, Hindi, Kannada, Konkani and 4 more
- Size
- 100M–1B rows
- Downloads this month
- 21.3K
- Likes
- 7
How to load it
from datasets import load_dataset
ds = load_dataset("nvidia/Nemotron-Pretraining-Indic-Specialized-v1")