Dataset · nvidia
Synthetic corpus in 22 Indian languages from NVIDIA
nvidia/Nemotron-Pretraining-Indic-CC-v1
Huge text corpus of 1 to 10 billion records in 22 Indian languages for language model pre-training.
What's inside
A large-scale synthetic corpus from NVIDIA with a volume of 1 to 10 billion text records for pre-training language models in 22 Indian languages, including Assamese, Bengali, Bodo, Dogri, Gujarati, and Hindi. For twelve priority languages, the data is presented in both the original script and romanized Latin script, and for ten lower-resource languages, in the original script. The dataset is one of two parts of the Nemotron-Pretraining-Indic corpus.
What you can do with it
- Pre-training and continued pre-training of LLMs to support Indian languages
- Improving the understanding of transliterated Latin texts for Indian languages
- Tasks
- Text generation
- Languages
- Assamese, Bangla, Bodo, Dogri, Gujarati, Hindi, Kannada, Konkani and 4 more
- Size
- 1B–10B rows
- Downloads this month
- 5.5K
- Likes
- 6
- Access
- on request on Hugging Face
How to load it
from datasets import load_dataset
ds = load_dataset("nvidia/Nemotron-Pretraining-Indic-CC-v1")