Dataset · ai4bharat
Sangraha: corpus of Indian languages
ai4bharat/sangraha
Largest cleaned corpus for pre-training across 22 Indian languages, 251 billion tokens.
What's inside
Sangraha is the largest high-quality cleaned corpus for pre-training in Indian languages. It contains 251 billion tokens across 22 languages, including Assamese, Bengali, Gujarati, English, Hindi, and Kannada. Data is sourced from curated resources, existing multilingual corpora, and large-scale translations; the card includes a paper and cleaning pipelines.
What you can do with it
- For pre-training LLMs on Indian languages
- For continued pretraining of a multilingual model
- For evaluating cleaning quality and language coverage
- Tasks
- Text generation
- Languages
- Assamese, Bangla, Gujarati, English, Hindi, Kannada, Kashmiri, Malayalam and 4 more
- Size
- 100M–1B rows
- Downloads this month
- 81.5K
- Likes
- 90
Models trained on it
- ai4bharat/Cadence-FastToken classification 3.2K
- udgamlabs/Vemana-BaseText generation 1.3K
- soketlabs/pragna-1bText generation 1K
- kushal-sathyanarayan/Sirigannada-2B-baseText generation 907
- mradermacher/Sirigannada-2B-base-GGUF 820
- mradermacher/shivneri-marathi-llm-7b-v0.1-i1-GGUF 720
- ai4bharat/CadenceToken classification 695
- mradermacher/hindi-llm-340m-instruct-GGUF 511
How to load it
from datasets import load_dataset
ds = load_dataset("ai4bharat/sangraha")