Skip to content

Dataset · ai4bharat

Sangraha: corpus of Indian languages

ai4bharat/sangraha

Largest cleaned corpus for pre-training across 22 Indian languages, 251 billion tokens.

What's inside

Sangraha is the largest high-quality cleaned corpus for pre-training in Indian languages. It contains 251 billion tokens across 22 languages, including Assamese, Bengali, Gujarati, English, Hindi, and Kannada. Data is sourced from curated resources, existing multilingual corpora, and large-scale translations; the card includes a paper and cleaning pipelines.

What you can do with it

  • For pre-training LLMs on Indian languages
  • For continued pretraining of a multilingual model
  • For evaluating cleaning quality and language coverage
Tasks
Text generation
Languages
Assamese, Bangla, Gujarati, English, Hindi, Kannada, Kashmiri, Malayalam and 4 more
Size
100M–1B rows
Downloads this month
81.5K
Likes
90

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("ai4bharat/sangraha")