Skip to content

Dataset · ai4bharat

Parallel corpus for English and 11 Indian languages

ai4bharat/samanantar

49.6M sentence pairs for training translation systems between English and major Indian languages.

What's inside

Samanantar is a large collection of parallel texts between English and 11 Indian languages: Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi, Tamil, and Telugu. The corpus contains a total of 49.6 million sentence pairs. Data is presented in text format.

What you can do with it

  • Training bilingual and multilingual machine translation models
  • Aligning cross-lingual representations in embeddings
  • Fine-tuning LLMs for Indian languages
Tasks
Text generation, Translation
Languages
English, Assamese, Bangla, Gujarati, Hindi, Kannada, Malayalam, Marathi and 4 more
Size
10M–100M rows
Downloads this month
2.5K
Likes
49

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("ai4bharat/samanantar")