Dataset · ai4bharat
Parallel corpus for English and 11 Indian languages
ai4bharat/samanantar
49.6M sentence pairs for training translation systems between English and major Indian languages.
What's inside
Samanantar is a large collection of parallel texts between English and 11 Indian languages: Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi, Tamil, and Telugu. The corpus contains a total of 49.6 million sentence pairs. Data is presented in text format.
What you can do with it
- Training bilingual and multilingual machine translation models
- Aligning cross-lingual representations in embeddings
- Fine-tuning LLMs for Indian languages
- Tasks
- Text generation, Translation
- Languages
- English, Assamese, Bangla, Gujarati, Hindi, Kannada, Malayalam, Marathi and 4 more
- Size
- 10M–100M rows
- Downloads this month
- 2.5K
- Likes
- 49
Models trained on it
- toonist/AnuLM-Translate-400MText generation 418
- mradermacher/Misal-1B-base-v0.1-GGUF 383
- mradermacher/Misal-7B-base-v0.1-GGUF 311
- mradermacher/gujarati-finetune-llama3b-GGUF 176
- Varsha00/finetuned-opusmt-en-to-ta 29
- smallstepai/Misal-7B-base-v0.1Text generation 24
- smallstepai/Misal-1B-base-v0.1Text generation 20
- Varsha00/t5-small-en-to-ta 20
How to load it
from datasets import load_dataset
ds = load_dataset("ai4bharat/samanantar")