Dataset · HuggingFaceFW
Parallel texts in 500+ languages
HuggingFaceFW/finetranslations
Over 1 trillion tokens of parallel text in English and 500+ languages for machine translation.
What's inside
Parallel texts in English and 500+ languages, over 1 trillion tokens. Generated by translating FineWeb2 data into English using Gemma3 27B. The synthetic pipeline is deployed via datatrove with inference managed on VLLM.
What you can do with it
- Train machine translation models.
- Fine-tune multilingual models for low-resource languages.
- Evaluate translation across 500+ languages.
- Tasks
- Text generation, Translation
- Languages
- Abkhazian, abq, abs, acm, adh, adi, Adyghe, Tunisian Arabic and 4 more
- Size
- 1B–10B rows
- Downloads this month
- 27.3K
- Likes
- 310
Models trained on it
- polyglot-tagger/multilabel-language-identificationText classification 212
- quickmt/quickmt-uk-enTranslation 209
- freococo/MyanmarTranslator-210M-ProTranslation 175
- polyglot-tagger/language-identificationToken classification 139
- quickmt/quickmt-en-ukTranslation 59
- DerivedFunction/polyglot-tagger-v2.2Token classification 41
- quickmt/quickmt-is-enTranslation 39
- msmth/Source-1Text classification 36
How to load it
from datasets import load_dataset
ds = load_dataset("HuggingFaceFW/finetranslations")