Dataset · wmt
WMT19 parallel texts for machine translation
wmt/wmt19
A large six-language multilingual corpus for training machine translation systems, containing 100M to 1B rows.
What's inside
The classic WMT19 competition dataset includes 100 million to 1 billion text records in English, German, French, Finnish, Czech, and Gujarati. The description explicitly notes that some Common Crawl data contains unaligned sentences and English text insertions in non-English files, which the organizers do not plan to fix.
What you can do with it
- Training and benchmarking machine translation models between European languages and Gujarati
- Pre-training multilingual language models (subject to additional noise filtering)
- Testing cleaning and alignment algorithms for parallel corpora
- Tasks
- Translation
- Languages
- Czech, German, English, Finnish, French, Gujarati, Kazakh, Lithuanian and 2 more
- Size
- 100M–1B rows
- Downloads this month
- 7.2K
- Likes
- 51
Models trained on it
- mradermacher/NanoTranslator-immersive_translate-0.5B-GGUF 195
- UNI-PIC/Retail-TW-Nemotron-3-Nano-4B-BaseText generation 29
- UNI-PIC/Retail-TW-Qwen3.5-4B-BaseText generation 24
- UNI-PIC/Retail-TW-Qwen3.5-0.8B-BaseText generation 23
- UNI-PIC/Retail-TW-Qwen3-4B-BaseText generation 21
- Varine/opus-mt-zh-en-modelText generation 14
- Mxode/NanoTranslator-immersive_translate-365MTranslation 14
- Mxode/NanoTranslator-immersive_translate-0.5BTranslation 10
How to load it
from datasets import load_dataset
ds = load_dataset("wmt/wmt19")