Skip to content

Dataset · wmt

WMT19 parallel texts for machine translation

wmt/wmt19

A large six-language multilingual corpus for training machine translation systems, containing 100M to 1B rows.

What's inside

The classic WMT19 competition dataset includes 100 million to 1 billion text records in English, German, French, Finnish, Czech, and Gujarati. The description explicitly notes that some Common Crawl data contains unaligned sentences and English text insertions in non-English files, which the organizers do not plan to fix.

What you can do with it

  • Training and benchmarking machine translation models between European languages and Gujarati
  • Pre-training multilingual language models (subject to additional noise filtering)
  • Testing cleaning and alignment algorithms for parallel corpora
Tasks
Translation
Languages
Czech, German, English, Finnish, French, Gujarati, Kazakh, Lithuanian and 2 more
Size
100M–1B rows
Downloads this month
7.2K
Likes
51

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("wmt/wmt19")