Skip to content

Dataset · Helsinki-NLP

Translated FineWeb-Edu Educational Corpus

Helsinki-NLP/fineweb-edu-translated

Over 36.7 million FineWeb-Edu documents automatically translated into 36 languages for training multilingual LLMs and translation systems.

What's inside

Contains educational web texts from the FineWeb-Edu corpus translated by OPUS-MT and HPLT-MT models. Version v1.0 contains 36,704,000 documents (over 28 billion English tokens and more than 960 billion total tokens) in 36 languages, including Bosnian, Bulgarian, Catalan, Czech, Danish, and German. Documents are aligned across all languages.

What you can do with it

  • Pre-training multilingual language models
  • Train and test machine translation systems
  • Cross-lingual alignment of text representations
Tasks
Translation, Text generation
Languages
Bosnian, Bulgarian, Catalan, Czech, Danish, German, Greek, English and 4 more
Size
1B–10B rows
Downloads this month
184K
Likes
28

How to load it

from datasets import load_dataset

ds = load_dataset("Helsinki-NLP/fineweb-edu-translated")