Dataset · Helsinki-NLP
Translated FineWeb-Edu Educational Corpus
Helsinki-NLP/fineweb-edu-translated
Over 36.7 million FineWeb-Edu documents automatically translated into 36 languages for training multilingual LLMs and translation systems.
What's inside
Contains educational web texts from the FineWeb-Edu corpus translated by OPUS-MT and HPLT-MT models. Version v1.0 contains 36,704,000 documents (over 28 billion English tokens and more than 960 billion total tokens) in 36 languages, including Bosnian, Bulgarian, Catalan, Czech, Danish, and German. Documents are aligned across all languages.
What you can do with it
- Pre-training multilingual language models
- Train and test machine translation systems
- Cross-lingual alignment of text representations
- Tasks
- Translation, Text generation
- Languages
- Bosnian, Bulgarian, Catalan, Czech, Danish, German, Greek, English and 4 more
- Size
- 1B–10B rows
- Downloads this month
- 184K
- Likes
- 28
How to load it
from datasets import load_dataset
ds = load_dataset("Helsinki-NLP/fineweb-edu-translated")