Skip to content

Dataset · epfml

Curated texts for 100-language pretraining

epfml/FineWeb2-HQ-Plus

Large corpus of the top 10% FineWeb2 documents across 100 languages for high-quality language model pretraining.

What's inside

An improved version of FineWeb2-HQ containing the top 10% highest quality web documents from FineWeb2 (v2.1.1) across 100 languages (metadata notes German, French, Polish, Portuguese, Spanish, and Italian). Selection was performed using an XLM-RoBERTa embedding-based quality classifier. The dataset size is between 100 million and 1 billion records.

What you can do with it

  • Pretraining and continued pretraining of multilingual LLMs
  • Train custom web text quality filters and classifiers
  • Compare model training efficiency on cleaned versus raw corpora
Tasks
Text generation
Languages
German, French, Polish, Portuguese, Spanish, Italian, Chinese, Dutch and 4 more
Size
100M–1B rows
Downloads this month
25.8K
Likes
1

How to load it

from datasets import load_dataset

ds = load_dataset("epfml/FineWeb2-HQ-Plus")