Dataset · epfml
Curated texts for 100-language pretraining
epfml/FineWeb2-HQ-Plus
Large corpus of the top 10% FineWeb2 documents across 100 languages for high-quality language model pretraining.
What's inside
An improved version of FineWeb2-HQ containing the top 10% highest quality web documents from FineWeb2 (v2.1.1) across 100 languages (metadata notes German, French, Polish, Portuguese, Spanish, and Italian). Selection was performed using an XLM-RoBERTa embedding-based quality classifier. The dataset size is between 100 million and 1 billion records.
What you can do with it
- Pretraining and continued pretraining of multilingual LLMs
- Train custom web text quality filters and classifiers
- Compare model training efficiency on cleaned versus raw corpora
- Tasks
- Text generation
- Languages
- German, French, Polish, Portuguese, Spanish, Italian, Chinese, Dutch and 4 more
- Size
- 100M–1B rows
- Downloads this month
- 25.8K
- Likes
- 1
How to load it
from datasets import load_dataset
ds = load_dataset("epfml/FineWeb2-HQ-Plus")