Dataset · epfml
FineWeb2-HQ: High-quality corpus for pretraining
epfml/FineWeb2-HQ
A quality-filtered subset of FineWeb2 across 20 languages for faster language model pretraining.
What's inside
This is a text pretraining corpus selected from FineWeb2: for each language, the top 10% of documents were taken based on an XLM-RoBERTa embedding classifier score that identifies structured, knowledge-rich texts. According to the card, it covers 20 languages, with ru, zh, des, ja, es, and fr specified in the metadata. The size according to metadata ranges from 100 million to 1 billion.
What you can do with it
- Pretraining or continued pretraining of a small LLM in Russian and other languages
- LoRA fine-tuning for text generation
- Verifying corpus filtering quality
- Tasks
- Text generation
- Languages
- Russian, Chinese, German, Japanese, Spanish, French, Italian, Portuguese and 4 more
- Size
- 100M–1B rows
- Downloads this month
- 49.3K
- Likes
- 89
Models trained on it
- athenista/slm-tavenel-124mText generation 1.2K
- ufakai/ufakzeka-1-baseText generation 861
- mainlp/copy-first-translate-laterText generation 807
- Rinstitute/Rina-Juvenile-Exp-2B-GGUFText generation 547
- mradermacher/Villanova-2B-Base-2603-i1-GGUF 465
- Ertugrul1611/AKIL-2.0-107MText generation 369
- Dokitek/Dokichat-100M-InstructText generation 367
- mittwald/Kolibri-1-mixed-NVFP4-FP8Text generation 323
How to load it
from datasets import load_dataset
ds = load_dataset("epfml/FineWeb2-HQ")