Skip to content

Dataset · epfml

FineWeb2-HQ: High-quality corpus for pretraining

epfml/FineWeb2-HQ

A quality-filtered subset of FineWeb2 across 20 languages for faster language model pretraining.

What's inside

This is a text pretraining corpus selected from FineWeb2: for each language, the top 10% of documents were taken based on an XLM-RoBERTa embedding classifier score that identifies structured, knowledge-rich texts. According to the card, it covers 20 languages, with ru, zh, des, ja, es, and fr specified in the metadata. The size according to metadata ranges from 100 million to 1 billion.

What you can do with it

  • Pretraining or continued pretraining of a small LLM in Russian and other languages
  • LoRA fine-tuning for text generation
  • Verifying corpus filtering quality
Tasks
Text generation
Languages
Russian, Chinese, German, Japanese, Spanish, French, Italian, Portuguese and 4 more
Size
100M–1B rows
Downloads this month
49.3K
Likes
89

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("epfml/FineWeb2-HQ")