Skip to content

Dataset · epfml

Curated FineWeb texts for LLM pretraining

epfml/FineWeb-HQ

Top 10% highest-quality and structured web texts from the FineWeb corpus for language model training.

What's inside

Contains selected English web documents ranging from 1 billion to 10 billion records. The data represents the best 10% of the FineWeb corpus, filtered by an XLM-RoBERTa embedding-based neural classifier to find structured, knowledge-rich texts. The dataset is presented in tabular and text formats.

What you can do with it

  • Pretraining large language models from scratch
  • Continual pretraining of LLMs to improve knowledge quality
  • Researching data filtering methods for model training
Tasks
Text generation
Languages
English
Size
1B–10B rows
Downloads this month
368.1K
Likes
36

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("epfml/FineWeb-HQ")