Dataset · epfml
Curated FineWeb texts for LLM pretraining
epfml/FineWeb-HQ
Top 10% highest-quality and structured web texts from the FineWeb corpus for language model training.
What's inside
Contains selected English web documents ranging from 1 billion to 10 billion records. The data represents the best 10% of the FineWeb corpus, filtered by an XLM-RoBERTa embedding-based neural classifier to find structured, knowledge-rich texts. The dataset is presented in tabular and text formats.
What you can do with it
- Pretraining large language models from scratch
- Continual pretraining of LLMs to improve knowledge quality
- Researching data filtering methods for model training
- Tasks
- Text generation
- Languages
- English
- Size
- 1B–10B rows
- Downloads this month
- 368.1K
- Likes
- 36
Models trained on it
- fromziro/ZeroS-Qana-5MText generation 1.1K
- CodeSoft/sorbet-v2-25mText generation 849
- mainlp/copy-first-translate-laterText generation 807
- AxiomicLabs/GPT-S-5MText generation 716
- StentorLabs/Portimbria-150MText generation 560
- AxiomicLabs/GPT-S-1.4MText generation 317
- sz14/cRia-LM-75MText generation 193
- Flexan/StentorLabs-Portimbria-150M-GGUFText generation 173
How to load it
from datasets import load_dataset
ds = load_dataset("epfml/FineWeb-HQ")