Skip to content

Dataset · airtrain-ai

Cleaned FineWeb-Edu with embeddings

airtrain-ai/fineweb-edu-fortified

Large English FineWeb-Edu corpus after precise deduplication, featuring embeddings and row repetition counters.

What's inside

This is a derived dataset from FineWeb-Edu: precise deduplication has been applied across the corpus, and the occurrence count has been calculated for each row. Embeddings were generated using the TaylorAI/bge-micro model. The data contains text strings, tabular features, and a count column; scale is 100 million to 1 billion examples.

What you can do with it

  • Fine-tune or pre-train English LLMs on a cleaned corpus
  • Train text quality classifiers on embeddings
  • Research deduplication and repetition in web corpora
Tasks
Text generation
Languages
English
Size
100M–1B rows
Downloads this month
9.2K
Likes
71

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("airtrain-ai/fineweb-edu-fortified")