Dataset · airtrain-ai
Cleaned FineWeb-Edu with embeddings
airtrain-ai/fineweb-edu-fortified
Large English FineWeb-Edu corpus after precise deduplication, featuring embeddings and row repetition counters.
What's inside
This is a derived dataset from FineWeb-Edu: precise deduplication has been applied across the corpus, and the occurrence count has been calculated for each row. Embeddings were generated using the TaylorAI/bge-micro model. The data contains text strings, tabular features, and a count column; scale is 100 million to 1 billion examples.
What you can do with it
- Fine-tune or pre-train English LLMs on a cleaned corpus
- Train text quality classifiers on embeddings
- Research deduplication and repetition in web corpora
- Tasks
- Text generation
- Languages
- English
- Size
- 100M–1B rows
- Downloads this month
- 9.2K
- Likes
- 71
Models trained on it
- TurkuNLP/finnish-modernbert-baseFill mask 821
- TurkuNLP/finnish-modernbert-tinyFill mask 394
- TurkuNLP/finnish-modernbert-large-shortFill mask 226
- TurkuNLP/finnish-modernbert-largeFill mask 162
- TurkuNLP/finnish-modernbert-tiny-eduFill mask 146
- TurkuNLP/finnish-modernbert-base-shortFill mask 144
- TurkuNLP/finnish-modernbert-tiny-short-eduFill mask 143
- TurkuNLP/finnish-modernbert-tiny-shortFill mask 130
How to load it
from datasets import load_dataset
ds = load_dataset("airtrain-ai/fineweb-edu-fortified")