Dataset · jrast
Raw Web Corpus for Deduplication Benchmarks
jrast/web-corpus-text-raw
Nearly 5M HTML texts without deduplication to reproduce and verify LSH cleaning algorithms.
What's inside
A snapshot of a web corpus comprising 4,998,441 documents stored across 432 parquet files. The table contains only the document identifier, extracted HTML text, and a character count. LSH or content-based deduplication was intentionally omitted to serve as a fixed baseline for reproducing cleaning pipelines.
What you can do with it
- Testing and benchmarking text LSH deduplication algorithms
- Debugging web corpus filtering pipelines prior to LLM pre-training
- Size
- 1M–10M rows
- Downloads this month
- 24.4K
- Likes
- 0
How to load it
from datasets import load_dataset
ds = load_dataset("jrast/web-corpus-text-raw")