Skip to content

Dataset · jrast

Raw Web Corpus for Deduplication Benchmarks

jrast/web-corpus-text-raw

Nearly 5M HTML texts without deduplication to reproduce and verify LSH cleaning algorithms.

What's inside

A snapshot of a web corpus comprising 4,998,441 documents stored across 432 parquet files. The table contains only the document identifier, extracted HTML text, and a character count. LSH or content-based deduplication was intentionally omitted to serve as a fixed baseline for reproducing cleaning pipelines.

What you can do with it

  • Testing and benchmarking text LSH deduplication algorithms
  • Debugging web corpus filtering pipelines prior to LLM pre-training
Size
1M–10M rows
Downloads this month
24.4K
Likes
0

How to load it

from datasets import load_dataset

ds = load_dataset("jrast/web-corpus-text-raw")