Skip to content

Dataset · tiiuae

English web corpus from CommonCrawl

tiiuae/falcon-refinedweb

A filtered and deduplicated English web corpus from CommonCrawl for training language models.

What's inside

This is a large English web text dataset collected by TII from CommonCrawl with strict filtering and large-scale deduplication. The card notes that models trained on RefinedWeb show performance on par with or better than those trained on curated data. The dataset is distributed under the ODC-By 1.0 license and includes links, making it convenient for multimodal scenarios as well.

What you can do with it

  • Pre-training or fine-tuning English LLMs on web texts
  • Comparing web data filtering quality
  • Training a tokenizer or testing robustness against web noise
Tasks
Text generation
Languages
English
Size
100M–1B rows
Downloads this month
25.3K
Likes
979

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("tiiuae/falcon-refinedweb")