Dataset · tiiuae
English web corpus from CommonCrawl
tiiuae/falcon-refinedweb
A filtered and deduplicated English web corpus from CommonCrawl for training language models.
What's inside
This is a large English web text dataset collected by TII from CommonCrawl with strict filtering and large-scale deduplication. The card notes that models trained on RefinedWeb show performance on par with or better than those trained on curated data. The dataset is distributed under the ODC-By 1.0 license and includes links, making it convenient for multimodal scenarios as well.
What you can do with it
- Pre-training or fine-tuning English LLMs on web texts
- Comparing web data filtering quality
- Training a tokenizer or testing robustness against web noise
- Tasks
- Text generation
- Languages
- English
- Size
- 100M–1B rows
- Downloads this month
- 25.3K
- Likes
- 979
Models trained on it
- tiiuae/falcon-7bText generation 126.8K
- tiiuae/falcon-7b-instructText generation 55.8K
- chandar-lab/NeoBERTEmbeddings 36.7K
- openlm-research/open_llama_3b_v2Text generation 14.7K
- tiiuae/falcon-40b-instructText generation 12.2K
- tiiuae/falcon-40bText generation 11.8K
- stabilityai/stablelm-3b-4e1tText generation 11.4K
- vilsonrodrigues/falcon-7b-instruct-shardedText generation 11.1K
How to load it
from datasets import load_dataset
ds = load_dataset("tiiuae/falcon-refinedweb")