Dataset · BramVanroy
Common Crawl web texts with Creative Commons license
BramVanroy/CommonCrawl-CreativeCommons
A massive corpus of web pages (100 million to 1 billion records) in six European languages with Creative Commons license markup.
What's inside
The dataset represents raw web page dumps from Common Crawl, annotated with Creative Commons license information from links in page tags or metadata. Volume ranges from 100 million to 1 billion text records in English, German, French, Italian, Afrikaans, and Frisian. The author warns that automated license detection may yield false positives.
What you can do with it
- Pre-training language models on legally clean data
- Filtering large web corpora by license type
- Training text generation models in European languages
- Tasks
- Text generation
- Languages
- Afrikaans, German, English, French, Western Frisian, Italian, Dutch, Spanish and 4 more
- Size
- 100M–1B rows
- Downloads this month
- 1.8K
- Likes
- 43
How to load it
from datasets import load_dataset
ds = load_dataset("BramVanroy/CommonCrawl-CreativeCommons")