Skip to content

Dataset · BramVanroy

Common Crawl web texts with Creative Commons license

BramVanroy/CommonCrawl-CreativeCommons

A massive corpus of web pages (100 million to 1 billion records) in six European languages with Creative Commons license markup.

What's inside

The dataset represents raw web page dumps from Common Crawl, annotated with Creative Commons license information from links in page tags or metadata. Volume ranges from 100 million to 1 billion text records in English, German, French, Italian, Afrikaans, and Frisian. The author warns that automated license detection may yield false positives.

What you can do with it

  • Pre-training language models on legally clean data
  • Filtering large web corpora by license type
  • Training text generation models in European languages
Tasks
Text generation
Languages
Afrikaans, German, English, French, Western Frisian, Italian, Dutch, Spanish and 4 more
Size
100M–1B rows
Downloads this month
1.8K
Likes
43

How to load it

from datasets import load_dataset

ds = load_dataset("BramVanroy/CommonCrawl-CreativeCommons")