Dataset · opendatalab
AI-ready multilingual Common Crawl
opendatalab/AICC
Large multilingual corpus from Common Crawl, cleaned for training language models.
What's inside
AICC is a Common Crawl-based dataset prepared for AI training. The card specifically highlights the updated CC-MinerU-Code corpus: 4.58 million examples extracted from the full Common Crawl; each record contains the language, code language, and Markdown with code blocks. The dataset is multilingual, with metadata indicating a size of 1B to 10B examples.
What you can do with it
- Pre-train a multilingual LLM on web texts
- Train a code model on CC-MinerU-Code
- Evaluate Common Crawl cleaning
- Tasks
- Text generation
- Languages
- Multilingual
- Size
- 1B–10B rows
- Downloads this month
- 9.2K
- Likes
- 117
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("opendatalab/AICC")