Skip to content

Dataset · opendatalab

AI-ready multilingual Common Crawl

opendatalab/AICC

Large multilingual corpus from Common Crawl, cleaned for training language models.

What's inside

AICC is a Common Crawl-based dataset prepared for AI training. The card specifically highlights the updated CC-MinerU-Code corpus: 4.58 million examples extracted from the full Common Crawl; each record contains the language, code language, and Markdown with code blocks. The dataset is multilingual, with metadata indicating a size of 1B to 10B examples.

What you can do with it

  • Pre-train a multilingual LLM on web texts
  • Train a code model on CC-MinerU-Code
  • Evaluate Common Crawl cleaning
Tasks
Text generation
Languages
Multilingual
Size
1B–10B rows
Downloads this month
9.2K
Likes
117

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("opendatalab/AICC")