Skip to content

Dataset · stanford-oval

Cleaned CommonCrawl world news archive

stanford-oval/ccnews

A massive multilingual corpus of 600 million news articles from 2016 to June 2024.

What's inside

Contains about 600 million cleaned and deduplicated news articles in over 100 languages from around the world, extracted from CCNews WARC files. The language for each article was automatically determined during processing using a DataTrove-like pipeline. Text and visual modalities are also specified in the metadata.

What you can do with it

  • Large-scale pre-training of multilingual language models
  • Training models for news classification and fact-based question answering
  • Analyzing media trends and public sentiment across different languages
Tasks
Text classification, Question answering, Text generation
Languages
Multilingual, Afrikaans, Amharic, Arabic, Assamese, Azerbaijani, Belarusian, Bulgarian and 4 more
Size
100M–1B rows
Downloads this month
8.1K
Likes
36

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("stanford-oval/ccnews")