Dataset · stanford-oval
Cleaned CommonCrawl world news archive
stanford-oval/ccnews
A massive multilingual corpus of 600 million news articles from 2016 to June 2024.
What's inside
Contains about 600 million cleaned and deduplicated news articles in over 100 languages from around the world, extracted from CCNews WARC files. The language for each article was automatically determined during processing using a DataTrove-like pipeline. Text and visual modalities are also specified in the metadata.
What you can do with it
- Large-scale pre-training of multilingual language models
- Training models for news classification and fact-based question answering
- Analyzing media trends and public sentiment across different languages
- Tasks
- Text classification, Question answering, Text generation
- Languages
- Multilingual, Afrikaans, Amharic, Arabic, Assamese, Azerbaijani, Belarusian, Bulgarian and 4 more
- Size
- 100M–1B rows
- Downloads this month
- 8.1K
- Likes
- 36
Models trained on it
- mradermacher/Cicikus-v3-1.4B-i1-GGUF 874
- mradermacher/Cicikus-v3-1.4B-Opus4.6-Powered-i1-GGUF 578
- mradermacher/Cicikus-v3-1.4B-GGUF 448
- mradermacher/Cicikus-v3-1.4B-Opus4.6-Powered-GGUF 157
- Kervyn-AI/Kervyn-Embeddings-100M-PreviewSentence similarity 79
- pthinc/Cicikus-v3-1.4BText generation 27
- pthinc/Cicikus-v3-1.4B-Opus4.6-PoweredText generation 23
How to load it
from datasets import load_dataset
ds = load_dataset("stanford-oval/ccnews")