Dataset · PleIAs
Largest open text corpus
PleIAs/common_corpus
An open multilingual corpus of 2.27 trillion tokens spanning books, press, science, law, and code.
What's inside
Common Corpus is the largest openly licensed text dataset, featuring 2,267,302,720,836 tokens according to its card. It contains books, newspapers, scientific articles, government and legal documents, code, and other texts. Created by Pleias and partners, supported languages include English, French, German, Chinese, Italian, and Spanish.
What you can do with it
- Pretraining a base language model
- Fine-tuning LoRA on books and scientific texts
- Evaluating a multilingual model on open data
- Languages
- English, French, German, Chinese, Italian, Spanish, Japanese, Polish and 4 more
- Size
- 10K–100K rows
- Downloads this month
- 201.7K
- Likes
- 462
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("PleIAs/common_corpus")