Skip to content

Dataset · PleIAs

Largest open text corpus

PleIAs/common_corpus

An open multilingual corpus of 2.27 trillion tokens spanning books, press, science, law, and code.

What's inside

Common Corpus is the largest openly licensed text dataset, featuring 2,267,302,720,836 tokens according to its card. It contains books, newspapers, scientific articles, government and legal documents, code, and other texts. Created by Pleias and partners, supported languages include English, French, German, Chinese, Italian, and Spanish.

What you can do with it

  • Pretraining a base language model
  • Fine-tuning LoRA on books and scientific texts
  • Evaluating a multilingual model on open data
Languages
English, French, German, Chinese, Italian, Spanish, Japanese, Polish and 4 more
Size
10K–100K rows
Downloads this month
201.7K
Likes
462

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("PleIAs/common_corpus")