Skip to content

Dataset · mlfoundations

4T-token DCLM-baseline pretraining corpus in parquet

mlfoundations/dclm-baseline-1.0-parquet

Massive cleaned English text corpus (3 billion documents) in parquet format for LLM pretraining.

What's inside

A complete parquet-converted copy of the famous DCLM-baseline dataset. It contains approximately 3 billion English documents totaling 4 trillion tokens (between 1 billion and 10 billion records). The dataset is designed for basic pretraining of language models from scratch and delivers strong benchmark results in the 7B class.

What you can do with it

  • Pretraining base language models from scratch
  • Continued pretraining on high-quality English texts
Languages
English
Size
1B–10B rows
Downloads this month
24K
Likes
57

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("mlfoundations/dclm-baseline-1.0-parquet")