Dataset · mlfoundations
4T-token DCLM-baseline pretraining corpus in parquet
mlfoundations/dclm-baseline-1.0-parquet
Massive cleaned English text corpus (3 billion documents) in parquet format for LLM pretraining.
What's inside
A complete parquet-converted copy of the famous DCLM-baseline dataset. It contains approximately 3 billion English documents totaling 4 trillion tokens (between 1 billion and 10 billion records). The dataset is designed for basic pretraining of language models from scratch and delivers strong benchmark results in the 7B class.
What you can do with it
- Pretraining base language models from scratch
- Continued pretraining on high-quality English texts
- Languages
- English
- Size
- 1B–10B rows
- Downloads this month
- 24K
- Likes
- 57
Models trained on it
- bartowski/INTELLECT-1-Instruct-GGUFText generation 2.7K
- mradermacher/INTELLECT-1-i1-GGUF 1.7K
- mradermacher/Apex-2-Base-i1-GGUF 1.4K
- QuantFactory/INTELLECT-1-GGUFText generation 1.2K
- mradermacher/INTELLECT-1-Instruct-i1-GGUF 1.1K
- DALabCommunity/Haidass1.5-143M 923
- Manprot/MP-01-100M-PreviewText generation 916
- aethertp/PicoLM-V4-85MText generation 888
How to load it
from datasets import load_dataset
ds = load_dataset("mlfoundations/dclm-baseline-1.0-parquet")