Dataset · allenai
Dolma 3 raw text pool with 9 trillion tokens
allenai/dolma3_pool
A massive unfiltered English corpus of web pages, research papers, and code for pre-training large language models.
What's inside
The dataset from the Allen Institute for AI contains over 9 trillion English tokens gathered from diverse web sources, academic publications, computer code, and other texts. This is the raw data pool of the Dolma 3 project prior to quality selection and mix proportioning (the final training mixture for Olmo 3 7B and 32B is released separately).
What you can do with it
- Researching data cleaning, filtering, and deduplication pipelines for LLMs
- Assembling custom training mixtures for language model pre-training
- Tasks
- Text generation
- Languages
- English
- Size
- 10B–100B rows
- Downloads this month
- 25K
- Likes
- 42
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("allenai/dolma3_pool")