Skip to content

Dataset · allenai

Dolma 3 raw text pool with 9 trillion tokens

allenai/dolma3_pool

A massive unfiltered English corpus of web pages, research papers, and code for pre-training large language models.

What's inside

The dataset from the Allen Institute for AI contains over 9 trillion English tokens gathered from diverse web sources, academic publications, computer code, and other texts. This is the raw data pool of the Dolma 3 project prior to quality selection and mix proportioning (the final training mixture for Olmo 3 7B and 32B is released separately).

What you can do with it

  • Researching data cleaning, filtering, and deduplication pipelines for LLMs
  • Assembling custom training mixtures for language model pre-training
Tasks
Text generation
Languages
English
Size
10B–100B rows
Downloads this month
25K
Likes
42

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("allenai/dolma3_pool")