Skip to content

Dataset · allenai

Dolma 3 mixed pre-training dataset

allenai/dolma3_mix-150B-1025

150B-token training data sample of Dolma 3 from web, code, math, and scientific documents.

What's inside

A pre-training data sample including Common Crawl web pages (121B tokens), olmOCR scientific PDF documents (19.9B), Stack-Edu code (11.1B), arXiv articles, FineMath pages, and Wikipedia.

What you can do with it

  • Pre-training and fine-tuning large language models (LLMs)
  • Data quality research for generative models
Tasks
Text generation
Languages
English
Size
10M–100M rows
Downloads this month
9.2K
Likes
12

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("allenai/dolma3_mix-150B-1025")