Dataset · allenai
Dolma 3 mixed pre-training dataset
allenai/dolma3_mix-150B-1025
150B-token training data sample of Dolma 3 from web, code, math, and scientific documents.
What's inside
A pre-training data sample including Common Crawl web pages (121B tokens), olmOCR scientific PDF documents (19.9B), Stack-Edu code (11.1B), arXiv articles, FineMath pages, and Wikipedia.
What you can do with it
- Pre-training and fine-tuning large language models (LLMs)
- Data quality research for generative models
- Tasks
- Text generation
- Languages
- English
- Size
- 10M–100M rows
- Downloads this month
- 9.2K
- Likes
- 12
Models trained on it
- asciibase64/Llama-3.2-1B-dolma3_mixText generation 458
- Nat1an/cerulean-lab5-distill128-3b-j559819Text generation 413
- Nat1an/cerulean-lab5-distill128-3b-j561140Text generation 47
- Nat1an/cerulean-lab5-muon-pilot-j511584Text generation 18
- Nat1an/cerulean-lab5-muon-3b-j539809Text generation 17
- Nat1an/cerulean-lab5-distill3b-teacher-3b-j586139Text generation 16
How to load it
from datasets import load_dataset
ds = load_dataset("allenai/dolma3_mix-150B-1025")