Dataset · allenai
Training data mix for the Olmo 3 7B model
allenai/dolma3_mix-6T-1025-7B
A multi-billion English text corpus from AllenAI designed to replicate Olmo 3 7B training.
What's inside
A massive English text corpus of 1B to 10B records compiled by the AllenAI institute. The authors warn that this slice is strictly created for replicating the Olmo 3 7B model, with some scientific PDF texts from olmOCR removed and replaced with the [REMOVED] tag.
What you can do with it
- Replicating experiments and pre-training of the Olmo 3 7B model
- Studying the training data composition of large language models
- Analyzing the impact of data filtering and removal on pre-training quality
- Tasks
- Text generation
- Languages
- English
- Size
- 1B–10B rows
- Downloads this month
- 83.6K
- Likes
- 58
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("allenai/dolma3_mix-6T-1025-7B")