Skip to content

Dataset · allenai

Training data mix for the Olmo 3 7B model

allenai/dolma3_mix-6T-1025-7B

A multi-billion English text corpus from AllenAI designed to replicate Olmo 3 7B training.

What's inside

A massive English text corpus of 1B to 10B records compiled by the AllenAI institute. The authors warn that this slice is strictly created for replicating the Olmo 3 7B model, with some scientific PDF texts from olmOCR removed and replaced with the [REMOVED] tag.

What you can do with it

  • Replicating experiments and pre-training of the Olmo 3 7B model
  • Studying the training data composition of large language models
  • Analyzing the impact of data filtering and removal on pre-training quality
Tasks
Text generation
Languages
English
Size
1B–10B rows
Downloads this month
83.6K
Likes
58

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("allenai/dolma3_mix-6T-1025-7B")