Skip to content

Dataset · allenai

OLMo 2 pre-training set (November 2024)

allenai/olmo-mix-1124

Pre-training corpus for OLMo-2-1124: DCLM-Baseline, Arxiv, pes2o, StarCoder, and others.

What's inside

This is a dataset for pre-training OLMo-2-1124 models. The main part is DCLM-Baseline without additional filtering; Arxiv, pes2o, starcoder, and Algebraic-stack are also included. The card lists tokens, bytes, and the number of documents by component. The language is specified as English, with a size ranging from 1B to 10B.

What you can do with it

  • Train or continue pre-training LLMs
  • Reproduce OLMo 2 pre-training
  • Study the composition of the open pre-training mix
Tasks
Text generation
Languages
English
Size
1B–10B rows
Downloads this month
16.6K
Likes
93

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("allenai/olmo-mix-1124")