Skip to content

Dataset · allenai

OLMo2 Annealing Data Mix

allenai/dolmino-mix-1124

High-quality mix of web texts, articles, encyclopedias, and code for the second training stage of OLMo2.

What's inside

A DOLMino mix for the second stage (annealing) of OLMo2 training. It includes high-quality DCLM HQ web pages, Flan, Pes2o scientific articles, Wiki, StackExchange, and other sources, with sizes specified in tokens, bytes, and documents in the table. English language, provided in tabular and text formats.

What you can do with it

  • Fine-tuning a large language model in the style of OLMo2
  • Comparing high-quality corpus mixing recipes
  • Pre-training baselines on web text and STEM
Tasks
Text generation
Languages
English
Size
100M–1B rows
Downloads this month
24.4K
Likes
105

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("allenai/dolmino-mix-1124")