Dataset · allenai
OLMo 2 pre-training set (November 2024)
allenai/olmo-mix-1124
Pre-training corpus for OLMo-2-1124: DCLM-Baseline, Arxiv, pes2o, StarCoder, and others.
What's inside
This is a dataset for pre-training OLMo-2-1124 models. The main part is DCLM-Baseline without additional filtering; Arxiv, pes2o, starcoder, and Algebraic-stack are also included. The card lists tokens, bytes, and the number of documents by component. The language is specified as English, with a size ranging from 1B to 10B.
What you can do with it
- Train or continue pre-training LLMs
- Reproduce OLMo 2 pre-training
- Study the composition of the open pre-training mix
- Tasks
- Text generation
- Languages
- English
- Size
- 1B–10B rows
- Downloads this month
- 16.6K
- Likes
- 93
Models trained on it
- allenai/OLMo-2-1124-7B 43K
- allenai/OLMo-2-1124-13B 10.1K
- marin-community/marin-8b-baseText generation 2.9K
- marin-community/marin-32b-baseText generation 1.8K
- HPLT/NorOLMo-13BText generation 1.2K
- mradermacher/Unbound-v1.12.0-27B-i1-GGUF 1.1K
- UW/OLMo2-8B-SuperBPE-t180kText generation 856
- mradermacher/Mira-v1.12.1-27B-i1-GGUF 846
How to load it
from datasets import load_dataset
ds = load_dataset("allenai/olmo-mix-1124")