Dataset · allenai
OLMo2 Annealing Data Mix
allenai/dolmino-mix-1124
High-quality mix of web texts, articles, encyclopedias, and code for the second training stage of OLMo2.
What's inside
A DOLMino mix for the second stage (annealing) of OLMo2 training. It includes high-quality DCLM HQ web pages, Flan, Pes2o scientific articles, Wiki, StackExchange, and other sources, with sizes specified in tokens, bytes, and documents in the table. English language, provided in tabular and text formats.
What you can do with it
- Fine-tuning a large language model in the style of OLMo2
- Comparing high-quality corpus mixing recipes
- Pre-training baselines on web text and STEM
- Tasks
- Text generation
- Languages
- English
- Size
- 100M–1B rows
- Downloads this month
- 24.4K
- Likes
- 105
Models trained on it
- allenai/OLMo-2-1124-7B 43K
- allenai/OLMo-2-1124-13B 10.1K
- allenai/OLMoE-1B-7B-0125Text generation 5K
- marin-community/marin-8b-baseText generation 2.9K
- mradermacher/Marco-Mini-Global-Base-i1-GGUF 2.2K
- marin-community/marin-32b-baseText generation 1.8K
- mradermacher/Apex-2-Base-i1-GGUF 1.4K
- mradermacher/Marco-Nano-Base-i1-GGUF 1.3K
How to load it
from datasets import load_dataset
ds = load_dataset("allenai/dolmino-mix-1124")