Dataset · IFM
Math corpus for LLM pre-training
IFM/MegaMath
An open English mathematical dataset exceeding 300 billion tokens for pre-training language models.
What's inside
MegaMath is an open math pre-training corpus, part of TxT360 from LLM360. Gathered from various math-oriented sources, it contains over 300 billion tokens. One of the steps involved re-extracting mathematical documents from Common Crawl with HTML optimization, fasttext filtering, and deduplication in English.
What you can do with it
- Pre-training mathematical LLMs
- Continuing pre-training on math data
- Evaluating the impact of math corpora on reasoning
- Tasks
- Text generation
- Languages
- English
- Size
- 100M–1B rows
- Downloads this month
- 60.8K
- Likes
- 144
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("IFM/MegaMath")