Skip to content

Dataset · IFM

Math corpus for LLM pre-training

IFM/MegaMath

An open English mathematical dataset exceeding 300 billion tokens for pre-training language models.

What's inside

MegaMath is an open math pre-training corpus, part of TxT360 from LLM360. Gathered from various math-oriented sources, it contains over 300 billion tokens. One of the steps involved re-extracting mathematical documents from Common Crawl with HTML optimization, fasttext filtering, and deduplication in English.

What you can do with it

  • Pre-training mathematical LLMs
  • Continuing pre-training on math data
  • Evaluating the impact of math corpora on reasoning
Tasks
Text generation
Languages
English
Size
100M–1B rows
Downloads this month
60.8K
Likes
144

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("IFM/MegaMath")