Skip to content

Dataset · openbmb

290B-token math pre-training corpus

openbmb/UltraData-Math

290 billion tokens of English and Chinese math data across three tiers for LLM pre-training.

What's inside

A large-scale math dataset for LLM pre-training with a total volume exceeding 290B tokens. It is divided into three tiers: L1 contains 170.5 billion tokens of web corpus, L2 includes 33.7 billion tokens of curated high-quality data, and L3 comprises 88 billion tokens of cleaned multi-format data. Available in English and Chinese, these data were used in the math pre-training of the MiniCPM model series.

What you can do with it

  • Continue pre-training LLMs on mathematical data
  • Train math-reasoning models using L2 and L3 data
  • Fine-tune models from the MiniCPM series
Tasks
Text generation
Languages
English, Chinese
Size
100M–1B rows
Downloads this month
34K
Likes
365

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("openbmb/UltraData-Math")