Dataset · openbmb
290B-token math pre-training corpus
openbmb/UltraData-Math
290 billion tokens of English and Chinese math data across three tiers for LLM pre-training.
What's inside
A large-scale math dataset for LLM pre-training with a total volume exceeding 290B tokens. It is divided into three tiers: L1 contains 170.5 billion tokens of web corpus, L2 includes 33.7 billion tokens of curated high-quality data, and L3 comprises 88 billion tokens of cleaned multi-format data. Available in English and Chinese, these data were used in the math pre-training of the MiniCPM model series.
What you can do with it
- Continue pre-training LLMs on mathematical data
- Train math-reasoning models using L2 and L3 data
- Fine-tune models from the MiniCPM series
- Tasks
- Text generation
- Languages
- English, Chinese
- Size
- 100M–1B rows
- Downloads this month
- 34K
- Likes
- 365
Models trained on it
- openbmb/MiniCPM5-2BText generation 923.4K
- openbmb/MiniCPM5-2B-GGUFText generation 678.9K
- openbmb/MiniCPM5-2B-DSparkText generation 306.8K
- openbmb/MiniCPM5-1BText generation 227.2K
- openbmb/MiniCPM5-2B-MLXText generation 137.4K
- openbmb/MiniCPM5-1B-GGUFText generation 35.9K
- openbmb/MiniCPM5-1B-MLXText generation 29.8K
- openbmb/MiniCPM5-2B-BaseText generation 11K
How to load it
from datasets import load_dataset
ds = load_dataset("openbmb/UltraData-Math")