Dataset · OpenSQZ
2.46T STEM Tokens for Pre-training
OpenSQZ/AutoMathText-V2
A massive cleaned corpus on STEM, mathematics, code, and reasoning in English and Chinese.
What's inside
AutoMathText-V2 is a corpus of 2.46 trillion tokens of high-quality deduplicated text covering web content, mathematics, code, reasoning, and STEM. The languages are en and zh, size 1B<n<10B, with tabular and text formats. The data was gathered for model pre-training and fine-tuning.
What you can do with it
- Pre-train a STEM-oriented LLM
- Fine-tune a model on mathematics and code
- Evaluate a large-scale corpus for reasoning tasks
- Tasks
- Text generation, Question answering
- Languages
- English, Chinese
- Size
- 1B–10B rows
- Downloads this month
- 60.9K
- Likes
- 78
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("OpenSQZ/AutoMathText-V2")