Skip to content

Dataset · OpenSQZ

2.46T STEM Tokens for Pre-training

OpenSQZ/AutoMathText-V2

A massive cleaned corpus on STEM, mathematics, code, and reasoning in English and Chinese.

What's inside

AutoMathText-V2 is a corpus of 2.46 trillion tokens of high-quality deduplicated text covering web content, mathematics, code, reasoning, and STEM. The languages are en and zh, size 1B<n<10B, with tabular and text formats. The data was gathered for model pre-training and fine-tuning.

What you can do with it

  • Pre-train a STEM-oriented LLM
  • Fine-tune a model on mathematics and code
  • Evaluate a large-scale corpus for reasoning tasks
Tasks
Text generation, Question answering
Languages
English, Chinese
Size
1B–10B rows
Downloads this month
60.9K
Likes
78

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("OpenSQZ/AutoMathText-V2")