Dataset · MineAITechnology
20B-token corpus for MineAI pre-training
MineAITechnology/mine-20B-Tokens
Cleaned multilingual corpus (English, Urdu, Arabic, Hindi, code, and math) in parquet format totaling 19.34 billion tokens.
What's inside
The dataset for pre-training the MineAI 2B model includes 16.2 million documents (19.34 billion training and 19 million validation tokens) under the Apache-2.0 license. It contains cleaned texts in English, Urdu, Arabic, and Hindi, code in 9 languages, and math, organized into parquet files with quality scores and token counts.
What you can do with it
- Pre-training compact language models from scratch
- Language adaptation of LLMs to Urdu, Arabic, and Hindi while retaining coding and math skills
- Tasks
- Text generation
- Languages
- English, Urdu, Arabic, Hindi
- Size
- 10M–100M rows
- Downloads this month
- 711
- Likes
- 7
How to load it
from datasets import load_dataset
ds = load_dataset("MineAITechnology/mine-20B-Tokens")