Skip to content

Dataset · MineAITechnology

20B-token corpus for MineAI pre-training

MineAITechnology/mine-20B-Tokens

Cleaned multilingual corpus (English, Urdu, Arabic, Hindi, code, and math) in parquet format totaling 19.34 billion tokens.

What's inside

The dataset for pre-training the MineAI 2B model includes 16.2 million documents (19.34 billion training and 19 million validation tokens) under the Apache-2.0 license. It contains cleaned texts in English, Urdu, Arabic, and Hindi, code in 9 languages, and math, organized into parquet files with quality scores and token counts.

What you can do with it

  • Pre-training compact language models from scratch
  • Language adaptation of LLMs to Urdu, Arabic, and Hindi while retaining coding and math skills
Tasks
Text generation
Languages
English, Urdu, Arabic, Hindi
Size
10M–100M rows
Downloads this month
711
Likes
7

How to load it

from datasets import load_dataset

ds = load_dataset("MineAITechnology/mine-20B-Tokens")