Skip to content

Dataset · m-a-p

Bilingual corpus of 4.69 trillion tokens

m-a-p/Matrix

English-Chinese open corpus for language model pretraining, listed as 4.69 trillion tokens in the card.

What's inside

Open dataset for pretraining in English and Chinese. According to the card, it contains 4,690 billion tokens and is gathered from multiple sources, including Common Crawl dumps. Metadata indicates a size of 1B-10B; format is text.

What you can do with it

  • Pretraining a bilingual LLM
  • Continued pretraining of an English-Chinese model
  • Training or evaluating a tokenizer
Tasks
Text generation
Languages
English, Chinese
Size
1B–10B rows
Downloads this month
78.9K
Likes
189

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("m-a-p/Matrix")