Dataset · m-a-p
Bilingual corpus of 4.69 trillion tokens
m-a-p/Matrix
English-Chinese open corpus for language model pretraining, listed as 4.69 trillion tokens in the card.
What's inside
Open dataset for pretraining in English and Chinese. According to the card, it contains 4,690 billion tokens and is gathered from multiple sources, including Common Crawl dumps. Metadata indicates a size of 1B-10B; format is text.
What you can do with it
- Pretraining a bilingual LLM
- Continued pretraining of an English-Chinese model
- Training or evaluating a tokenizer
- Tasks
- Text generation
- Languages
- English, Chinese
- Size
- 1B–10B rows
- Downloads this month
- 78.9K
- Likes
- 189
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("m-a-p/Matrix")