Dataset · AdaMLLab
Cleaned Japanese corpus for LLM pre-training
AdaMLLab/JpnMix
A large Japanese corpus of 100M to 1B documents with quality filtering and MinHash deduplication.
What's inside
A Japanese language model pre-training corpus (for arXiv:2512.18834) compiled from five open Japanese datasets. The data underwent special quality filtering and cross-dataset deduplication, and the repository is divided into three subsets: quality_filtered, minhash_deduped, and matched (documents appearing in two or more sources).
What you can do with it
- Pre-training and continual pre-training of LLMs in Japanese
- Studying the impact of deduplication and quality filtering on language model training
- Tasks
- Text generation
- Languages
- Japanese
- Size
- 100M–1B rows
- Downloads this month
- 4K
- Likes
- 5
How to load it
from datasets import load_dataset
ds = load_dataset("AdaMLLab/JpnMix")