Skip to content

Dataset · AdaMLLab

Cleaned Japanese corpus for LLM pre-training

AdaMLLab/JpnMix

A large Japanese corpus of 100M to 1B documents with quality filtering and MinHash deduplication.

What's inside

A Japanese language model pre-training corpus (for arXiv:2512.18834) compiled from five open Japanese datasets. The data underwent special quality filtering and cross-dataset deduplication, and the repository is divided into three subsets: quality_filtered, minhash_deduped, and matched (documents appearing in two or more sources).

What you can do with it

  • Pre-training and continual pre-training of LLMs in Japanese
  • Studying the impact of deduplication and quality filtering on language model training
Tasks
Text generation
Languages
Japanese
Size
100M–1B rows
Downloads this month
4K
Likes
5

How to load it

from datasets import load_dataset

ds = load_dataset("AdaMLLab/JpnMix")