Skip to content

Dataset · JeanKaddour

MiniPile: A compact Pile subset

JeanKaddour/minipile

6 GB of cleaned English text from The Pile for data-efficient model training.

What's inside

An English text corpus of about 6 GB. This is a subsample of deduplicated The Pile, filtered by embeddings: documents are clustered using k-means, and low-quality clusters are removed. Created to evaluate models under low-data conditions.

What you can do with it

  • Training data-efficient language models
  • Benchmarking sample efficiency
  • Pre-training a small model on a clean Pile-like corpus
Tasks
Text generation, Fill mask
Languages
English
Size
1M–10M rows
Downloads this month
3.6K
Likes
155

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("JeanKaddour/minipile")