Dataset · JeanKaddour
MiniPile: A compact Pile subset
JeanKaddour/minipile
6 GB of cleaned English text from The Pile for data-efficient model training.
What's inside
An English text corpus of about 6 GB. This is a subsample of deduplicated The Pile, filtered by embeddings: documents are clustered using k-means, and low-quality clusters are removed. Created to evaluate models under low-data conditions.
What you can do with it
- Training data-efficient language models
- Benchmarking sample efficiency
- Pre-training a small model on a clean Pile-like corpus
- Tasks
- Text generation, Fill mask
- Languages
- English
- Size
- 1M–10M rows
- Downloads this month
- 3.6K
- Likes
- 155
Models trained on it
- BEE-spoke-data/smol_llama-101M-GQAText generation 1.1K
- BEE-spoke-data/smol_llama-220M-GQAText generation 1K
- HelpingAI/HelpingAI2-6BText generation 917
- Locutusque/TinyMistral-248MText generation 863
- HelpingAI/HelpingAI2-9BText generation 856
- mradermacher/llama3-42b-v0-i1-GGUF 711
- bartowski/HelpingAI2-9B-GGUFText generation 698
- mradermacher/HelpingAI2-9B-i1-GGUF 695
How to load it
from datasets import load_dataset
ds = load_dataset("JeanKaddour/minipile")