Dataset · DKYoon
SlimPajama 6 billion token sample
DKYoon/SlimPajama-6B
An English corpus of approximately 6 billion tokens for training and testing language models.
What's inside
This is a 10% sample from the train/chunk1 portion of the cerebras/SlimPajama-627B dataset; in total, about 6 billion tokens and 5,489,000 rows. The validation and test parts are also sampled. After unpacking, the dataset takes about 24 GB, while the original corpus is over 2 TB.
What you can do with it
- Pre-training small language models
- LoRA fine-tuning for text generation
- Testing the pipeline on a reduced copy of SlimPajama
- Tasks
- Text generation
- Languages
- English
- Size
- 1M–10M rows
- Downloads this month
- 10.4K
- Likes
- 67
Models trained on it
- KvcHub/KaiLu3Text generation 140
- zongyueq/LLoPS-360MText generation 111
- zongyueq/LLoPS-1BText generation 109
- KvcHub/KaiLu-3-ProText generation 77
- brivangl/qwenar-0.6bEmbeddings 22
- pszemraj/sail-slimpajama-6B-32768-BPE-tokenizer 0
- Kirili4ik/ICAE-for-SWE-agents 0
- Abigail45/Llama-Smol-DeepSWE-Preview-NanBeige-ClaudeText generation 0
How to load it
from datasets import load_dataset
ds = load_dataset("DKYoon/SlimPajama-6B")