Skip to content

Dataset · DKYoon

SlimPajama 6 billion token sample

DKYoon/SlimPajama-6B

An English corpus of approximately 6 billion tokens for training and testing language models.

What's inside

This is a 10% sample from the train/chunk1 portion of the cerebras/SlimPajama-627B dataset; in total, about 6 billion tokens and 5,489,000 rows. The validation and test parts are also sampled. After unpacking, the dataset takes about 24 GB, while the original corpus is over 2 TB.

What you can do with it

  • Pre-training small language models
  • LoRA fine-tuning for text generation
  • Testing the pipeline on a reduced copy of SlimPajama
Tasks
Text generation
Languages
English
Size
1M–10M rows
Downloads this month
10.4K
Likes
67

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("DKYoon/SlimPajama-6B")