Dataset · roneneldan
Short synthetic stories for LLM training
roneneldan/TinyStories
Synthetic short stories with a limited English vocabulary for training and evaluating language models.
What's inside
The dataset contains synthetic short stories generated by GPT-3.5 and GPT-4 using a restricted vocabulary. The texts are in English, ranging from 1 to 10 million examples. Main files include TinyStories-train.txt and tinystories-valid.txt for training and validation, alongside the tinystories_all_data.tar.gz archive. According to the card, it was used to train TinyStories-1M/3M/8M/28M/33M/1Layer-21M models.
What you can do with it
- Training a small language model or LoRA for story generation
- Evaluating generation quality and validation loss on a simple vocabulary
- Researching model training on synthetic data
- Tasks
- Text generation
- Languages
- English
- Size
- 1M–10M rows
- Downloads this month
- 216.4K
- Likes
- 1.3K
How to load it
from datasets import load_dataset
ds = load_dataset("roneneldan/TinyStories")