Skip to content

Dataset · roneneldan

Short synthetic stories for LLM training

roneneldan/TinyStories

Synthetic short stories with a limited English vocabulary for training and evaluating language models.

What's inside

The dataset contains synthetic short stories generated by GPT-3.5 and GPT-4 using a restricted vocabulary. The texts are in English, ranging from 1 to 10 million examples. Main files include TinyStories-train.txt and tinystories-valid.txt for training and validation, alongside the tinystories_all_data.tar.gz archive. According to the card, it was used to train TinyStories-1M/3M/8M/28M/33M/1Layer-21M models.

What you can do with it

  • Training a small language model or LoRA for story generation
  • Evaluating generation quality and validation loss on a simple vocabulary
  • Researching model training on synthetic data
Tasks
Text generation
Languages
English
Size
1M–10M rows
Downloads this month
216.4K
Likes
1.3K

How to load it

from datasets import load_dataset

ds = load_dataset("roneneldan/TinyStories")