tiny-stories-1m
An ultra-lightweight 1M parameter language model
2 min read
The tiny-stories-1m model is a tiny generative AI model with just 1 million parameters (around 4 MB) that generates children's stories at the byte level. It was trained from scratch in just 2 hours on two CPU cores using the TinyStories V2 dataset, which consists of stories generated by GPT-4. The architecture is a byte-level GPT featuring a 256-character vocabulary, a width of 128, and 5 pre-LN blocks. The model is useful for research, experiments with CPU training, and analyzing how miniature language systems work.
Key features
- CPU training: Built and trained in just 2 hours on two CPU cores without using powerful graphics cards.
- Minimal size: Weighs only 4 MB and contains 1,018,624 parameters.
- Byte level: Works directly with text bytes using a 256-character vocabulary.
- Transparency: The repository provides fully open training scripts, evaluation files, and all recorded logs.
What you can do
- Run generation: Use the provided PyTorch script to generate short text snippets and stories right on a standard processor.
- Study the architecture: Explore the source code of the compact byte-level GPT (
gpt.py) to understand the fundamentals of building language models. - Benchmarking: Test the model on bits per byte metrics and compare its results with simple n-grams.
Example
import torch
from safetensors.torch import load_file
from gpt import GPT
m = GPT(128, 5, 4); m.load_state_dict(load_file("model.safetensors")); m.eval()
x = torch.tensor([list(b"Once upon a time")])
with torch.no_grad():
for _ in range(200):
nxt = torch.multinomial((m(x[:, -256:])[0, -1] / 0.8).softmax(-1), 1)
x = torch.cat([x, nxt[None]], 1)
print(bytes(x[0].tolist()).decode("utf-8", "replace"))Read more
DetailsBenchmarks: 1.1242 bits per byte on validation texts, 99.48% real words.
Hardware requirements: Standard Linux CPU, no graphics card required.
Limitations: 256-byte context, story plots lose coherence over time, characters can change.
Training details: AdamW optimizer, 6250 steps, cosine learning rate schedule.










Comments