Hedwig 97.6M: A Compact Model for Stories and Grammar Editing
3 min readupdated
Hedwig 97.6M is a miniature 97.6M-parameter ChatGPT-style language model (LLM) built and trained from scratch using pure PyTorch, without high-level frameworks like HF Trainer or Lightning. It can write short English fairy tales and correct grammar and tone in casual text. Based on a modern Transformer architecture (12 layers, RoPE, RMSNorm, SwiGLU, SDPA) with a custom 8,192-token BPE tokenizer, it goes through pre-training, SFT (instruction fine-tuning), and Anchored DPO (preference alignment). Thanks to its small footprint and quantized versions (8-bit and 4-bit weight compression), the model runs completely offline even on laptops—achieving 54 tokens per second on an Apple M4 chip.
Key features
- Built from scratch in PyTorch: The entire pipeline—from the tokenizer and pre-training on 159M tokens (TinyStories, FineWeb-Edu, and Project Gutenberg classics) to DPO and quantization—is implemented without third-party wrappers.
- Extreme compactness: In the base
bfloat16format, the checkpoint weighs 195.6 MB, while theint8andint4quantized versions take up just 98.2 MB and 55.2 MB, respectively. - Two primary modes: The model is fine-tuned on 30,930 instruction pairs from the TinyStories-Instruct and CoEdIT datasets for story generation and basic editing tasks.
- Fast local inference: Optimized for offline execution on consumer hardware, including Apple Silicon Macs.
What you can do
- Local story generator: Embed the checkpoint into an offline application or script to generate simple English stories without internet access.
- Basic short-text correction: Use it for quick grammar and style edits in short, everyday messages.
- LLM architecture study: Explore the open-source training code in pure PyTorch and visualize layer activity using the included Brain X-Ray UI.
Example
Prompt: Write a story about a boy named Harry Potter.
Model response (54 tok/s offline on Apple M4): Harry Potter was a humble boy who liked to play. One day, he decided to go out and play. He walked down the street, laughing and having so much fun. Suddenly, Harry saw something shiny in a store window...
Read more
DetailsMetrics: The final
brain_dpo.pt checkpoint showed a Val/Test Loss of 1.235 and a 95.0% saved win-rate during the DPO stage (peaking at 96.7%); for the int4 version, the loss was 1.4118.Training: Took about 66 minutes on a single NVIDIA A40 GPU (48 GB) and cost roughly $0.65.
Limitations: With 97.6M parameters, this is an educational model for stories rather than a general-purpose assistant; in long work emails, it may cut off text or miss tense agreement.
Additional info: The release also introduces a larger
hedwig-300m variant, and the source code is available in the author's GitHub repository.









Comments