Transformer-134M-BabiStories
A Baseline Model for Architecture Comparisons
2 min readupdated
Built to study and benchmark architectures for the Memory Mosaics v2 project, this compact research model was trained from scratch on the BabiStories children's stories dataset. Rather than relying on pretrained weights from large models like Llama or Mistral, it builds from the ground up while using the Mistral tokenizer. With 134 million parameters, it serves as a standard Transformer baseline to evaluate the new Memory Mosaics architecture, targeting NLP researchers who study the efficiency of different architectural approaches.
Key features
- Architecture: 12 layers with a width of 768, using RMSNorm, causal self-attention with RoPE, and SwiGLU FFN.
- Trained from scratch: 15,502 steps (127 million tokens total) using the AdamW optimizer in bf16 precision.
- Metrics: Achieves a final held-out validation loss of 1.7824 for direct comparison against its paired Memory Mosaics v2 model.
- Efficiency: Trained in 21 minutes on a single RTX 5090 GPU at roughly 102,883 tokens per second.
What you can do
- Run text generation: Use the loading script to test prompt continuations and generate stories.
- Compare architectures: Benchmark training speed, VRAM usage, and loss metrics directly against Memory Mosaics v2.
- Study small LLMs: Explore how compact language models behave when trained on specialized datasets.
Example
Run text generation using the script from the repository:
git clone [https://github.com/facebookresearch/MemoryMosaics](https://github.com/facebookresearch/MemoryMosaics)
git -C MemoryMosaics checkout c2ca0e88c5c89fc18a0a9465d6f7a1c3c8ea8faf
pip install torch safetensors tokenizers pyyaml huggingface_hub
wget [https://huggingface.co/Eureka-COM/Transformer-134M-BabiStories/resolve/main/load_and_generate.py](https://huggingface.co/Eureka-COM/Transformer-134M-BabiStories/resolve/main/load_and_generate.py)
python load_and_generate.py --upstream ./MemoryMosaics --repo Eureka-COM/Transformer-134M-BabiStories --prompt "Once upon a time, a little fox"Read more
DetailsParameters: 134M (134,105,856)
Context: 512 tokens, 32,000 vocabulary
VRAM usage: 7.4 GB peak (PyTorch, batch 16 × 512)
License: Apache-2.0










Comments