Rambley-V3-50M
an ultra-compact experimental language model
1 min readupdated
Created by glossia, Rambley-V3-50M is a 50-million-parameter language model (LLM) trained from scratch on roughly 36B tokens. The architecture packs in a range of cutting-edge techniques, including Virtual Head Attention, Sliding Window Attention (SWA), HoPE positional embeddings, and the Muon optimizer. Designed for research and experimentation with tiny model architectures, it brings together modern industry innovations at a miniature scale.
Key features
- Trained natively in BF16 for maximum stability.
- Uses modern Muon and AdamW optimizers paired with an AIR LR scheduler.
- Leverages Virtual Head Attention and headwise gated attention to keep parameter counts low.
- Incorporates Identity Hyperconnections and learned attention sinks.
What you can do
- Run the pre-training codebase to study custom architectural pipelines.
- Explore how ultra-small LLMs behave when equipped with advanced attention and optimization mechanisms.
Example
The model card does not include a ready-to-run inference script, though you can run the model with standard Hugging Face libraries alongside the author's pre-training repository.
Read more
DetailsParameters: ~50M (~49.7M)
Dataset: ~36B tokens
Context length: up to 2048 tokens
Key benchmarks: ARC-Challenge (acc 0.2116), Hellaswag (acc 0.3021), PIQA (acc 0.6496)










Comments