MiniGrimm
an ultra-compact 48M-parameter LLM with a deep architecture
2 min readupdated
MiniGrimm is an experimental language model weighing in at just 48.29M parameters (42.00M excluding embeddings), trained from scratch by Bayhakl1. Instead of a standard wide layout, it uses a "tall and narrow" 28-layer Llama-style architecture trained with the Muon optimizer. Despite its tiny footprint and 512-token context window, the model went through SFT (Supervised Fine-Tuning on dialogues and tasks) and can hold short English conversations and solve basic arithmetic step by step. It is useful for developers and researchers experimenting with ultra-lightweight architectures and running models locally on low-end hardware.
Key features
- Deep and narrow layout: 28 layers with a hidden dimension of
d_model = 384, incorporating GQA (Grouped-Query Attention), QK-Norm, RoPE, and SwiGLU. - Trained from scratch with Muon: Pretrained on ~12.55B tokens (FineWeb-Edu, Cosmopedia-v2, Wikipedia, math, and dialogues) using a combination of the Muon and AdamW optimizers.
- Custom tokenizer: A 16,384-token Byte-Level BPE vocabulary that splits numbers into single digits for more accurate arithmetic.
- Supervised fine-tuning: Fine-tuned on the GSM8K, SmolTalk, UltraChat-200k, Tülu-3, and No Robots datasets with loss masking applied strictly to assistant responses.
What you can do
- Run locally on a CPU: At under 50M parameters, the model runs effortlessly even without a GPU for quick tests of conversational pipelines.
- Test step-by-step arithmetic: Use greedy decoding (
do_sample=False) to evaluate its reasoning on simple word problems. - Study deep-and-thin networks: Use it as a baseline or reference when exploring unconventional layer proportions and the Muon optimizer in the
transformerslibrary.
Example
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "Bayhakl1/MiniGrimm"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)
prompt = "<|user|>If I have 3 apples and buy 5 more, then eat 2, how many apples do I have?<|end|><|assistant|>"
inputs = tokenizer(prompt, return_tensors="pt")
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=150)[0], skip_special_tokens=True))
Read more
DetailsBenchmarks: ArithMark 3.0 — 38.1% (Raw) / 38.2% (Norm), SciQ — 70.7%, PIQA — 61.7%, Winogrande — 52.4%, ARC-Easy — 48.6%, HellaSwag — 29.1%.
Specs: 48.29M total parameters (~42.00M non-embedding), strictly limited to a 512-token context window.
Limitations: English only, prone to factual hallucinations, and requires the
trust_remote_code=True flag when loading.









Comments