Petal-1-150M
a compact 152M-parameter language model trained from scratch
2 min readupdated
Petal-1-150M is an ultralight English LLM (large language model) by GautierP, trained completely from scratch. The root of the repository holds an SFT (supervised fine-tuning) version built as a compact assistant for short answers and basic arithmetic, while a pretrained base model sits in the base/ folder. On benchmarks, the base model beats Pythia-160m across most tasks, though it trails SmolLM2-135M. It is a solid fit for experimenting with local inference on low-end hardware, as well as training and fine-tuning compact architectures.
Key features
- Architecture with 18 layers, 768 hidden dimension, 12 attention heads, 2048 FFN, RoPE, and tied embeddings, totaling 152,596,224 parameters.
- Uses a digit-splitting BPE tokenizer with a 32,768-token vocabulary and a 2048-token context length.
- Includes two checkpoints in a single repository: the final conversational assistant and the pretrained base in the
base/directory. - Fully open-source training (Mini-LLM) and evaluation (Mini-Eval) code released under a permissive MIT license.
What you can do
- Run it locally via the
transformerslibrary to prototype simple chatbots with short conversation histories (keeping the system prompt and the last 4 turns is recommended). - Use the base checkpoint from
base/and the open Mini-LLM pipeline as a starting point for your own fine-tuning experiments on ultra-small models.
Example
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("GautierP/Petal-1-150M", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("GautierP/Petal-1-150M", trust_remote_code=True)
prompt = "System: You are Petal, a small AI assistant.\nUser: What is 2+2?\nAssistant:"
inputs = tok(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=64)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:]))
Read more
DetailsBenchmarks (base model, zero-shot): HellaSwag — 0.336, ARC-Easy — 0.479, PIQA — 0.628, SciQ — 0.685, Wikitext — 0.890 bits/byte.
Limitations: English only; due to its small size (150M), the model lacks encyclopedic knowledge, is prone to hallucinations, and handles only basic arithmetic.
Safety: The model has not undergone safety tuning, and refusal rates are minimal.










Comments