ullm-base: A compact 360M-parameter base LLM trained from scratch
2 min read
A 360-million-parameter base language model (LLM) trained from scratch using a minimalist GPT implementation called ullm by author mattiasz. It features a classical decoder-only transformer architecture and is designed solely for text continuation in English, lacking conversational formatting or out-of-the-box instruction following. Unlike ready-to-use chatbots, this checkpoint serves as a transparent foundation for exploring how language models work internally and for subsequent supervised fine-tuning. The model is useful for developers and researchers who need lightweight, readable code and open weights for experimentation with training.
Key features
- 24-layer decoder-only architecture incorporating modern components: RMSNorm, RoPE positional encodings, and SwiGLU MLP activation.
- Trained from scratch on 7 billion tokens from the FineWeb-Edu educational dataset (
sample-10BT). - Utilizes the GPT-2 tokenizer (
tiktoken) with added special tokens<|user|>,<|assistant|>, and<|end|>(vocabulary size of 50,304). - Runs locally on NVIDIA GPUs (CUDA), Apple Silicon chips (MPS), or standard CPUs.
What you can do
- Use the model as a playground for custom supervised fine-tuning (SFT) to turn it into a specialized chatbot.
- Study the mechanics of modern transformers in practice by linking
safetensorsweights with the readable code of theullmrepository. - Solve basic English text generation or question-answering tasks via few-shot prompting (for instance, by providing multiple pairs of
Q: ... A: ...).
Example
git clone [https://github.com/mAzurkovic/ullm.git](https://github.com/mAzurkovic/ullm.git) && cd ullm && uv sync
hf download mattiasz/ullm-base --local-dir checkpoints/ullm-base
uv run ullm complete "The capital of France is" --model checkpoints/ullm-base
Read more
DetailsParameters: 359,842,816; context window: 1024 tokens; weights saved in fp32.
Training: 13,351 steps on 4 × NVIDIA H100 SXM GPUs using the AdamW optimizer.
Limitations: supports English only, prone to repetition, drifting off-topic, and confidently hallucinating facts; no safety alignment.
Important: weights load exclusively through
ullm; the transformers library is not supported. License: Apache-2.0.









Comments