mini-llm-base
an open-source language model for text completion
2 min readupdated
This 124-million-parameter model was trained from scratch on 2.5 billion tokens of educational web text (Mini FineWeb-Edu and Cosmopedia). It is built for text continuation rather than question answering. Its architecture mirrors the GPT-2 small layout with four upgrades: RoPE (rotary position embeddings), SwiGLU (activation function), QK normalization, and no biases. It is a solid resource for developers and researchers learning how to build and train language models from scratch in pure PyTorch.
Key features
- Trained on 2.5B tokens of educational data
- Custom GPT-2-based architecture with RoPE and SwiGLU
- 512-token context window
- Weights released under the MIT license
What you can do
- Run inference using scripts from the GitHub repository
- Explore a pure PyTorch model training pipeline
- Test text generation quality with built-in presets
Example
from mini_llm import checkpoint
from mini_llm.inference import PRESETS, generate_stream
from mini_llm.tokenizer import EOT, get_tokenizer
model, info, device = checkpoint.load("mini-llm-base")
enc = get_tokenizer()
prompt = [EOT] + enc.encode_ordinary("The water cycle is the process by which")
s = dict(PRESETS["balanced"])
tokens = [ev["id"] for ev in generate_stream(model, prompt, device, 100, s.pop("temperature"), stop_token=EOT, sampling=s)]
print(enc.decode(tokens))Read more
DetailsBenchmarks: HellaSwag 32.3, LAMBADA 20.3, ARC-Easy 46.8
Hardware requirements: trained on Colab L4 and A100
Limitations: small model size; frequently generates inaccurate facts
Training details: 2 stages, 183,100 steps total










Comments