Firstlight-124M
a language model built from scratch in PyTorch
2 min readupdated
This repository contains two versions of a compact 124M-parameter language model written in pure PyTorch using the GPT-2 architecture. The base model was trained from scratch on the FineWeb-EDU dataset (9.83 billion tokens), while the chat version underwent SFT (supervised fine-tuning) on the smol-smoltalk dataset for conversational use. Depending on the version, it either completes text or answers questions in a chat format. The project is a great fit for learning how pretraining and SFT work under the hood, or for running experiments with small models.
Key features
- Decoder-only transformer architecture (GPT-2): 12 layers, 12 attention heads, 768 embedding dimension, and a 1024-token context window.
- Includes both a base model for text completion and a chat model for Q&A.
- Pretrained on 9.83 billion tokens from FineWeb-EDU.
- Fine-tuned (SFT) on 403,748 conversations from the smol-smoltalk dataset.
What you can do
- Experiment with pretraining and SFT workflows on small language models.
- Use it as a starting point for preference tuning (DPO) or LoRA experiments.
- Compare the base and chat models side by side in an interactive Hugging Face Space.
Example
Here is how to load and run the model using the script from the repository's Space:
import shutil
from huggingface_hub import hf_hub_download
shutil.copy(hf_hub_download("adarsh-007/firstlight-chat", "model.py", repo_type="space"), "model.py")
from model import load_model, render_chat, stream
chat = load_model("chat")
ids = render_chat([{"role": "user", "content": "What is photosynthesis?"}])
for reply in stream(chat, ids, max_new_tokens=300, temperature=0.7, top_k=50):
pass
print(reply)
Read more
DetailsBenchmarks: HellaSwag (acc_norm) — 30.30% for the base model and 29.81% for the chat version (compared to 29.55% for OpenAI GPT-2 124M).
Limitations: The model tends to confidently hallucinate facts, mix up numbers, and repeat itself in longer outputs.
License: MIT.










Comments