tiny-dlms-rhoa-small
a compact model for intent classification
2 min read
The tiny-dlms-rhoa-small model is a tiny decision-making system that evaluates a list of options for a given situation and picks the best one. The architecture is written from scratch in PyTorch without using the standard transformers library and contains only 0.72M parameters. It is based on pre-training on 2 GB of web text followed by fine-tuning on four classification datasets. It uses RMSNorm, RoPE (positional encoding method), and SwiGLU (a type of neural network activation function). The project was created as an educational demonstration for developers interested in the inner workings of transformers.
Key features
- An architecture with 2 layers and a width of 64 with 4 attention heads, written entirely in PyTorch.
- A byte-level BPE tokenizer with a vocabulary of 8,192 tokens, trained on web texts.
- Two-stage training: masked pre-training on 2 GB of FineWeb-Edu data and fine-tuning on benchmarks.
- Lightweight: only 0.72M weights for fast CPU performance.
What you can do
- Integrate it into lightweight text classification or intent recognition pipelines.
- Run the local example from the repository to evaluate choice probabilities using the decide() method.
- Use the source code for educational purposes to understand the inner workings of the tokenizer and transformer.
Example
from tiny_dlm.data import Decision, collate
from tiny_dlm.hub import from_pretrained
model, tokenizer, config = from_pretrained("liamsi/tiny-dlms-rhoa-small")
context = "i lost my card yesterday , what should i do ?"
options = ["lost or stolen card", "change pin", "card arrival"]
idx, cand_mask, _ = collate([Decision(context, options, options[:1])], tokenizer)
choice, probs = model.decide(idx, cand_mask)
print(options[choice.item()], probs[0].tolist())
Read more
DetailsMetrics on held-out tests: bAbI — 81.4%, CLINC150 — 67.6%, Banking77 — 83.0%, CommonsenseQA — 17.6%.
Training: 14,000 steps on 479M tokens (FineWeb-Edu).
Hardware requirements: minimal thanks to the weight of 0.72M parameters.










Comments