Kestrel-34M-anneal-instruct
ultra-compact LLM with a recurrent Titans architecture
2 min readupdated
This model generates responses to user prompts in a conversational format with an extremely small memory footprint—just 34.3 million parameters. Created by lillycham, it is the supervised fine-tuning (SFT) stage of the Kestrel-34M-anneal-base model, calculating loss exclusively on assistant turns. Unlike standard transformers, the project relies on a custom Looped Titans-MAG architecture featuring a neural memory module and looped passes through its middle layers. It is ideal for researchers and hobbyists exploring unconventional ultra-small architectures and their training methods.
Key features
- Custom
KestrelForCausalLMarchitecture based on Looped Titans-MAG (1 input layer, 4 middle layers looped 3 times, and 1 output layer). - Built-in memory mechanism with a 256-token attention window, 64-token memory chunks, and 4 memory heads.
- 2,000-step instruction fine-tuning using Muon and AdamW optimizers, reaching a final val loss of 1.805.
- Compact 16,384-token vocabulary and a hidden state size of 512.
What you can do
- Test the behavior and generation quality of the Titans-MAG architecture at an ultra-small scale locally via
transformers. - Use it as a lightweight base checkpoint for custom fine-tuning experiments on narrow tasks.
Example
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "lillycham/Kestrel-34M-anneal-instruct"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)
prompt = tokenizer.apply_chat_template([{"role": "user", "content": "Hello!"}], tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt")
out = model.generate(inputs.input_ids, max_new_tokens=64, temperature=0.7, top_p=0.9)
Read more
DetailsParameters: 34,262,272 (~34.3M)
Metrics at step 2,000: train loss 1.706, val loss 1.805
Training: batch size 8 (grad_accum 4), sequence length 1024, bf16 precision on CUDA
Running the model requires
trust_remote_code=True









Comments