Momentum-121M
experimental hybrid language model with search
2 min readupdated
Momentum-121M is a small 121M parameter research language model (65M active per token) built on an unusual hybrid architecture. It combines recurrent Gated DeltaNet layers (state management), compressed attention MLA layers (Multi-head Latent Attention for precise retrieval), and a sparse mixture-of-experts (MoE) layer. The model was trained from scratch on FineWeb-Edu and underwent supervised fine-tuning (SFT) for a small chat format and tool calling via web_search(query) with source citations. It is intended for experiments and research rather than as a general-purpose assistant.
Key features
- Hybrid architecture: Combines 9 Gated DeltaNet layers, 3 MLA layers, and sparse MoE (48 experts + 1 shared).
- Tool support: Can call the
web_search(query)function and respond with[n]style citations. - Trained from scratch: Pre-trained on 1000M tokens of FineWeb-Edu, with a long-context warmup and an SFT stage.
- Compact size: Only 121 million parameters for local runs and studying new architectures.
What you can do
- Architecture testing: Run the model locally using custom code and explore how the hybrid Gated DeltaNet and MLA layers work.
- Search integration: Test the model's performance with an external search engine and citation generation based on provided text snippets.
Example
import json, torch
from safetensors.torch import load_file
from transformers import AutoTokenizer
from momentum import Cfg, MomentumLM
cfg = Cfg(**json.load(open("config.json")))
model = MomentumLM(cfg); model.load_state_dict(load_file("model.safetensors")); model.eval()
tok = AutoTokenizer.from_pretrained("gpt2")Read more
DetailsParameters: 121M (65M active)
Tokenizer: GPT-2 BPE (vocab 50257)
Limitations: English language only, no safety tuning, and context up to 1024 tokens
License: Code Apache-2.0, weights — other (research use)










Comments