Atom Zero 49M: A miniature language assistant focused on local RAG
2 min read
Atom Zero 49.2M is a compact 49.2-million parameter decoder-only transformer model built to run on modest hardware like an RTX 3050 (6 GB). Rather than trying to replace giant general-purpose models or relying on broad native knowledge, it compensates with a built-in RAG (Retrieval-Augmented Generation — searching for relevant documents to answer questions) system and deterministic tools. The project is tailored for building a private local assistant to handle texts, notes, and simple tasks without telemetry or cloud dependencies.
Key features
- Decoder-only transformer architecture with 49.2 million parameters and a 128-token context
- Built-in hybrid RAG (SQLite FTS5 + MiniLM vector search) for precise fact retrieval
- A set of local tools for handling time, arithmetic, and task lists
- Complete local privacy with zero telemetry, running on CPU or CUDA
What you can do
- Deploy a local assistant for daily notes and document management on low-spec hardware
- Set up search across personal text files and URLs with source citations
- Connect local tools for safe computations and task management
- Use an optional provider for seamless integration with larger models when needed
Example
Running model verification and benchmarks from the command line:
python research/benchmark_raw.py \
--model checkpoints/transformer_behavior_50m_scaled \
--out research/results/raw_knowledge_scaled_40k.jsonRead more
DetailsBenchmarks: validation chat bpb dropped to 1.3815 (an improvement over the base 1.5032)
Hardware requirements: PyTorch, CUDA or CPU, tested on an RTX 3050 (6 GB)
Limitations: 128-token context, English language, weak general reasoning without RAG
Training details: 6 layers, 768 hidden dimension, 12 attention heads, trained on 175M clean tokens plus a mix










Comments