Nantiro-150M-5.6B
Ultra-fast ternary language model for CPUs
2 min read
This compact 150M-parameter model uses ternary weights (where values are almost entirely −1, 0, or +1), allowing it to run significantly faster than alternatives directly on a central processing unit. The model was trained on just 5.6 billion text tokens and delivers a record-breaking 1,100–1,500 tokens per second on a standard Mac CPU (Apple M2–M4) without using a graphics card. Thanks to a custom engine and small file size (only 124 MB), it is ideal for experimenting with running local AI on laptops and low-end PCs.
Key features
- Ternary architecture: model weights consist almost entirely of −1, 0, and +1 values (1.58-bit approach).
- Extreme speed: delivers 1,100 to 1,500 tokens per second on an Apple Silicon CPU without GPU assistance.
- Tiny size: the weight file takes up only 124 MB, fitting into the RAM of any device.
- High energy efficiency: consumes minimal power during text generation on a CPU.
What you can do
- Run a local chat: deploy a fast language model on your Mac for interactive conversation and coding in the terminal.
- Run benchmarks: test generation speed and response accuracy on your own CPU using built-in scripts.
- Integrate into lightweight apps: use the model as a fast and efficient AI assistant on devices without powerful graphics cards.
Example
pip install -U huggingface_hub
hf download Nantiro/Nantiro-150M-5.6B --local-dir Nantiro-150M-5.6B
cd Nantiro-150M-5.6B
bash mac/chat.sh
Read more
DetailsParameters: 150M, ternary weights (−1, 0, +1)
Training: 5.6B text tokens + 113.5M chat fine-tuning tokens
Context: 1,024 tokens
Requirements: Mac with Apple Silicon processor (M1 and newer), macOS 13+ (launchers for NVIDIA, Intel, and AMD in development)
License: CC BY-NC 4.0 (non-commercial use only)










Comments