Meron Nano MoE 2.26M
an ultra-compact language model for edge devices
2 min readupdated
Meron Nano MoE 2.26M is a tiny base language model with just 2.26M parameters (only 910K active per token). Built on split dynamic Mixture-of-Experts (MoE) routing and recurrent parameter-sharing loops, it packs strong efficiency on math and science tasks, outperforming larger counterparts with fewer active parameters. The model is aimed at researchers as well as Edge AI and microcontroller developers.
Key features
- High efficiency: Scores 15.45 on the Open SLM Intelligence Index for models under 10M parameters.
- Dynamic MoE: Separates math and language compute streams to minimize interference.
- Recurrent reasoning: Runs 2 physical layers across 2 passes for an effective depth of 4 layers.
- Tiny footprint: Takes around 9 MB in FP32 and under 600 KB with 4-bit quantization.
What you can do
- Run on microcontrollers: Deploy on memory-constrained hardware like ARM Cortex-M or ESP32.
- Local math and logic: Use it as an offline engine for basic arithmetic and logic tasks without internet access.
- Architecture research: Use it as a benchmark when studying ultra-small language models and MoE setups.
Example
import torch
import torch.nn.functional as F
from safetensors.torch import load_file
from transformers import AutoTokenizer
# Load tokenizer and model weights
tokenizer = AutoTokenizer.from_pretrained("path/to/Meron-Nano-MoE")
weights = load_file("path/to/Meron-Nano-MoE/model.safetensors")
print(f"Loaded {len(weights)} weight tensors successfully.")
# Example arithmetic inference prompt
prompt = "Sam had 43 rubber bands. A friend gave Sam 19 more. Afterward, Sam had"
tokens = tokenizer.encode(prompt, return_tensors="pt")
print(f"Encoded prompt shape: {tokens.shape}")Read more
Details• Architecture: Dynamic MoE, 2 physical layers with recurrent loops
• Training dataset: 1.355B tokens (Nemotron-ClimbMix, Cosmopedia v2, scientific texts)
• Benchmarks: ArithMark-3.0 — 59.30%, PIQA — 58.49%, ARC-Challenge — 29.77%
• Limitations: base model without chat tuning, limited to English and narrow factual knowledge










Comments