Aurora2-400M-A20M: a compact open-source MoE model
2 min readupdated
Aurora2-400M-A20M is a decoder-only base language model (generating text from preceding tokens) with roughly 400M total parameters, of which only 19M are active per token. Built on a Mixture-of-Experts (MoE) architecture, it features 32 routed experts and 1 shared expert per layer, and was trained from scratch on 100B tokens using a single GPU. Because it is a raw base model without instruction-tuning or RLHF (reinforcement learning from human feedback), it is intended for research or further post-training.
Key features
- MoE architecture: 32 routed experts and 1 shared expert across each of the 12 layers.
- Tokenizer with a 16,384-token vocabulary, trained from scratch.
- Trained on a 100B-token dataset mix covering web text, math, and encyclopedias.
- Combines Muon and AdamW optimizers for efficient training.
What you can do
- Study the behavior of compact sparse MoE models and router dynamics.
- Use it as a lightweight base for fine-tuning on domain-specific tasks.
- Run inference on a consumer GPU with BF16 support via a custom Transformers implementation.
Example
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "AuroraAI-Research/Aurora2-400M-A20M"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
repo,
subfolder="fp16",
trust_remote_code=True,
torch_dtype=torch.float16,
device_map="auto",
).eval()
inputs = tokenizer("The Capital of France is", return_tensors="pt").to(model.device)
with torch.inference_mode():
outputs = model.generate(
**inputs,
max_new_tokens=128,
do_sample=True,
temperature=0.5,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Read more
DetailsBenchmarks: HellaSwag (acc_norm: 38.36), PIQA (acc_norm: 66.92), ARC-Challenge (acc_norm: 26.11).
Context length: 2,048 tokens.
Limitations: base model only, no conversational assistant capabilities, English-focused.
Hardware: single GPU with BF16 support (training took about 36 hours on an RTX Pro 6000).










Comments