Kaze-4.7B-A530M-MIT: A compact Sparse MoE language model for local devices
2 min readupdated
Kaze-4.7B-A530M-MIT is an experimental foundational language model trained from scratch by developer Kineyo under the MIT license. The model uses a Sparse Mixture-of-Experts architecture, meaning that out of a total size of about 4.72B parameters, only about 528M parameters are activated per token. It is optimized for local and private work on edge devices (CPU, Linux, and Android), and also comes in a ready-to-use GGUF format for llama.cpp and Kofu_Mobile. The model focuses on research into MoE architectures, as well as Spanish-language tasks and code generation.
Key features
- Sparse MoE architecture: contains 1 shared expert and 31 routed experts (top_k = 2) with the SwiGLU activation function.
- Hardware efficiency: around 528M active parameters per token lower computing resource requirements during inference.
- Local execution: full support for running on CPU, Linux, and in the Kofu_Mobile mobile app thanks to pre-converted weights in GGUF format.
- Context and tokenization: supports a context length of up to 4096 tokens and uses a custom Byte-Level BPE vocabulary of 32,768 tokens.
What you can do
- CPU and terminal inference: run a quantized version of the model locally using llama.cpp for private text tasks.
- Mobile app integration: use the model in Android projects through the Proyecto Kofu ecosystem.
- Architecture research: study the training and operation features of compact Sparse MoE models built from scratch.
Example
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "shebxyd/Kaze-4.7B-A530M-MIT"
tok = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id, dtype=torch.bfloat16, device_map="auto")
prompt = "<|user|>\nHola, ¿quién eres y para qué sirves?\n<|assistant|>\n"
inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128, temperature=0.65, top_p=0.9, repetition_penalty=1.18, do_sample=True)
print(tok.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
Read more
DetailsParameters: ~4.72B total / ~528M active
Layers: 23 layers, d_model = 1024, GQA 16Q / 4KV
Limitations: experimental model trained on an MIT-licensed corpus, with an emphasis on Spanish and code
File formats: BF16 safetensors and Q4_K_M GGUF quantization










Comments