Qwen3-8B-5layer-MoE
a compact MoE drafter for faster generation
2 min read
This model is a specialized drafter for speculative decoding (a method for speeding up large language model generation), built on top of Qwen3-8B using SpecForge (DSpark). Its architecture features just 5 target layers, where each FFN (feed-forward network layer) is converted into a Mixture of Experts with 8 experts of width 24576 and two active experts per token. It has a total of 12.45B parameters with 3.39B active. The project is tailored for researchers and engineers working on LLM (large language model) inference speed optimization.
Key features
- Architecture specifications: 5 target layers (1/9/17/25/33) with a projector and Markov head.
- MoE structure: 8 experts without a shared expert, top-2 routing.
- Parameters: 12.45B total and 3.39B active parameters per token.
- Training: 1 epoch on the perfectblend dataset with an 8192 context and the qwen3-instruct template.
What you can do
- Inference speedup: Use the model as a drafter for speculative decoding of Qwen3-8B.
- Pipeline optimization: Deploy the model using a special inference library fork for MoE drafters.
- Research: Study the behavior of 5-layer MoE architectures in text generation tasks.
Example
Running and serving the model requires a special package from the SpecForge repository fork (dev-moe-serving branch) with support for Qwen3MoeDSparkModel.
Read more
DetailsBase model: Qwen/Qwen3-8B
Architecture: 5 layers, MoE 8x24576, top-2
Training: 1 epoch, lr 6e-4, 9867 steps on GB300 nodes
Requirements: dev-moe-serving package for inference










Comments