EchoMoE-Qwen3-30B-A3B
an optimized language model for fast distributed inference
2 min readupdated
This is a variant of the MoE (Mixture of Experts, an approach where only a subset of parameters is activated for each token) architecture adapted with EchoMoE technology to cut inter-node data transfer. Instead of sending full hidden states, the model transmits a compressed delta from a reference state that refreshes every few layers. Built on Qwen3-30B-A3B and trained via distillation (transferring knowledge from a larger or more accurate model), it is built for engineers and researchers scaling and speeding up large language models across multi-GPU clusters.
Key features
- Cuts inter-node network traffic by 58% under expert parallelism
- Delivers up to a 22.6% throughput gain over the unmodified base model across 4 nodes
- EchoMoE architecture configuration: anchor checkpoints every 4 layers, a 512-dimensional dispatch code, and a 768-dimensional combine code
- 32,768-token context length with bf16 precision support
What you can do
- Deploy the model in vLLM with the EchoMoE plugin on a multi-GPU cluster for high-throughput serving
- Run local inference on a single GPU with at least 80 GB of VRAM using the transformers library via remote code
- Benchmark throughput and network overhead across different expert parallelism configurations
Example
An example of running the model in Python using the transformers library on a single bf16-capable GPU:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "McGill-NLP/EchoMoE-Qwen3-30B-A3B-s4-r512-c768"
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda")
tokenizer = AutoTokenizer.from_pretrained(repo)
messages = [{"role": "user", "content": "Give me a short introduction to large language models."}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Read more
DetailsBenchmarks (average across 18 tests): 63.1% (3.8 points below the base teacher model).
Hardware requirements: bf16 weights fit on a single 80 GB VRAM GPU for standalone inference; distributed mode requires a multi-H100 cluster.
License: Apache 2.0.










Comments