qwen2.5-3b-instruct-eagle3
speeding up Qwen2.5-3B generation in vLLM
2 min readupdated
A lightweight draft head for speculative decoding (a speedup method where a small model predicts upcoming tokens and the main model quickly verifies them) targeting Qwen/Qwen2.5-3B-Instruct. Built by ressf on the EAGLE-3 architecture, this 0.47B-parameter adapter uses a single decoder layer and a 32,000-token vocabulary. Unlike running the base model standalone, this add-on boosts token generation speed by up to 22% under light load while keeping output quality virtually untouched. It is a solid fit for developers deploying Qwen2.5-3B-Instruct on local or server GPUs with low concurrent traffic.
Key features
- +18% to +22% generation speedup across 1 to 16 concurrent clients (benchmarked on an RTX 5070 Ti 16 GB in FP16).
- Trained via the
speculatorslibrary on 9,618 UltraChat dialogues regenerated by Qwen2.5-3B-Instruct itself. - Mean acceptance length of 1.52 tokens, peaking when predicting 1 speculative token per step.
- Subsample accuracy stays close to baseline: 0.580 vs. 0.585 on both GSM8K and IFEval.
What you can do
- Speed up a local vLLM 0.29.0 inference server for chatbots or assistants handling low concurrency.
- Use it as a ready-made reference and baseline for speculative decoding experiments paired with
TRITON_ATTN.
Example
VLLM_USE_V2_MODEL_RUNNER=0 vllm serve Qwen/Qwen2.5-3B-Instruct --dtype half --attention-backend TRITON_ATTN \
--speculative-config '{"method":"eagle3","model":"","num_speculative_tokens":1,"attention_backend":"TRITON_ATTN"}'Read more
DetailsSpeed (output tok/s on RTX 5070 Ti 16 GB): +20% (1 client), +22% (2–8 clients), +18% (16), +11% (32), −4% (64).
Benchmarks (with head / without): GSM8K 0.580 / 0.585, IFEval 0.580 / 0.585, MMLU-Pro 134.4 / 136.7 out of 350.
Limitations: runs slower than the base model at high batch sizes (64 clients) and on the FP8-quantized model at 32–64 clients.
License: Qwen Research License (includes the base model's embedding matrix).










Comments