Mistral-7B-Instruct-v0.3 EAGLE3 draft
a draft checkpoint for speculative decoding
2 min readupdated
This is a specialized EAGLE-3 draft model built to speed up text generation for mistralai/Mistral-7B-Instruct-v0.3. It uses speculative decoding—where a lightweight auxiliary model quickly drafts several tokens ahead for the main model to verify in a single pass—boosting throughput without sacrificing quality or altering the output distribution. Trained on AWS Trainium2 across the UltraChat, Magicoder, and GSM8K datasets, the checkpoint has 293.6M trainable parameters and is optimized for the speculators format. It is a solid drop-in option for developers and engineers running Mistral-7B services who want lower inference latency.
Key features
- EAGLE-3 architecture: speculative decoding that verifies every drafted token against the target model while preserving its exact output distribution.
- Faster inference: cuts per-token latency by 1.5–2.1x under low-to-moderate server load.
- Curated training data: trained on 61.2M tokens generated directly by Mistral-7B-Instruct-v0.3 using AWS Trainium2.
- vLLM compatibility: works out of the box with standard speculative decoding libraries.
What you can do
- Speed up local or cloud deployments: plug the checkpoint into a vLLM pipeline alongside
mistralai/Mistral-7B-Instruct-v0.3to cut response latency. - Benchmark on coding and math: maximize token acceptance lengths on programming tasks (Magicoder) and multi-step reasoning (GSM8K).
- Deploy on AWS Trainium2: run the model using the vllm-neuron framework with tensor parallelism.
Example
Running speculative decoding with vLLM:
vllm serve mistralai/Mistral-7B-Instruct-v0.3 \
--speculative-config '{"method": "eagle3", "model": "", "num_speculative_tokens": 3}'Read more
DetailsBenchmarks: mean acceptance length reaches 2.70 on UltraChat, 3.25 on Magicoder, and 3.16 on GSM8K (at k=3).
Hardware: trained on a single trn2.3xlarge node (AWS Trainium2).
Limitations: reported speedups apply to greedy decoding; efficiency drops when sampling with temperature > 0. Optimized primarily for English, code, and math.










Comments