Gemma 4 31B LiLiCorr Drafter
generation accelerator for Gemma 4
2 min readupdated
A lightweight 2.58B-parameter draft model built to speed up google/gemma-4-31B-it via speculative decoding (where a small, fast model predicts several tokens ahead and the larger model verifies them in a single pass). Trained from scratch by MeganEFlynn, it pairs a 5-layer Qwen3 decoder backbone with a LiLiCorr head. The model generates 7 speculative tokens per step (block size 8), cutting response latency without sacrificing the base model's output quality. It is aimed at developers and server operators running Gemma 4 31B in production on vLLM who want to squeeze out maximum generation speed.
Key features
- Speculative decoding: predicts an 8-token block (7 speculative tokens per step) using greedy proposal and verification against the base model's hidden states.
- Hybrid architecture: combines a 5-layer Qwen3 decoder (hidden size 5376, 32 attention heads, 2048 sliding-window attention) with a 2-layer LiLiCorr head (top-k 8, width 1024).
- High prediction accuracy: after epoch 0 validation, overall accuracy (full_acc) reached 0.6048, with an expected accepted length (lilicorr_eal) of 4.7257 tokens.
- Compact footprint: the bf16
model.safetensorsweights file takes up just 5.16 GB.
What you can do
- Speed up Gemma 4 31B inference: plug it in as a draft model in a supported vLLM build or via the
speculatorslibrary to lower latency in chatbots and APIs. - Fine-tune or benchmark epoch 1: use this checkpoint (
checkpoint_bestafter epoch 0) to benchmark speed on your own prompts, or wait for the final epoch 1 release.
Example
This model cannot generate text standalone and only runs paired with the google/gemma-4-31B-it verifier. To run it in vLLM, set this repository as your draft model using the draft_model method and a block size of 8, after replacing the local verifier path in config.json with the Hugging Face Hub model ID.
Read more
DetailsValidation (13,925 prompt groups): per-position accuracy across positions 1–7 was 0.8613 / 0.7433 / 0.6528 / 0.5800 / 0.5179 / 0.4636 / 0.4133, with a loss of 1.4685.
Training: 1,774,091 rows (2.14B tokens) from the
nm-research/gemma-4-31b-it-responses-open-perfectblend dataset over 13,383 steps (epoch 0 of 2).Limitations: requires the
speculators library (lilicorr-clean / gemma4-perfectblend-lilicorr-clean branch) or a patched vLLM build with the lilicorr algorithm registered.









Comments