Qwen2.5-Coder-3B-AWQ-Pile-Q4_0-NPU2
optimized code completion model for AMD Ryzen AI processors
2 min readupdated
This is a custom quantized build of Qwen/Qwen2.5-Coder-3B (3B parameters) tailored for local fill-in-the-middle (FIM) code completion on AMD XDNA 2 NPU hardware accelerators via the FastFlowLM (FLM) framework. Standard uncalibrated 4-bit quantization degrades output quality too heavily on this hardware, so the author applied AWQ (activation-aware weight quantization) using the Pile dataset before repacking the weights to match the engine's requirements. It is built for developers and laptop owners with Ryzen AI 300 series or newer processors who want to run code autocomplete directly on their integrated NPU.
Key features
- Base model: built on the popular Qwen2.5-Coder-3B language model tuned for coding tasks.
- AWQ calibration: calibrated on the Pile corpus prior to Q4_0 quantization to preserve generation accuracy.
- NPU support: packaged specifically to run on integrated AMD XDNA 2 neural processing units.
- FIM integration: appends special fill-in-the-middle tokens at the end of the sequence and uses a pass-through chat template.
What you can do
- Local code autocomplete: host a local code completion server on your AMD NPU without taxing a discrete GPU.
- IDE integration: point your code editor to the local endpoint for fast, on-the-fly line and function suggestions.
- Performance benchmarking: test code generation speed and quality directly on Ryzen AI accelerator hardware.
Example
Start the local model server via the flm-add utility and send a test FIM request:
uv tool install flm-add
flm-add nickto/Qwen2.5-Coder-3B-AWQ-Pile-Q4_0-NPU2 \
--tag qwen2.5-coder:3b-awq-pile-q4_0 --xclbin-from Qwen2.5-3B-Instruct-NPU2
export FLM_CONFIG_PATH="$HOME/.config/flm/model_list.json" FLM_XCLBIN_PATH="$HOME/.config/flm"
flm serve qwen2.5-coder:3b-awq-pile-q4_0
curl -s http://localhost:52625/v1/completions -H 'Content-Type: application/json' \
-d '{"model":"qwen2.5-coder:3b-awq-pile-q4_0","prompt":"<|fim_prefix|>def add(a, b):\n <|fim_suffix|>\n\nprint(add(1, 2))<|fim_middle|>","max_tokens":32,"temperature":0}'Read more
DetailsSAFIM pass@1 benchmark (control / block / api): this build scores 54.7 / 39.5 / 65.5, compared to 5.3 / 3.9 / 18.1 without AWQ on FLM.
Hardware: AMD Ryzen AI processor with XDNA 2 architecture (Ryzen AI 300 series or newer).
Environment: FastFlowLM v1.0.11.
License: Qwen RESEARCH LICENSE AGREEMENT (Alibaba Cloud, non-commercial use only).










Comments