jaygensi/sd-vae-encoder-npu - Snapdragon NPU VAE encoder
2 min readupdated
This repository contains a Stable Diffusion VAE half (the encoder) converted into a QNN (Qualcomm Neural Network) binary context to run on Snapdragon mobile chips. The model is designed for the LATAS app on Android to run image-to-image generation directly on the device NPU. The encoder is quantized (compressed to 8-bit weights and 16-bit activations) and compiled for various Hexagon chip generations, from Snapdragon 8 Gen 2 to 8 Elite. A single encoder file is compatible with three base models: SD 1.5, SD 2.1, and DreamShaper 8.
Key features
- w8a16 quantization: weights compressed to 8 bits and activations to 16 bits for efficient hardware execution.
- Snapdragon chip support: separate binary files available for Hexagon v73, v75, v79, and v81 versions.
- Efficiency: the encoder loads into the NPU only while submitting a photo, keeping resources free the rest of the time.
- Compatibility: works with popular SD 1.5, SD 2.1, and DreamShaper 8 models.
What you can do
- Android integration: use binary files in the LATAS mobile app for NPU execution.
- Smartphone image-to-image: run Stable Diffusion-based photo transformation directly on mobile processors.
- Script inspection: explore the ONNX export, quantization, and compilation process in the LATAS repository.
Example
The model consists of compiled binary files (for example, vae-encoder-sm8650.bin) to be loaded via the QNN SDK into a mobile processor with a fixed 512x512 image input.
Read more
DetailsBenchmarks: cosine similarity 1.0000 on simulator
Limitations: strict input size of 512x512 pixels
License: CreativeML Open RAIL-M










Comments