Laya on Qualcomm NPU
binaries for fast decision-model inference on Qualcomm chips
2 min readupdated
This repository provides precompiled QNN context binaries (specialized context binary files for the Qualcomm AI Runtime engine) for the Laya decision-making model, targeting the Hexagon NPU on the Qualcomm QCS8550 chip in fp16 (half-precision floating-point) precision. The original model weights were converted with zero degradation in output accuracy—every decision matches the CPU baseline identically. This delivers a 15–39x speedup on embedded hardware compared to running on a standard device CPU, making it ideal for developers building embedded systems and edge AI applications on the Qualcomm platform.
Key features
- Massive speedup: Inference runs 15–39x faster depending on context length and checkout.
- Exact accuracy: Matches CPU baseline outputs across 48/48 tests with zero loss in decision quality.
- Ready-to-use binaries: Includes three precompiled variants covering the English, multilingual, and typed-decision models.
- Sustained efficiency: Maintains steady throughput over five-minute runs without throttling or latency degradation.
What you can do
- Deploy to edge hardware: Run the decision-making model directly on Qualcomm QCS8550 devices such as the Thundercomm RB5 Gen2.
- Run via Python: Use ready-made classes from the companion repository for fast text classification and task routing.
- Customize builds: Use the open-source export and compilation scripts to adapt the model to custom requirements or target environments.
Example
Sample on-device inference using the dedicated router from the open-source repository:
from laya_npu import NPURouter # from the code repository
router = NPURouter() # one NPUAgent per models/.bin
print(router.predict("We were billed twice for March. Refund it today or we cancel.", {
"department": {"type": "choice", "instructions": "Which department should handle this?",
"criteria": {"billing": "invoices, payments, refunds",
"technical": "bugs, outages, errors", "other": "everything else"}},
"churn_risk": {"type": "noul", "instructions": "Does the customer threaten to leave?"}}))Read more
DetailsBenchmarks: At a 128-token context length, the English variant runs in 49 ms (a 27x speedup over CPU); at 512 tokens, it completes in 148 ms (a 31x speedup).
Hardware requirements: Targets the Qualcomm QCS8550 SoC (Hexagon HTP V73) and requires QAIRT 2.51 or newer.
Limitations: Maximum batch size is 1, choice count is capped at 32 options per query, and binaries are tightly coupled to the specific SDK version and target chip.
License: Apache-2.0 (inherited from the original Laya model).










Comments