LPCompact-Qwen3.5-9B
attention cache compactor for long contexts
1 min readupdated
This compact 393,473-parameter selector compresses the attention cache of the base Qwen/Qwen3.5-9B model. By scoring position importance from attention values across specific layers, it retains critical keys and values alongside recurrence and convolution states. This lets you handle long contexts far more efficiently without significant accuracy loss—ideal for developers and researchers working with large text volumes and extended sessions.
Key features
- Cache compression: reduces the memory footprint needed to store long-context history.
- Flexible policy: the default 8x policy keeps the first and last 512 positions, with a minimum floor of 1,024 positions.
- Tunable compression: set the ratio parameter to save even more memory.
- Broad compatibility: supports BF16, NF4, and compatible GGUF weights.
What you can do
- Integrate into pipelines: use the latentport library for prefill and generation sessions with a reduced cache.
- Process large documents: run search tasks across code repositories and structured data over long histories.
Example
import latentport as lp
with lp.load_model("Qwen/Qwen3.5-9B", backend="bf16") as model:
state = model.prefill(history)
child = state.compact()
answer = model.session(child).ask("Continue the task.")
Read more
DetailsRequires LatentPort 0.4.0 or newer.
Compression starts at 65,536 physical positions by default.
Training included a warmup phase on 16 histories with 256 AdamW updates, followed by main training on 32 histories.










Comments