Qwen3.5-0.8B-Python-SoftMoE-v2
a soft MoE Python expert without fine-tuning
2 min read
Qwen3.5-0.8B-Python-SoftMoE-v2 is a compact Python and PEP expert built on top of the 0.87B-parameter dense model Qwen/Qwen3.5-0.8B. Its main feature is weight-space steering (modifying weights directly without gradient training), which isolates a domain direction through an activation scan and injects it into the model weights. Unlike the base version, it includes an embedding-based router that automatically switches Python-related queries to this expert block. It is useful for developers and researchers doing local code analysis and working with specifications.
Key features
No fine-tuning: weights are modified via weight-space steering using a gradient-free activation scan.- Python specialization: domain prompts are sourced from the official Python EPUB documentation and PEPs (30 chunks).
- Built-in router: an embedding-margin router automatically activates the expert for on-topic queries.
- Preserved general skills: HumanEval scores remain on par with the base model without degrading general capabilities.
What you can do
- Run local experiments and Ivanna Lab pipelines for code generation and PEP checks.
- Use it as a lightweight component to answer questions about Python syntax and specifications.
- Compare the behavior of the base Qwen3.5-0.8B and the soft MoE expert on code translation tasks.
Example
When given a query related to Python syntax or PEPs (such as questions about specifications from the official documentation), the built-in embedding router automatically selects this expert over the base model to strictly follow style guidelines.
Read more
Details• Base model: Qwen/Qwen3.5-0.8B (0.87B parameters)
• Mode and parameters: steer, α = 0.1, layers [15, 16, 17, 18, 20, 21, 22, 23]
• Benchmark: code2text_python (CodeXGLUE) BLEU-4 = 1.0534
• Method: Weight-Space Programming (MIT CSAIL, 2026)










Comments