statetrace-qwen3.5-0.8b-transcoders
per-layer transcoders for dissecting Qwen3.5-0.8B
2 min read
A set of 24 per-layer transcoders (PLTs — modules for interpreting a model's internal computations) by Fealtyy, built for analyzing the Qwen/Qwen3.5-0.8B language model. The base model uses a hybrid architecture with 18 Gated DeltaNet layers and 6 classic full-attention layers. Each transcoder reads an MLP block's input and predicts its output across 16,384 sparse JumpReLU features, letting you build attribution graphs (circuit tracing) and track decision logic inside the network. This tool is aimed at mechanistic interpretability researchers working with the statetrace and circuit-tracer libraries.
Key features
- Covers all 24 layers of Qwen3.5-0.8B with a width of 16,384 features per layer (16x the 1024 residual stream dimension).
- Sparsity of ~48 active features per token (L0) and 0% dead features thanks to the AuxK loss function.
- Fully compatible with the
circuit-tracertranscoder format: weight matrices are saved inlayer_{i}.safetensorsat fp32 precision alongsideconfig.yaml. - High attribution graph accuracy on test prompts, averaging a 0.72 replacement score and a 0.93 completeness score.
What you can do
- Build attribution graphs for short prompts in
statetraceto see which internal features fire during response generation. - Compare how the hybrid Gated DeltaNet layers behave versus standard full-attention layers (layers 3, 7, 11, 15, 19, and 23) inside Qwen3.5-0.8B.
- Load individual layers directly via
safetensorsfor custom MLP feature labeling and analysis experiments.
Example
from statetrace.replacement import ReplacementModel
from statetrace.attribution import attribute
rm = ReplacementModel.from_pretrained(
"Qwen/Qwen3.5-0.8B",
"Fealtyy/statetrace-qwen3.5-0.8b-transcoders",
local_files_only=False,
)
graph = attribute("The Eiffel Tower is located in the city of", rm)
Read more
DetailsTrained on 399.6M tokens per layer from the FineWeb-Edu dataset (1024-token context) in 3.5 hours on a single H200 GPU (~25 GB).
Metrics on 32,736 validation tokens: baseline model cross-entropy is 2.855, rising to 3.464 (+0.61 nats) when swapping in all 24 transcoders simultaneously; mean FVU across layers is 0.230 (ranging from 0.070 on layer 23 to 0.355 on layer 9).
Limitations: trained exclusively on English educational web text (performance on code, chat, and other languages was not measured); features do not include automated text labels; these are per-layer (PLT) rather than cross-layer (CLT) transcoders.










Comments