statescope-qwen3.5-0.8b
State dictionaries and MLP for analyzing Qwen3.5-0.8B
2 min readupdated
The statescope dataset and toolset for the Qwen3.5-0.8B model includes two dictionaries per layer (for stream hidden states and MLP transformations) along with a Python reader. The library lets you study internal language model computations through sparse state and transformation dictionaries, tracking signal propagation between tokens and layers. The release is useful for interpretability researchers, engineers, and anyone exploring transformer internal mechanisms in practice.
Key features
- Layer-wise dictionaries: 24 layers with detailed state configurations and weights in safetensors format.
- State flow analysis: breaking down the residual stream into active groups of directions.
- MLP transformations: decomposing MLP layers into a sum of small gated linear operations.
- Ready-to-use scripts: built-in utilities for manifest verification and reproducing string length experiments.
What you can do
- Activation analysis: reconstruct hidden states (activations) and check the operation of individual attention heads or GatedDeltaNet.
- Computation interpretation: modify group codes or transformation gates to study their impact on the final model output.
- Run checks: use the verification script to quickly re-verify all 144 weight files of the release.
- Python integration: plug in the reader to debug model behavior directly within PyTorch and Transformers pipelines.
Example
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from statescope.reader import StateVocab, PublicMolt, capture
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-0.8B")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-0.8B", dtype=torch.bfloat16, device_map="cuda").eval()
input_ids = torch.tensor([tokenizer("The committee met on Thursday to review the budget.")["input_ids"]], device="cuda")
V = StateVocab("/state/L9", "cuda")
M = PublicMolt("/mlp/L9", "cuda")
acts = capture(model, 9, input_ids)
x_hat = V.reconstruct(acts["resid_post"], "resid_post.9")
gate, y_hat = M.forward(acts["resid_mid"])
Read more
DetailsTotal file size: ~27.4 GiB (146 files, including manifest and configurations).
State architecture: 16,384 groups per layer, 4 directions per group, 24 active groups per token.
MLP dictionaries: 3,968 transformations per layer.
Environment requirements: Python 3.12+, torch 2.11.0+cu128, transformers 5.20.2.
Related releases: an analogous version is available for Qwen3.5-4B.










Comments