Nla-qwen3-8b-layer18
activation verbalizer for Qwen3-8B
1 min readupdated
This model is an activation verbalizer (a tool for interpreting AI models) built on a single-layer natural language autoencoder trained from Qwen/Qwen3-8B. It takes a layer-18 hidden-state vector at a specific token and translates it into a plain-language text explanation. The repository includes the post-reinforcement learning (RL) verbalizer, while the sft folder stores the supervised fine-tuning weights. It is aimed at researchers working on interpretability and analyzing how language models operate internally.
Key features
- Built on Qwen/Qwen3-8B, targeting layer 18
- Fine-tuned with a rank-64 LoRA (a lightweight adapter on top of the base model)
- Trained for 150 reinforcement learning (RL) iterations to improve generation quality
- Reaches 63% explained variance on held-out data
What you can do
- Inspect internal vector representations and activations in Qwen3-8B
- Generate natural-language interpretations of layer-18 hidden states
- Compare explanation quality before and after the RL stage
Example
import re, torch, yaml
from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoTokenizer
torch.set_grad_enabled(False)
AV = snapshot_download("camila-blank/nla-qwen3-8b-layer18", ignore_patterns=["sft/*"])
Read more
DetailsMetric: 63% explained variance after RL (vs. 43% before RL)
Requirements: GPU with bfloat16 and CUDA support
Limitations: not a general-purpose language model
Training: 1 epoch of SFT and 150 RL iterations










Comments