Subspace NLA: Decoding hidden states of Qwen3-8B
2 min read
The Subspace NLA (Natural Language Autoencoder) model is designed for AI model interpretability, allowing you to decompose Qwen3-8B layer activations into separate semantic components. The system works alongside a Block-Sparse Featurizer (BSF), which splits the 24th layer's activation into 16 active blocks within 64-dimensional subspaces. A verbalizer generates short text descriptions for each component, while a reconstructor assembles them back into a vector. This tool is useful for researchers and developers studying the internal decision-making mechanisms of LLMs (large language models).
Key features
- Autoencoder construction: breaking down layer activations into independent subspaces
- Verbalization: generating text descriptions for each active block
- Vector reconstruction back into the model space using a trained projector
- Fine-tuning via GRPO (Group Relative Policy Optimization) with a GRPO reward for projection accuracy
What you can do
- Run text token analysis using the quick-start script for the model
- Explore internal vector representations and semantic components of Qwen3-8B activations
- Check vector reconstruction quality using the FVE (Fraction of Variance Explained) metric
Example
python scripts/hf_quickstart.py --nla asher577/subspace-nla-qwen3-8b --text "The capital of France is"Read more
DetailsBenchmarks: FVE for the entire activation is 0.489 with a cosine similarity of 0.857
Requirements: runs on top of the base Qwen/Qwen3-8B model
Training: GRPO for 2,400 steps on a mixture of chat and web text
Limitations: requires the mandatory use of the paired BSF featurizer










Comments