Instella-MoE-16B-A3B-Think-SOMPOA-heresy
uncensoring AMD's new MoE model
2 min readupdated
This is an uncensored version of the Instella-MoE-16B-A3B-Think reasoning language model, created by MuXodious using the Heretic tool. Built on an MoE architecture (Mixture-of-Experts, where only a fraction of weights activate per token), it engages 2.8B out of 15.86B total parameters. Unlike AMD's original release, which rejected every provocative test prompt, this build responds to almost anything without losing its core reasoning abilities. It's a solid fit for local AI enthusiasts looking for a fast thinking model free of strict built-in filters.
Key features
- Almost zero refusals: Refusals dropped from 101/101 in the base model down to 1/101 in the final checkpoint (Trial 756).
- Intact reasoning quality: The model held its ground on the PIQA benchmark, scoring 0.7312 (
acc,none) and 0.7432 (acc_norm,none) versus 0.7231 and 0.7459 for the base model on an H100 server. - Minimal weight drift: KL divergence (a measure of how far the weights shifted from the original model) is just 0.1280, preserving the base model's style and capabilities.
- Instella-MoE architecture: Features 64 experts (6 active and 2 shared per token), 27 decoder layers, and Gated Multi-head Latent Attention (Gated MLA).
What you can do
- Hook it up via the
transformerslibrary to a local assistant for tasks requiring step-by-step chain-of-thought reasoning without triggering censorship guardrails. - Use it for English text generation and roleplay scenarios where AMD's standard checkpoint refuses to engage.
- Compare its behavior directly against the original
amd/Instella-MoE-16B-A3B-Thinkwhen researching refusal ablation techniques (SOMPOA / Heretic).
Example
To keep the model working properly and trigger its reasoning mode reliably, the author used the following response prefix during Heretic processing:
--response-prefix "<|Assistant|>\n\n\n"
Read more
DetailsProcessed with Heretic v2.2.0.dev0 using SOMPOA settings (published build is Trial 756).
H100 test results: Refusals 1/101, KL divergence 0.1280, PIQA acc_norm 0.7432.
Merging and export were run on a rented H100 GPU server; the author notes that multi-GPU configurations affect the final metrics.










Comments