Safety_Tuning-Qwen3.5-4B-Base
interim safety training checkpoint
2 min readupdated
This model is an intermediate training stage (600 optimizer steps) for the safety alignment of the Qwen3.5-4B-Base base language model using the GRPO (Group Relative Policy Optimization) algorithm. Training was conducted on a diverse mixture-8 dataset featuring tasks in secure code handling, complex advice, and prompt processing. The checkpoint retains the original base architecture and vision module weights, loading standardly through the transformers library.
Key features
- Based on Qwen3.5-4B-Base with 4.66 billion parameters.
- Uses the GRPO algorithm with group normalization of binary rewards.
- Trained on a dataset mixture (mixture-8) covering safety benchmarks and code generation.
- Maintains compatibility with original vision module and MTP head weights.
What you can do
- Test model behavior on code safety tasks using the tool rollouts feature.
- Experiment with response generation using the enabled thinking mode (`enable_thinking=true`).
- Compare response quality across different training stages (for example, step 600 versus the base version).
Example
from transformers import AutoTokenizer, Qwen3_5ForConditionalGeneration
model_id = "thkim0305-SNU/Safety_Tuning-Qwen3.5-4B-Base-GRPO-Mixture8-binary-lr1e-6-kl0.001-p15-k8-step600"
tok = AutoTokenizer.from_pretrained(model_id)
model = Qwen3_5ForConditionalGeneration.from_pretrained(model_id, dtype="bfloat16", device_map="auto")
messages = [{"role": "user", "content": "Explain why phishing emails are dangerous."}]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=True)
out = model.generate(**tok(text, return_tensors="pt").to(model.device), max_new_tokens=512)
print(tok.decode(out[0], skip_special_tokens=True))
Read more
DetailsAlgorithm: GRPO with PPO clip 0.2
Framework: Slime + Megatron-LM, bf16
KL coefficient: 0.001
Limitations: this is an intermediate training checkpoint, not a final model.










Comments