hal-rock/qwen3-4b-maze-grpo-fft
Qwen3-4B model fine-tuned with reinforcement learning in a maze
2 min read
This model is a fully fine-tuned version of Qwen3-4B-Instruct-2507 (a 4-billion-parameter language model) using the Dr. GRPO algorithm (reinforcement learning method for language models). The experiment explores how reinforcement learning (RL (reinforcement learning)) changes the model's environment predictions in a custom emoji maze. The model outputs step predictions for navigating through tiles. It is designed for AI researchers and studying language model training dynamics.
Key features
- Four checkpoints: trained to a peak at step 25 and to the final state at step 95 with different reward configurations.
- Maze environment: a 100x100 grid where the agent must make moves across tiles with gold and mold.
- Wind factor: the simulation includes a probability of random agent displacement.
- Training method: Dr. GRPO (group relative policy optimization algorithm) with AdamW steps and fp32 weights under bf16.
What you can do
- Explore behavior: test the model's ability to navigate character-based mazes.
- Compare checkpoints: analyze the difference between the training peak at step 25 and the final state at step 95.
- Run inference: load specific checkpoints (main or swap) via transformers to analyze predictions.
Example
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "hal-rock/qwen3-4b-maze-grpo-fft"
tok = AutoTokenizer.from_pretrained(repo, subfolder="main/step-25")
model = AutoModelForCausalLM.from_pretrained(repo, subfolder="main/step-25", dtype=torch.bfloat16)
Read more
Details• Rewards: +20 for gold, -10 for mold, -0.1 per step.
• Training: 64 mazes with 64 rollouts per step, lr 3e-6.
• Limitations: instability after step 25, rise in action entropy, and drop in rewards by step 95.










Comments