Qwen3-1.7B-Countdown-DrGRPO-512tok: training a compact LLM to solve number puzzles
3 min read
The model is a fully fine-tuned version of the Qwen3-1.7B language model (1.72 billion parameters), trained to solve the arithmetic puzzle Countdown: using 3 or 4 given numbers, you need to form an expression with + - * / operations to reach a target number. Author calmcam used the Dr.GRPO method (a reinforcement learning variant without the penalty for deviating from the base model) as part of an experiment comparing the GRPO and Dr.GRPO algorithms under a strict response length limit of 512 tokens. Unlike the base version, which often failed to fit within the thinking token limit and made mistakes, the fine-tuned model learned to reason more compactly and increased accuracy from 15.7% to 57.5%. The checkpoint will be useful for RL (reinforcement learning) researchers and developers studying reasoning chain compression in small models.
Key features
- 3.6x accuracy increase: on the test set, first-try accuracy increased from 15.7% for the base Qwen3-1.7B to 57.5%, and the pass@4 metric (at least one correct answer out of 4 attempts) reached 66.1%.
- Compact reasoning: the average generation length decreased from 496 to 382 tokens, and the share of responses hitting the 512-token limit dropped from 81% to 44%.
- Comparison with GRPO: the checkpoint is part of a series of four models trained on identical data — the Dr.GRPO version with a 512-token limit slightly trails the classic GRPO-based counterpart (61.7%).
- Fast training: the model underwent full-parameter fine-tuning in just 80 steps with a fixed budget for reasoning and the final response.
What you can do
- Compare the behavior of the GRPO and Dr.GRPO algorithms on identical math problems by matching this checkpoint with neighboring models from the author's series.
- Use the model as a ready-made Countdown puzzle solver in educational bots or local scripts with inference via
transformersorvLLM. - Study the impact of a strict token limit on the quality of reasoning chains (thinking mode) in compact models of the Qwen3 family.
Example
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "calmcam/Qwen3-1.7B-Countdown-DrGRPO-512tok"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="bfloat16", device_map="auto")
messages = [{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Using the numbers [44, 19, 35], create an equation that equals 98... Keep your reasoning under 512 tokens."}]
out = model.generate(**tok(tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False), return_tensors="pt").to(model.device), max_new_tokens=512, temperature=1.0, top_p=1.0, stop_strings=[""], tokenizer=tok)
Read more
DetailsCountdown-Tasks-3to4 benchmark (1024 tasks): accuracy 57.5% (1 attempt), 85.1% on tasks with 3 numbers, and 29.9% on tasks with 4 numbers (mean@4).
Training: 80 steps (16 prompts × 8 generations per step), AdamW optimizer, lr 7e-06, bfloat16 format.
Limitations: the model is sensitive to the prompt format (
and tags are required) and when run without explicit generation parameters, it does not stop at automatically.









Comments