Qwen3.5-9B Equational Theory
training on ground-truth-free notes
2 min readupdated
A full fine-tune of Qwen3.5-9B (9.65B parameters) from the U-WIN project, built as an experiment in learning equational theory. The model was fine-tuned on 105.6M tokens: core study notes plus synthetic breakdowns that Qwen3.5-9B generated itself after analyzing its own problem-solving attempts without access to ground-truth answers. Only the text layers (427 tensors) were updated, leaving the vision module untouched from the original Qwen3.5-9B. This checkpoint is aimed at researchers exploring LLM self-training and the internalization of mathematical knowledge.
Key features
- Trained on 100M tokens of base study notes (167,705 texts) and 5.6M tokens of rollout notes generated without reference solutions.
- Uses a KL penalty (0.01 x KL) against the base model on a frozen pool of trajectories to prevent capability degradation.
- Vision components (348 tensors) remain frozen—only the text backbone of the
AutoModelForImageTextToTextarchitecture was updated. - Includes two releases in the repository: an intermediate checkpoint after epoch one (
epoch-1) and the final model after epoch two (main/epoch-2).
What you can do
- Study how training on noisy, self-generated notes impacts performance (roughly 8% contain an incorrect final verdict).
- Compare equational reasoning quality across the base
Qwen/Qwen3.5-9B, epoch 1, and epoch 2 checkpoints. - Load the weights with
transformersfor local VLM/LLM fine-tuning experiments.
Example
from transformers import AutoModelForImageTextToText, AutoProcessor
model_id = "U-WIN/qwen35-9b-equational-theory-sair-notes100m-noref-rollout-notes-ntp-kl-0p01"
processor = AutoProcessor.from_pretrained(model_id, revision="main")
model = AutoModelForImageTextToText.from_pretrained(model_id, revision="main", dtype="auto", device_map="auto")Read more
DetailsTraining: 2 epochs (1,554 steps) on 8x GH200 in 150 minutes, learning rate 5e-06, sequence length 17,408 tokens.
Validation: held-out loss on study notes dropped from 1.245 (base) to 0.277; loss on base trajectories dropped from 0.145 to 0.126.
Limitations: the model has not been evaluated on any public benchmarks, and multimodal performance has not been tested.










Comments