DRV-Qwen3-1.7B-loop3-avg3
a language model with looped layers
2 min readupdated
An experimental 1.72B-parameter base LLM (large language model) that spends more compute per token without increasing the total parameter count. Built on Qwen3-1.7B-Base and further fine-tuned by CharlesDDDD on ~20B tokens (web text, PDFs, code, and agent trajectories), its core design loops a central block of 12 layers (8 through 19) multiple times in a row using shared weights, with the final checkpoint produced by averaging three independent training runs. A solid fit for researchers and developers exploring depth-recurrent architectures and inference-time compute scaling.
Key features
- Repeating layer block: Layers 0–7 and 20–27 run once, while the middle 12-layer block executes in a loop (trained on 3 passes, though the loop count can be adjusted at generation time).
- Three-run weight averaging: The checkpoint is an element-wise weight average of three independent runs (
loop3-run1,run2,run3), achieving lower loss than any single run on its own. - Learnable gate (
model.loop_gate): Each subsequent pass receives a learned blend of the layer-7 output and the previous loop; when zeroed out, the architecture matches standard Qwen3. - Early exit support: Training computed loss after every intermediate loop, allowing the model to generate coherent text even when run with fewer passes.
What you can do
- Benchmark inference scaling: Compare text and code generation quality across different loop counts (1 to 8) via the
num_loopsparameter. - Use as a fine-tuning base: Run SFT (supervised fine-tuning) on top of the recurrent architecture and compare it against standard
Qwen3-1.7B-Base. - Explore model merging: Test linear interpolation across the three original checkpoints from the author's collection.
Example
from transformers import AutoModelForCausalLM, AutoTokenizer
name = "CharlesDDDD/DRV-Qwen3-1.7B-loop3-avg3"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, dtype="bfloat16", trust_remote_code=True).cuda()
ids = tok("The capital of France is", return_tensors="pt").input_ids.cuda()
out = model.generate(ids, max_new_tokens=20, num_loops=3)
Read more
DetailsHeld-out test loss: 1.1139 at 3 passes (vs. 1.1827 at 1 pass and 1.1460 for a non-looped control model).
Training setup: 16K context length, AdamW optimizer, 3e-4 peak LR.
Memory: Each pass maintains its own KV cache, making cache usage roughly 1.9x higher than standard Qwen3-1.7B at 3 loops.
Limitations: Base model without instruction tuning; requires
trust_remote_code=True (tested on transformers 5.14); vLLM and other inference engines do not support this architecture yet.









Comments