rooster-coder-rwkv7-1.5b
coding adapter for the RWKV7 architecture
2 min readupdated
A new LoRA (a small add-on on top of a base model) adapter and merged versions for RWKV/RWKV7-G1k-1.5B-20260930, trained on a coder profile from the Qwen4 Max reasoning corpus. The model brings a step-by-step think-style reasoning approach to the RWKV architecture. The release includes the original adapter weights as well as converted GGUF and MLX formats for running locally.
Key features
- Base RWKV7 architecture (1.5B parameters) with seamless execution support.
- Trained on structured reasoning traces (think-tag format).
- Full support for running via peft or ready-to-use quantized GGUF and MLX builds.
- Optimized for fp32 during training for stability on a Tesla T4 GPU.
What you can do
- Generate code and answer technical questions with logical reasoning.
- Run locally on a laptop via llama.cpp in Q4_K_M or Q6_K GGUF formats.
- Use the MLX version for fast performance on Apple Silicon devices.
Example
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "AJ-Gonzalez/rooster-coder-rwkv7-1.5b"
base = AutoModelForCausalLM.from_pretrained(
"RWKV/RWKV7-G1k-1.5B-20260930", trust_remote_code=True, dtype=torch.float32)
model = PeftModel.from_pretrained(base, repo, trust_remote_code=True)
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
text = tok.apply_chat_template(
[{"role": "user", "content": "Write a Python LRU cache."}],
tokenize=False, add_generation_prompt=True, thinking=True)
ich = torch.tensor([tok.encode(text)])
out = model.generate(ids, max_new_tokens=512, do_sample=False)
print(tok.decode(out[0][ids.shape[1]:]))
Read more
DetailsBenchmarks and specs: train loss 1.6735, eval loss 1.9399.
VRAM requirements: peak usage during training reached 11.43 GiB on a Tesla T4.
Limitations: trained on a small dataset of 70 rows, so the model picks up coding and reasoning style rather than acting as a full-fledged general-purpose assistant.
Available artifacts: GGUF (Q4_K_M, Q6_K) and MLX 4-bit for local inference.










Comments