Artsy RLHF
A LoRA for drawing via code generation
2 min readupdated
The final checkpoint of the Artsy project is a LoRA adapter (a lightweight add-on on top of a base model) built on Qwen/Qwen3.5-4B by Rashik24. Instead of generating pixels directly, the model writes Python code and calls drawing environment tools, while an external environment handles rendering with preview support. Trained across three stages (SFT → RLVR → RLHF using rewards from the HPSv3 human preference model), the adapter significantly outperforms the base Qwen3.5-4B in reliability and the visual quality of the final illustrations. It is a great fit for agentic system developers, multimodal LLM researchers, and anyone experimenting with procedural graphics generation through code.
Key features
- Code-based graphics generation: The model outputs Python scripts and tool calls, and the final image is produced when they execute in a sandbox.
- Optimized for render quality: A final RLHF stage across 1,000 prompts scored with HPSv3 boosts aesthetics without degrading coding ability.
- Iterative workflow: Retains full support for generation, visual previewing, and final result submission.
- Updated output-head architecture: Must be loaded via the PEFT library while keeping the base model's separate input embeddings intact.
What you can do
- Integrate into Artsy and ArtCanvas: Use it as your main checkpoint for creating illustrations from complex text prompts.
- Embed into an agentic pipeline (mini-SWE): Run model-generated code inside an isolated sandbox with a preview-based self-verification loop.
- Compare SFT, RLVR, and RLHF training stages: Test how preference optimization (PPO + HPSv3) affects the valid render rate.
Example
from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration
from peft import PeftModel
base_id = "Qwen/Qwen3.5-4B"
rev = "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a"
processor = AutoProcessor.from_pretrained(base_id, revision=rev)
base = Qwen3_5ForConditionalGeneration.from_pretrained(base_id, revision=rev, dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(base, "Rashik24/rlhf-experiment10-genai1000-linear-epoch1-kl-budget-recovery").eval()
Read more
DetailsBenchmark (150 complex GenAI-Bench prompts, 12-step limit): 149/150 valid images and an average HPSv3 score of 0.539, compared to 102/150 and 0.094 for the base Qwen3.5-4B.
Training: Rank-128 LoRA, 32 updates (1 epoch) on 1,000 prompts, batch size 32, 4 trajectories per prompt, lr 1e-5, KL penalty 0.01.
Limitations: Requires Artsy instructions, the ArtCanvas API, and a sandbox; you must pass enable_thinking=False in the chat template.
Compatibility: Tested with Transformers 5.5.4 and PEFT 0.21.3.










Comments