Qwen3-VL-8B InteriorGS
VLM camera navigation in 3D scenes
2 min readupdated
A fine-tune of the Qwen3-VL-8B-Instruct multimodal model (8.77B parameters) by yinbq, trained for active spatial navigation inside indoor scenes from the InteriorGS dataset. Given the current camera feed, its coordinates, and a spatial prompt, the model outputs a sequence of movement commands to reach the target viewpoint. Unlike the base model, this checkpoint was fine-tuned with reinforcement learning — RL (rewarding the model for valid navigation steps) via PPO — giving it a grasp of object distances, occlusions, and frame composition. Built for researchers working on embodied AI, 3D navigation, and automated viewpoint selection in virtual interiors.
Key features
- Handles complex spatial tasks: object distance, keeping targets in frame, occlusion, projective relationships, screen occupancy ratio, and equidistance.
- Runs in a multi-turn conversational loop (up to 12 turns), receiving a fresh camera view and pose at each step while preserving full conversation history.
- Outputs discrete control commands inside a
tag (supporting multiple actions per turn):move_forward,move_backward,move_left,move_right,turn_left,turn_right. - Weights converted from fp32 to
bf16with the original Qwen3-VL tokenizer, processor, and chat template left intact.
What you can do
- Hook the model up to the InteriorGS indoor simulator to automatically search for camera angles matching a target object layout.
- Deploy it as a baseline navigation agent in 3D interiors that require step-by-step camera movement driven by visual feedback.
- Benchmark spatial planning gains after SFT and PPO against the stock
Qwen3-VL-8B-Instructmodel.
Example
import torch
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration
repo = "yinbq/qwen3vl8b-interiorgs-reward-rw_s2tcsw3_100-bf16"
processor = AutoProcessor.from_pretrained(repo)
model = Qwen3VLForConditionalGeneration.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")
Read more
DetailsTraining: SFT on trajectories generated by Qwen3-VL-8B-Instruct, followed by RL (PPO with step-level scoring, seed 20261021, step 100 checkpoint).
Metrics (InteriorGS v10, greedy, success rate): validation (413 tasks) — 53.75%, test (292 tasks) — 47.60%, overall (705 tasks) — 51.21%.
Inference: chat template without reasoning mode (
no_think), temperature=0, top_p=1; unparseable outputs count as a failed step.Compatibility: requires a
transformers release with Qwen3-VL support (tested on 4.57).









Comments