microduck-skill-agent
LoRA adapters for robot control in MuJoCo
2 min readupdated
A set of LoRA adapters (lightweight add-ons for a base model) that turns the multimodal Qwen3-VL-4B-Instruct model into an action planner for a small walking "microduck" robot in the MuJoCo simulator. At each step, the model takes a camera frame and a text instruction in English or Chinese, then outputs a specific JSON command (such as move, turn, approach, explore, or return_home). Rather than acting as a direct motor controller, it handles only high-level skill selection and works alongside algorithms from the project repository (depth mapping and A* pathfinding). It is built for robotics researchers and developers exploring VLM (vision-language model) navigation in virtual environments.
Key features
- 11 selectable skills: predicts a single action from
move,turn,approach,search,explore,return_home,stop,done,clarify,unsupported, orbypass_return. - Bilingual commands: understands instructions in English and Chinese grounded in the live camera feed.
- Four versions in one repository: includes
lang-v1–lang-v4subfolders, withlang-v2serving as the main release with the strongest results. - Lightweight fine-tuned add-on: packs just 655,360 trainable parameters (LoRA rank 8) on top of a 4-bit quantized Qwen3-VL-4B-Instruct base.
What you can do
- Test VLM navigation in MuJoCo: run the agent in multi-room simulated worlds to locate objects and reach targets (for example, "walk to the leftmost ball and return").
- Compare training iterations: load different subfolders (
lang-v1–lang-v4) to see how dataset scale impacts skill-selection accuracy. - Study hybrid architectures: use the project as a reference for pairing a language model with classical algorithms like A* and frontier-based map exploration.
Example
import torch
from transformers import AutoModelForImageTextToText, BitsAndBytesConfig
from peft import PeftModel
base = "Qwen/Qwen3-VL-4B-Instruct"
quant = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16)
model = AutoModelForImageTextToText.from_pretrained(base, quantization_config=quant, device_map="auto")
model = PeftModel.from_pretrained(model, "atlasmarkz/microduck-skill-agent", subfolder="lang-v2")Read more
Detailslang-v2 benchmarks on test layouts (seeds 99000, 99100): strict accuracy 61/74 (82.4%), outcome-equivalent accuracy 65/74 (87.8%).Training: 300 steps on 3,074 examples (for
lang-v2) took roughly 13 minutes on a single RTX 3060 (8 GiB).Limitations: intended strictly for simulation (do not run on physical hardware); requires a specific prompt format and the skill executors from the
atlasmarkz/microduck-qwen repository.License: Apache-2.0.










Comments