obstacle-avoidance-vla
robot navigation via text commands
2 min readupdated
A set of checkpoints for training a virtual Mecanum-wheeled rover to find target objects and avoid obstacles in the Genesis simulator. The model takes a text instruction (such as "drive to the red cylinder") and searches for the goal in a room filled with barriers and similar distractor objects. The control policy is built on the VLA (Vision-Language-Action — a model linking vision, text, and actions) lerobot/smolvla_base, while top-down view variants add a server powered by SmolVLM2-500M-Video-Instruct to generate a short route. Compared to baseline imitation learning, pairing reinforcement learning (PPO) with top-down camera navigation boosts room completion rates from 74% to 98%. This repository is useful for Embodied AI developers and students looking to reproduce experiments without long training runs from scratch.
Key features
- Step-by-step progression: the archive includes 4 weight variants, ranging from baseline imitation learning (Behaviour Cloning) to reinforcement learning fine-tuning (PPO) and setups paired with an external camera.
- Two-component architecture: in advanced versions, the rover handles local control via
smolVLAwhile an externalSmolVLM2server analyzes the top-down view and sends route tokens. - Modular PPO weights: the
best.ptfiles contain only the updated components (action expert, critic, and jitter size) and load directly on top of the baseline imitation checkpoints. - Fewer collisions: adding top-down routing and PPO training reduced obstacle contact rates from 62% to 19%.
What you can do
- Run ready-made simulator tests: place the weights into the
obstacle_avoidanceproject folders and run evaluations immediately in notebooks 04–07 without hours of training. - Compare navigation approaches: test the difference between front-camera-only control and a hybrid setup using an external VLM server.
- Use as a base for fine-tuning: take the pretrained
smolVLAheads to experiment with your own scenarios in the Genesis simulator.
Example
# Запуск теста PPO-машинки с навигацией от сервера верхней камеры
python scripts/car06_tests.py --car runs/checkpoints/car06_R_long_s0.pt \
--ppo runs/rl/car07_route_ppo_s0/best.pt --map runs/checkpoints/server_map_long_s0.pt \
--out nb07_results.json
Read more
DetailsMetrics (128 test rooms): imitation — 74% success (62% collisions), PPO — 76% (43%), top-down route — 94% (22%), route + PPO — 98% success and 96% when the target is hidden at the start (19% collisions).
Base models
lerobot/smolvla_base and HuggingFaceTB/SmolVLM2-500M-Video-Instruct download automatically on startup.Limitations: training was conducted exclusively on synthetic data in the Genesis simulator using a single seed; the models have not been tested on a physical robot.










Comments