smolvla-so101-piano-flow
model for controlling a piano-playing robot
2 min readupdated
This compact vision-language-action system (a multimodal model that combines video, text, and physical actions) was trained with the LeRobot library on top of lerobot/smolvla_base. Built for the so_follower robot, it controls physical movements using feeds from three cameras and joint state data. In practice, it recognizes and presses specific keys on a musical instrument based on target notes. It is useful for robotics researchers and developers working with physical hardware on LeRobot.
Key features
- Compact SmolVLA architecture designed to run on consumer hardware.
- Combines data from three cameras (front, ears) for spatial orientation.
- Trained on a 300-episode dataset of piano key-pressing tasks.
- Full support for inference and fine-tuning using LeRobot ecosystem tools.
What you can do
- Run the control policy on a physical so_follower robotic arm to play keys.
- Fine-tune the model on your own motion datasets using LeRobot utilities.
- Visualize the training dataset through the project's official Spaces.
Example
Running the pre-trained control policy to press a target key:
lerobot-rollout \
--strategy.type=base \
--robot.type=so_follower \
--robot.port= \
--robot.cameras="{ : {type: opencv, index_or_path: , width: 640, height: 480, fps: 30}, : {type: opencv, index_or_path: , width: 640, height: 480, fps: 30}}" \
--policy.path=PawelLach/train_2026-11-17_22-11-36 \
--task="play re, the second key" \
--duration=60Read more
DetailsBase model: lerobot/smolvla_base (0.45B parameters)
License: apache-2.0
Training: 10,000 steps, batch size 64, adamw optimizer, LR 0.0001
Dataset: 300 episodes, 25,308 frames at 30 FPS










Comments