SmolVLA for four sentences with two sponges on an SO-101 arm
fine-tuned robotics policy for a robotic arm
2 min readupdated
This specialized control policy for the SO-101 robotic arm sorts red and blue sponges into left and right trays based on text commands. Built on lerobot/smolvla_base (0.45B parameters) using LeRobot 0.6.0, it processes dual-camera feeds alongside current joint positions to output control actions. It is a great fit for developers and researchers experimenting with robotics and vision-language-action (VLA) models on accessible hardware.
Key features
- Trained on 96 teleoperated demonstrations from the so101_part2 dataset (24 episodes per command across four prompts).
- Completed 46,000 training steps with a batch size of 8 on a single RTX 3060 (12 GB) without image augmentation.
- Kept the vision-language backbone (the core network architecture used for feature extraction) frozen during training.
- Supports real-time inference (running a trained model to generate predictions) with dynamic action chunking.
What you can do
- Integrate it into desktop robotic arm projects to execute natural language instructions.
- Test the policy on real hardware to automate object sorting and pick-and-place tasks.
- Reuse the dataset and training configuration to adapt the policy to new command phrasings and different objects.
Example
Sample command for the arm: "Put the blue sponge in the left tray." The model parses the text and visual positions of the objects, then guides the arm to the target sponge.
Read more
DetailsBenchmarks: 34 out of 36 successful trials on the four base sentences; 11 out of 16 on unseen rephrased commands.
Hardware requirements: single RTX 3060 (12 GB) for training.
Inputs: two cameras (top and side views, 640×480 at 30 fps) and six arm joint positions.
Notes: evaluated using the final training checkpoint.










Comments