pi05-so101-cvpr2027-realworld-general-fullft-30k-step30000
SO-101 Robotic Arm Control
2 min readupdated
A full fine-tune of the 3.62B-parameter lerobot/pi05_base Vision-Language-Action (VLA — an AI model connecting vision, text commands, and motor control) model for the SO-101 manipulator. It takes images from two RGB cameras and 6 current joint/gripper coordinates as input, outputting a 50-step sequence of target actuator positions to execute a text instruction. The checkpoint was trained on 120 real-world demonstrations covering three core tasks: pick-and-place, pushing a cube, and stacking a cube onto another cube. The repository bundles FP32 weights, the tokenizer, normalization processors, and loading code, so no separate base model is required.
Key features
- Full fine-tuning of all parameters across both the Vision-Language Model (VLM) and the action generation module from
lerobot/pi05_base. - Processes two video streams (
topandfrontat 224×224 resolution) alongside 6 degrees of freedom for the manipulator. - Predicts a 50-step trajectory (with a 10-step execution horizon) in 10 denoising steps.
- Self-contained package: includes FP32 weights, tokenizer, normalization configs, and the
experiment_code/load_export.pyscript.
What you can do
- Deploy on a physical SO-101 manipulator (provided calibration and measurement units match) to pick and place a doll into a box, push a cube onto a black zone, and stack a wooden cube onto a Rubik's cube.
- Use the checkpoint as a ready-to-go starting point in the LeRobot framework for further fine-tuning on custom episodes or benchmarking against the intermediate
step15000checkpoint.
Example
The model takes a text instruction, frames from the top and front cameras, and 6 joint/gripper values, returning a 50-step trajectory. Sample prompts from the training dataset:
— Pick up the doll and place it in the box.
— Push the cube onto the black zone.
— Pick up the wooden cube and place it on the Rubik's cube.
Read more
DetailsTraining data: 120 episodes (37,435 frames), with 40 demonstrations for each of the 3 tasks; 30 demonstrations were held out for validation and excluded from training.
Training parameters: 30,000 steps (cosine lr schedule from 2.5e-5 to 2.5e-6, 1,000 warmup steps, batch size 8, AdamW, bf16 autocast, weights saved in FP32).
Limitations: real-world robot success metrics are not reported by the author; optimizer state is not included; the
front input maps to the left_wrist_0_rgb slot.









Comments