ACT policy for SO-101 robotic arm - picking and placing a cube
2 min read
The model implements the ACT (Action Chunking with Transformers, a robot control algorithm) policy based on the LeRobot library for the SO-101 robotic arm. It is trained to pick a green cube from a fixed table position and place it into a box, handling scenarios where other cubes are already inside the box. The model accepts video input from two cameras (top-down and wrist-mounted) along with current joint states, and outputs a sequence of target positions for the robot's 6 joints.
Key features
- Training for SO-101: the model is built to control the specific manipulator model via the LeRobot framework.
- Multimodal input: uses data from two cameras (640x480 at 30 fps) and 6-joint parameters.
- Fine-tuned scenarios: trained on demonstrations with cubes already present inside the target box.
- Hardware integration: runs using standard rollout utility tools for real robotics tests.
What you can do
- Run on a robotic arm: apply the policy to automate object picking on compatible SO-101 hardware.
- Test in simulation: verify the playback of recorded manipulator movement trajectories.
- Integrate into a setup: use alongside external color detectors to implement conditional sorting scenarios.
Example
lerobot-rollout --strategy.type=base --policy.path=MaxPrain/act_pick_place_green_v3_40000 --device=mps \
--robot.type=so101_follower --robot.port= --robot.id= --robot.max_relative_target=10 \
--robot.cameras="{ top: {type: opencv, index_or_path: 1, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30} }" \
--task="pick the green cube and place it in the box" --fps=30 --duration=12Read more
DetailsUsed the LeRobot 0.7.0 framework and the MaxPrain/pick_place_green_v3 dataset (30 episodes).
Training took 40,000 steps via Hugging Face Jobs (a10g-small).
Important: the model reproduces a fixed coordinate-based trajectory and does not recognize color directly.










Comments