twanghcmut/molmoact2-so101-left-arm-task3
robot arm control model
2 min readupdated
This model is a fully fine-tuned version of the MolmoAct2 architecture designed to control the left arm of an SO101 robot. The checkpoint combines three different object manipulation tasks, and the desired behavior is selected using a language instruction. Created using the LeRobot library, the model targets practical applications in robotics and autonomous manipulators, making it useful for researchers and developers working with robotic hardware and VLA (Vision-Language-Action) algorithms.
Key features
- Trained on three combined cube and box manipulation tasks.
- Uses the
molmoact2policy from the LeRobot ecosystem. - Outputs absolute joint targets in chunks of 30 steps.
- Accepts image inputs from two cameras (main overhead and wrist cameras).
What you can do
- Run control of the SO101 left arm in simulation or on real hardware.
- Test executing cube-moving and drawer-opening tasks using text prompts.
- Integrate the model into control pipelines via the LeRobot or VLA-RT libraries.
Example
from lerobot.policies.molmoact2.modeling_molmoact2 import MolmoAct2Policy
from lerobot.policies.factory import make_pre_post_processors
from lerobot.configs.policies import PreTrainedConfig
repo = "twanghcmut/molmoact2-so101-left-arm-task3"
policy = MolmoAct2Policy.from_pretrained(repo).to("cuda").eval()
pre, post = make_pre_post_processors(PreTrainedConfig.from_pretrained(repo), pretrained_path=repo)
batch = {
"observation.images.top": top, # (1, 3, 360, 640) float in [0, 1]
"observation.images.wrist": wrist, # (1, 3, 480, 640) float in [0, 1]
"observation.state": state, # (1, 6) float32, raw joint units
"task": "Take the red cube out of the steel pot and place it into the orange hollow cube.",
}
actions = post(policy.predict_action_chunk(pre(batch))) # (1, 30, 6) absolute joint targets
Read more
Details• Base model: allenai/MolmoAct2-SO100_101 (full fine-tune, frozen embeddings)
• Training: 24,000 steps, batch size 32, 1x H100 80GB (took 23.7 hours)
• Limitations: 159 episodes used entirely for training (no held-out split), specific drawer-opening task has a higher error rate
• Data: 30 fps, state and action dimensionality 6 (shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll + gripper)










Comments