khanhnd61/pop-pretrain-so101-community
PoP policy checkpoints for the SO-101 robot
2 min read
This repository provides pretraining checkpoints and fine-tuned policy weights for the PoP (Pointer Policy) robotic control system designed for the SO-101 manipulator. Instead of text instructions, the model takes a 2D object coordinate, a 2D target coordinate, and a short skill code, predicting joint actions based on images from fixed and wrist-mounted cameras. Built on a DINOv2 ViT-S/14 visual encoder and an adaLN-zero transformer decoder (roughly 28M parameters) by khanhnd61, this project suits robotics engineers and researchers exploring manipulator control without large language models.
Key features
- Pretrained on 10,639 episodes: The visual encoder and control head (27.74M parameters) were trained via behavioral cloning on the HuggingFaceVLA/community_dataset_v2 dataset.
- Pointers instead of text: The model accepts 2D object and target points with Fourier features and a skill code instead of text task descriptions.
- Baseline comparison: The card includes L1 validation error benchmarks for PoP alongside a text-conditioned baseline using a T5-Efficient-Tiny encoder on the identical dataset.
- Four manipulation tasks: Towel folding, drawer opening with block placement, cup stacking, and mug pushing toward a target.
What you can do
- Test on real SO-101 hardware: Run the fine-tuned checkpoints on an SO-101 manipulator to evaluate real-world policy performance, as the card currently provides demonstration validation errors only.
- Compare conditioning methods: Use the published weights and logs to analyze trajectory prediction accuracy when specifying goals via coordinate points versus text prompts.
- Build custom tasks: Use the pretrained step_200000.pt checkpoint as a foundation to fine-tune on your own 6-axis manipulator demonstrations.
Example
import torch
ckpt = torch.load("finetune/261121-so101-pop-s0/step_040000.pt", map_location="cpu", weights_only=False)
ckpt["ema"] # веса модели (state dict)
ckpt["norm"] # нормализация состояний и действий
ckpt["instructions"] # токенизированные инструкции (для текстовых прогонов)
ckpt["config"] # параметры обученияRead more
DetailsThe best L1 validation error at step 40k for PoP is 0.292 (0.284 without pretraining), with minimum error reached between steps 4k and 6k (0.271–0.278).
Base architecture: DINOv2 ViT-S/14 (22.06M) plus an adaLN-zero transformer decoder (5.68M), trained in bf16 with a batch size of 64.
Limitations: Model architecture code is not included in the repository, checkpoints have not yet been validated via real-world robot deployment, and overfitting occurs by step 40k.










Comments