SyncWorld-mv3-strip384-views-first-selv2-rt2h
a world model for bimanual robots with smart context selection
2 min readupdated
This is an intermediate checkpoint (iteration 2,000 of 20,320) of the SyncWorld generative world model from developer yyuncong. Given the current robot camera frame and 16 upcoming actions for two robotic arms, the model predicts the next 16 frames of video. It builds on mvSV3p weights (update 20,000), fine-tuned on RoboTwin 2.0 simulation data. The standout addition in this version is a trainable context selector (selector v2) using an OR sibling rule, which automatically pulls the most relevant past recording clips from a massive library to guide the generator without leaking future information.
Key features
- Three-camera layout: Each frame is a 1536×384 strip made of three 512×384 tiles—one front-facing head camera and two wrist-mounted cameras.
- Trainable context selector (v2): A 15-feature linear scorer evaluates candidates from a 17.6M-clip library based on trajectory similarity, poses, camera alignment (within 10° and 10 cm), and DINOv2-base visual features.
- Leakage prevention (OR sibling rule): When searching for similar examples across variations of the same episode (counterfactual runs), clips where at least one arm is still following the target episode's trajectory are filtered out.
- Bimanual control (bimanual_gripper): At each step, the model takes 14 parameters—displacements, Euler angles, and gripper openness for both the left and right arms.
What you can do
- Benchmark retrieval quality: Compare the trainable selector against control checkpoints (fixed-weight
equalwand baselinerecent8) on a held-out split of RoboTwin 2.0 tasks. - Simulate bimanual manipulation: Test visual action-outcome predictions for aloha-agilex, arx-x5, franka, and ur5 robots in dual-arm tasks.
- Study selector training dynamics: Inspect the
router.npzweights (version 4, trained on a pool of 2,837 queries) to see which context features matter most for accurate video generation.
Example
export WM_TTT_SIBLING_OVERRIDE=wm_ttt/handoff/rt2h_sibling_or_v1.npz
python wm_ttt/handoff/selector_eval_theta.py router.npz --out theta.jsonRead more
DetailsTraining: 4 nodes × 8 NVIDIA B200 GPUs (32 streams, HSDP), bf16 with fp32 master weights, AdamW (LR 5e-5).
Data: 45,184 episodes (225,001 recordings) across 50 RoboTwin 2.0 tasks; frame rate — 20.83 Hz (conditioning at 15 fps).
Limitations: Intermediate stage (1,968 generator updates out of 20,000); running it requires the
mv_strip_rt2h_v1_s4_views_sib_grip_anch_wrist library, the query_context_bimanual.py script, and the mandatory WM_TTT_SIBLING_OVERRIDE environment variable.









Comments