CONTROL Resonant IDM
recovering gamepad inputs from gameplay video
2 min readupdated
An IDM (Inverse Dynamics Model, a model that infers actions from visual changes) created by XvKuoMing specifically for CONTROL Resonant. It analyzes gameplay footage and predicts, frame by frame, which of 21 buttons were pressed and the positions of both analog sticks. Under the hood, it pairs a fine-tuned google/siglip2-large-patch16-256 vision encoder with a 4-layer temporal transformer (329M parameters total). Unlike standard video classifiers, the model processes an 8-frame sliding window to auto-label footage lacking controller logs so it can be used to train game-playing AI agents.
Key features
- Predicts the state of 21 gamepad buttons and 4 axes across both analog sticks in the [-1, 1] range on a per-frame basis.
- Built on SigLIP 2 Large (256×256 px) with patch pooling and a 4-layer transformer with 8 attention heads.
- Cuts analog stick prediction error by 35–40% compared to a zero-input baseline (assuming sticks remain centered).
- Supports both running directly from Python and fast CLI video labeling with output saved to
.npz.
What you can do
- Automatically label CONTROL Resonant playthrough recordings to train AI bots via behavioral cloning.
- Analyze playstyles and input frequency for primary action buttons (triggers and face buttons) from existing gameplay videos.
- Use the architecture and pipeline as a reference to build custom IDM models for other third-person action games.
Example
from huggingface_hub import snapshot_download
import sys
folder = snapshot_download("XvKuoMing/control-resonant-idm")
sys.path.insert(0, folder)
from idm import load, label_video
model = load(folder)
out = label_video(model, "gameplay.mp4") # buttons, probs, sticks
Read more
DetailsMetrics across 3 test sessions (88 minutes): mean stick error of 0.086–0.121 (versus 0.142–0.180 for the baseline), F1 scores on frequent buttons up to 0.58–0.68 (left trigger) and 0.50–0.62 (X, A, and B buttons).
Training: 6.4 hours of single-player gameplay (controller logs sampled at 1000 Hz), trained for 8,000 steps with a batch size of 16; the released checkpoint is from step 7,000.
Requirements and input: video is resampled to 60 fps and resized to 256×256 without cropping (HUD must remain visible);
label_video uses roughly 12 GB of RAM per hour of footage, so longer recordings should be split into chunks.Limitations: works only with CONTROL Resonant; rarely used buttons (D-pad, stick clicks, Start) are barely recognized, and predictions degrade in menus and cutscenes.










Comments