Shot Reader 8B
Video shot analysis into a ready 3D scene for Blender
2 min readupdated
The Shot Reader 8B model was created by Voyager based on Qwen3-VL-8B-Instruct (an 8.77 billion parameter multimodal language model) and is designed for reverse engineering video shots. It analyzes a continuous video segment and outputs a detailed JSON report: camera height, tilt angle, focal length, movement, as well as the positions, sizes, and trajectories of objects and environment elements. The resulting data makes it possible to directly recreate the scene in Blender or pass it to a coding agent. The new model outperforms major closed models in camera parameter estimation accuracy on test benchmarks and runs locally on consumer graphics cards.
Key features
- Purpose: reading camera parameters, object positions, and movement from video footage.
- Camera accuracy: outperforms several large models in determining camera height and movement trajectory.
- Efficiency: optimized to run locally or via vLLM on a single accelerator.
- Output format: a structured JSON document with shot specifications, timings, and coordinates in meters.
What you can do
- Pre-visualization: quickly obtain editable 3D blockouts from reference video clips for setting up lighting and angles.
- Controlled AI video: use camera parameters and trajectories as precise control inputs for generative pipelines.
- Library analysis: massively tag video archives by camera movement types, angles, and mise-en-scène.
- Scene generation: pass the ready JSON descriptor to scenographic agents for automated environment assembly.
Example
Running a video shot processing script with specified target objects to track:
python shot_reader.py clip.mp4 \
--subject "P01: person (woman in a black polo)" \
--subject "O01: dog"Read more
DetailsMetrics (on 300 test shots): camera height and movement accuracy is 69% (compared to 44–46% for competitors). Processing time per shot is 5.9 sec on an H100 (or 0.76 sec in vLLM batch mode).
Hardware: tested on an RTX 5080 (16 GB VRAM) using 4-bit quantization (about 25 seconds per shot including model loading).
Limitations: real-world video events and movement trajectories are more challenging for the model; strictly processes one continuous shot at a time.
Training: fine-tuned (LoRA r64 on attention and MLP layers) on synthetic data from procedural 3D scenes and annotated archival frames.










Comments