mars-dfd-hr
deepfake and manipulated video detector
2 min readupdated
This model detects whether a video has been digitally manipulated or AI-generated, outputting a manipulation probability (P(manipulated)) for the entire clip. Built on a frozen OpenAI CLIP ViT-L/14 encoder paired with trainable MoE adapters (Mixture-of-Experts, an architecture with multiple specialized expert sub-networks), this reimplementation was released by researchers at the Vector Institute as part of the MARS paper. Unlike basic frame-by-frame classifiers, it processes every frame at 448×448 resolution across two scales simultaneously—as a 224×224 global view and a 2×2 grid of 224×224 patches—while using token and layer routers in the upper layers for key feature selection and early exit. It is well suited for content moderation developers, media verification specialists, and security researchers.
Key features
- Dual-scale frame analysis: Every second frame is resized to 448×448 and processed simultaneously as a full image and as four 224×224 tiles.
- Standalone ONNX graph: Alongside the
model.safetensorsweights (415 MB), the repo includes a self-containedmodel.onnxfile (1.63 GB) with the CLIP encoder baked in, letting you run inference via ONNX Runtime without PyTorch. - Dynamic routing: Token and layer routers in blocks 21–23 skip unnecessary computation on simpler regions.
- Permissive license: The adapter weights are released under Apache 2.0 (the base encoder is fetched separately from OpenAI under its own license).
What you can do
- Drop incoming video checks into a moderation or anti-fraud pipeline using a single
model.onnxfile on CPU or GPU. - Use the model as an out-of-the-box baseline to benchmark your own deepfake detectors on standard datasets like FaceForensics++ and Celeb-DF v2.
- Calibrate the
fake_probthreshold on your own validation set to filter out synthetic videos from various generators.
Example
import numpy as np
import onnxruntime as ort
from huggingface_hub import hf_hub_download
session = ort.InferenceSession(
hf_hub_download("vector-institute/mars-dfd-hr", "model.onnx"),
providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
)
# frames: (1, 64, 3, 448, 448) float32 в [-1, 1]; n_valid: число реальных кадров
prob = session.run(["fake_prob"], {"frames": frames, "n_valid": np.array([64], dtype=np.int64)})[0][0]
Read more
DetailsBenchmarks (video-level AUROC): FaceForensics++ — 99.80%, DeepfakeTIMIT — 99.59%, Celeb-DF v2 — 93.89%, GenVideo — 79.48%, 7-dataset average — 90.03%.
Training: 3 epochs on FaceForensics++ (c23) and synthetic GTA-V scenes (SAIL-VOS 3D) on 4× NVIDIA A40.
Input: 64-frame clips (sampling every 2nd frame) at 448×448 resolution.
Limitations: Output probabilities are uncalibrated, so binary classification requires tuning a threshold on your own data; the ONNX export uses float32 instead of bfloat16.










Comments