MARS: Multiscale Appearance-Residual Streams for Video Forgery Detection
MARS is a full-frame video forgery detector that covers both facial manipulation (face swapping, reenactment) and fully generated video. It reads one frozen Perception Encoder (PE-Core-L) in two ways:
- A residual path compares the encoder's intermediate patch features with smoothed copies of themselves, across space and time at several scales, and a temporal transformer models how these differences develop over the clip.
- A semantic path classifies sampled frames with the complete encoder, tuning only its own copies of the LayerNorm parameters.
A gate mixes the two paths per clip and returns P(manipulated) for the clip.
Files
| file | contents | size |
|---|---|---|
model.safetensors |
trained weights; the frozen PE-Core-L encoder is not included and is downloaded from timm on first use | 270 MB |
config.yaml |
model and data settings, read by the mars package |
2 KB |
model.onnx |
complete graph, frozen encoder included; runs with ONNX Runtime alone, float32 | 1.59 GB |
model.fp16.onnx |
the same graph in float16, inputs and outputs still float32; less GPU memory | 0.80 GB |
usual_ranges.json |
percentiles of the ONNX video score on the authentic and manipulated test videos, with model.onnx |
3 KB |
usual_ranges.fp16.json |
the same with model.fp16.onnx |
3 KB |
serving.json |
measured GPU memory and time per clip with ONNX Runtime (both files, batch 1 and 2), opset, outputs, encoder chunk, ONNX Runtime version | 1 KB |
This is the checkpoint behind the MARS results in the paper.
Input
Every clip is built the same way, whichever file you use:
- Decode the video as RGB and keep every second frame (0, 2, 4, ...).
- Cut the kept frames into consecutive clips of 64. A short last clip (or a short video) is
padded by repeating its last frame;
n_validis the number of real frames. - Resize each frame to 336 x 336 (square, no crop, bicubic with antialiasing), scale to [0, 1], then to [-1, 1].
- The video score is the mean P(manipulated) over its clips.
Scores are not calibrated. If you need a decision, choose a threshold on validation data from your own setting.
Use with safetensors (PyTorch)
Requires the code repository:
git clone https://github.com/VectorInstitute/mars.git && cd mars && uv sync
import torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from mars.config import load_yaml
from mars.data.sampling import clips_for_video
from mars.data.transforms import preprocess
from mars.data.video import count_frames, read_frames
from mars.models.factory import build_detector
folder = snapshot_download("vector-institute/mars", allow_patterns=["config.yaml", "model.safetensors"])
raw = load_yaml(f"{folder}/config.yaml")
detector = build_detector(raw["model"]["name"], raw)
model = detector.build_network() # downloads the frozen encoder on first use
missing, unexpected = model.load_state_dict(load_file(f"{folder}/model.safetensors"), strict=False)
assert not unexpected and all(key.startswith("encoder.") for key in missing) # only the frozen encoder is absent
model = model.cuda().eval()
data = detector.cfg.data
def score(path: str) -> float:
"""Mean P(manipulated) over the clips of one video."""
probs = []
for clip in clips_for_video(path, count_frames(path), data.n_frames, data.frame_stride):
frames = preprocess(read_frames(path, list(clip.frame_indices)), data.image_size)[None].cuda()
valid = (torch.arange(data.n_frames) < clip.n_valid)[None].cuda()
with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
logits = model(frames, valid).logits
probs.append(logits.float().softmax(-1)[0, 1].item())
return sum(probs) / len(probs)
print(score("video.mp4"))
Use with ONNX (no PyTorch, no code repository)
pip install onnxruntime-gpu av numpy pillow huggingface_hub # or onnxruntime for CPU
| name | shape, type | meaning |
|---|---|---|
input frames |
(B, 64, 3, 336, 336), float32 | B clips in [-1, 1] |
input n_valid |
(B,), int64 | number of real frames in each clip, 1 to 64 |
output fake_prob |
(B,), float32 | P(manipulated) for each clip |
output logits |
(B, 2), float32 | authentic and manipulated logits |
output frame_contrib |
(B, 64), float32 | signed contribution of each frame to the clip's fake-minus-real logit margin; positive pushes toward manipulated, 0 on padded frames |
output patch_contrib |
(B, 64, 24, 24), float32 | signed contribution of each frame's 14 x 14-pixel patches through the residual path, same units, 0 on padded frames |
output gate |
(B, 2), float32 | weight given to the residual and the semantic path; the two sum to 1 |
The batch size B is free. Clip length, frame stride and image size are also stored in the file's metadata.
Both files take the full 64-frame clip. Inside the graph the encoder processes 16 frames of
each clip at a time; frames are encoded independently, so this lowers memory without
changing the outputs. On an A100, one clip peaks at about 6.8 GiB of GPU memory with
model.onnx and 2.9 GiB with model.fp16.onnx (details in serving.json).
The float16 video scores are within 0.003 of float32 on UADFV and Celeb-DF. Use model.onnx
on CPU, or as the source for a TensorRT engine.
The contributions are those behind the paper's figures and faithfulness tests: the clip's
margin (logit of fake_prob) equals the sum of frame_contrib plus a per-clip constant.
A frame's frame_contrib is the sum of its patch_contrib cells, plus, for the 16 frames
the semantic path samples, that path's share.
usual_ranges.json gives the percentiles (0 to 100) of the video score
on the authentic and on the manipulated test videos listed under Results, computed with
model.onnx; usual_ranges.fp16.json is the same for model.fp16.onnx.
Use them to place a score in context, not as a threshold.
import av
import numpy as np
import onnxruntime as ort
from huggingface_hub import hf_hub_download
from PIL import Image
session = ort.InferenceSession(
hf_hub_download("vector-institute/mars", "model.onnx"), # or "model.fp16.onnx" for less GPU memory
providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
)
meta = session.get_modelmeta().custom_metadata_map
T, STRIDE, S = (int(meta[key]) for key in ("n_frames", "frame_stride", "image_size"))
def to_input(rgb: np.ndarray) -> np.ndarray:
"""(H, W, 3) uint8 RGB to (3, S, S) float32 in [-1, 1]."""
x = rgb.astype(np.float32) / 255.0
if x.shape[:2] != (S, S):
channels = [Image.fromarray(np.ascontiguousarray(x[..., c])).resize((S, S), Image.BICUBIC) for c in range(3)]
x = np.clip(np.stack([np.asarray(c) for c in channels], axis=-1), 0.0, 1.0)
return ((x - 0.5) / 0.5).transpose(2, 0, 1)
def score(path: str) -> float:
"""Mean P(manipulated) over the clips of one video."""
with av.open(path) as container:
kept = [to_input(f.to_ndarray(format="rgb24")) for i, f in enumerate(container.decode(video=0)) if i % STRIDE == 0]
probs = []
for start in range(0, len(kept), T):
clip = kept[start : start + T]
n_valid = len(clip)
clip = clip + [clip[-1]] * (T - n_valid)
inputs = {"frames": np.stack(clip)[None].astype(np.float32), "n_valid": np.array([n_valid], dtype=np.int64)}
probs.append(float(session.run(["fake_prob"], inputs)[0][0]))
return float(np.mean(probs))
print(score("video.mp4"))
Neither ONNX file runs in bfloat16 like the evaluation behind Results, so scores differ
slightly from that table. To use the GPU, ONNX Runtime needs CUDA and cuDNN; if PyTorch with
CUDA is installed, import torch before creating the session also works.
Training
| training data | FaceForensics++ (c23; original videos and Deepfakes, Face2Face, FaceSwap, NeuralTextures; official splits) and GTA-V (rendered sequences from SAIL-VOS 3D, all synthetic) |
| validation | FaceForensics++ validation split; the checkpoint is the epoch with the best video AUROC (the last of 3) |
| clips | 64 frames, every second frame, full frames at 336 x 336 |
| encoder | vit_pe_core_large_patch14_336.fb, frozen; the semantic path tunes its own LayerNorm copies |
| optimization | AdamW, learning rate 1e-4, weight decay 1e-4, batch 32 clips, step decay x0.5 every 1,000 steps, 3 epochs, bfloat16, seed 0 |
| hardware | 4 x NVIDIA A40 |
Results
Video-level AUROC and average precision (%), trained on FaceForensics++ and GTA-V. A video's score is the mean over its clips.
| test set | videos | AUROC | AP |
|---|---|---|---|
| FaceForensics++ (test split) | 700 | 98.98 | 99.74 |
| Celeb-DF v2 | 518 | 91.22 | 94.73 |
| DeepfakeTIMIT | 960 | 99.95 | 99.97 |
| UADFV | 98 | 96.58 | 97.29 |
| HiFiFace (paired with FF++ targets) | 280 | 84.05 | 85.95 |
| GenVideo (validation split, 10 generators) | 18,286 | 93.44 | 92.60 |
| DF26 (10 generators) | 2,691 | 75.81 | 95.76 |
| average | 91.43 |
License
The weights in this repository are released under Apache 2.0. The frozen encoder is Meta's Perception Encoder (PE-Core-L), downloaded from timm under its own license (Apache 2.0).
Citation
@misc{radwan2026marsmultiscaleappearanceresidualstreams,
title={MARS: Multiscale Appearance-Residual Streams for Video Forgery Detection},
author={Ahmed Y. Radwan and Shaina Raza},
year={2026},
eprint={2611.12508},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2611.12508},
}
- Downloads last month
- 6
Model tree for vector-institute/mars
Base model
timm/vit_pe_core_large_patch14_336.fb