thesteven123/dfd-detector

DFD research detectors (general image, AI-face, face-swap video)

Three calibrated deepfake / AI-generated-media detectors from a research demo, served together behind a conservative router. Research use only. Not for enforcement decisions. Uncertain cases are meant to go to human review, and every model here sends a large share of inputs there by design.

Folder	What it detects	Input
general_all/	AI-generated / synthetic whole images	image (224 px centre crop)
face_t2i_b3/	AI-generated (text-to-image) faces	largest detected face, 1.8x square crop
video_v3/	Face-swap videos	8 time-sampled face crops per clip (YuNet, 1.3x square)

Each folder holds best.pt (LoRA + head weights only; the frozen backbone is facebook/dinov2-base), calibration.json (temperature + split-conformal thresholds), config.yaml, and the training/eval reports.

Architecture

• Backbone: DINOv2-base (frozen) with LoRA (rank 8) on the last 3 blocks; features = layer-normed CLS + mean patch token from layers 4, 8, 10, 12.
• Head: variational information bottleneck (latent 128) + linear classifier.
• Video: a temporal adapter over per-frame features, 8 frames per clip.
• Training augmentation: platform transcoding emulation (resize, JPEG/WebP re-compression, blur, noise, colour jitter, beauty-filter simulation) to survive social-media processing.
• Calibration: temperature scaling + split-conformal thresholds fit only on a held-out calibration split (target FPR 1%, per-generator FNR 2%). Output is three-way: fake (p >= tau_fake), real (p < tau_real), otherwise abstain (route to review). If the calibration file is missing or was fit for a different crop mode, every verdict must be treated as abstain.

Results

All numbers come from the reports in this repo or the project build log. "Single run" means one training run with no confidence interval.

general_all (Tiny-GenImage, all 7 generators)

• Generalization is measured with the same recipe under nested leave-one-generator-out (7 rotations; the test generator is never seen in training): mean AUC 0.915, mean TPR@1%FPR 0.42; conformal FPR / FNR / abstain about 1.2% / 6.6% / 45%. The weakest unseen generator is Midjourney (AUC 0.741).
• This exact checkpoint is trained on all 7 generators, so its own validation numbers are in-distribution (7,000 images): AUC 0.963, TPR@1%FPR 0.61, conformal FPR 0.97%, abstain 42%. [single run]
• A pre-registered ablation (attention-MIL patch pooling) did not improve generalization (mean dAUC -0.0002, 95% CI [-0.0028, +0.0023]).

face_t2i_b3 (AI-generated faces)

Trained on DeepFakeFace (IMDB-WIKI reals with SD1.5 / inpainting / InsightFace fakes), SFHQ-T2I (SDXL; DALL-E 3 used only for checkpoint selection) and FFHQ real portraits. Flux was held out entirely. [single run]

• Held-out Flux: AUC 0.988, TPR@1%FPR 0.85; SFHQ-val Flux dev / pro / schnell caught 86% / 91% / 93%.
• FairFace (10,954 real faces): FPR 0.58% overall, review burden 18.7%. Worst race: Middle Eastern 0.99%; worst race x gender: Middle Eastern male 1.35%; ages 0-2 / 3-9: 0.5% / 0.4%. No race group is significantly above 1% (exact binomial + Holm); Wilson 95% CIs are in eval_external_fairface.json.
• Not a face-swap detector: WildDeepfake frames are at chance (AUC 0.53) and 41% of in-the-wild face-swap frames get a confident real. In-the-wild real-face FPR is 2.4% on WildDeepfake, above the 1% target.

video_v3 (face-swap video; WildDeepfake + Celeb-DF v2)

Warm-started from an earlier WildDeepfake-only model; Celeb-DF fakes capped at 2:1 per target identity. Checkpoint chosen on validation only; each test split scored once. [single run]

Test split	AUC	TPR@1%FPR	Real called fake	Fake called real	Abstain
WildDeepfake official test (806 clips)	0.877	0.283	0.5%	1.0%	75.4%
Celeb-DF v2 official test list (518 clips)	0.845	0.171	1.1%	0.0%	83.4%
Talking-head animation (TalkingHeadBench) vs CelebV-HQ reals (1,179)	0.441	0.007	0.0%	41.4%	60.4%

• Celeb-DF's official split shares swapped-in identities with train, so part of the Celeb-DF score is in-domain.
• Does not detect talking-head animation (below-chance AUC; 41% of those fakes score real).

Intended use and how it is served

Research and demonstration of calibrated, abstaining deepfake detection under social-media processing. In the demo, a router runs the general detector on the whole image and the face detector on the largest face (at least 64 px and 1% of the image). Videos also get the clip model when a face is found in at least half the sampled frames. Fusion is conservative: fake if any specialist says fake; real only if every specialist that ran says real; otherwise review. Fusing can only raise the false-positive rate (up to about the sum of the specialists' FPRs).

Out of scope

• Automated takedowns, legal, journalistic or identity decisions; any use without human review.
• Talking-head / audio-driven animation, modern text-to-video (Sora/Kling/Wan-class), audio deepfakes.
• Real-world face swaps in still images (no model here handles them).
• Commercial use (see licences).

Training data and licences

Data	Use	Licence / terms
Tiny-GenImage (TheKernel01/Tiny-GenImage)	general model	CC-BY-NC-SA 4.0
DeepFakeFace (OpenRL/DeepFakeFace)	face model	Apache-2.0; IMDB-WIKI reals research-only
SFHQ-T2I	face model	MIT (source repository)
FFHQ (gaunernst/ffhq-1024-wds)	face model	CC-BY-NC-SA 4.0
WildDeepfake	video model	Apache-2.0
Celeb-DF v2	video model	non-commercial research only, no redistribution, citation required
FairFace, TalkingHeadBench, CelebV-HQ	evaluation only	CC-BY 4.0 / MIT / research terms

Because several sources are non-commercial (and FFHQ / Tiny-GenImage are ShareAlike), these weights are released under CC-BY-NC-SA 4.0 and must not be used commercially. No dataset media is redistributed here.

Usage

The inference code (dfd package: dfd.inference.pipeline.DetectionPipeline, dfd.inference.router.MultiDetector / VideoClipScorer) lives in the project repository.

from dfd.inference.pipeline import DetectionPipeline
from PIL import Image

pipe = DetectionPipeline("general_all/best.pt", None, "auto", crop_mode="center")  # reads calibration.json beside it
res = pipe.score_pil(Image.open("photo.jpg").convert("RGB"))
print(res.verdict, res.p_fake)  # "fake" / "real" / "abstain"

Checkpoints load with torch.load(..., weights_only=True); they contain only tensors and plain metadata.

Limitations

• Most face and video results are single runs; only the image LOGO protocol has spread across rotations.
• Abstention is high (42-83%), which is the price of the 1% FPR target.
• Calibration was fit on curated datasets; in-the-wild real content (low-resolution, re-compressed video faces) can exceed the 1% FPR target, as WildDeepfake real faces show for the face model (2.4%).
• New generators appear constantly; thresholds need recalibration and the models need retraining.