Delving into Sequential Patches for Deepfake Detection
Anonymous Author(s)
AffiliationAddress email
Abstract
Recent advances in face forgery techniques produce nearly visually untraceabledeepfake videos, which could be harmful to society with malicious intentions.
Totackle this problem, researches have been devoted to deepfake detection.
While theimportance of local low-level cues and temporal information has been verified inthis task, previous methods struggle in achieving generalizability across deepfakemethods and robustness towards image post-processings.
In this work, we proposethe L ocal- & T emporal-aware T ransformer-based Deepfake D etection ( LTTD )framework, which adopts a local-to-global learning protocol with a particular focuson the valuable temporal information within local sequences.
Specifically, wepropose a Local Sequence Transformer (LST), which models the temporal consistency on sequences of restricted spatial regions, where low-level informationis hierarchically enhanced with shallow layers of learned 3D filters.
Based on thelocal temporal embeddings, we then achieve the final classification in a global contrastive way.
Extensive experiments on popular datasets validate that our approacheffectively spots local forgery cues and achieves state-of-the-art performance.
Introduction
With the development of face forgery methods [ the enormous fake videos(a.k.a deepfakes) on the Internet have raised non-neglectable concerns on the privacy preservation andinformation security.
For example, we have witnessed the on-going misinformation spreading 1 .
Tothis end, researches have been devoted to the reliable tagging of deepfakes to block the propagationof malicious information.
However, it is still an open problem due to the limited generalization abilityof detection methods and the continuous advances in deepfake creation.
Earlier studies [7, 32, 4, 14, 15, 60, 42, 66] devote efforts to enhance general Convolutional Neural
Networks (CNN) for identifying clear semantic distortions in deepfakes.
Recent methods payattention to the temporal inconsistency problem [6, 31, 21, 54, 46, 56, 45, 5], yet most of them fallinto semantic motion understanding (e. g., abnormal eye blinking, phoneme-viseme mismatches,aberrant landmark fluctuation).
These methods are able to learn specific forgery patterns, however,the remarkable visual forgery cues are expected to be gradually eliminated during the continuousarmy race between forgers and detectors.
As a result, the generalizability of previous methods istypically unsatisfactory when encountering deepfakes generated by unseen techniques.
On the other hand, the importance of low-level information is identified for tackling the generalizationproblem.
A group of studies are developed using hand-made low-level filters (e. g., DCT [41, 20, 27],steganalysis features [63], SRM [36]) to better capture subtle differences between generated texturesand the natural ones. However, methods depending on recognizable low-level patterns would become [Table] war
[Table]
https://www.snopes.com/fact-check/
[Table] russian- [Table] less effective on degraded data with commonly applied post-processing procedures like visualcompression [35, 21, 62], which indicates the lack of robustness .
As a result, we rethink the appropriate representation that can ensure both generalizability androbustness in deepfake detection.
We take the inspiration from Liu et al.
[35], which indicatesthat deepfakes can be directly distinguished from pieces of skin patches.
Containing little uniformsemantics, networks are forced to look into the low-level details within local patches, which preventsoverfitting to global semantic cues, making learned patterns more generalizable .
In addition, as thecreation of deepfakes inevitably relies on frame re-assembling, the substantial temporal differencesalso arise locally during the independent local modifications of every forged frame.
Such temporalinconsistency will not be easily erased during common perturbations, as the underlying temporalpatterns are less affected by spatial interference, making the low-level modeling more robust .
Thus in this work, we propose the L ocal- & T emporal-aware T ransformer-based Deepfake D etection( LTTD ) framework, which particularly focuses on patch sequence modeling in deepfake detectionwith Transformers [18].
Detailedly, we divide the 3D video information spatially into independentlocal regions, which can be referred to as “Group Patch Sequences” (Fig. 1).
Encouraged by therecent success of vision transformer [18], we formulate the patch sequence modeling problem ina self-attention style and propose a Local Sequence Transformer (LST) module.
Benefiting fromthe attention mechanism, LST is not constrained by the receptive field in CNNs, allowing for betterlearning of both long and short span temporal patterns.
In addition to that, we hierarchically injectlow-level results from shallow 3D convolutions after self-attention layers to enhance the low-levelfeature learning at multiple scales.
After feature extraction of the LST, we take the temporal tokensof all local regions to predict the final result using the proposed Global Contrastive Classification(GCC) module and also explicitly impose local contrastive supervision using temporal-pooled patchembeddings with a Cross-Patch Inconsistency (CPI) loss.
Ablations on these designs show non-trivialimprovements.
Our contributions can be summarized as follows: 1) We propose to address deepfake detectionfrom the perspective of low-level temporal pattern learning, which enjoys better generalizability androbustness.
We propose the Local- & Temporal-aware Transformer-based Deepfake Detection(LTTD) framework, which adopts a local-to-global learning protocol with a focus on the temporalinformation within patch sequences.
3) Quantitative experiments show that our approach achievesthe state-of-the-art generalizability and robustness.
And qualitative results further illustrate theinterpretability of our approach.
Related work
2.1 Deepfake detection
In recent years, we witness great progress in deepfake detection, where numerous forgery spottingmodels are proposed successively to address the practical demands of the application.
In the earlierstage, methods [1, 2, 11, 23, 24, 26, 64] are built with a major emphasis on spotting semantic visualartifacts with sophisticated model designs.
Dang et al.
[14] design a segmentation task optimizedwith
the classification backbone simultaneously to predict the forgery regions.
Zhao et al.
[60]take the deepfake detection task as a fine-grained image classification [58] problem, and enforce themodel to attend to widespread potential forgery regions.
While they both achieve plausible resultswhen tested in data within the same distribution of train set, the cross-dataset evaluation shows poorgeneralizability.
Nevertheless, generalizability is considered to be the first point should it be designed for practicalapplication scenarios, since we never have a chance to foresee the attackers’ movements.
More works[51, 10, 44] begin to notice the vital problem.
Wang et al.
[51] argue that the lack of generalizabilityis due to overfitting to significant semantic visual artifacts, and propose a dynamic data argumentationschema to relieve the issue.
But considering the CNN always takes the downsampled semanticrepresentation for final classification, their method tends to only focus on more semantic visualartifacts.
Another group of works [29, 10, 61, 41, 20, 27, 36, 34, 35, 9] dig deeper into the fundamentaldifferences of deepfakes from the generation process.
With the expectation that forgery methodswill gradually improve, those works propose to identify deepfakes from low-level image featuresrather than semantic visual clues, as the latter are disappearing in the latest deepfakes.
[29, 10, 61]both notice the content-independent low-level features that can uniquely identify their sources and the identity swapping will destroy the origin consistency.
Li et al.
[29] propose to detect thoselow-level features across facial boundary and Zhao et al.
[61] and Chen et al.
[10] turn to learnthe spatial-local inconsistencies.
Other methods look for clues in the frequency domain.
With DCT transform, [41, 20, 27] fuse the low-level frequency pattern learning into CNN to improvethe generalizability.
Liu et al.
[34] instead theoretically analyze that phase information are moresensitive to upsampling, therefore, such low-level feature is more crucial than high-level semanticinformation for our task.
The finds from [35] further encourage us to detect deepfakes from localpatches, where the generated images are easy to be isolated even only one small skin patch is given.
However, there is still a vital issue that low-level features should be even less robust to real-worlddistortions.
Experiments in [35] show that simple smoothing could impair the performance of morethan 20%.
Drastic performance drops of [29] on suspected videos with degradation also indicate theweakness of low-level pattern learning.
Different from current works, we propose to rely on temporallow-level changes with learnable shallow filters to better cope with real-world degradations.
Although temporal methods [6, 31, 21, 54, 45, 62] are also studied, most of they fall into the visualanomalous pattern learning such abnormal eye blinking [31], non-synchronized lip movements [6, 21],inconsistent facial movements [62], etc.
In the foreseeable further or even good cases of currentarts, we could hardly find these significant patterns.
In contrast, we explore the substantial temporalinconsistency of independently frame-wise generation at local patches, which is more generalizableand also more robust to common perturbations, as the underlying temporal patterns are less affected.
2.2 Vision transformer
Vaswani et al.
[50] first propose to using only self-attention, multilayer perceptron ( MLP ), and layernorm to establish a new canonical form, coined transformer, for natural language processing (NLP).
Promoted by the great advancement in NLP, researchers of vision communities also start to explore thepotential of transformer designs.
ViT [18] could be the first success to apply pure typical transformerin image classification.
After that, many researches also extend transformer into different vision tasks,(e.
g., semantic segmentation [19], action recognition [40, 30], video understanding [8, 59], scenegraph generation [13], object detection [65, 55, 38]) and achieve exceptional performance.
Recently, transformer structure is also adopted by a few works [62, 17, 52] in deepfake detection.[52] stack multiple ViTs [18] with different spatial embedding size.
With the motivation to learnidentity, transformer with extra token is proposed in [17].
Besides, The closest work to us, [62] adoptfew self-attention layers to aggregate successive frames’ features, which are completely downsampledsemantic embeddings.
While in our method, transformer is introduced to achieve patch-sequentialtemporal learning in a restricted spatial receptive field with a totally different purpose to identifylow-level temporal inconsistency.
Approach
In this section, we will elaborate our proposed L ocal- & T emporal-aware T ransformer-based Deepfake D etection ( LTTD ) framework after formally define the problem in section 3.1.
We describe
Local Sequence Transformer ( LST ) in section 3.2.
Then, the Cross-Patch Inconsistency ( CPI ) loss isintroduced with the Global Contrastive Classification ( GCC ) in section 3.3 and end up with details ofmodel optimization in section 3.4.
3.1 Problem statement
Given one image x ∈ R C × H × W , we spatially split it into local patches { x ip ∈ R C × P 2 | i =1 , 2 , ..., N } , where C is the number of image channels, P is the local patch size, and N = HW/P 2is the patch number.
In the original setting of ViT [18], the separated patches can be directlysent into the Transformer blocks for semantic understanding.
Differently, in our task, we takea video clip v ∈ R C × T × H × W as input, where the extra T indicates the clip length, i. e., with T successive frames.
We also split each frame into independent patches for better low-levelforgery pattern learning, but with an additional temporal dimension.
The patch set is represented ass = { x t,ip ∈ R C × P 2 | t = 1 , 2 , ..., T ; i = 1 , 2 , ..., N } .
Thus the total patch number becomes T × N .
After that, our proposed LTTD leverages the patch locality with simple learnable filters and powerful temporal token after LST … … . . .
LST LST
Share Weights
Sequence Length
W i d t h
H i gh
Num. of Patch
Sequences … … …
RealorFake
Group Patch Sequences
LST forward
CPI&GCC … 1 2 *
Transformer Encoder Layer
Conv3d&Maxpooling
LET-stage
Flatten&Linear …
Flatten&Linear
Conv3d&Maxpooling
Flatten&Linear
Local Sequence Embedding
Data Flow patch embedding temporal
[Figure]
temporal token
CPI &
GCC . . .
Temporal Avgpooling . . .
TransformerEncoder Layer . . .
[class]
FC
Real or Fake . . .
[class]
GCC . . .
[Figure] [Figure]
[Figure]
self-attention operations to explore the low-level temporal inconsistency of deepfakes, and finallygive a prediction (fake or real) based on global contrast across the whole spatial region.
3.2 Local Sequence Transformer
As we discussed previously, in order to learn the low-level temporal patterns, we feed the local patchesat the same spatial position into the proposed Local Sequence Transformer (LST) for further temporalencoding, i. e., the input of our LST is a patches set like s i
= { x t,ip ∈ R C × P 2 | t = 1 , 2 , ..., T } , wherei ∈ { 1 , 2 , ..., N } , and s i includes frame patches at the spatial location of i .
We show the details of the LST in the middle of Fig.
1, which is divided into two parts: the Local
Sequence Embedding and the Low-level Enhancement Transformer stages.
In both parts, low-leveltemporal enhancement is consistently introduced by shallow 3D convolutions, which is mainly comingfrom two thoughts: 1) voxels at one specific spatial location maybe not well aligned consideringpossible camera shaking, and the 3D filter with the inductive bias of 3D neighborhood structure isable to well handle this kind of situations; 2) the initial patch embedding always projects the patch asa whole without more fine-grained locality emphasis, we make up for this using shallow convolutionsto enhance low-level feature learning at multiple scales.
And the details will be formally defined later.
Local Sequence Embedding .
In addition to commonly used linear patch embedding, we combine3D convolutions at the first beginning to enhance later low-level temporal modeling.
More formally,we define this part as follows:
[Equation]
where in Eq.
(1), E s is a trainable linear projection with dimension of D , and so is E 0 s in Eq.
(3), butthe difference is that E 0 s acts on patches after temporal filtered in Eq.
The k in Eq.
(2) denotes thepooling kernel size, which is always set to 2 for more efficient training, and the pooling also plays arole for multi-granularity locality emphasis as discussed previously.
And C t in Eq.
(3) is the numberof the used temporal filters in Conv3d , which is set to 64.
Finally, the embedding is given by Eq.
(4),where x itemp is a learnable temporal token, σ indicates sigmoid function, z t,i
=
zs i ( t ) · σ ( zt i ( t ))represents the patch sequence embedding of spatial location i at timestamp t , and learnable positionembedding E pos is also kept, but with different meaning of temporal indicator.
Low-level Enhancement Transformer stage .
Then, patch sequence embeddings at different spatiallocations are independently fed into multiple Low-level Enhancement Transformer stages (LETstages) for further temporal modeling.
Similar to the 3D filters we used before, we propose to enhancethe temporal modeling of patch sequence with aids of shallow spatial-temporal convolution at multiplescales.
Given the input embeddings at stage- l as z il =
[ z itemp ; z 1 ,i ; z 2 ,i ; ... ;
z T,i ] , with multiheadedself-attention ( MSA ) [18], MLP , and Layernorm ( LN ), we formally define one LET-stage as:
[Equation]
where En in Eq.
(5) denotes one Transformer block [18], and En 3 in Eq.
(6) indicates cascadedstacking three layers together.
Overall, the proposed LST is formed by a Local Sequence Emebeddingstage and three cascaded Low-level Enhancement Transformer stages.
Discussion .
Compared with our designs, one straight thought might be “just leave the work toself-attention”, since theoretically patches can progressively find the most relative patches at the samespatial room for temporal modeling.
However, this is nearly impracticable considering both short andlong span temporal information is important to our task [62] and the self-attention operation has aquadratic complexity with respect to the number of patches.
In contrast, by independently modelingthe patch sequences with weight-shared LST, we not only reduce the time complexity of the straightthrought from O ( T 2 · N 2 ) to O ( T · N 2 ) , but also explicitly avoid semantic modeling of features likefacial structure and always focus on low-level temporal learning.
3.3 Cross-Patch Inconsistency loss and Global Contrastive Classification
After modeling the temporal relation of sequential patches in LST, we have a set of temporalembeddings of all spatial locations { z i | i = 1 , 2 , ..., N } .
However, how to give a final decisionis still nontrivial, since different spatial regions should represent different temporal features withdifferent local changes, e. g., pixel variation in the background region is usually less dramatic thanthat of the mouth region.
In addition, there normally exist real regions (usually the background) indeepfakes.
Thus, directly adopting an overall binary classification loss on all patches is not the bestchoice.
Instead, we propose to identify the inconsistency by global contrast, because forgery partsshould retain heterogeneous temporal pattern changes compared with the real ones.
Concretely, weachieve this goal through the Cross-Patch Inconsistency loss and the proposed Global Contrastive
Classification.
Cross-Patch Inconsistency loss .
Given the feature set after LST { z i | i = 1 , 2 , ..., N } , we leave thefirst temporal token at all spatial locations ( { z itemp ∈ R D | i = 1 , 2 , ..., N } ) for latter classification,and use the remaining patch embeddings ( { z t,i ∈
R D | t
= 1 , 2 , ..., T ; i = 1 , 2 , ..., N } ) in this part.
We first reduce the temporal dimension of all local regions as:
[-Tt =
 z t,i .-][Equation]
Then we calculate the dense cosine similarities of all regions as:
[Equation]
According to the simplest thought that real region should be similar to the real one to some extent,and vise versa.
We propose to impose a intra-frame contrastive supervision as:
[Equation]
where µ is a tolerance margin which we set it to 0.1, that allows more diverse feature representationsin binary classes.
And sim gt ∈ R N × N is the ground truth similarity matrix that generated from themodification mask sequence m o ∈ R T × H × W as follows:
[Equation]
where π indicates interpolation operation. Note that the modification mask should be normalize torange of (0 , 1) in advance.
As for the real clip without mask, the sim gt = 1 N × N .
Global Contrastive Classification .
For the final classification, we want to determine the finalprediction based on the overall consistency.
With only binary label supervision, here we again askself-attention for help, and insert a final class token x class ∈ R D in to earlier LST output temporaltokens as [ x class ; z 1 temp ; z 2 temp ; ..., z Ntemp ] .
Based on those location-specific temporal tokens, weadopt additional three Transformer blocks (similar to Eq.
(6)) to achieve global location-contrastivemodeling.
Then the final prediction is given by a fully connection layer taking the [ class ] embeddingas input.
For the classification loss, we simply use the binary cross-entropy loss denoted as L BCE .
3.4 Training details
And finally, we can train our LTTD in an end-to-end manner with the weighted sum of the two lossesas:
[Equation]
14) where the parameter λ is used to balance the two parts and set to 10 − 3 .
The spatial input size H × Wand patch size P is set to 224 × 224 and 16, respectively.
It is worth noting that the division into suchsmall patches has considerably suppress the overall semantic features.
The embedding dimension D is set to 384. For temporal dimension T , we set it to 16. For optimizer, we use Adam [25] withthe initial learning rate of 10 − 4 .
When the model performance no longer improves significantly, wegradually decay the learning rate until convergence.
Four NVIDIA A100 GPUs are used.
Experiments
4.1 Setups
Datasets .
Our experiments are conducted based on several popular deepfake datasets including
FaceForensics++ ( FF++ ) [42], DeepFake Detection Challenge dataset ( DFDC ) [16], CelebDF-V2( CelebDF ) [33], FaceShift dataset ( FaceSh ) [28], and DeeperForensics dataset ( DeepFo ) [22]. FF++ is one of the most widely used dataset in deepfake detection, which contains 1000 real videoscollected from Youtube and 4000 fake videos generated by four different forgery methods including
Deepfakes [1], FaceSwap [2], Face2Face [48] and NeuralTextures [47]. To simulate the real-worldstream media environment, FF++ also provides three versions with different compression rates, whichare denoted by raw (no compression), HQ (constant rate quantization parameter equal to 23), and LQ(the quantization parameter is set to 40), respectively. Based on the 1000 real videos of FF++, FaceShis a later published dataset containing 1000 fake videos, which are generated by a more sophisticatedface swapping technique.
DeepFo is a large-scale dataset for real-world deepfake detection. Toensure better quality and diversity, the authors make the source videos in a controlled scenario withpaid actors.
More impressively, a new face swapping pipeline considering temporal consistencyis proposed to generate deepfakes with more “natural” low-level temporal features.
DFDC is amillion-scale dataset used in the most famous deepfake challenge [3]. Following previous works[21], we use more than 3000 videos in the private test set for cross-dataset evaluation in this paper. In addition, CelebDF is one of the most challenging dataset, which is generated using an improveddeepfake technique based on videos of celebrities.
Data preprocessing .
All the used datasets are published in a full-frame format, thus, most of thedeepfake detection methods will crop only the face regions in advance as input.
However, with amotivation to learn the low-level temporal patterns, cropping face regions in advance will lead toartificially introduced shaking due to the independent face detection.
Therefore, in our method, wecrop the face regions using the same bounding box after randomly determine the clip range in-the-fly,where the box is detected by MTCNN [57].
Evaluation metrics .
Following previous works [42, 21, 29], we use binary classification accuracy(ACC) and Area Under the Receiver Operating Characteristic Curve (AUC) as evaluation metrics.
4.2 Generalizability evaluation
For practical application, generalizability should be one of the most concerned properties.
Nevertheless, it is usually the Achilles’ heel of most deepfake detectors, since deepfakes generated bydifferent forgery methods (in different datasets) hold different kinds of forgery cues, and overfittingon semantic visual artifacts of the train set can easily lead to cross-dataset evaluation collapse.
As weshow the performance of models trained on FF++ HQ and tested on four unseen datasets in Table.
1,many methods do not perform satisfactorily.
In contrast, our approach outperforms all the recentlypublished novel detectors, and achieves a new state-of-the-art of 91.8 AUC% averaged from the fourdatasets.
Note PatchForensics [9] also focuses on local patches, but only narrowed the perceptivefield by truncating the CNN without considering the relations between patches globally, it showslimited generalizability.
In addition, FTCN-TT reports comparable results, where they also leveragethe self-attention mechanism.
But different from our low-level temporal view, they use the semanticfeatures of the whole frame for prediction.
Thus, when the visual artifacts are less distinguishable in CelebDF and DFDC, our method surpasses it by a clear margin.
A similar situation in LipForensics,which is proposed to leverage pretrain priors [37] of high-level semantic understanding for deepfakedetection, with temporal regularity considered, they also achieve comparable results but show suboptimal performance on CelebDF and DFDC.
Moreover, PCL+I2G and Face X-ray also focus onlow-level learning, but without considering temporal properties, they exhibit insufficient robustnessand perform poorly in DFDC, where the videos are filmed under very different circumstances andhave been perturbed to some extent. [Table]
perturbations
Another vital property for practical application is robustness.
For network transmission, videos arealways compressed.
Also, consider possible attacking against the detectors, we evaluate our approachon different types of perturbed videos.
As previous works [21, 22], we use the script 2 on FF++ and generate seven types of perturbations atfive levels.
As we show the comparison diagram in Fig.
2, the seven perturb methods include colorsaturation change, color contrast change, block-wise noise, gaussian noise, gaussian blur, pixelation,and video compression.
Examples of these perturbations can be found in this site.
[Figure]
Comparing with other methods, our approach shows excellent robustness against most of the perturbations.
As for the type of gaussian noise at a relatively higher level, the underlying low-level patternsare severely disturbed that our method can not work properly.
We also present the precise numbersof two state-of-the-arts in Table.
2, where the results are averaged from all five levels.
In general,
Face X-ray detects low-level boundary artifacts, and LipForensics focuses on high-level semanticfeatures.
Thus, the former performs better in block-wise noise type and the latter shows better resultsin others.
In contrast to Face X-ray, we detect low-level patterns in spatial-temporal space, which isbetter resistant to perturbations, and thus achieves better robustness evaluation performances. Method [Table] Face
[Table]
ray [29] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
LipForensics [21]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] 4.4 Ablations In this section, we compare our method with related baselines and several alternative designs. 1) Xception is the commonly used backbone in deepfake detection; 2) ViT is the most famous visiontransformer backbone, where we use the “small” version with embedding dimension of 384; 3) LTTDw/
o LST indicates the model that replace the proposed LST with commonly used Patch Embeddingand Transformer blocks [18]; 4) LTTD w/o CPI is our L 3 TD framework trained without using
L CPI (Eq. (13)); 5) LTTD w/o GCC represents the model that we replace the GCC module witha simple fully connected classification layer after average pooling the temporal embeddings fromall spacial locations.
From Table.
3, all the models achieve nearly perfect results on FF++.
Whilein the cross-dataset setting, our model with elaborately designed modules exhibits much strongergeneralizabilty, and the three components consistently boost the best results.
4.5 Visualization
To intuitively demonstrate the reason that our approach is more generalizable, we compare the featurerepresentations before the final classification in Fig.
Models are trained on FF++ HQ and tested on
[Table] four subsets of FF++ (Deepfakes, Face2Face, FaceSwap, and NeuralTextures) respectively.
Due tothe abundant inductive bias of the convolution, Xception clearly split the four kinds of deepfakes intofour clusters, even only binary labels are used in training.
This shows that the strong CNN tendsto overfit on method-specific artifacts generated by different deepfake methods, although plausibleresults are achieved in in-dataset testing, it is harmful to generalize to unseen deepfakes.
In addition,similar phenomena can be found in the ViT results, but due to less inductive bias introduced, thefour clusters are not as clearly divided as Xception.
In contrast, the features of our approach showa completely different outline, where the real videos are also clearly separated, but the remainingfour types of deepfakes are compacted in a unified manifolds.
We attribute this phenomenon toour low-level temporal learning, which can distinguish deepfakes from real depending on morefundamental low-level temporal inconsistencies, thus achieving the best generalizability.
[Figure]
Conclusion and discussion
Conclusion .
In this paper, we propose a reliable framework to address the practical problem ofdeepfake detection, which emphasizes on the low-level temporal patterns of sequential patches in therestricted spatial region, but with a whole-range temporal receptive field using Transformer blocks.
In addition, we make the final classification in a more general global-contrastive way.
Thus bettergeneralizability and robustness are achieved to better support deepfake detection in real-world scenes.
Moreover, qualitative results further verify that low-level temporal information can lead to strongergeneralizability, which could also be a guideline for developing better approaches in the further.
Limitation .
Our approach establishes a new state-of-the-art in deepfake detection based on the findingthat low-level temporal information can be more helpful. But considering the continuous advancesin deepfake creation and adversarial training, the performance of our approach when encounteringlow-level and temporal adversarially enhanced deepfakes in the future is yet unclear.
Ethical Concern .
Our work is proposed to mitigate the potential negative social impacts of otherworks, but there is still a risk of facilitating the forgers to propose better deepfake methods throughadversarial training.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No](b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No]
[Table] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
[Table]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes]
[Table]
(b) Did you mention the license of the assets?
[Yes] The FF++ dataset and FaceSh dataset [Table]
are released under the FaceForensics Terms of Use.
The CelebDF dataset is released
[Table]
under the Terms to Use Celeb-DF.
The use of DeepFo dataset is bounded by the Terms
[Table]
of Use: DeeperForensics-1.0 Dataset.
[Table] (c) Did you include any new assets either in the supplemental material or as a URL?
[No]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[Yes] The datasets we use are face-related open-
[Table]
source datasets that contain personally identifiable information.
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A] [Table]