Delving into Sequential Patches for Deepfake Detection
Anonymous Author(s)
AffiliationAddress email
Abstract
Recent advances in face forgery techniques produce nearly visually untraceabledeepfake videos, which could be leveraged with malicious intentions.
As a result, researchers have been devoted to deepfake detection.
Previous studies haveidentified the importance of local low-level cues and temporal information in pursuit to generalize well across deepfake methods, however, they still suffer fromrobustness problem against post-processings.
In this work, we propose the L ocal-& T emporal-aware T ransformer-based Deepfake D etection ( LTTD ) framework,which adopts a local-to-global learning protocol with a particular focus on thevaluable temporal information within local sequences.
Specifically, we propose a
Local Sequence Transformer (LST), which models the temporal consistency on sequences of restricted spatial regions, where low-level information is hierarchicallyenhanced with shallow layers of learned 3D filters.
Based on the local temporalembeddings, we then achieve the final classification in a global contrastive way.
Extensive experiments on popular datasets validate that our approach effectivelyspots local forgery cues and achieves state-of-the-art performance.
Introduction
With the development of face forgery methods [1, 2, 10, 22, 23, 25, 59] an enormous amount offake videos (a.k.a deepfakes) have raised non-neglectable concerns on privacy preservation andinformation security.
To this end, researches have been devoted to the reliable tagging of deepfakesin order to block the propagation of malicious information.
However, it is still an open problem dueto the limited generalization of detection methods and the continuous advances in deepfake creation.
Earlier studies [6, 31, 3, 13, 14, 55, 41, 61] devote efforts to enhance general Convolutional Neural
Networks (CNN) for identifying clear semantic distortions in deepfakes.
Recent methods payattention to the temporal inconsistency problem [5, 30, 20, 50, 45, 52, 44, 4], yet most of themfall into semantic motion understanding (e. g., detecting abnormal eye blinking, phoneme-visememismatches, aberrant landmark fluctuation).
These methods are able to learn specific forgery patterns,however, the remarkable visual forgery cues are expected to be gradually eliminated during thecontinuous army race between forgers and detectors.
As a result, the generalizability of previousmethods is typically unsatisfactory when encountering deepfakes generated by unseen techniques.
On the other hand, the importance of low-level information is identified for tackling the generalizationproblem.
A group of studies are developed using hand-made low-level filters (e. g., DCT [40, 19, 26],steganalysis features [58], SRM [35]) to better capture subtle differences between generated texturesand the natural ones.
However, methods depending on recognizable low-level patterns would becomeless effective on degraded data with commonly applied post-processing procedures like visualcompression [34, 20, 57], which indicates the lack of robustness .
As a result, we rethink the appropriate representation that can ensure both generalizability androbustness in deepfake detection.
We take the inspiration from Liu et al.
[34], which indicates that deepfakes can be directly distinguished from pieces of skin patches.
Containing little uniformsemantics, networks are forced to look into the low-level details within local patches.
Such practiceprevents network from overfitting to global semantic cues, making learned patterns more generalizable .
In addition, as the creation of deepfakes inevitably relies on frame re-assembling, the substantialtemporal differences also arise locally during the independent local modifications of forged frames.
The underlying temporal patterns are less affected by spatial interference, thus, such local temporalinconsistency will not be easily erased during common perturbations, making the low-level modelingmore robust .
Motivated by the observations above, in this work, we propose the L ocal- & T emporal-aware
T ransformer-based Deepfake D etection ( LTTD ) framework, which particularly focuses on patchsequence modeling in deepfake detection with Transformer [17].
Detailedly, we divide the 3D videoinformation spatially into independent local regions (as shown in Fig. 1).
Encouraged by the recentsuccess of vision transformer [17], we formulate the patch sequence modeling problem in a selfattention style and propose a Local Sequence Transformer (LST) module, which operates on sequencesof local patches.
Benefited from the attention mechanism, LST is not constrained by the receptivefield in CNNs, allowing for better learning of both long and short span temporal patterns.
In addition,we hierarchically inject low-level results from shallow 3D convolutions after self-attention layers toenhance the low-level feature learning at multiple scales.
Such design particularly emphasizes on thelow-level information from a temporal consistency perspective.
After modeling each sequence oflocal patches in LST, two questions remain to be solved: 1) how to model their inherent relationshipsand 2) how to aggregate their information for final deepfake prediction.
We explicitly impose globalcontrastive supervision on patch embeddings with a Cross-Patch Inconsistency (CPI) loss.
Thenthe final predictions is given in the Global Contrastive Classification (GCC) module with follow up
Transformer blocks.
Ablations on these designs show non-trivial improvements.
Our contributions can be summarized as follows: 1) We propose to address deepfake detectionfrom the perspective of low-level temporal pattern learning, which enjoys better generalizability androbustness.
We propose the Local- & Temporal-aware Transformer-based Deepfake Detection(LTTD) framework, which adopts a local-to-global learning protocol with a focus on the temporalinformation within patch sequences.
3) Quantitative experiments show that our approach achieves thestate-of-the-art generalizability and robustness.
Qualitative results further illustrate the interpretabilityof our approach.
Related work
2.1 Deepfake detection
In recent years, we witness great progress in deepfake detection, where numerous forgery spottingmodels are proposed successively to address the practical demands of the application.
In the earlierstage, methods [6, 31, 3, 14, 41, 61] are built with a major emphasis on spotting semantic visualartifacts with sophisticated model designs.
Dang et al.
[13] design a segmentation task optimizedwith
the classification backbone simultaneously to predict the forgery regions.
Zhao et al.
[55] takethe deepfake detection task as a fine-grained image classification problem, and enforce the modelto attend to widespread potential forgery regions.
While these methods achieve satisfied in-datasetresults, the cross-dataset evaluation shows poor generalizability.
Nevertheless, generalizability is considered to be the first point should it be designed for practicalapplication scenarios, since we never have a chance to foresee the attackers’ movements.
More works[48, 9, 43] begin to notice the vital problem.
Wang et al.
[48] argue that the lack of generalizability isdue to overfitting to significant semantic visual artifacts, and propose a dynamic data argumentationschema to relieve the issue.
But considering the CNN always takes the downsampled semanticrepresentation for final classification, their method tends to only focus on more semantic visualartifacts.
Another group of works [28, 9, 56, 40, 19, 26, 35, 33, 34, 8] dig deeper into the fundamentaldifferences of deepfakes from the generation process.
With the expectation that forgery methods willgradually improve, those works propose to identify deepfakes from low-level image features ratherthan semantic visual clues, as the latter are disappearing in the latest deepfakes.
[28, 9, 56] bothnotice the content-independent low-level features that can uniquely identify their sources and theidentity swapping will destroy the origin consistency.
Li et al.
[28] propose to detect those low-levelfeatures across facial boundary and Zhao et al.
[56] and Chen et al.
[9] turn to learn the spatial-local inconsistencies.
Other methods look for clues in the frequency domain.
With DCT transform,[40, 19, 26] fuse the low-level frequency pattern learning into CNN to improve the generalizability.
Liu et al.
[33] instead theoretically analyze that phase information are more sensitive to upsampling,therefore, such low-level feature is more crucial than high-level semantic information for our task.
The finds from [34] further encourage us to detect deepfakes from local patches, where the generatedimages are easy to be isolated even only one small skin patch is given.
However, there is still a vitalissue that low-level features should be even less robust to real-world distortions.
Experiments in [34]show that simple smoothing could impair the performance of more than 20%.
Drastic performancedrops of [28] on suspected videos with degradation also indicate the weakness of low-level patternlearning.
Different from current works, we propose to rely on temporal low-level changes withlearnable shallow filters to better cope with real-world degradations.
Although temporal methods [5, 30, 20, 50, 44, 57] are also studied, most of they fall into the visualanomalous pattern learning such abnormal eye blinking [30], non-synchronized lip movements [5, 20],inconsistent facial movements [57], etc.
In the foreseeable further or even good cases of currentarts, we could hardly find these significant patterns.
In contrast, we explore the substantial temporalinconsistency of independently frame-wise generation at local patches, which is more generalizableand also more robust to common perturbations, as the underlying temporal patterns are less affected.
2.2 Vision transformer
Vaswani et al.
[47] first propose to using only self-attention, multilayer perceptron ( MLP ), and layernorm to establish a new canonical form, coined transformer, for natural language processing (NLP).
Promoted by the great advancement in NLP, researchers of vision communities also start to explore thepotential of transformer designs.
ViT [17] could be the first success to apply pure typical transformerin image classification.
After that, many researches also extend transformer into different vision tasks,(e.
g., semantic segmentation [18], action recognition [39, 29], video understanding [7, 54], scenegraph generation [12], object detection [60, 51, 37]) and achieve exceptional performance.
Recently, transformer structure is also adopted by a few works [57, 16, 49] in deepfake detection.[49] stack multiple ViTs [17] with different spatial embedding size.
With the motivation to learnidentity, transformer with extra token is proposed in [16].
Besides, The closest work to us, [57] adoptfew self-attention layers to aggregate successive frames’ features, which are completely downsampledsemantic embeddings.
While in our method, transformer is introduced to achieve patch-sequentialtemporal learning in a restricted spatial receptive field with a totally different purpose to identifylow-level temporal inconsistency.
Approach
In this section, we will elaborate our proposed L ocal- & T emporal-aware T ransformer-based Deepfake D etection ( LTTD ) framework after formally define the problem in section 3.1.
We describe
Local Sequence Transformer ( LST ) in section 3.2.
Then, the Cross-Patch Inconsistency ( CPI ) loss isintroduced with the Global Contrastive Classification ( GCC ) in section 3.3.
3.1 Problem statement
Given one image x ∈ R C × H × W , we reshape it into a sequence of flattened 2D patches { x i ∈
R C · P 2 | i = 1 , 2 , ..., N } , where C is the number of image channel, P is the patch size, and N =
HW/P 2 is the patch number.
In the original setting of ViT [17], the separated patches can be directlysent into the Transformer blocks for semantic understanding.
Differently, in our task, we take avideo clip v ∈ R T × C × H × W as input, where the extra T indicates the clip length, i. e., with Tsuccessive frames.
We also split each frame into independent patches for better low-level forgerypattern learning, but with an additional temporal dimension.
The flattened patch set is represented ass = { x t,i ∈ R C
· P 2 | t = 1 , 2 , ..., T ; i = 1 , 2 , ..., N } , where x t,i stands for the flattened patch at thei -th spatial region of the t -th frame.
Thus the total patch number becomes T · N .
After that, ourproposed LTTD leverages the patch locality with simple learnable filters and powerful self-attentionoperations to explore the low-level temporal inconsistency of deepfakes, and finally give a prediction(fake or real) based on global contrast across the whole spatial region.
temporal token after LST … … . . .
LST LST
Share Weights
Sequence Length
W i d t h
H i gh
Num. of Patch
Sequences … … …
RealorFake
Group Patch Sequences
LST forward
CPI&GCC … 1 2 *
Transformer Encoder Layer
Conv3d&Maxpooling
LET-stage
Flatten&Linear …
Flatten&Linear
Conv3d&Maxpooling
Flatten&Linear
Local Sequence Embedding
Data Flow patch embedding temporal
[Figure]
temporal token
CPI &
GCC . . .
Temporal Avgpooling . . .
TransformerEncoder Layer . . .
[class]
FC
Real or Fake . . .
[class]
GCC . . .
[Figure] [Figure]
[Figure]
3.2 Local Sequence Transformer
As we discussed previously, in order to learn the low-level temporal patterns, we feed the localpatches at the same spatial position into the proposed Local Sequence Transformer (LST) for furthertemporal encoding, i. e., the input of our LST is a set like
s i = { x t,i ∈
R C · P 2 | t = 1 , 2 , ..., T } , wherei ∈ { 1 , 2 , ..., N } , and s i includes frame patches at the spatial location of i .
We show the details of the LST in the middle of Fig.
1, which is divided into two parts: the Local
Sequence Embedding and the Low-level Enhancement Transformer stages.
In both parts, low-leveltemporal enhancement is consistently introduced by shallow 3D convolutions given three intuitions:1)
Learnable shallow filters would be better than hand-crafted filters [41,20,36] in capturing low-levelinformation in complex situations.
Voxels at one specific spatial location may not be well alignedtemporally considering camera movements.
Thus 3D filters which covers 3D neighborhood structuresare more suitable than 2D ones when handling this kind of situation.
The patch embedding in
Transformers always projects the patch as a whole without more fine-grained locality emphasis.
Wemake up for this using shallow convolutions to enhance low-level feature learning at multiple scales.
Local Sequence Embedding .
In addition to commonly used linear patch embedding, we involve 3Dconvolutions at the beginning to enhance low-level temporal modeling.
More formally, we define thispart as follows:
[Equation]
where in Eq.
(1), E s is a trainable linear projection with dimension of D , and so is E 0 in Eq.
(3), butthe difference is that E 0 acts on patches after temporal filtered in Eq.
(2), i.e., the enhanced patch representations at the initial stage y t,i 0 .
The k in Eq.
(2) denotes the pooling kernel size, which isalways set to 2 for multi-granularity locality emphasis as discussed previously.
The the poolingalso leads to more efficient training.
And C t in Eq.
(3) is the number of the used temporal filters in
Conv3d , which is set to 64.
Finally, the embedding is given by Eq.
(4), where x itemp is a learnabletemporal token, σ indicates sigmoid function, z t,i
=
zs i 0 ( t ) · σ ( zt i 0 ( t )) represents the patch sequenceembedding of spatial location i at timestamp t , and learnable position embedding E pos is also kept,but with different meaning of temporal indicator.
Low-level Enhancement Transformer stage .
Then, patch sequence embeddings at different spatiallocations are independently fed into multiple Low-level Enhancement Transformer stages (LETstages) for further temporal modeling.
Similar to the 3D filters we used before, we propose toenhance the temporal modeling of patch sequence with aids of shallow spatial-temporal convolutionat multiple scales.
Given one Transformer block ( Trans ) [17] defined as:
[Equation]
where MSA and LN denote multiheaded self-attention and LayerNorm, respectively.
With the inputembeddings at stage l − 1 : z il − 1 =
[ z itemp ; z 1 ,i ; z 2 ,i ; ... ;
z T,i ] , and the enhanced patch representations:[ y 1 ,il − 1 ; y 2 ,il − 1 ; ... ; y T,il − 1 ] , we formally define one LET-stage as:
[Equation]
z
[Equation]
where Trans 3 in Eq.
(6) indicates cascaded stacking three blocks together.
Overall, the proposed
LST is formed by a Local Sequence Emebedding stage and three cascaded Low-level Enhancement
Transformer stages.
Discussion .
Compared with our designs, one straight thought might be “just leave the work toself-attention”, since theoretically patches can progressively find the most relative patches at the samespatial room for temporal modeling.
However, this is nearly impracticable considering both short andlong span temporal information is important to our task [57].
Also, the self-attention operation has aquadratic complexity with respect to the number of patches.
In contrast, by independently modelingthe patch sequences with weight-shared LST, we not only reduce the time complexity of the straightthrought from O ( T 2 · N 2 ) to O ( T · N 2 ) , but also explicitly avoid semantic modeling of features likefacial structure.
3.3 Cross-Patch Inconsistency loss and Global Contrastive Classification
After modeling the temporal relation of sequential patches in LST, we have a set of temporalembeddings of all spatial locations { z i | i = 1 , 2 , ..., N } .
However, how to give a final decision isstill nontrivial, since the embeddings at different spatial location would represent different levels oftemporal changes.
For example, pixel variation in the background region is usually less dramaticthan that of the mouth region.
In addition, there normally exist non-edited regions (usually thebackground) in deepfakes.
Thus, directly adopting an overall binary classification loss on all patchesis not the best choice.
Instead, we propose to identify the inconsistency by global contrast, becauseforgery parts should retain heterogeneous temporal patterns compared with the real ones.
Concretely,we achieve this goal through the Cross-Patch Inconsistency loss and the proposed Global Contrastive
Classification.
Cross-Patch Inconsistency loss .
Given the feature set after LST { z i | i = 1 , 2 , ..., N } , we leave thefirst temporal token at all spatial locations ( { z itemp ∈ R D | i = 1 , 2 , ..., N } ) for latter classification,and use the remaining patch embeddings ( { z t,i ∈
R D | t
= 1 , 2 , ..., T ; i = 1 , 2 , ..., N } ) in this part.
We first reduce the temporal dimension of all local regions as: f i = 1 T (cid:80)
Tt =
z t,i .
Then we calculatethe dense cosine similarities of all regions as:
[Equation]
The sequence of real regions will certainly depict a “natural” variation, while the sequence of fakeregions formed by re-assembling will be different.
According to the simplest thought that temporalfeatures of real regions should be similar to the real ones, and vice versa.
We propose to impose aintra-frame contrastive supervision as:
[Equation]
where µ is a tolerance margin which we set it to 0.1, that allows more diverse feature representationsin binary classes.
And sim gt ∈ R N × N is the ground truth similarity matrix that generated from themodification mask sequence m o ∈ R T × H × W as follows:
[Equation]
The modification masks m o are generated by simply subtracting the fake frame from the correspondingreal one, which should be normalize to range of (0 , 1) in advance.
Thus, the value range of sim gt isconsistent with cosine similarity, i.e., ( − 1 , 1) .
As for the real clip without mask, the sim gt = 1 N × N .
Global Contrastive Classification .
For the final classification, we want to determine the finalprediction based on the overall consistency.
We first insert a final class token x class ∈ R D in toearlier LST output temporal tokens as [ x class ; z 1 temp ; z 2 temp ; ..., z Ntemp ] .
Based on those locationspecific temporal tokens, we then adopt additional three Transformer blocks (similar to Eq.
(6)) toachieve cross-patch temporal information aggregation.
Then the final prediction is given by a fullyconnection layer taking the [ class ] embedding as input.
With such practice, our model is empoweredto identify the different correlations between real and fake sequences.
For the classification loss, wesimply use the binary cross-entropy loss denoted as L BCE .
And finally, we can train our LTTD in an end-to-end manner with the two losses as:
[Equation]
(14) where the parameter λ is used to balance the two parts and empirically set to 10 − 3 .
Experiments
4.1 Setups
Datasets .
Our experiments are conducted based on several popular deepfake datasets including
FaceForensics++ ( FF++ ) [41], DeepFake Detection Challenge dataset ( DFDC ) [15], CelebDF-V2( CelebDF ) [32], FaceShift dataset ( FaceSh ) [27], and DeeperForensics dataset ( DeepFo ) [21]. FF++(HQ) is used as train set and the remaining four datasets are used for generalization evaluation.
Pleasefind more details of these datasets in the supplementary materials.
Data preprocessing .
All the used datasets are published in a full-frame format, thus, most of thedeepfake detection methods crop the face regions in advance.
However, with a motivation to learnthe low-level temporal patterns, cropping face regions in advance will lead to artificially introducedjittering due to the independent face detection.
Therefore, in our method, we crop the face regionsusing the same bounding box after randomly determine the clip range on-the-fly, where the box isdetected by MTCNN [53].
We leave more details of this processing in the supplementary materials.
Implementation details .
The spatial input size H × W and patch size P is set to 224 × 224 and 16,respectively.
It is worth noting that the division into such small patches has considerably suppress theoverall semantic features.
The embedding dimension D is set to 384.
The kernel size of Conv3d of
LST is set to 3 × 3 × 3 .
For temporal dimension T , we empirically set it to 16 and provide morediscussion in ablations.
We use only the first 128 frames of each video in the experiments, thus thefinal prediction is averaged from 8 clips.
For optimizer, we use Adam [24] with the initial learningrate of 10 − 4 .
When the model performance no longer improves significantly, we gradually decay thelearning rate until convergence.
Four NVIDIA A100 GPUs are used in our experiments.
Evaluation metrics .
Following previous works [41, 20, 28], we use binary classification accuracy(ACC) and Area Under the Receiver Operating Characteristic Curve (AUC) as evaluation metrics.
4.2 Generalizability evaluation
For practical application, generalizability should be one of the most concerned properties.
Nevertheless, it is usually the Achilles’ heel of most deepfake detectors.
Since deepfakes generated bydifferent forgery methods (in different datasets) hold different kinds of forgery cues, and overfittingon semantic visual artifacts of the train set can easily lead to cross-dataset evaluation collapse.
As weshow the performance of models trained on FF++ and tested on four unseen datasets in Table.
1, manymethods do not perform satisfactorily.
In contrast, our approach outperforms all the recently publishednovel detectors, and achieves a new state-of-the-art of 91.9 AUC% averaged from the four datasets.
[Table]
Note PatchForensics [8]also focuses on localpatches, but only narrowed the perceptive fieldby truncating the CNNwithout considering the relations between patchesglobally, it shows limitedgeneralizability.
In addition, FTCN-TT reportscomparable results, wherethey also leverage the selfattention mechanism.
Butdifferent from our lowlevel temporal view, theyuse the semantic featuresof the whole frame for prediction.
Thus, when the visual artifacts are less distinguishable in CelebDFand DFDC, our method surpasses it by a clear margin.
A similar situation in LipForensics, which isproposed to perform deepfake detection using pretrain priors [36] of high-level semantic understanding.
With temporal regularity considered, they also achieve comparable results but show suboptimalperformance on CelebDF and DFDC.
Moreover, PCL+I2G and Face X-ray also focus on low-levellearning.
However, without considering temporal properties, they exhibit insufficient robustness andperform poorly in DFDC, in which the videos are filmed under very different circumstances and havebeen perturbed to some extent.
4.3 Robustness to perturbations
Another vital property for practical application is robustness.
For network transmission, videos arealways compressed.
Also, consider possible attacking against the detectors, we evaluate our approachon different types of perturbed videos.
[Figure]
As previous works [20, 21], we use the script 1 on FF++ and generate seven types of perturbations
As we show the comparison diagram in Fig.
2, the seven perturb methods include colorsaturation change, color contrast change, block-wise noise, gaussian noise, gaussian blur, pixelation,and video compression.
Examples of these perturbations can be found at this site.
Comparing with other methods, our approach shows excellent robustness against most of the perturbations.
As for the type of gaussian noise at a relatively higher level, the underlying low-level patterns are severely disturbed that our method can not work properly.
We also present the precise numbersof two state-of-the-arts in Table.
2, where the results are averaged from all five levels.
In general,
Face X-ray detects low-level boundary artifacts, and LipForensics focuses on high-level semanticfeatures.
Thus, the former performs better in block-wise noise type and the latter shows better resultsin others.
In contrast to Face X-ray, we detect low-level patterns in spatial-temporal space, which isbetter resistant to perturbations, and thus achieves better robustness evaluation performances.
[Table]
Module effects .
We first compare our method with related baselines and several alternative designs.1) Xception is the commonly used backbone in deepfake detection; 2) ViT is the most famousvision transformer backbone, where we use the “small” version with embedding dimension of 384;3) LTTD w/o LST indicates the model that replace the proposed LST with commonly used Patch
Embedding and Transformer blocks [17]; 4) LTTD w/o CPI is our L 3 TD framework trained withoutusing L CPI (Eq.
(13)); 5) LTTD w/o GCC represents the model that we replace the GCC modulewith a simple fully connected classification layer after average pooling the temporal embeddings fromall spacial locations.
From Table.
3, all the models achieve nearly perfect results on FF++.
Whilein the cross-dataset setting, our model with elaborately designed modules exhibits much strongergeneralizabilty, and the three components consistently boost the best results.
[Table] [Table]
Visualization .
To intuitively demonstrate the reason that our approach is more generalizable, wecompare the feature representations before the final classification in Fig.
Models are trained on
FF++ and tested on four subsets of FF++ (Deepfakes, Face2Face, FaceSwap, and NeuralTextures)respectively.
Due to the abundant inductive bias of the convolution, Xception clearly split the fourkinds of deepfakes into four clusters, even only binary labels are used in training.
This shows thatthe strong CNN tends to overfit on method-specific artifacts generated by different deepfake methods,although plausible results are achieved in in-dataset testing, it is harmful to generalize to unseendeepfakes.
In addition, similar phenomena can be found in the ViT results, but due to less inductivebias introduced, the four clusters are not as clearly divided as Xception.
In contrast, the features ofour approach show a completely different outline, where the real videos are also clearly separated,but the remaining four types of deepfakes are compacted in a unified manifold.
We attribute thisphenomenon to our low-level temporal learning, which can distinguish deepfakes from real dependingon more fundamental low-level temporal inconsistencies, thus achieving the best generalizability.
Sequential input .
As we mentioned previously, we empirically set the clip length to 16.
Onthe one hand, too short a clip does not ensure enough temporal information, e. g., 4 consecutive frames are almost identical to each other.
On the other hand, too long a clip is
TSNE vis Vit: 2022_0508_2300 Xcep: 2022_0118_1400 PST: 2022_0512_
ViT Xception
LTTD (ours)
L3TD (ours) not necessarily better: 1) A longer clip costs more memory; accordingly, a smaller batchsize also slow down training convergence or worse.
We found some of the videos in thetest datasets contained scene switching.
Considering the FPS of 24, a video clip in our experiments will not last 1 second, thus greatly avoiding the scene-switching problem.
Here,we further analyze how clip sampling will affect the low-level temporal learning of the proposed LTTD.
Two hyper-parameters are investigated: clip length and frame sampling space.
[Table] [Table]
From Table 4, the first 4 rows show that cliplength has little effect on the results.
We believethis is closely related to the idea of low-leveltemporal learning, which does not require thevideo clip to last long enough (e. g., 3 seconds) but only adequate frames to extract thelow-level temporal patterns.
Sparse sampling isanother practice to aggregate more content in asingle clip.
When we expand the frame interval,the performance degrades considerably.
Thisphenomenon suggests that sparse sampling isdetrimental to the learning of low-level temporal patterns even when more motion content isincluded.
These findings demonstrate that LTTD is distinctly different from related temporal-basedmodels, since the low-level temporal learning is specially designed for deepfake detection.
Conclusion and discussion
Conclusion .
In this paper, we propose a reliable framework to address the practical problem ofdeepfake detection, which emphasizes on the low-level temporal patterns of sequential patches in therestricted spatial region, but with a whole-range temporal receptive field using Transformer blocks.
In addition, we make the final classification in a more general global-contrastive way.
Thus bettergeneralizability and robustness are achieved to better support deepfake detection in real-world scenes.
Moreover, qualitative results further verify that low-level temporal information can lead to strongergeneralizability, which could also be a guideline for developing better approaches in the future.
Limitation .
Considering the continuous advances in deepfake creation and adversarial training, theperformance of our approach when encountering low-level and temporal adversarially enhanceddeepfakes in the future is yet unclear.
Moreover, although our method shows favorable performance compared to recent works, it stillrequires intensive labor in order to handle real-world scenarios.
This is a commonly shared limitationthat we do not know if the detectors are calibrated well for real-world deployment.
In addition toidentifying deepfakes, how we can ensure the predictions are credible remains an open problem,hindering the application of all deepfake detectors.
Ethical Concern .
Our work is proposed to mitigate the potential negative social impacts of otherworks, but there is still a risk of facilitating the forgers to propose better deepfake methods throughadversarial training.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No]
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] The FF++ dataset and FaceSh dataset
are released under the FaceForensics Terms of Use.
The CelebDF dataset is released
under the Terms to Use Celeb-DF.
The use of DeepFo dataset is bounded by the Terms
of Use: DeeperForensics-1.0 Dataset.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[Yes] The datasets we use are face-related open-
[Table]
source datasets that contain personally identifiable information.
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A] [Table]