V IDEO F ORGERY D ETECTION U SING M ULTIPLE C UES ON F USION OF E FFICIENT N ET AND S WIN T RANSFORMER
Anonymous authors
Paper under double-blind review
A BSTRACT
The rapid development of video processing technology makes it easy for people to forge videos without leaving visual artifacts.
The spread of forged videos may lead to moral and legal consequences and pose a potential threat to people’s lives and social stability.
So it is signiﬁcant to identify deepfake video information.
Although the previous detection methods have achieved high accuracy, the generalization is poor when facing unprecedented data in the real scene.
There are three fundamental reasons.
The ﬁrst is that capturing the general clue of artifacts is difﬁcult.
The second is that selecting the appropriate model is challenging in speciﬁc feature extraction.
The third is that exploiting fully and effectively the extracted features is hard.
We ﬁnd that the high-frequency information in the image and the texture in the shallow layer of the model expose the subtle artifacts.
The optical ﬂow of the real video has variations while the optical ﬂow of the deepfake video has rarely variations.
Furthermore, consecutive frames in the real video have temporal consistency.
In this paper, we propose a dual-branch video forgery detection model named ENST, which integrates parallelly and interactively EfﬁcientNet-B5 and Swin Transformer.
Speciﬁcally, EfﬁcientNet-B extracts the artifacts information of high frequency and texture in the shallow layer of the model.
Swin Transformer captures the subtle discrepancies between optical ﬂows.
To extract more robust face features, we design a new loss function for EfﬁcientNet-B5.
In addition, we also introduce the attention mechanism into
EfﬁcientNet-B5 to enhance the extracted features.
We conduct test experiments on FaceForensics++ and Celeb-DF (v2) datasets, and comprehensive results show that ENST has higher accuracy and generalization, which is superior to the most advanced methods.
I NTRODUCTION
Video synthesis techniques based on deep learning Karras et al.
(2017; 2018); Thies et al.
(2019) have reduced the threshold for people to manipulate videos, and the high-quality deepfake videos that are indistinguishable to the human eye are increasingly available Deepfakes (2020); Faceswap (2019); Fakeapp (2020).
Although this technology has been applied to Cineﬂex, Crytek, and driverless technology, which has had a positive effect on society.
Inevitably, it is illegally abused by some malicious users to forge indistinguishable false information, which seriously affects the level of public trust in the government and society, causes irreparable social problems, and can even pose a serious threat to politics.
Many deepfake detection methods Wang et al.
; Kim et al.
; Jeon et al.
(2020) have been proposed to address this problem and signiﬁcant progress has been made.
Previous works explore forgery patterns based on manual features Ferrara et al.
(2012); Pan et al.
; Cozzolino et al.
and then classify them by zooming in on the subtle differences between the real and forged images.
While most of the recent learning-based approaches Cozzolino et al.
(2017); Chollet (2017a) deﬁne the problem as a binary classiﬁcation of an image or video with a process of ﬁrst extracting the global features of the image using a neural network and then feeding them to a classiﬁer to distinguish between genuine and faked images.
However, with the increasing reﬁnement of deep generative methods, the generated forged videos are becoming increasingly difﬁcult to discriminate, especially some low-quality ones with only a few subtle visual artifacts.
Most existing detection methods still focus on speciﬁc artifacts, leading to signiﬁcant challenges in the generalization and effectiveness of detection systems in realistic scenarios.
This creates an urgent need to discover subtle cues that are more critical for detection, and then to extract and incorporate appropriate modules with efﬁcient backbone networks to enhance these artifact features and combine them effectively for forgery detection.
After systematic analysis, we ﬁnd three fundamental reasons for the low generalization of the model.
The ﬁrst is the difﬁculty of capturing general artifact cues and the limitations of the dataset in terms of quantity and quality.
The second is the limitations of selecting a suitable network model for speciﬁc feature extraction.
The third is the limitations of fully and effectively utilizing the extracted features.
In terms of feature cues, we ﬁnd that although the generated videos are realistic with few artifacts in the texture, the frequency domain, especially at high frequency, still reveals signiﬁcant differences, even for low-quality videos, and many forgery methods are designed to generate videos that are slightly defective to human vision, with few methods targeting the frequency domain.
In addition, it is found that texture information located at the shallow level of the detection network rather than the higher-level semantic information is particularly critical for detection.
To eliminate the effect of luminance in RGB on texture feature extraction, we convert the images from RGB to YCbCr and remove the luminance (Y) channel, and combine Cb and Cr channels with high frequencies.
In addition, we ﬁnd a characteristic that the optical ﬂow of the real video changes signiﬁcantly while the optical ﬂow of the deepfake video has little change, and coupled with the temporal consistency of the continuous frames of the real video, we choose the high frequency, texture, and optical ﬂow as the feature cues for video forgery detection.
For features such as high frequency and texture, which require attention to local details, we choose
EfﬁcientNet-B5 Tan & Le (2019), a CNN class network that is superior in local feature extraction, as one of the backbone networks, which has higher accuracy and performance compared with other CNNs.
For features such as optical ﬂow, which reﬂect the inter-frame temporality, we choose
Swin Transformer Liu et al.
(2021), a Transformer class network that is superior in global representation, as another backbone network, which applies to all scales of images, simple and ﬂexible in computation, and shows excellent performance compared with other Transformers.
In addition, we introduce the attention mechanism and a new loss function for EfﬁcientNet-B5 to extract more robust face features.
In summary, in this paper, we propose a novel multiple-cue video forgery detection model ENST using a combination of high frequency, low-level texture, and optical ﬂow cues, based on the fusion of EfﬁcientNet-B5 and Swin Transformer, which fully integrates multiple key features and uses an advanced backbone network for extraction.
To demonstrate the effectiveness of ENST, we conduct extensive experiments on FaceForensics++ Rssler et al.
and
Celeb-DF (v2) Li et al.
(2019), as well as a comparison of various settings in an ablation study.
Comprehensive experiments indicate that ENST has excellent competitive properties.
The contributions of this paper are as follows:
We investigate the current video forgery detection methods in-depth and systematically, and ﬁnd three reasons for the poor generalization ability of these methods, as well as discover several artifact cues in forgery detection, and use these cues to design an efﬁcient network structure for the above reasons.
We propose a novel network structure ENST, which can utilize the local feature extraction ability of EfﬁcientNet-B5 and the global relationship sensing ability of Swin Transformer, and integrate them to complement their abilities during data processing.
In addition, we introduce the attention module and a new loss function for EfﬁcientNet-B5 to enhance the ability to extract features.
Our experiments on FaceForensics++ and Celeb
-DF (v2) demonstrate that ENST achieves superior classiﬁcation performance and generalization compared to other state-of-the-art methods.
The rest of this paper is organized as follows.
Section 2 presents related work in the ﬁeld of video forgery detection and describes the motivation for the proposed ENST.
Section 3 details the speciﬁcs of the proposed model.
Section 4 describes the experimental results and analysis.
Section 5 concludes this paper.
R ELATED W ORK 2. V IDEO F AKE C UES
In recent years, various deep learning forgery methods have produced synthetic images or videos of faces
Suwajanakorn et al.
(2017; 2015) that have caused some degree of harm to society.
The problem of deep forgery detection has received extensive attention in the vision ﬁeld, and many forgery detection methods have been proposed to detect such forged products.
Some methods focus on mining speciﬁc artifacts generated by the forgery process, such as color space Li et al.
McCloskey & Albright (2018) and shape cues.
Many deep learning methods Marra et al.
Yu et al.
; Zhou et al.
(2018) use deep neural networks to extract high-level semantic information from the spatial domain and then classify a given image or video.
Some methods convert the image from the spatial domain to the frequency domain Stuchi et al.
Franzen (2018), capturing some information that is valuable for forgery detection.
Stuchi et al.
(2017) employ a ﬁxed set of ﬁlters to extract different ranges of frequency information and then adopt a fully connected layer to obtain classiﬁcation results.
Durall et al.
(2019) extract the frequency domain information using the
DFT transform and average the amplitudes of the different frequency bands.
Other methods extract statistical features, such as capturing features from spatial textures Pan et al.
; Conotter (2011) and transform domain coefﬁcient distributions Chen et al.
; Lyu & Farid (2005).
Video detection methods can discover differences in the temporal dimension of video frames Sabir et al.
Guera & Delp (2018), such as Wang & Dantcheva (2020) using spatial and motion information, and Chintha et al.
Amerini et al.
; Caldelli et al.
(2021) using optical ﬂow for video forgery detection.
While the above methods are limited to speciﬁc cues, we take full advantage of these cues.
2. D EEPFAKE D ETECTION S TRUCTURES
Many deep learning models have been proposed to extract features and discover forgery patterns directly without targeting a speciﬁc forgery cue or artifact, classify real and fake images or videos, and implement deep forgery detection created by random methods.
EfﬁcientNet Tan & Le (2019) achieves higher accuracy with a larger network by combining a composite model expansion method with neural structure search techniques to optimize the dimensions of network depth, network width, and input image resolution, which has higher accuracy and performance compared to other advanced
CNNs.
Each layer of Swin Transformer computes self-attention within the window of the transform, enabling cross-window connectivity for all scales of the image.
Swin Transformer is ﬂexible and efﬁcient, with only linear complexity for image size.
Convolution module in CNN is good at extracting local details, and as the layers get deeper, the attention ﬁeld will be wider and more suitable for image processing.
But to capture global information, detection model often requires stacking many convolutional layers.
While attention in Transformer is good at grasping the whole, but requires a large amount of data for training.
Conformer Peng et al.
(2021) uses a parallel and complementary approach, combining both CNN and Transformer branches to preserve the features extracted by both branches, to achieve performance beyond CNN and ViT Dosovitskiy et al.
(2020) with comparable parameter complexity.
P ROPOSED M ETHOD
In this section, we outline the principles and details of the different parts of the proposed method according to the data processing process.
It is challenging for most methods to extract sufﬁciently comprehensive features with suitable network models and then integrate them effectively for forgery detection.
Inspired by Conformer Peng et al.
(2021), we choose two models EfﬁcientNet-B5 and Swin Transformer with more efﬁcient performance among CNNs and Transformers, as baselines and combine them for video forgery detection.
EfﬁcientNet-B5 extracts local features of frames, such as high frequency features and texture features by using the characteristic of focusing well on local features.
Swin Transformer extracts artifact features from the optical ﬂow map between frames to obtain the global representation of frames, where the optical ﬂow features responding to the temporal consistency between frames are used.
We design a parallel interactive deep forgery detection architecture ENST to learn a potential representation that can distinguish between real and forged faces, and the pipeline of ENST is shown in
Figure 1.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
3. P RE - PROCESSING
We ﬁrst extract the frames of the input video, and then use the Multitask Cascaded CNNs (MTCNN)
Zhang et al.
(2016) to detect and extract the faces present in each frame, crop the faces of each frame, then resize to 224×224 pixels.
Moreover, we normalize them to zero mean and unit variance, and ﬁnally obtain the extracted faces.
After that, we ﬁrst operate on a single frame in the video, and the face in frame i is input to two branches after the basic feature extraction process.
3. E FFICIENT N ET -B5 B RANCH
We exploit the structurally improved EfﬁcientNet-B5 branch to extract the deepfake information in the high frequency and texture, and enhance it to ﬁnally obtain the features of a fully connected linear layer.
We introduce the speciﬁc details in the following subsections
. 3.2. F EATURE
The motion process of faces is complex, and most forgery methods do not consider sufﬁciently precise global constraints during forgery, which can easily lead to missing facial texture details of the generated faces and inconsistencies among features in diverse domains.
As the skin tones in RGB images are more inﬂuenced by the luminance, we convert the extracted faces from RGB to YCrCb by separating and removing the Y channel which denotes luminance.
In addition, considering the important effect of high frequency on forgery detection, we convert the image from the spatial domain to the frequency domain by DCT operation, and then use a high-pass ﬁlter to extract the high frequency component, which was merged with the two channels of Cr and
Cb to form a three-dimensional 224×224×3 feature tensor.
Then it was fed into EfﬁcientNet-B5 to extract the detail components in high frequency and the ﬁne artifacts of the shallow texture.
3.2. L OSS
We assume that real and fake faces in videos have distinguishable feature distributions, and samples of different classes are clustered together.
To extract robust face features and to distinguish the video distributions of real and fake faces, instead of using the more common softmax and cross-entropy, we combine softmax loss, additive angular margin loss (ArcFace) Deng et al.
and single-center loss (SCL) Li et al.
(2021) as the loss function of EfﬁcientNet-B5 for extracting features.
ArcFace and SCL have some similarities, both are used to compress intra-class tightness and enhance interclass diversity.
And we combine these objective functions to improve the accuracy of the extracted features.
ArcFace
ArcFace is an improvement on SphereFace Liu et al.
(2017) in terms of feature vector normalization and additive angular spacing, forcing a boundary between the distance of a sample to its class center and the distance of a sample to other class centers in a corner space.
It improves interclass separability, and also enhancing intraclass tightness and interclass diversity, allowing the model to learn features that are highly distinguishable between real and fake faces features, resulting in a more robust classiﬁcation for forgery detection.
ArcFace loss is deﬁned as:
[Equation]
where N is the batch size, n is the class number, m is an additive angular margin penalty between x i and W y i , θ j is the angle between the weight W j and the feature x
i , W j is the j -th column of the weight W , and s is the radius of a hypersphere where the learned embedding features are thus distributed on.
Single-center Loss (SCL)
The purpose of SCL is to minimize the distance from the real face to the center point while maximizing the distance from the fake face to the center point, so that the network can learn more subtle forgery information and reduce the optimization difﬁculty.
SCL is deﬁned as:
[Equation]
where M nat is the average Euclidean distance of the true face representation from the center point
C , and M man is the average Euclidean distance of the false face representation from the center point
C .
The Euclidean distance m is related to the arithmetic square root of the feature dimension D .
To facilitate the setting of the hyper-parameter m , the boundary is designed here as m √ D .
In addition, considering that SCL is based on mini-batch samples and focuses directly on feature representation, while softmax loss can focus on the global and on how to map the feature representation to the discrete label space.
So, we use the global information retained by softmax loss to guide the update of center points in SCL and increase the robustness of training.
In general, to reconcile the local feature representation and the global update, we use a combination of three loss functions, The total loss function is deﬁned as:
[Equation]
where α and β are hyper-parameters or weights that adjust the balance between L softmax , L Arcface , and L sc .
They are used to provide a relatively efﬁcient and ﬂexible total loss function.
3.2. A TTENTION
The attention mechanism allows models to focus on and fully learn crucial information, and is widely adopted as a module in models in natural language processing and computer vision Jie et al.
; Huang et al.; Vaswani et al.
We take EfﬁcientNet-B5 as the extraction model for artifact features in high frequency and low layer textures.
To focus more on the artifacts in the feature maps, inspired by Bonettini et al.
, we insert the attention module between the MBConv layers of EfﬁcientNet-B5, as shown in Figure 2,
[Table]
[Figure]
Figure 2 shows the effect of adding only one attention module.
In the practical experiment, we add the attention module sequentially from front to back between layers to compare the effect of the attention mechanism on the overall detection performance of the model.
First, the high frequency and texture information of the frames are combined and fed to EfﬁcientNet-B5.
Forging the images will result in overlapping areas of high frequency and texture information, which can be a guide to the extraction of texture information.
In addition, the attention module can notice these subtle artifacts in the shallow layers.
The ﬁnal output is an enhanced fusion feature of high frequencies and textures.
3. S WIN T RANSFORMER B RANCH
Considering that the optical ﬂow of the real video changes, while the optical ﬂow of the faked video rarely changes.
In addition, the distance between parts of the face and the head motion vary continuously and regularly between frames of the real video, while most forgery methods seldom concern to ensure the continuity between frames of the deep forgery video by effective modeling, which leads to the temporal inconsistency of the consecutive frames.
To exploit this temporal variation difference, we ﬁrst divide the video into 0 ∼ N consecutive frames, and extract the optical ﬂow of frame i and frame i + 1 as the optical ﬂow of frame i by using PWCNet Sun et al.
Then we input them into Swin Transformer-A to obtain the patch embeddings of the middle layer, where Swin Transformer-A represents the ﬁrst three stages in Swin Transformer.
3. F EATURE I NTERACTION M ODULE
We introduce the feature of complementary fusion by adding a feature interaction module.
In order to solve the problem of mismatching the size of the feature map in the Efﬁcient-B5 branch and the patch embeddings in the Swin Transformer branch, we use a special conversion operation, in which a 1×1 convolution is used to align the dimensionality of the feature map with the number of channels of the patch embeddings.
Then, the spatial dimensions are aligned by the downsampled module.
Finally, the feature map are added to the patch embedding.
In addition, the global context is gradually exported from the Swin Transformer branch to the feature map in Efﬁcient-B5 to enhance the global perception capability of the Efﬁcient-B5 branch.
We adopt a similar operation by ﬁrst upsampling the patch embedding to align the spatial scales.
Then a 1× convolution is used to align the number of channels of the patch embeddings with the dimensionality of the feature map.
Moreover, considering that the forgery operation causes changes in various aspects of the forgery region, the same regions of different features extracted have deﬁnite similarities, so this parallel and complementary fusion of the two branches also enables the two branches to guide each other and focus on the local artifact that is instrumental for forgery detection.
3. F EATURE C ONCATENATION
This part is the combination of all face region features of frame i extracted from two branches connected to form a feature concatenation of frame i , which includes the extracted high frequency, texture features, and patch embeddings.
The above operations are performed for each frame in sequence to obtain the feature concatenations of the 0 ∼ N frames of the videos.
After resizing, they are transformed into individual patches, which are combined into N patches, and then transformed into a new patch embedding, and then input it to Swin Transformer-B, then followed by the linear layer and softmax layer, ﬁnally output the classiﬁcation prediction of the whole video, where Swin Transformer-B represents the last stage in Swin Transformer.
It is worth mentioning that, after “ i -th frame” and before “All frames features” in Figure 1 represent the operation process of the i -th frame alone, and the rest of Figure 1 represents the operation process of all frames.
E XPERIMENTS 4. D ATASETS
In this paper, we use FaceForensics++ to train and verify the performance of our proposed model
ENST, and Celeb-DF (v2) to verify the generalization of the model.
4.1. F ACE F ORENSICS ++
FaceForensics++ is a facial forgery video dataset, including original videos and videos generated by the four most representative face manipulation methods (Face2Face, FaceSwap, DeepFakes and
NeuralTextures), each category has 1000 forgery videos, and the extracted frames contain more than 500,000 forged images in each category.
According to different compression levels, the compression factors can be classiﬁed as c0(raw), c23(medium) and c40(high).
As with the setup in
FaceForensics++, we split the dataset in each category into a ﬁxed train set, a validation set and a test set containing 720, 140 and 140 videos, respectively.
In the following experiments, we use two compression levels of c23(medium) and c40(high) for the data.
By default, the metric are Acc and
AUC.
4.1. C ELEB -DF ( V 2)
Celeb-DF (v2) contains both real and fake videos from Youtube.
Unlike the FaceForensics++, the fake videos in Celeb-DF (v2) are further segmented to handle issues such as color inconsistency and low-frequency smoothing, and have online playback-level high-resolution pictures quality.
CelebDF (v1) contains 795 fake videos.
Celeb-DF (v2) is an extension of Celeb-DF (v1) and includes 590 original videos collected from YouTube with different age, race, and gender themes, andcorresponding fake videos.
Celeb-DF (v2) has more diverse and challenging data, which is beneﬁcial to verify the generalization of the proposed method.
4. S ETTINGS
Based on the parameters of Conformer by default, we combine the parameters in both branches of
EfﬁcientNet-B5 and Swin Transformer.
ENST uses an optimizer of AdamW.
The momentum is 1.0.
The weight decay factor is 0.05.
The batch size is 256.
The epoch is 300.
The initial learning rate is 0.001.
The LR scheduler is decaying with a cosine schedule.
4. C OMPARISON OF METHODS
We compare the proposed method ENST with previous high-performance video forgery detection models on FaceForensics++, and the results are shown in Table 1.
Comparing with some advanced methods, we ﬁnd that although c40 reduces the AUC by 6.50 and the accuracy by 10.57 compared to c23, it still outperforms other methods.
We analyze that although the compression loses the details of image texture, the extracted high-frequency information and the optical ﬂow information between
[Table]
video frames have excellent robustness to the compressed information, which can well resist the compression interference and maintain the robustness of the model
[Table]
In order to verify whether ENST has superior generalization in realistic scenarios that are not included in the train set
[Table]
we put the model trained on FaceForensics++ on Celeb-DF (v2) for experiments on transfer learning
[Table]
we use the ﬁne-tuning model, i
[Table]
e., freeze parts of the convolutional layers of the pre-trained model, train the remaining convolutional layers, and after the model converges, reduce its learning rate and ﬁne-tune the layers.
We randomly select 40 samples in Celeb-DF (v2) with FaceForensics++ to train the model to prevent catastrophic forgetting, and the results are shown in Table 2.
The experimental results show that ENST has a better transfer capability and the ability to learn new types of samples.
It shows superior generalization on new types of data.
4. A BLATION S TUDY
To verify the effectiveness of each part of ENST, we conduct an ablation study and evaluate the network on FaceForensics++.
4.4. C OMPARISON OF LOSS
[Table]
On FaceForensics++, we compare the experimental results of using different loss functions when extracting features in EfﬁcientNet-B5, as shown in Table 3.
First, we compare the classiﬁcation results of Arcface and SCL individually, after which softmax is added to guide the improvement of the global classiﬁcation.
Finally the three losses are combined.
The experimental results indicate that the highest accuracy is achieved when softmax, Arcface, and SCL are integratively used.
[Table]
C OMPARSION OF ATTENTION
On FaceForensics++, we start from the previous of the EfﬁcientNet-B5 structure and add attention modules at different stages, and the experimental results are shown in Table 4, which shows that the accuracy increases gradually with the addition of attention modules.
However, when continuing to add at the 4th stage, the accuracy has slightly decreased, and when continuing to join in the 5th stage, the accuracy rate shows a signiﬁcant decrease, which is consistent with our speculation.
Our analysis suggests that the artifacts are mainly concentrated in the shallow network of EfﬁcientNet-B5, and the shallow feature extraction can provide enough critical information for the whole classiﬁcation network to classify, while the deep feature extraction is of little signiﬁcance.
Furthermore, the increase of the attention module will increase the computational volume and affect the overall performance improvement.
C ONCLUSION
In this paper, we systematically analyze the reasons for the low generalization of the current forgery video detection system and discover some subtle but critical cue features in forgery detection.
In order to combine the discovered cues and improve the generalization problem, we propose a network structure ENST with parallel interaction fusion of two branches, EfﬁcientNet-B5 and Swin Transformer.
We also introduce the attention mechanism and add a new loss function to EfﬁcientNet-B to further focus on and enhance the artifact features and extract more robust face features.
Comprehensive experiments indicate that ENST has better performance and generalization than other advanced methods.