E FFECTIVE AND R OBUST D ETECTION OF A DVERSARIAL E XAMPLES VIA B ENFORD -F OURIER C OEFFICIENTS
Anonymous authors
Paper under double-blind review
A BSTRACT
Adversarial examples have been well known as a serious threat to deep neural networks (DNNs).
To ensure successful and safe operations of DNNs on realworld tasks, it is urgent to equip DNNs with effective defense strategies.
In this work, we study the detection of adversarial examples, based on the assumption that the output and internal responses of one DNN model for both adversarial and benign examples follow the generalized Gaussian distribution (GGD), but with different parameters ( i.e. , shape factor, mean, and variance).
GGD is a general distribution family to cover many popular distributions ( e.g. , Laplacian, Gaussian, or uniform).
It is more likely to approximate the intrinsic distributions of internal responses than any speciﬁc distribution.
Besides, since the shape factor is more robust to different databases rather than the other two parameters, we propose to construct discriminative features via the shape factor for adversarial detection, employing the magnitude of Benford-Fourier coefﬁcients (MBF), which can be easily estimated using responses.
Finally, a support vector machine is trained as the adversarial detector through leveraging the MBF features.
Through the
Kolmogorov-Smirnov (KS) test (Massey Jr (1951)), We empirically verify that:
1) the posterior vectors of both adversarial and benign examples follow GGD; 2) the extracted MBF features of adversarial and benign examples follow different distributions.
Extensive experiments in terms of image classiﬁcation demonstrate that the proposed detector is much more effective and robust on detecting adversarial examples of different crafting methods and different sources, in contrast to state-of-the-art adversarial detection methods.
I NTRODUCTION
Deep neural networks (DNNs) have achieved a remarkable success in many important applications, such as image classiﬁcation, face recognition, object detection, etc .
In the meanwhile, DNNs have been shown to be very vulnerable to adversarial examples.
However, many real-world scenarios have very restrictive requirements about the robustness of DNNs, such as face veriﬁcation for login, or semantic segmentation in autonomous driving.
Adversarial examples are a serious threat to the applications of DNNs to these important tasks.
Since many kinds of adversarial attack methods have been proposed to fool DNNs, it is more urgent to equip effective defensive strategies to ensure the safety of deep models in real-world applications.
However, defense seems to be more challenging than attack, as it has to face adversarial examples from unknown crafting methods and unknown data sources.
Typical defensive strategies include adversarial training, adversarial de-noising, and adversarial detection.
Compared to the former two strategies, adversarial detection is somewhat more cost-effective, as it often does not need to re-train or modify the original DNN model.
There are two main challenges for adversarial detection.
(1) The adversarial examples are designed to camouﬂage themselves to be close to the corresponding benign examples in the input space.
Then, where and how to extract the discriminative information to train the detector?
The data sources and the generating methods of adversarial examples are often inaccessible to the detector.
In this case, the detector can be stably effective across different data sources and different attack methods?
In other words, a good adversarial detector is required to be not only effective to distinguish adversarial and benign examples, but also robust to different data sources and attack methods.
To satisfy the ﬁrst requirement of effectiveness, we utilize the other principle of crafting adversarial examples that the outputs between benign and adversarial examples should be large, to encourage the change of the ﬁnal prediction.
It means that the imperceptible difference between benign and adversarial examples in the input space is enlarged along the DNN model, leading to the signiﬁcant difference in the output space.
Inspired by this fact, we assume that the output or the responses of internal layers of the DNN model should include the discriminative information for benign and adversarial examples.
A few works have attempted to extract different types of discriminative features from the output or the internal responses, such as kernel density (KD)
(Feinman et al.
and the local intrinsic dimensionality (LID) (Ma et al.
(2018)), etc .
To achieve the robustness, the extracted discriminative features should model the intrinsic difference between adversarial and benign examples, rather than the difference from the changes of data sources or attack methods.
Many existing methods have shown the effectiveness to some extent on detecting adversarial examples of speciﬁc data sources and attack methods.
However, their robustness, especially across different data sources, has not been well studied and veriﬁed.
In this work, we propose a novel detection method based on the assumption that the internal responses of both adversarial and benign examples follow the generalized Gaussian distribution (GGD) (Varanasi & Aazhang (1989)), but with different parameters, including the shape factor, mean, and variance.
The rationale behind this assumption is that GGD covers many popular distributions with varied shape factors (such as Laplacian, Gaussian, or uniform distribution), such that
GGD is more likely to approximate the intrinsic response distributions rather than one speciﬁc distribution.
Moreover, mean and variance of GGD may vary signiﬁcantly with respect to different classes and data sources, even for benign examples, while the shape factor is more robust.
For example, the mean and variance of two Gaussian distributions could be totally different, but their shape factors are the same ( i.e. , 2).
Thus, we propose to use the shape factor as an effective and robust discrimination between adversarial and benign examples.
However, it is difﬁcult to exactly estimate the shape factor in practice.
We resort to the magnitude of Benford-Fourier coefﬁcients (Pasquini et al.
(2014)), which is a function of the shape factor.
It can be easily estimated using internal responses, according to the deﬁnition of Fourier transform.
Then, the magnitudes estimated from internal responses of different convolutional layers are concatenated as a novel representation.
Finally, a support vector machine (SVM) (Vapnik (2013)) is trained using the new representations as the adversarial detector.
Extensive experiments carried out on several databases verify the effectiveness and robustness of the proposed detection method.
To further verify the rationale of our assumption, we present the empirical analysis through the Kolmogorov-Smirnov test (KS test)
(Massey Jr (1951)).
The KS test veriﬁes that 1) the posterior vectors of both adversarial and benign examples predicted by the
CNN model follow the distribution of GGD, but with different parameters, and 2) the MBF features of adversarial and benign examples follow different distributions, and the MBF features of adversarial examples crafted from different attack methods follow the same distribution, as well as that the MBF features of adversarial/benign examples from different data sources follow the same distribution.
Moreover, we visualize the statistics ( i.e. , mean ± standard deviation) of the extracted
MBF features for adversarial and benign examples.
The visualization reveals the distinct difference between adversarial and benign examples.
These empirical analysis demonstrates the effectiveness and robustness of the proposed MBF detector.
R ELATED WORK
In this section we focus on reviewing existing works about adversarial detection, while those of other adversarial defensive strategies are out of scope of this work.
The general idea of most existing detection methods is learning or constructing a new representation to discriminate adversarial and benign examples, utilizing the outputs or immediate responses of an original classiﬁcation network.
Li & Li (2017) trained a cascading classiﬁer based on the principal component analysis (PCA)
(Pearson (1901)) statistics of responses from each convolutional layer of the defended convolutional neural network (CNN) model.
An example is recognized as benign if all single classiﬁers of the cascade predict it as benign, otherwise adversarial.
Lu et al.
(2017) proposed SafetyNet by adding a RBF-SVM classiﬁer to detect adversarial examples, at the end of the original classiﬁcation network.
Metzen et al.
(2017) proposed a detection network along with an original classiﬁcation network, which takes the internal responses of the original network as inputs.
It shows effectiveness on detecting adversarial examples generated by simple attacks (such as FGSM
(Goodfellow et al.
and JSMA (Papernot et al.
(2016))), while performs much worse when facing more advanced attacks (such as C&W (Carlini & Wagner (2017a))).
It tells that this method is sensitive to attack methods.
Grosse et al.
(2017) attempted to detect adversarial examples using the statistical test of maximum mean discrepancy (MMD).
Although above detection methods show effectiveness on some attack methods and some databases, but a thorough evaluation presented in
Carlini & Wagner (2017b) has shown that these methods are sensitive to attack methods or databases, and they can be somewhat easily invaded by new attacks.
Some recent works proposed to utilize neighboring samples in the same database to construct a better representation of a current sample.
Feinman et al.
(2017) deﬁned two metrics based on the responses of the ﬁnal hidden layer of the classiﬁcation neural network, including kernel density estimation (KDE) and Bayesian neural network uncertainty (BU).
If the metric score of KDE/BU is lower/higher than a pre-deﬁned threshold, then the example is predicted as adversarial.
Ma et al.
(2018) utilized the local intrinsic dimensionality (LID) to measure the characterization of adversarial regions of DNNs.
LID describes the distance between one example and its k -nearest neighboring sample in the feature space of immediate responses of the original classiﬁcation network.
The distances computed from different layers are concatenated as the example representation, which is then used to train a shallow classiﬁer to discriminate adversarial and benign examples.
Zheng &
Hong (2018) deﬁned the intrinsic hidden state distribution (IHSD) of the responses of the original classiﬁcation network to model different classes.
The Gaussian mixture model (GMM) was used to approximate IHSD of each class.
Then, the posterior probability of one sample assigned to
GMM is computed as the metric.
If the probability is lower than a pre-deﬁned threshold, then it is recognized as adversarial.
Lee et al.
(2018) computed the class-conditional Gaussian distribution of the responses of the original classiﬁcation network based on the whole training set.
Then, the
Mahalanobis distance between one sample and its nearest class-conditional Gaussian distribution is used as the metric for detection.
If the distance is larger than a pre-deﬁned threshold, then it is detected as adversarial.
Compared to some aforementioned single-representation-based detection methods, these joint-representation-based methods showed better performances on some databases.
However, the detection cost for each example is much higher, as the responses of its neighboring samples should also be computed.
Besides, since the representation is highly dependent on the neighbors or all training examples, the detection performance may be sensitive to data sources, which will be studied in later experiments.
There are also some other approaches that do not construct representations from the responses of the original classiﬁcation network.
Hendrycks & Gimpel (2016) adopted PCA statistics to discriminate adversarial and benign images, independent of any DNN model.
However, the study presented in Carlini & Wagner (2017b) has demonstrated that this method works for MNIST but not for CIFAR-10, and PCA statistics are not robust features to detect adversarial images.
Pang et al.
(2018) proposed a novel loss called reverse cross-entropy (RCE) to train the classiﬁcation network, such that the distance measured by kernel density (Feinman et al.
between adversarial and benign examples could be enlarged.
Samangouei et al.
(2018) proposed Defense-GAN to model the distribution of benign examples using a generative adversarial network (GAN).
If the Wasserstein distance between one example and its corresponding example generated by the GAN is larger than a pre-deﬁned value, then it is detected as adversarial.
However, these above three methods are much more costly than other detection methods.
P RELIMINARIES 3. G ENERALIZED G AUSSIAN D ISTRIBUTION
Assume that a random variable X ∈ R d follows the generalized Gaussian distribution (GGD) (Varanasi & Aazhang (1989)).
Then, its probability density function (PDF) is formulated with two positive parameters, including the shape factor c and the standard deviation σ , as follows
[Equation]
where β
= 1 σ (cid:0) Γ(3 /c ) Γ(1 /c ) (cid:1) 12 and A = βc 2Γ(1 /c ) , with Γ( · ) being the Gamma function.
Note that the mean parameter µ is omitted above, as µ has no relation with the shape of distribution and we set it aswithout loss of generality.
A nice characteristic of GGD is that it covers many popular distributions with varied shape factors.
For example, when c = 1 , then it becomes the Laplacian distribution; when c = 2 , then it is the Gaussian distribution with a variance of σ 2 ; when c → + ∞ , then it is speciﬁed as a uniform distribution on ( −√ 2 σ, √ 2 σ ) .
3. B ENFORD -F OURIER C OEFFICIENTS
Although generalized Gaussian distribution (GGD) is able to cover a bunch of distributions, it is hard to depict the exact forms of GGD precisely.
To this end, we further deﬁne a random variable
Z = log 10 |X| mod 1 for detecting and distinguishing different form of GGD, of which the PDF is formulated by means of Fourier Series as P´erez-Gonz´alez et al.
(2007), with the fundamental period being ﬁxed as 2 π ,
[Equation]
where z ∈ [0 , 1) corresponds to the domain of random variable Z , the phase of Fourier Series is explained as φ n = arctan (cid:0) − B n A n (cid:1) , and the magnitude denotes | a n | = (cid:112)
A 2 n + B 2 n .
a n = | a n |· e jφ n denotes the n -th Fourier coefﬁcient of P Z ( z ) evaluated at 2 πn , and its deﬁnition is
[Equation]
a n is also called as Benford-Fourier coefﬁcient.
Note that a n is a complex number, and its magnitude can be calculated as follows
[Equation]
Note that | a n | gets smaller as n ∈ N increases.
And, an interesting property of | a n | is that it only depends on the shape factor c , while is independent of the parameter σ .
Thus, one set of the absolute values of Benford-Fourier coefﬁcients {| a n |} n ∈ N correspond to one identical c , i.e. , one identical special distribution of GGD.
In other words, we could use {| a n |} n ∈ N as features or representations to discriminate different special distributions of GGD.
However, if it is often difﬁcult to know or even estimate the shape factor c , we cannot compute the value of | a n | .
But fortunately, recalling that a n is the n -th Fourier coefﬁcient of P Z ( z ) evaluated at 2 πn , we can derive an easy estimation.
Speciﬁcally, assume that x = { x 1 , . . .
, x M } is a set of
M i.i.d.
points sampled from GGD with the same shape factor c .
Then, the corresponding BenfordFourier coefﬁcients can be estimated as follows (Pasquini et al.
[Equation]
The gap between ˆ a n and a n is analyzed in Theorem 1.
It tells that ˆ a n gets closer to a n as M increases.
For clarity, we ﬁrstly introduce a few notations:
[Equation]
| is a random variable with X obeying the generalized Gaussian distribution, and ˆ a n is an observation of the random variable Y = 1 M (cid:80) Mm
=1 T m .
Due to the space limit, the proof will be presented in supplementary material .
Theorem 1 Assume that the estimation error ε n = ˆ a n − a n is an observation of the random variable
E = Y− a n .
|E| follows the Rayleigh distribution (Papoulis & Pillai (2001)), of which the probability density function (PDF) is formulated as
[Equation]
And the expectation and variance are respectively shown as follows:
[Equation]
which implies that the estimation error ε n gets closer to 0 as the number of samples M increases.
A DVERSARIAL D ETECTION VIA B ENFORD -F OURIER C OEFFICIENTS 4. T RAINING P ROCEDURE OF A DVERSARIAL D ETECTOR
There are three stages to train the proposed adversarial detector, including: 1) building a training set based on benign images; 2) extracting novel representations of the training set via Benford-Fourier coefﬁcients; 3) training a SVM classiﬁer as the adversarial detector.
They will be explained in details sequentially.
And, the overall training procedure is brieﬂy summarized in Algorithm 1.
Build a training set .
Firstly, we collect N clean images { x 1 , . . .
, x N } , which can be correctly predicted by f θ .
Then, we adopt one adversarial attack method ( e.g. , C&W (Carlini & Wagner (2017a)) or BIM ( Kurakin et al.
to generate one adversarial image corresponding to each clean image.
The crafted N adversarial examples are denoted as { ˆ x 1 , . . .
, ˆ x N } .
Besides, to avoid that the noisy image (polluted by some kind of non-malicious noises but still can be correctly predicted by f θ ) is incorrectly detected as adversarial, we also craft one noisy image by adding small random
Gaussian noises onto each clean image.
These N noisy examples are denoted as { ¯ x 1 , . .
, ¯ x N } .
Note that, hereafter benign examples include both clean and Gaussian noisy examples.
Consequently, we obtain one training set with 3 N examples, denoted as D tr = { ( x i , − 1) , (¯ x i , − 1) , (ˆ x i , +1) } i =1 ,...,N .
Extract novel representations .
We ﬁrstly feed the i -th training image from D tr into f θ .
We concatenate all response entries of the l -th layer in f θ to obtain one vector r li .
Then, we estimate the corresponding Benford-Fourier coefﬁcients according to Eq.
(5), as follows
[Equation]
where M li indicates the length of r li .
The magnitude of (ˆ a n ) li is computed as follows
[Equation]
Then, we extract one T -dimensional feature vector a li =
[ | (ˆ a 1 ) li | , . .
, | (ˆ a T ) li | ] ∈ R T + for the i -th training image from the l -th layer.
We set T = 16 in experiments, as | (ˆ a n ) li | of larger n is too small for discrimination.
Finally, we concatenate the feature vectors of all layers to form a long vector ˆ a i
=
[ˆ a 1 i ; . .
; ˆ a Li ] ∈ R TL + , with L being the number of layers in f θ .
Consequently, we obtain a novel representation of the training set D tr , denoted as ˆ A tr = { (ˆ a i , ± 1) } i =1 ,..., 3 N , where the label of a i ( i.e. , +1 or − 1 ) is directly obtained from D tr .
Train an adversarial detector .
Finally, we train a binary SVM classiﬁer based on ˆ A tr .
The trained
SVM classiﬁer will serve as the adversarial detector for the CNN model f θ .
Testing .
One novel testing example is ﬁrstly predicted as adversarial or not by the trained adversarial detector .
If adversarial, then it is rejected; otherwise, it is fed into f θ to predict its class label.
Remark .
Note that in the derivation of a n (see Eq.
(3)), the mean parameter of GGD is set to 0 .
In experiments, we calculate the mean values of internal-layer responses of all networks for every image, and ﬁnd that mean parameters at most layers are close to 0, while the mean parameters at a few layers could be large.
However, the mean value is subtracted from each response entry when we extract the MBF features in our experiments.
Thus, the derived a n is applicable to our task.
Besides, according to Theorem 1, the estimation error of (ˆ a n ) li is inversely proportional to M li .
It tells that the coefﬁcient estimated from the larger-sized layer is more accurate.
In many neural networks ( e.g. , AlexNet (Krizhevsky et al.
, the response sizes of high layers get smaller, which means the less accurate estimation.
However, we believe that the discrimination between adversarial and benign examples in higher layers is more evident than that in lower layers.
There is a trade-off between estimation accuracy and discrimination.
This is why we concatenate the estimated magnitudes of all layers together to construct the novel representation.
4. E XPERIMENTAL S ETTINGS
Databases and network architectures We conduct experiments on three databases, including
CIFAR-10 (Krizhevsky et al.
, SVHN (Netzer et al.
(2011)), and a subset of ImageNet (Deng et al.
In terms of CIFAR-10 and SVHN, we adopt the same settings as the compared method
LID (Ma et al.
Speciﬁcally, a 33-layer network pre-trained on the training set of CIFARachieves 82 .
37% accuracy on the testing set with 10 , 000 benign images; a 19-layer network pretrained on the training set of SVHN achieves 92 .
6% accuracy on the testing set with 26 , 032 benign images.
Then, we add a small noise drawn from N (0 , σ 2 ) on each testing image, with σ being the similar level of the (cid:96) 2 norm of adversarial perturbations on the same database.
If both the benign and its noisy image can be correctly predicted by the classiﬁcation network, then it is picked out for training the detection.
We ﬁnally collect 8 , 175 and 23 , 862 benign images from CIFATR-10 and SVHN, respectively.
These images are randomly partitioned to the 80% training set and the 20% testing set, used for the training and testing of the detector.
We also collect a subset from ImageNet, including 800 benign images of 8 classes ( snowbird, spoonbill, bobtail, Leonberg, hamster, proboscis monkey, cypripedium calceolus, and earthstar ).
The 100 images of each class contain 50 testing images and 50 randomly selected training images.
We ﬁne-tune the checkpoints of both AlexNet and VGG pre-trained on ImageNet 1 on these 800 images to achieve 100% accuracy.
Then, 785 benign images are kept for detection, as both their noisy images and themselves can be correctly predicted by both the ﬁne-tuned AlexNet and VGG models.
These 785 images are then randomly partitioned totraining and 305 testing images used for detection.
For each database, as described in Section 4.1, one noisy and one adversarial image are generated for each benign image; then, all of benign, noisy, and adversarial images are used for detection.
Attack methods We adopt four popular adversarial attack methods to craft adversarial examples, including basic iterative method (BIM (Kurakin et al.
, CarliniWagnerL2Attack (CW-L (Carlini & Wagner (2017a))), DeepFool ( (Moosavi-Dezfooli et al. (2016))), and random projected gradient descent (R-PGD (Madry et al.
They are implemented by Foolbox 2 .
Compared detection methods We compare with three state-of-the-art and open-sourced adversarial detection methods, including KD+BU 3 (Feinman et al.
, Mahalanobis distance 4 (M-D) (Lee et al.
(2018)), and LID 5 (Ma et al.
Note that another recent work called I-defender (Zheng & Hong (2018)) is not compared, as its code is not available.
To ensure the fair comparison, the
SVM classiﬁer is trained with all compared methods, implemented by the ﬁtcsvm 6 function in MATLAB.
There are two important hyper-parameters in LID, i.e. , the size of mini-batch and the number of neighbors.
On CIFAR-10 and SVHN, they are respectively set as 100 and 20, as suggested in Ma et al.
(2018); on ImageNet, as there are only 400 benign training images, they are respectively set as 50 and 20 in experiments.
Moreover, we ﬁnd that there are some unfair settings in the implementations of compared methods.
For example, KD+BU utilizes the extra 50 , 000 images of CIFAR-10 to compute the kernel density of each training and testing image; M-D also uses these extra images to compute the mean and co-variance of GMM.
Since extra images of the similar distribution with the training images are often unavailable, we believe that extra images should not be used to ensure the fair comparison.
Thus, extra images are not used for KD+BU and M-D in our experiments.
Besides,
[Table] [Table]
[Table]
LID utilizes other benign testing images as neighborhoods to extract features for each testing image.
It is unfair to utilize the information of benign or adversarial for neighboring testing images.
In our experiments, we use benign training images as neighborhoods for each testing image.
Three comparison cases and evaluation metrics We conduct experiments of three cases, including: 1) non-transfer , both training and testing adversarial examples are crafted by the same attack method; 2) attack-transfer , both training and testing adversarial examples are crafted by different attack methods; 3) data-transfer , both training and testing adversarial examples are crafted by the same attack method, but the data sources of training and testing benign examples are different.
Two widely used metrics are used to evaluate the detection performance, including area under the receiver operating characteristics (AUROC), and detection accuracy, which is the diagonal summation of the confusion matrix, using 0 .
5 as the threshold of the posterior probability.
Higher values of both metrics indicate better performance.
4. R ESULTS
Detection results in the non-transfer case are shown in Table 1.
The proposed MBF method shows the best performance in all cases, and is much superior to all compared methods.
LID performs the second-best in most case.
KD+BU gives somewhat good detection performance on CIFARand SVHN, but performs very poor on ImageNet.
It tells that KD+BU is very sensitive to different databases and networks.
M-D gives almost the degenerate results.
Note that the results of M-D on
[Table] [Table]
CIFAR-10 and SVHN reported in (Lee et al.
(2018)) are very high, but their networks ( i.e. , ResNet (He et al. (2016)) and DenseNet (Huang et al. (2017)) are different from that used in our experiments.
It implies that M-D may be suitable for very deep neural networks, but not for shallower networks.
Detection results in the attack-transfer case evaluated by AUROC and accuracy are presented in
Tables 2 and 3, respectively.
MBF still shows much better performance in all transfer cases than all compared methods, and the changes among detecting different attacks are very small.
It veriﬁes the robustness of MBF to different attacks.
We also conduct a data-transfer experiment on ImageNet.
Speciﬁcally, we collect extra 365 images of 8 classes (same with the classes used for the detection training, see Section 4.2) through searching the class names in Baidu and Facebook.
These 365 benign and their noisy images can be correctly predicted by both the ﬁne-tuned AlexNet and VGG models.
The results on these images are shown in Table 4.
MBF still shows the best performance.
And, compared to the corresponding results in
Table 1, the AUROC/accuracy scores of MBF on detecting different attacks change very gently.
It veriﬁes the robustness of MBF to different data sources.
4. A DAPTIVE A TTACK A GAINST A DVERSARIAL D ETECTION
Similar to Carlini & Wagner (2017b) and Ma et al.
(2018), we also present an adaptive white-box attack, of which the goal is not only to fool the classiﬁer, but also to evade the adversarial detector.
It measures how easy to bypass the detector.
Speciﬁcally, by combining the CW-L2 attack (Carlini & Wagner (2017a)) and the proposed MBF detector, the adaptive attack is formulated as follows:
[Equation]
(9) where x denotes the clean input, y indicates the ground-truth label.
l ( x adv , y ) is the Hinge loss.
D ( f θ ( x adv )) represents the extracted features by the the detector D (speciﬁed later), from f θ ( x adv ) .
The minimization of the (cid:96) 1 term (cid:107) D ( f θ ( x adv ))
− D ( f θ ( x ))
(cid:107) 1 encourages the detection features of adversarial and benign examples to be close, such that the detector could be evaded.
We conduct the above adaptive attack on both CIFAR-10 and SVHN.
Speciﬁcally, 1000 clean examples are randomly selected from the test set of each database.
For each generated adversarial example, if it fails to fool the classiﬁer f θ , we record 1 .
The attack failure rate over all 1000 adversarial examples is computed as the metric.
The higher rate indicates that it is more difﬁcult to evade the detector.
Since LID performs much better than KD+BU and M-D in above experiments, here we only compare LID and the proposed MBF detector.
For clarity, the detection features are only extracted from the soft-max layer for both LID and MBF.
The original CW-L2 attack ( i.e. , without the (cid:96) 1 term) is also presented as the baseline.
With the same setting in (Ma et al.
, the trade-off parameter α is determined by binary search within the range (cid:2) 10 − 3 , 10 6 (cid:3) .
The results are shown in
Table 5.
The attack failure rates of MBF on both databases are much higher than those of LID.
It tells that it is more difﬁcult to evade MBF than LID.
4. H YPOTHESIS T EST VIA K OLMOGOROV -S MIRNOV T EST
Kolmogorov-Smirnov test (KS test)
(Massey Jr (1951)) is a non-parametric test method in statistics, to test whether a sample follows a reference probability distribution (one-sample KS test), or
[Table] [Table]
[Table]
whether two samples follow the same distribution (two-sample KS test).
Speciﬁcally, in one-sample
KS test, the distance between the empirical distribution function of one sample and the cumulative distribution function of the reference probability distribution is measured.
Then, the p -value corresponding to the obtained distance is computed.
If the p -value is larger than the signiﬁcance level α
(here we set α = 0 . 05 ), then the null hypothesis that the sample follows the reference distribution is accepted; otherwise, rejected.
Similarly, in two-sample KS test, the distance between the empirical distribution functions of two samples is computed.
If the corresponding p -value is larger than α , then it accepts that two samples follow the same distribution.
The KS test conducted below is implemented by the python function scipy.stats.ks 2 samp 7 .
Hypothesis test 1 .
Here we verify that whether the posterior vectors of both adversarial and benign examples follow GGD.
We denote the posterior vector of one adversarial example as p adv , and that of one benign example as p ben .
The distribution of GGD is denoted as P GGD .
Then, we conduct two one-sample KS tests, including:
• H1.
The test of adversarial examples: H 0 : p adv ∼ P GGD-adv ; H 1 : p adv (cid:54)∼ P GGD-adv .
H1.
The test of benign examples: H 0 : p ben ∼ P GGD-ben ; H 1 : p ben (cid:54)∼ P GGD-ben .
In H1.1 , the reference distribution P GGD-adv is ﬁrstly estimated from p adv , using the estimated method proposed in (Lasmar et al.
To alleviate the uncertainty of the estimation, we draw 500 samples from the estimated P GGD-adv .
Then, we conduct the two-sample KS test between p adv and these 500 samples respectively.
The average p -value over 1000 tests is recorded.
The mean of the average p -values over the whole database is reported.
H1.2 is conducted similarly.
The results tested on ImageNet-AlexNet are shown in Table 6.
Due to the space limit, results on other databases and models will be presented in the supplementary material .
In all cases, the p -values are larger than the signiﬁcance level 0 .
Hence, we can conclude that the posterior vectors of both adversarial and benign examples follow GGD .
However, note that the parameters of their corresponding
GGD are different, which will be veriﬁed in the following test.
Hypothesis test 2 .
Here we verify that whether the extracted MBF features of adversarial and benign examples follow the same empirical distribution.
We denote the MBF feature vector of one adversarial example as m adv , and the corresponding empirical distribution is denoted as ˆ P adv-MBF .
Similarly, we deﬁne m ben and ˆ P ben-MBF for benign examples.
Then, we conduct the following four two-sample KS tests:
• H2.
The test between adversarial and benign examples: H 0 :
ˆ P adv-MBF =
ˆ P ben-MBF ; H 1 : ˆ P adv-MBF (cid:54) =
ˆ P ben-MBF .
• H2.2 The test between adversarial examples crafted from different attack methods: H 0 :
ˆ P attack-1adv-MBF = ˆ P attack-2adv-MBF ; H 1 : ˆ P attack-1adv-MBF (cid:54) =
ˆ P attack-2adv-MBF .
• H2.3 The test between adversarial examples from different data sources ( i
e. , train, test and out-of-sample set): H 0 :
ˆ P source-1adv-MBF = ˆ P source-2adv-MBF ; H 1 : ˆ P source-1adv-MBF (cid:54) =
ˆ P source-2adv-MBF .
(a
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
(e) DeepFool 2 4 6 8 10 12 14 16 0.4 0.5 0.
(f) R-PGD
• H2.
The test between benign examples from different data sources ( i.e. , train, test and out-of-sample set): H 0 : ˆ P source-1ben-MBF = ˆ P source-2ben-MBF ; H 1 : ˆ P source-1ben-MBF (cid:54) =
ˆ P source-2ben-MBF .
In above four two-sample KS tests, we test on the 16-dimensional MBF features extracted from the soft-max layer.
Since the implementation scipy.stats.ks 2 samp cannot compare two vectors, we compare the feature of each dimension separately, then report the average p -value over all dimensions.
Speciﬁcally, when comparing two sets of samples, we ﬁrstly concatenate the feature of each dimension across all samples in the same set, leading to 16 long vectors for each set.
Then, each pair of two long vectors corresponding to the same dimension from two sets is compared by KS test.
The average p -value over all 16 dimensions is reported.
The p -values of H2.1 are shown in Table 7 (see the column “(clean, BIM)”).
The p -values on both train and test set are 0 .
Thus, the hypothesis His rejected, i.e. , the MBF features of adversarial and benign examples follow different distributions.
The p -values of H2.2 are shown in Table 7.
We pick two groups of attack methods, i.e. , (BIM,
DeepFool) and (CW-L2, R-PGD).
The p -values of (BIM, DeepFool) are 0, while the p -values of (CW-L2, R-PGD) are larger than 0 .
It demonstrates that the distributions of adversarial examples crafted from different attack methods are possible to be different.
The p -values of H2.3 are shown in Table 8.
The p -values of all types of adversarial examples exceed 0.05.
Thus, the MBF features of adversarial examples from different data sources follow the same distribution.
The p -values of
H2.4 are shown in Table 8.
Only the p -value of “(train set, out-of-sample set)” of clean examples is slightly lower than 0.05, while the values of other cases exceed 0.05.
Thus, in most cases, the MBF features of benign examples from different data sources follow the same distribution.
From above analysis, we obtain the following conclusions: 1) The extracted MBF features of adversarial and benign examples follow different empirical distributions.
It explains why MBF features are effective for detecting adversarial and benign examples; 2) The extracted MBF features of adversarial/benign examples from the train, test and out-of-sample sets follow the same empirical distribution.
Although the extracted MBF features of adversarial examples crafted from different attack methods may not follow the same empirical distribution, the signiﬁcant difference between benign and different adversarial distributions can still lead to the good detection performance in the attack-transfer case.
It explains why MBF features are robust across different attack methods and different data sources.
Moreover, we visualize the statistics of each dimension of MBF features, i.e. , mean and standard deviation, as shown in Fig.
These visualizations also support above conclusions.
Due to space limit, more KS tests and visualizations on different databases and networks will be presented in the supplementary material .
C ONCLUSION
This work has proposed a novel adversarial detection method, dubbed MBF.
The assumption behind is that the internal responses of the classiﬁcation network of both adversarial and benign examples follow the generalized Gaussian distribution (GGD), but with different shape factors.
The magnitude of Benford-Fourier coefﬁcient is a function w.r.t.
the shape factor, and can be easily estimated based on responses.
Thus, it can serve as the discriminative features between adversarial and benign examples.
The extensive experiments conducted on several databases, as well as the empirical analysis via KS test, demonstrate the superior effectiveness and robustness to different attacks and different data sources of the proposed MBF method, over state-of-the-art detection methods.
A A PPENDIX A. P ROOF OF T HEOREM
Proof 1 Assume that all the variables {T m } m =1 ,...,M are independent and identically distributed.
Applying the central limit theorem Rosenblatt (1956) to the real and imaginary parts of Y [Equation] we can obtain that both parts asymptotically follow the Gaussian distribution
[Equation]
) where
[Equation] [Equation]
The PDF of Y can be rewritten as follows:
[Equation]
Thus
[Equation]
we obtain
[Equation]
Besides
[Equation]
the pseudo variance Goodman (1963) of T 1 is
[Equation]
[Equation]
Correspondingly,
[Equation]
Since ( a 2 n − a 2 n ) is bounded [Equation] we have lim M →∞ J E , E = 0 .
Thus, we obtain that the random variable
E follows circularly-symmetric complex Gaussian distribution, because the sufﬁcient and necessary condition is that mean value and pseudo variance equal zero Goodman (1963).
It implies that both the real and imaginary part of E follow the same Gaussian distribution and they are independent.
Thus, the magnitude of this complex random variable follows the Rayleigh distribution Papoulis &
Pillai (2001), and the probability density function of |E| can be formulated as
[Equation]
(17) where s is the scale parameter.
Knowing the properties of the Rayleigh distributionPapoulis & Pillai (2001), we have:
[Equation]
Utilizing the fact that | a n | 2 is close to 0 when n is a modest number, we obtain that D ( E ) ≈ 1 M , leading to s 2 = 1 2 M .
Then, we obtain
[Equation]
It is easy to observe that both E (cid:0) |E| (cid:1) and D (cid:0) |E| (cid:1) are close to zero when the number of samples M increases.
It implies that the estimation error ε n gets closer to 0 as M increases.
[Table]
[Table]
[Table]
(c) BIM 2 4 6 8 10 12 14 16 0.4 0.5 0.
(d) CW-L 2 4 6 8 10 12 14 16 0.4 0.5 0.
(e) DeepFool 2 4 6 8 10 12 14 16 0.4 0.5 0.
(f) R-PGD
[Figure] [Figure]
(a) clean 2 4 6 8 10 12 14 16 0.4 0.
[Figure]
(b) noisy 2 4 6 8 10 12 14 16 0.4 0.5 0.
(c) BIM 2 4 6 8 10 12 14 16 0.4 0.5 0.
(d) CW-L 2 4 6 8 10 12 14 16 0.4 0.5 0.
(e) DeepFool 2 4 6 8 10 12 14 16 0.4 0.5 0.
(f) R-PGD (a
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table]
(e) DeepFool 2 4 6 8 10 12 14 16 0.4 0.5 0.
(f) R-PGD
[Table]
A. A DDITIONAL E MPIRICAL A NALYSIS
Here we present additional empirical analysis on more databases and networks, as shown in Tables 9, 10 and 11.
The p -values in most cases also supports the conclusions obtained in the main manuscript.
We also present more visualizations in Figs.
2, 3 and 4.
These visualizations also demonstrate the distinct difference of MBF features between adversarial and benign examples.
A. C OMPARISON BETWEEN D EFENSE -GAN
We adopt ofﬁcial code 8 of Defense-GAN (Samangouei et al.
and train an generative model on train set of MNIST dataset after 130 , 000 epochs (As the ofﬁcial code of Defense-GAN currently only supports three datasets including MNIST, Fashion-MNIST, and CelebA).
The classiﬁcation model attains an accuracy of 98.7% on the test set of MNIST dataset.
We still utilize four attack strategies, including BIM, CW-L2, DeepFool and R-PGD.
After removing images that are misclassiﬁed and failed to attack, we pick 6000 images to train SVMs based on our MBF features, and leave nearly 4000 images for testing.
There are two key hyper-parameters in Defense-GAN, including the iteration number in each GD run, and the total number of GD runs.
We have tried multiple settings of these two hyper-parameters.
Table 12 show the AUROC score on nearly 4000 images.
Our MBF method exceeds Defense-GAN by a large margin in quantitative performance.
A. D ETAILED P ARAMETERS OF A DVERSARIAL A TTACK
Four popular adversarial attack methods are adopted to craft adversarial examples, including basic iterative method (BIM), CarliniWagnerL2Attack (CW-L2), DeepFool, and random projected gradient descent (R-PGD).
We emphasize detailed parameters of these attack strategies for reproducibility, which are BIM (eps= 0 . 3 , stepsize= 0 .
05 , iterations= 10 ), CW-L2 (binary search steps= 5 , conﬁdence= 0 .
0 , learning rate= 0 . 005 , max iterations= 1000 ), DeepFool (max steps= 100 ), R-PGD (eps= 0 .
3 , stepsize= 0 .
01 , iterations= 40 ).
All of the attack strategies are implemented by Foolbox.