PPD: P ERMUTATION P HASE D EFENSE A GAINST A DVERSARIAL E XAMPLES IN D EEP L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep neural networks have demonstrated cutting edge performance on various tasks including classiﬁcation.
However, it is well known that adversarially designed imperceptible perturbation of the input can mislead advanced classiﬁers.
In this paper, Permutation Phase Defense (PPD), is proposed as a novel method to resist adversarial attacks.
PPD combines random permutation of the image with phase component of its Fourier transform.
The basic idea behind this approach is to turn adversarial defense problems analogously into symmetric cryptography, which relies solely on safekeeping of the keys for security.
In PPD, safe keeping of the selected permutation ensures effectiveness against adversarial attacks.
Testing
PPD on MNIST and CIFAR-10 datasets yielded state-of-the-art robustness against the most powerful adversarial attacks currently available.
I NTRODUCTION
Recent advancements in deep learning have brought Deep Neural Networks (DNNs) to securitysensitive applications such as self-driving cars, malware detection, face recognition, etc.
Although
DNNs provide tremendous performance in these applications, they are vulnerable to adversarial attacks (Szegedy et al., 2013; Goodfellow et al., 2014; Athalye & Sutskever, 2017).
Especially in computer vision applications, small adversarial perturbation of the input can mislead the state-ofthe-art DNNs while being imperceptible to human eye (Goodfellow et al., 2014; Nguyen et al., 2015;
Moosavi-Dezfooli et al., 2017; Akhtar & Mian, 2018).
Since the existence of adversarial examples was discovered (Szegedy et al., 2013), numerous attacks and defenses have been proposed.
Attacks typically try to push the image towards the decision boundaries of the classiﬁer (Goodfellow et al., 2014; Carlini & Wagner, 2016; Kurakin et al., 2016;
Dong et al., 2018).
Defenses, on the other hand, can be categorized into three groups: (a) adversarial training, (b) hiding the classiﬁer, and (c) hiding the input image.
Adversarial Training.
As a natural defense, Szegedy et al.
; Goodfellow et al.
(2014) suggested to augment the training set with adversarial examples to reshape the decision boundaries of the classiﬁer.
Adversarial training requires an attack algorithm to generate adversarial examples during training.
The resulting classiﬁer is shown to be robust against the same attack used during training (Goodfellow et al., 2014;
Madry et al., 2017; Tram`er et al., 2017).
However, it is still vulnerable to other attacks or even the same attack with more perturbation budget (Sharma & Chen, 2018; Madry et al., 2017).
Hiding the Classiﬁer.
The idea of hiding the classiﬁer has been investigated in various ways.
One naive way is to not reveal the weights of the neural network.
However, adversarial examples generated by a substitute model can still fool the hidden model (Papernot et al., 2017; Liu et al., 2016), a phenomenon known as transferability of adversarial examples across different structures.
Another way is to conceal the gradient of loss function to guard against attacks that take advantage of gradient (Buckman et al., 2018;
Song et al., 2017; Samangouei et al., 2018;
Guo et al., 2017).
However, all of these defenses are circumvented by approximately recovering the gradient (Athalye et al., 2018).
Hiding the Input Image.
Another general approach to diminish adversarial examples is to hide the input image.
Following this idea, Guo et al.
(2017) suggested using randomization to hide the input image by two methods: (a) image quilting: reconstruct image by replacing small patches with clean patches from and b) pixels and recover them by total minimization.
One fundamental restriction of these randomized defenses is that a large portion of pixels has to be preserved for correct classiﬁcation.
This leaves the door open for adversarial attacks.
Indeed, both of these defenses are completely broken (Athalye et al., 2018).
Given that the defenses proposed under the last two categories are bypassed, to the best of our knowledge, adversarial training with examples generated by Projected Gradient Descent (PGD) attack (Madry et al., 2017) remains current state-of-the-art.
However, its robustness is limited to speciﬁc attack types with certain strength.
Changing the attack type or exceeding the perturbation limit degrades the accuracy signiﬁcantly.
In this paper, Permutation Phase Defense (PPD) is proposed to mitigate adversarial examples.
Following the philosophy of hiding the input image, we suggest to “encrypt” the image by applying two transformations before feeding the image to the neural network: First, randomly shufﬂe the pixels of the image with a ﬁxed random seed hidden from the adversary.
Second, convert the permuted image to the phase of its two dimensional Fourier transform (2d-DFT) (see Figure 1).
This pipeline is used in both training and inference phases.
The Fourier transformation captures the freqnency at which adjacent pixels of the input change along both axes.
This frequency depends on the relational positions of the pixels rather than their absolute value.
On the other hand, the relational positions are already hidden from the adversary by using the permutation block.
The random seed plays the role of the key in cryptography.
PPD offers some advantages as follows:
• PPD outperforms current state-of-the-art defense in two ways: First, PPD is a general defense and is not restricted to speciﬁc attack types.
Second, accuracy of PPD does not drop drastically by increasing perturbation budget of adversary (Table 1).
• In contrast to image quilting and random pixel drop (Guo et al.
, PPD randomly shufﬂes all the pixels in the pixel domain.
Therefore, important pixels of the phase domain remain hidden because of their dependence on the relational positions of the pixels in the pixel domain.
• Increasing adversarial perturbation budget is expected to decrease accuracy.
PPD, however, can distribute perturbation over different pixels in the phase domain such that adversarial perturbation is not more effective than random noise (Figures 4 and 5).
P RELIMINARIES 2. N OTATION
Let ( X , d ) be a metric space where X = [0 , 1] H × W × C is the image space in the pixel domain for images with height H , width W and C channels and d : X × X → [0 , ∞ ) is a metric.
A classiﬁer h ( · ) is a function from X to label probabilities such that h i ( x ) is the probability that image x corresponds to label i .
Let c ∗ ( x ) be the true label of image x and c ( x ) =
arg max i h
i ( x ) be the label predicted by the classiﬁer.
2. A DVERSARIAL E XAMPLES
Intuitively speaking, adversarial examples are distorted images that visually resemble the clean images but can fool the classiﬁer.
Thus, given a classiﬁer h ( · ) , (cid:15) > 0 and image x ∈ X , adversarial image x (cid:48) ∈
X satisﬁes two properties: d ( x, x (cid:48) ) ≤ (cid:15) and c ( x (cid:48) )
(cid:54) =
c ∗ ( x ) .
Ideally, metric d should represent visual distance.
However, there is no clear mathematical metric for visual distance.
Therefore, to compare against a common ground, researchers have considered typical mathematical metrics induced by (cid:96) 2 and (cid:96) ∞ norms.
[Figure]
[Equation]
3. D EFENSE S TRUCTURE
We address the problem of defending against adversarial examples.
We hide the input space by two transformations preceding the main neural network (see Figure 1):
• Random permutation block: pixels of the input image are permuted according to a ﬁxed random permutation.
The permutation seed is ﬁxed and hidden from the adversary.
• Pixel to phase block: permuted image is converted to the phase of its 2-dimensional discrete Fourier transform (2d-DFT).
These two blocks precede the neural network in both training and testing phases.
Mathematically speaking, consider channel c of image x ∈ X .
Its 2d-DFT can be written as
[Equation]
(1) where l = 0 , · · · , H − 1 and k = 0 , · · · , W − 1 and x hw is ( h, w ) image pixel value.
F lk is a complex number that can be represented in the polar coordinates as F lk =
A lk exp( jϕ lk ) where
A lk ∈ [0 , ∞ ) is the magnitude and ϕ lk ∈ [ − π, π ) is the phase of F lk .
Let P =
[ − π, π ) H × W × C be the phase space and p : X → P be a function that maps each channel of the image in the pixel domain to its phase.
Let σ : X → X be a channel-wise permutation, i.e., pixels in each channel of the input image are permuted according to a ﬁxed permutation.
Note that same permutation is used for all channels and all images.
The PPD classiﬁer can then be written as
[Equation]
where f ( · ) is the neural network.
3. I NTUITION B EHIND THE D EFENSE
It is known that random noise is unable to fool a trained neural network (Fawzi et al., 2016).
Motivated by this fact, if we train the neural network in a domain that is hidden from adversary, adversarial perturbations will not affect more than random noise in the actual domain fed to the neural network and thus not fool the classiﬁer.
Following this idea, Fourier transformation is used to capture the frequency at which neighboring pixels change.
Indeed, the values of the image in the Fourier domain do not purely depend on the values of pixels in the pixel domain.
Instead, they depend on what pixels are adjacent in the pixel domain, while neighboring pixels are hidden from adversary by the random permutation block.
Therefore, adversary will have little hope to attack the classiﬁer due to almost no knowledge about the input space.
3.2. W HY RANDOM PERMUTATION IS NOT ENOUGH ?
We would like to emphasize that input transformations that partially hide the input space are not secure.
For example, total variance minimization (randomly drop pixels and replace them with total variance minimization), quilting (replacing images) (Guo et al., 2017) are all defeated (Athalye et al., 2018).
One possible explanation for this failure is that although the adversary does not have full knowledge of the actual image fed to the neural network (by patching) good the is still preserved.
Hence, it is possible for adversary to attack.
Similarly, we have observed that random permutation by itself (without the pixel2phase block) is not sufﬁcient to defeat adversary.
Adversarial images generated by attacking a vanilla neural network can substantially decrease the accuracy of a permuted network.
This is because important pixels (e.g. white pixels in MNIST) in the original domain will still remain important pixels in the permuted domain.
Therefore, regardless of the employed permutation, if adversary attacks important pixels of the original domain (e.g. by converting white pixels to gray pixels in MNIST), it has successfully attacked important pixels in the permuted domain.
3.2. W HY PHASE OF 2 D -DFT IS USED ?
It is well known that phase component of Fourier transform is crucial in visual perception of human (Oppenheim & Lim, 1980; Morrone & Burr, 1988).
Indeed, an image reconstructed from phase only preserves essential information about the edges (see Figure 2).
This suggests that phase carries key information about the image and thus it is possible to train neural networks to operate in the phase domain rather than the pixel domain.
One might ask what if both phase and magnitude are fed to the neural network by doubling the size of the input space?
Our effort to train such a model was not fruitful but we leave it as a future research direction.
E XPERIMENTS
In this section, we test PPD trained on MNIST and CIFAR-10 against state-of-the-art adversarial attacks.
MNIST is a dataset of 28x28 grayscale images of handwritten digits and consists of 60, training as well as 10,000 test images.
CIFAR-10 dataset includes 50,000 32x32 color training images in 10 classes as well as 10,000 test images.
We assume that full information about the defense architecture is available to the adversary except for the permutation seed.
In addition, adversary can probe the classiﬁer with its crafted images and use output probabilities to construct more complicated adversarial examples.
Unknown permutation seed prevents adversary from taking advantage of classiﬁer parameters, because if a wrong permutation seed is used, the classiﬁer acts like a random-guess classiﬁer.
So far we have not come up with an effective way to test PPD against attacks that have access to model parameters.
Therefore, we leave it as an untested claim that PPD is secure against attacks with full knowledge of classiﬁer parameters as long as the permutation seed is not revealed .
[Figure]
Furthermore, huge space of permutations precludes the possibility of guessing the permutation seed.
For 28x28 MNIST images, there are 784! > 10 1500 possible permutations which is signiﬁcantly more than the number of atoms in the universe 1 .
4. T RAINING PPD C LASSIFIER ON MNIST AND CIFAR-
PPD requires training the neural network in the permutation-phase domain.
Before evaluating the defense against attacks, we need to ensure that high accuracy on clean images is indeed achievable.
This is a challenging task.
The analysis in Figure 2 suggests that training in the phase domain must be feasible in principle since the crucial information of the image are preserved.
However, a permutation block preceding the pixel2phase block may make training difﬁcult.
In this section, we successfully train a 3-layer dense neural network of size 800x300x10 on MNIST and CIFARdatasets to attain 96% and 45% test accuracy, respectively (see Figure 3).
Moreover, our experiments show that ensemble of 10 models (with different permutation seeds) yields around 98% and 48% (see Figure 3) accuracy on clean test images of MNIST and CIFAR-10, respectively.
Note that the goal here is not to achieve the best possible classiﬁer trained in the permutation-phase domain, rather to show that even with a naive 3-layer dense neural network without any tricks such as data augmentation, deep residual nets, etc., it is possible to reach certain level of learning.
Moreover, as shown in the following subsections, even with such simple networks, PPD can attain the state-ofthe-art performance on adversarial examples on both MNIST and CIFAR-10 datasets.
We believe that better results on clean images automatically translate to better results on adversarial examples and encourage research community to start a new line of research on how to train more accurate classiﬁers in the permutation-phase domain.
4. R OBUSTNESS OF PPD A GAINST A TTACKS
Now let’s see how PPD behaves against state-of-the-art attacks.
Cleverhans library v2.1.0 (Papernot et al., 2018) is used for attack implementations.
Each attack requires a set of parameters as input to control the algorithm as well as the perturbation strength.
Various set of parameters for different attacks make it difﬁcult to compare attacks against each other.
To overcome this issue, we decided to deﬁne an average perturbation measure to evaluate adversary’s strength and provide a uniﬁed measure to compare attacks against each other:
[Equation]
where x i is the original image (vectorized) and x (cid:48)
i is the adversarial image (vectorized).
The norm used in this measure can be any norm, but we focus on (cid:96) ∞ and (cid:96) 2 norms (and call corresponding perturbations (cid:96) ∞ and (cid:96) 2 -perturbation, respectively) here as most attacks are speciﬁcally designed to perform well in these norms.
However, robustness of PPD is not restricted to speciﬁc type of norms.
Average perturbation measure is essential to make claims about a defense.
To elaborate more on this point, suppose that adversary can change any pixel by 0.5, which means it can modify any image to a fully gray image and fool any classiﬁer completely.
Therefore, to quantify the robustness of defenses, it is crucial to mention how much perturbation adversary is allowed to make.
Guo et al.
[Equation]
considered
[Equation]
as a perturbation measure.
However, division by (cid:107) x i (cid:107) 2 makes the perturbation measure asymmetric for dark (low value of (cid:107)
Implementation Detail:
A 3-layer dense neural network of size 800x300x10 is used to train on
MNIST and CIFAR-10 datasets.
The ﬁrst two layers are followed by relu activation and the output layer is followed by softmax.
Pixels are normalized to [0 , 1] for both datasets.
The pixel2phase block is implemented in Tensorﬂow so that the derivative of this transformation is available for attacks to use.
Given a certain level of perturbation strength, attacks parameters were tuned to provide maximum rate of fooling on the model used by adversary to generate adversarial examples.
For example, for iterative attacks such as MIM, BIM, PGD and CW, number of iterations was increased to a level above which it could not fool the network any further.
Parameter tuning provides a fair comparison among different attacks of certain strength and is missing from previous work.
Thus, in addition to reporting attack parameters, we encourage researchers to evaluate attacks using perturbation measure in (3) and then ﬁne tune parameters to get the best fooling rate under the ﬁxed distortion level.
This gives a common ground to compare different attacks.
Adversary’s Knowledge:
The permutation seed is not revealed to the adversary.
Thus, adversary is granted knowledge of training dataset and parameters (such as learning rate, optimizer, regularization parameters, etc.). Moreover, it can probe the network with input images of it’s choice and receive the output label probabilities.
4.2. A TTACKS
Adversaries try to limit perturbation with respect to speciﬁc norms with the goal of keeping the adversarial modiﬁcation imperceptible to human eye.
Most adversarial attacks proposed so far are designed to keep (cid:96) ∞ or (cid:96) 2 perturbations limited.
Thus, we restrict our evaluations to these two groups by considering FGSM (Goodfellow et al., 2014), BIM (Kurakin et al., 2016), MIM (Dong et al., 2018) and (cid:96) ∞ -PGD (Madry et al., 2017) as (cid:96) ∞ oriented attacks and CW (Carlini & Wagner, 2016), (cid:96) 2 -PGD (Madry et al., 2017), (cid:96) 2 -FGM (Goodfellow et al., 2014) as (cid:96) 2 oriented attacks.
However, we should emphasize that PPD robustness is not restricted to distortions with speciﬁc norms.
Conceptually, it is supposed to provide more general robustness.
This is in particular important because previously proposed adversarial training using (cid:96) ∞ -PGD (Madry et al., 2017) showed robustness to (cid:96) ∞ attacks of the same strength, but was broken by (cid:96) 1 oriented attacks (Sharma & Chen, 2018).
Other attacks such as LBFGS (Szegedy et al., 2013), DeepFool (Moosavi-Dezfooli et al., 2016), JSMA (Papernot et al., 2016) and EoT (Athalye & Sutskever, 2017) were not effective on PPD, so we ignored reporting the results here.
In addition, our efforts to attack PPD in the phase domain (i.e., adversary generates adversarial phase using a neural network trained in the phase of a permuted image and then combine the adversarial phase with benign magnitude to recover adversarial image in the pixel domain) were not fruitful, so we are not reporting them either.
[Table]
To get a sense of perturbation strength, note that if adversary can modify each pixel by 0 . 5 , it can fool any classiﬁer by converting the image to a complete gray image.
This translates to (cid:96) ∞
distortion of 0 . 5 , and (cid:96) 2 distortion of 14 and 27 .
71 for MNIST and CIFAR-10, respectively 2 .
Moreover, it is worth to compare the results with images distorted by random noise of same strength.
Random noise cannot fool the neural networks (Fawzi et al., 2016).
Thus, it gives a good understanding of how effective attacks are.
(cid:96) ∞ random noise of strength p is generated by adding ± p uniformly at random to each pixel.
To generate (cid:96) 2 random noise of strength p , a matrix with the same size as the image is generated with elements chosen iid from [ − 0 .
This matrix is then scaled (to satisfy (cid:96) 2 norm of p ) and added to the image.
Figures 4 and 5 show performance of PPD against (cid:96) ∞ and (cid:96) 2 attacks, respectively.
A group of
PPD models (each for a different permutation seed) is trained on training data.
Adversary is given one of these PPD models as well as the test data to generate adversarial examples.
The resulting adversarial images are then fed to the ensemble of other models to evaluate accuracy.
This setup is to meet the assumption that permutation seed is hidden from adversary.
Figures 4 and 5 show that except for large perturbations, adversarial attacks have not been more destructive than random noise distortion.
This supports the claim that PPD essentially hides salient pixels from adversary in a way that adversarial perturbation simply acts like random noise.
Note that although we perform experiments on 50 PPD models, ensemble of 10 of them is sufﬁcient to achieve this level of accuracy (see Figure 3).
Another attack scenario is Blackbox (Papernot et al., 2017) where adversary probes an ensemble of PPD models as a black box and tries to train a substitute model to mimic decision boundaries of the target model.
The substitute model is then used to craft adversarial images.
We found that Blackbox attack is effective on MNIST dataset.
Thus, a standard denoising preprocessing is used before feeding to PPD.
Table 1 provides corresponding numerical values.
4.2. C OMPARISON WITH M ADRY ET AL . (2017)
As a solid step to understand and guard against adversarial examples, Madry et al.
(2017) proposed
PGD as possibly the strongest (cid:96) ∞ attack and claimed that adversarial training with examples generated by PGD secures the classiﬁer against all (cid:96) ∞ attacks of the same strength.
Given that some other defenses (Dhillon et al., 2018; Kannan et al., 2018) are circumvented (Athalye et al., 2018;
Engstrom et al., 2018), to the best of our knowledge, adversarial training with PGD gives the current state-of-the-art on both MNIST and CIFAR-10, achieving 88.8% accuracy on (cid:96) ∞ attack of strength 0.3 for MNIST 3 and 44.7% on (cid:96) ∞ attack of strength 0.03 on CIFAR-10 4 .
With the same attack budget, our worst accuracy on MNIST and CIFAR-10 are 87.4% and 47.6%, respectively (Table 1).
Increasing (cid:96) ∞ attack budget to 0.4 for MNIST and 0.1 for CIFAR-10 brings Madry’s accuracy down to 0% and 10%, respectively (Figure 6 of Madry et al.
This is while PPD still gives worst case accuracy of 67.5% and 45.1%, respectively. [Table] [Table]
[Figure] [Figure] Figure On the other hand, Madry’s defense is not robust against (cid:96) 1 (Sharma & Chen, 2017) and (cid:96) 2 attacks.
In particular, (cid:96) 2 attack of strength 0.35 can decrease Madry’s accuracy down to 10% on CIFAR-
(Figure 6 of Madry et al.
This is while PPD retains signiﬁcant resistance to (cid:96) 2 perturbations (Table 1).
C ONCLUSION
In this paper, we proposed a novel approach that combines random permutations and piexel2phase as a solid defense against a large collection of recent powerful adversarial attacks.
Inspired by symmetric cryptography, which relies merely on safekeeping of the keys for security, concealing the selected permutation ensures the effectiveness of our adversarial defense.
We demonstrated the effectiveness of our approach by performing extensive experiments on the MNIST dataset and tried to extend this approach to the CIFAR-10 dataset with competitive results.
We argue that our approach can easily create multiple models that correspond to different permutations.
In addition, using ensemble of multiple models for ﬁnal prediction is an inevitable cost for an effective defense.
A. V ISUALIZATION OF P ERTURBED I MAGES
F G S M
Clean 0.
M I M
B
I M
P G D
U n i f
o r m
N o
i s e
F G M
Clean 0.
P G D
C W
U n i f
o r m
N o
i s e
F G S M
Clean 0.
M I M
B
I M
P G D
U n i f
o r m
N o
i s e
F G M
Clean 0.
P G D
C W
U n i f
o r m
N o
i s e