A P ERTURBATION A NALYSIS OF I NPUT T RANSFORMATIONS FOR A DVERSARIAL A TTACKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Many defenses for Convolutional Neural Networks are based on a simple observation that the adversarial examples are not robust and small perturbations to the attacking input often recover the desired prediction.
Intuitively, the adversarial examples occupy a very small cone in the decision space which is surrounded by a large area corresponding to the correct class.
While the intuition is simple, a detailed understanding of this phenomenon is missing from the research literature.
We identify a family of defense techniques that are based on the instability assumption.
The defenses include deterministic lossy compression algorithms and randomized perturbations to the input that all lead to similar gains in robustness.
We present a comprehensive experimental analysis of when and why perturbation defenses work and potential mechanisms that could explain their effectiveness (or ineffectiveness) in different settings.
I NTRODUCTION
The attacks on Convolutional Neural Networks, such as Carlini & Wanger (Carlini & Wagner, 2017) or PGD (Madry et al., 2017), generate strategically placed modiﬁcations that can be easily dominated by different types of perturbations resulting in correct predictions (Dziugaite et al., 2016; Roth et al., 2019).
This suggests that the standard adversarial examples are not robust.
Many defense techniques explicitly leverage this property and can be retrospectively interpreted as perturbations of the input images.
However, a detailed understanding of this phenomenon is lacking from the research literature including: (1) what types of perturbations work and what is their underlying mechanism, (2) whether all attacks exhibit this property, and (3) possible counter-measures attackers can employ to defeat perturbation defenses.
We can interpret a large number of recent defenses as a type of input perturbations, for example, feature squeezing (Xu et al., 2017), frequency or JPEG compression (Dziugaite et al., 2016), randomized smoothing (Cohen et al., 2019), and perturbation of network structure or the inputs randomly (JafarniaJahromi et al., 2018; Zhang & Liang, 2019;
Guo et al., 2017).
The defense techniques exhibit very similar gains in robustness.
To show it, we start with a simple model where every example is passed through a lossy channel (stochastic or deterministic) prior to model inference.
This channel induces a small perturbation to the input.
We optimize the perturbation to be small enough as not to affect the prediction accuracy on clean examples but large enough to dominate any adversarial attack.
We ﬁnd that this trade-off is surprisingly consistent across very different families of input perturbations, where the relationship between channel distortion (the L 2 distance between channel input and output) and robustness is very similar.
Why are some state-of-the-art attacks are sensitive to perturbation-based defenses?
We ﬁnd that many attacks execute an optimization procedure that ﬁnds an adversarial image that is very close to the original image in terms of of L 1 , L 2 , or L ∞ norm.
The resultant optimum, i.e., the adversarial image, tends to exhibit a higher level of instability than natural examples, which we demonstrate from the perspective of a ﬁrst-order and second-order analysis.
By instability we mean that small perturbations of the example can affect the prediction conﬁdences.
The uniﬁcation of perturbation-based defense also gives us some insight into how an attacker might avoid them.
Our experiments suggest that all the perturbation based defenses are vulnerable to the same types of attack strategies.
We argue that the optimization procedure in the attacker should ﬁnd the smallest distance from the original image that closes the recovery window.
In fact, we can devise a generic attacker that attacks a particularly strong lossy channel, based on the additive Laplace noise, and adaptive attacks designed on this channel are often successful against other defenses.
This result implies that for many input perturbation defenses the attacker need not be fully adaptive, i.e., they do not need to know exactly what kind of transformation is used to defend the network.
R ELATED WORK
Much of the community’s current understanding of adversarial sensitivity in neural networks is based on the seminal work by Szegedy et al.
Multiple contemporaneous works also studied different aspects of this problem, postulating linearity and over-parametrization as possible explanations (Goodfellow et al., 2014; Biggio et al., 2013).
Since the beginning of this line of work, the connection between compression and adversarial robustness has been recognized.
The main defense strategies include: the idea of defensive network distillation 1 (Papernot et al., 2015), quantizing inputs using feature squeezing (Xu et al., 2017), the thermometer encoding as another form of quantization (Buckman et al., 2018), JPEG compression harnessed by (Dziugaite et al., 2016;
Guo et al., 2017; Das et al., 2017; 2018;
Aydemir et al., 2018; Liu et al., 2019).
Other line of research leveraged connection between randomization and adversarial robustness: Pixel Deﬂection (Prakash et al., 2018), random resizing and padding of the input image (Xie et al., 2017), and total variance minimization (Guo et al., 2017).
In our work we unify the methods based on compression and randomization that are applied to the input images.
While all of the aforementioned defenses were later broken (Carlini & Wagner, 2017; Athalye et al., 2018), it is important to understand why these approaches afforded any form of robustness.
The community actually lacks consensus on this point: Szegedy et al.
(2014) suggest that neural networks have blind spots , Xu et al.
(2017) suggest that quantization makes the adversarial search space smaller,
Buckman et al.
(2018) suggest the linearity is the main culprit and quantized inputs break up linearity.
Zhang & Liang (2019) inject random Gaussian noise into an image and then discretize it.
This method ﬁts into the noisy channel framework with different deﬁnitions of C ( x ) .
They show improved performance of this combined model in the non-adaptive setting.
Our experiments ﬁnd that simply injecting Gaussian noise (or even Uniform or Laplace noise) is an equally effective defense method.
We also show that the discretization is not essential to good performance if the level of noise is appropriately tuned.
The imprecise channel defense in a neural network is also related to the idea of gradient masking or gradient obfuscation, i.e., a hard to differentiate layer (Papernot et al., 2017).
In this work, the backward pass computation is perturbed to make it difﬁcult for a gradient-based attack to synthesize an adversarial image (while the forward pass is kept the same).
We implement both non-adaptive attacks and adaptive that can observe the channel and take an approximate gradient through it.
We further focus our study on the families of white-box attacks proposed by Carlini &
Wagner (2017), and their adaptive variants.
Noise injection can be much more powerful than regularization or a dataset augmentation method.
The dropout algorithm can be seen as applying noise to the hidden units.
The dropout randomization (Feinman et al., 2017) was used to create a defense that was not completely broken and required a high distortion added to the adversarial examples (Carlini & Wagner, 2017).
Many new defenses propose randomization through noise injection without considering the adversarial training (Zhang &
Liang, 2019; Cohen et al., 2019).
The work on injection of noise into inputs and each of the layers of neural networks by Liu et al.
(2018) is a strong heuristic that led to defenses with theoretical guarantees.
The random smoothing provides a certiﬁed robustness up to a certain threshold of input distortion (and is not designed to be robust beyond the threshold) by utilizing inequalities from the differential privacy literature (Lecuyer et al., 2018).
Cohen et al.
(2019) improve the theoretical bounds of methods that randomly smooth the input examples.
Recent work focuses on combination of randomized smoothing with adversarial training and achieves state of the art in terms of the provable robustness (Salman et al., 2019).
L OSSY C HANNEL M ODEL
We consider convolutional neural networks that take w × h (width times height)
RGB digital images as input, giving an example space of X ∈ (255) w ×
h × 3 , where ( z ) denotes the integer numbers from 0 to z .
We consider a discrete label space of k classes represented as a conﬁdence value Y ∈ [0 , 1] k .
Neural networks are parametrized functions (by a weight vector θ ) between the example and label spaces f ( x ; θ ) :
X (cid:55)→ Y .
An adversarial input x adv is a perturbation of a correctly predicted example x that is incorrectly predicted by f .
[Equation]
The distortion is the (cid:96) 2 error between the original example and the adversarial one:
[Equation]
Approximating f ( · ) with a less precise version ¯ f ( · ) can counter-intuitively make it more robust (Dziugaite et al., 2016):
[Equation]
Intuitively, a lossy version of f introduces noise into a prediction which dominates the strategic perturbations found by an adversarial attack procedure.
It turns out that we can characterize a number of popular defense methodologies with this basic framework.
Let x be an example and f be a trained neural network.
Precise evaluation means running f ( x ) and observing the predicted label.
Imprecise evaluation involves ﬁrst transforming x through a deterministic or stochastic noise process C ( x ) =
C [ x (cid:48) | x ] , and then evaluating the neural network
[Equation]
We can think of C ( x ) as a noisy channel (as in signal processing).
The distortion of a C ( x ) is the expected (cid:96) 2 reconstruction error:
[Equation]
2 ] , which is a measure of how much information is lost passing the example through a channel.
This paper shows that there is a subtle trade-off between δ c and δ adv .
In particular, we can ﬁnd δ c such that δ c >> δ adv and f ( x ) =
f ( C ( x adv )) .
We show that compression and randomization based techniques exhibit this property.
E XAMPLE OF D ETERMINISTIC C HANNEL
When C ( x ) is deterministic it can be thought of as a lossy compression technique.
Essentially, we run the following operation on each input example:
[Equation]
One form of compression for CNNs is color-depth compression.
Most common image classiﬁcation neural network architectures convert the integer valued inputs into ﬂoating point numbers.
We abstract this process with the norm function that for each pixel n ∈ (255) maps it to a real number v ∈ [0 , 1] by normalizing the value and the corresponding denorm function that retrieves the original integer value (where (cid:98)(cid:101) denotes the nearest integer function) 2 :
[Equation]
This process is normally reversible v = norm ( denorm ( v )) , but we can artiﬁcially make this process lossy.
Consider a parametrized C ( · ) version of the color-depth compression function:
[Equation]
By decreasing b by ∆ b we reduce the ﬁdelity of representing v by a factor of 2 ∆ b (for the b bits of precision).
E XAMPLE OF S TOCHASTIC P ERTURBATION
The channel model is particularly interesting when C ( x ) is stochastic.
Randomization has also been noted to play a big role in strong defenses in prior work (Madry et al., 2017; Zhang & Liang, 2019;
Cohen et al., 2019).
For example, we could add independent random noise to each input pixel:
[Equation]
We consider two schemes, Gaussian (cid:15) ∼ N (0 , σ ) and additive Uniform noise (cid:15) ∼ U ( − B, B ) which add independent noise to each pixel.
One of the advantages of randomization is that an adversary cannot anticipate how the particular channel C will transform an input before prediction.
However, there is another subtle advantage to randomization.
Randomized approaches can partially recover their loss in accuracy due to imprecision by averaging over multiple instances of the perturbation.
In classiﬁcation problems, we can take the most frequent label seen after T perturbation trials:
[Equation]
3. P ERTURBATION A NALYSIS
While the intuition is that the channel’s perturbations dominate strategically placed distortions in an adversarial example, the underlying mathematical mechanism of why recovery is possible is not clear.
We start with the hypothesis that synthesized adversarial examples have unstable predictions–meaning that small perturbations to the input space can change conﬁdence values drastically.
How do we quantify instability?
Let f ( x ) be a function that maps an image to a single class conﬁdence value (i.e., a scalar output).
We want to understand how f ( x ) changes if x is perturbed by (cid:15) .
We can apply a Taylor expansion of f around the given example x :
[Equation]
where ∇ x f ( x ) denotes the gradient of the function f with respect to x and ∇ 2 x f ( x ) denotes the
Hessian of the function f with respect to x .
The magnitude of the change in conﬁdence is governed by the Taylor series terms in factorially decreasing importance.
(cid:107) (cid:15) (cid:107) 2 is exactly the distortion measure δ c described at the beginning of Section 3.1.
Thus, the expression is bounded in terms of the operator norm, or the maximal change in norm that could be induced, of each of the terms (see Appendix A):
[Equation]
As ∇ x f ( x ) is a vector, this is simply the familiar (cid:96) 2 norm, and for the second order term this is the maximal eigenvalue:
[Equation]
When M 1 and M 2 are larger this means there is a greater propensity to change the prediction for small perturbations.
We will show experimentally that for certain types of attacks the M 1 and Mvalues around adversarial examples exhibit signs of instability compared to those around natural examples–suggesting a mathematical mechanism of why recovery is possible.
E XPERIMENTS
Our experiments evaluate the efﬁcacy of imprecision based defenses in a number of different adversarial problem settings.
4. E XPERIMENTAL S ETUP
We run our experiments using ResNet-18 on CIFAR-10 and ResNet-50 on ImageNet dataset using
P-100 GPUs (16GB memory).
We explore a number of different attacks that are implemented in the foolbox library (Rauber et al., 2017).
In each experiment we measure the test accuracy (%), the conﬁdence of predictions, and distances between the original images and either their adversarial counterparts or the recovered images after applying one of the defenses.
We present our results for non-targeted attacks ; if the adversary is successful it induces any misclassiﬁcation.
We experiment with many gradient-based attacks provided in the foolbox library that explore different optimization algorithms and distance measures, for instance:
• LBFGS minimizes the distance between the input image and the adversarial example as well as the cross-entropy between the predictions for the adversarial and the input image; introduced by Szegedy et al.
(2014) and further extended in Tabacof & Valle (2015).
• Carlini-Wagner L 2 (C&W L 2 ) is a generalization of the LBFGS attack that is devised after exhaustive search over possible space of: norms, loss functions, box optimization procedures, etc.
(Carlini & Wagner, 2017).
• BIM L 1 is a modiﬁed version of the Basic Iterative Method that minimizes the L 1 distance (Kurakin et al., 2016).
• FGSM adds the sign of the gradient to the image, gradually increasing the magnitude until the image is misclassiﬁed Goodfellow et al.
• PGD L ∞ the Projected Gradient Descent Attack that is an iterative version of the FGSM attack; we use the version that minimizes the L ∞ distance (Madry et al., 2017).
We extend the Carlini-Wagner L 2 attack in its adaptive version so that the gradients are not obfuscated.
We approximate the gradients for the backward pass on the compression layers as an identity function, similarly to (He et al., 2017; Athalye et al., 2018).
More details on the setup can be found in the
Appendix, Section C.
1. P ERTURBATION D EFENSES G IVE S IMILAR G AINS I N R OBUSTNESS
In the ﬁrst experiment, we select 1000 random images from ImageNet and CIFAR-10 datasets.
For each of these images we generate an adversarial attack with popular white-box gradient-based methods.
Then, we run the adversarial images through different channels: Frequency Compression (FC), Color Depth reduction (CD), Uniform noise (Unif), Gaussian Noise (Gauss), SVD-based compression (SVD), Identity (Iden).
The identity channel just passes through the adversarial image with no modiﬁcation.
We measure the accuracy of f ( C ( x )) , which indicates the ability of the imprecise channels to recover the original label.
We present the results in Figure 1 and also in the
Supplement in Figure 7 (for all images in the test CIFAR-10 and dev ImageNet sets) and in Table(for different channel parameters and ﬁve attacks).
When there is an identity channel, the adversarial attack is always successful.
However, each of the imprecise channels is able to recover a substantial portion of original labels from the adversarial examples.
It is important to note that we are evaluating these attacks in the setting, where any mis-classiﬁcation is considered a success and the adversary is not aware of the defense.
Importantly, all the channels can be tuned to recover very similar maximum accuracy after attacks.
For example, all the channels can achieve about 85% accuracy for CIFAR-10 after nonadaptive PGD or C&W attacks.
This suggests that any form of imprecision with the right error magnitude is effective at defending against these types of attacks.
We observe that the attacks that incur higher distortions such as FGSM or LBFGS decrease the accuracy more than the iterative attacks such as C&W L 2 or PGD L ∞ .
This is because the iterative attacks ﬁnd the adversarial images that are closer to the original images in terms of the corresponding distance measure that they optimize for in the input space.
The key is to ensure that the error introduced by the imprecise channels is big enough to dominate the adversarial perturbations but small enough to generate valid predictions.
Figure 1 illustrates this relationship (see also a detailed analysis for a single image presented in Figure 9 in the Appendix).
For ﬁve different imprecise channels, we plot the channel distortion against the accuracy for CIFAR-10 and ImageNet datasets.
We analyze FC, CD, Unif,
Gauss, and SVD channels.
The curves are qualitatively similar in the low noise regime, but they show more differences when higher distortions are incurred.
The lowest accuracy across the datasets for high distortions is observed for content-preserving FC and SVD compression channels.
The Gaussian and Uniform channels have very similar trends.
They outperform other channels on ImageNet for
[Figure]
[Table]
large distortions but are less performant on low-resolution CIFAR-10 images, where the CD channel achieves higher accuracy.
2. A TTACKS ARE T RANSFERABLE
Many input transformation defenses are broken.
If the attacker has full knowledge of the defense, it is possible to construct an attack that is impervious to the defense.
This is called the adaptive setting.
Since the underlying mechanisms of input transformations are similar
[Table]
we ﬁnd that an attacker does not need to be fully adaptive .
The attacker can assume a particular strong defense and that same adversarial input often transfers to other defenses.
We narrow the attacker to a single adaptive step (for details see Section C.8 in the Appendix).
Even in this weak adaptive setting, the deterministic channels are fully broken but the randomized channels retain relatively high accuracy above 23.8%.
We show in the Appendix in Figure 14 that the randomized defenses can also be broken when the adversary is given an unlimited number of adaptive steps.
Laplace attacked images transfer the best to other defenses .
They decrease the accuracy of the defense models by at least 44.3% (for Laplace itself).
Table 1 shows that FC attacked images do not transfer well to other defenses; the maximum drop in accuracy of the model protected by other defenses is 12.26%.
Most adversarial images (against a given defense) transfer very well to the FC defense, i.e. an adversarial image against any defense (e.g. CD, SVD, Gauss, Uniform, or Laplace) is also adversarial against the FC defense.
The adversarial images generated against the Uniform defense show better transfer to other defenses in comparison to the adversarial images generated against the Gaussian defense.
This is because the higher noise level is applied in the Uniform defense.
We observe analogous trends for the ImageNet dataset and present the results in the supplement (Tables 6 and 7).
3. R ECOVERABLE R ANGE SHRINKS WITH HIGHER ATTACK DISTORTION
We use an example from the ImageNet dataset, the ResNet-50 model, and set the stochastic channel to the Gaussian noise.
We start from an adversarial example generated with the Carlini & Wagner (non-adaptive)
L 2 attack and for consecutive subplots (in the left to right and top to bottom sequence), we increase the attack strength and incur higher distortion of the adversarial image from the original image.
For a single plot, we increase the L 2 distance of the output from the channel to the adversarial example by increasing the Gaussian noise (controlled by parameter σ ).
For L 2 distances incurred by different noise levels, we execute 100 predictions.
In Figure 9 in the Appendix, we use the frequency count and report how many times the model predicts the original, adversarial or other class.
The plot shows what range of distances from the adversarial image reveal the correct class.
For the adversarial examples that are very close to the original image (e.g. adversarial distance of 0.006 for the top-left ﬁgure), the window of recovery (which indicates which strengths of the random noise can recover the correct label) is relatively wide, however, as we increase the distance of the adversarial image from the original image, the window shrinks and ﬁnally we are not enable to recover the correct label.
For attacks that incur more input distortion or the stronger white-box attacks, we can resort to other statistics as proposed in Roth et al.
They show that adversarial examples are much closer to the unperturbed sample than to any other neighbor and use the fact that the probability of the correct class increases faster than the probability of the highest other class when adding noise with a small to intermediate magnitude to the adversarial example.
4. C OMPARISON WITH OTHER METHODS
[Figure] [Figure]
In Figure 2 we present the accuracy of different channels as the strength of the C&W attack is systematically increased.
The strength is controlled by the c tradeoff-parameter that is used to set the relative importance of distance and conﬁdence.
We present a related approach which is the RSE (Random Self-Ensemble) network with 0.2 noise level in the ﬁrst layer and 0.1 noise level in the remaining layers (as recommended in Liu et al.
This defense does better for lower distortion levels (c value below 0.1) than other noisy channels, but then its accuracy deterioration is faster for higher distortion levels.
The Laplace channel gives the highest accuracy for high values of the c parameter (above 1.0).
The CD, FC, SVD, Gauss, and
Uniform channels show similar trends.
We also include a very simple channel that reduces brightness of an image by subtracting an arbitrary value from each pixel.
The comparison between very complex approaches and a simple input transformation is informative–as they largely follow the same trends.
5. A DVERSARIAL E XAMPLES A RE U NSTABLE
[Figure]
First-order analysis.
The key question is why adversarial examples are more sensitive to perturbations than natural inputs, when there is evidence that from an input perspective they are statistically indistinguishable.
Our experiments suggest that this sensitivity arises from the optimization process that generates adversarial inputs.
Based on the operator-norm analysis presented in Section 3.2, we plot in Figure 4 the L 2 norm of gradients w.r.t.
the original x org and adversarial x adv images for original (correct) c org and adversarial classes c adv .
The work by Simon-Gabriel et al.
(2019) also evaluates the norm of gradients of the network output with respect to its inputs.
They show that the adversarial examples are primarily caused by large gradients of the classiﬁer as captured via the induced loss.
At ﬁrst-order approximation in (cid:15) , an (cid:15) -sized
L 2 untargeted adversarial attack increases the loss L at point x by (cid:15) ||
∂ x L ( x, c org )
||
Analogously, at ﬁrst-order approximation in (cid:15) , an (cid:15) -sized
L 2 targeted adversarial attack decreases the loss L at point x by (cid:15) ||
∂ x L ( x, c adv )
|| 2 .
In our experiments, the recovered original images have lower norms of the gradients for both original and adversarial classes than the not recovered images.
Additionally, the gradient w.r.t.
original image for the original class is much smaller than the gradient w.r.t.
adversarial image for the adversarial class.
We also observe that the magnitude of the gradient increases smoothly as we systematically add more Gaussian noise to the original image (Figure 16).
For the adversarial image, the gradient w.r.t.
adversarial image and for adversarial class decreases as we increase the attack strength and the conﬁdence of adversarial predictions.
Second-order analysis.
Our results in Figure 3 show that the adversarial inputs lead to noticeably higher Hessian spectrum than the original inputs.
This suggests that the model predictions for the adversarial inputs are less stable than for the original images.
Thus, perturbations of the adversarial images with some form of noise can easily change the classiﬁcation outcome while the prediction for the original images are much more robust and do not lead to such unstable predictions.
C ONCLUSION , L IMITATIONS , AND F UTURE W ORK
The non-adaptive attacks are not robust since small changes to the adversarial input often recover the original label.
This is an obvious corollary to the very existence of adversarial examples that by deﬁnition are relatively close to correctly predicted examples in the input space.
Random perturbations of the input can dominate the strategically placed perturbations synthesized by an attack.
In fact, the results are consistent across both deterministic and stochastic channels that degrade the ﬁdelity of the input example.
From the perspective of the attacker, the recovery window can be closed to make the perturbation based recovery techniques ineffective.
A P ERTURBATION ANALYSIS : ADDENDUM
(cid:15) T ∇ x f ( x ) + 1 2 (cid:15) T ∇ 2 x f ( x ) (cid:15) + ... ≤ δ c M 1 ( x ) + 1 2 δ 2 c M 2 ( x ) + ...
M 1 ( x )
=
(cid:107)∇ x f ( x ) (cid:107) 2 M 2 ( x ) =
λ
max ( ∇ 2 x f ( x )) (1) (cid:15) T ∇ x f ( x ) ≤ δ c M 1 ( x )
From the Cauchy-Schwarz inequality: (cid:15) T ∇ x f ( x ) ≤ || (cid:15) ||
2 M 1 ( x ) || (cid:15)
||
2 M 1 ( x ) =
δ adv M 1 ( x ) ≤ δ c M 1 ( x ) ( since δ adv << δ c ) (2) ∇ 2 x f ( x ) (cid:15) ≤ δ 2 c M 2 ( x )
From the deﬁnition of maximum eigenvalue : λ max ≥ (cid:15) T ∇ 2 x f ( x ) (cid:15) (cid:15) T (cid:15) (cid:15) T ∇ 2 x f ( x ) (cid:15) ≤ || (cid:15)
||
22 λ max = δ 2 adv λ max ≤ δ 2 c λ max
B C OMPRESSION TECHNIQUES B. FFT- BASED COMPRESSION
We apply compression in the frequency domain to reduce the precision of the input images.
Let x be an input image, which has corresponding Fourier representation that re-indexes each tensor in the frequency domain:
[Equation]
This Fourier representation can be efﬁciently computed with an FFT.
The mapping is invertible x = F − 1 ( F ( x )) .
Let M f [ ω ] be a discrete indicator function deﬁned as follows:
[Equation]
M f [ ω ] is a mask that limits the F [ ω ] to a certain band of frequencies.
f represents how much of the frequency domain is considered.
The band-limited spectrum is deﬁned as, F [ ω ] · M f [ ω ] , and the band-limited ﬁlter is deﬁned as:
[Equation]
B. SVD- BASED COMPRESSION
Analogously to the FFT-based method, we decompose an image with SVD transformation and reconstruct its compressed version with dominant singular values.
The basis used in SVD are adaptive and determined by an image, as opposed to pre-selected basis used in FFT.
This can result in higher quality for the same compression rate in case of SVD, however it is more computationally intensive than FFT-based compression.
C A DDITIONAL EXPERIMENTS FOR W HITE - BOX A TTACKS C. A CCURACY OF P ERTURBATION D EFENSES ON C LEAN D ATA
One pitfall of the imprecise channel defense is that it introduces errors whether or not there are any adversarial examples.
The errors act as an upper-bound for the best possible test accuracy we can get under adversarial perturbations.
For frequency compression, color depth compression, and uniform noise injection, we compare the test accuracy for different levels of imprecision.
Table 2 shows the results for all test images from CIFAR-10 on the ResNet-18 architecture, for three of the imprecise channels, and for different noise settings.
[Table]
[Figure]
[Table]
[Figure]
The test accuracy of the models can be increased by training with compression, e.g., by using FFT based convolutions with 50% compression in the frequency domain increases the accuracy to 92.32%.
We present the results for six different noisy channels; three of them are compression based: FC,
CD, SVD, and other three add different type of noise: Gauss, Uniform, and Laplace.
For each of the compression based channels, we increase the compression rate systematically from 0 to about 90% (in case of the CD channel, the compression rate is computed based on how many bits are used per value).
For the noise based channels, we increase the strength of the noise by controlling the epsilon parameter (cid:15) (in case of the Gaussian noise, it corresponds to the sigma parameter σ ).
The full result is presented in Figure 5.
[Figure] [Figure]
rate, while for 7 bit integer the accuracy drops slightly due to imprecision.
Figure 6 presents the test accuracy (%) after four different attacks (3 different norms) and the recovery via the uniform noise stochastic channel in more detail.
C. C HANNEL DISTORTION VS ACCURACY
We show channel distortion vs accuracy for the C&W L 2 attack on the full CIFAR-10 test set and full ImageNet dev set in Figure 7.
We further show the performance of the defenses across different attacks and 1000 images in Table 3.
[Table]
C. C HANNEL ACCURACY ON CLEAN AND ADVERSARIAL EXAMPLES
We compare the accuracy of the perturbation channels on clean and adversarial examples in Figure 8.
We plot the results for the whole test set from CIFAR-10 and the whole validation set from ImageNet.
We use two deterministic channels (FFT compression denoted by FC and SVD compression) and two noisy channels (Gaussian and Uniform noise).
We use C&W and PGD attacks.
The FC Clean label denotes that we pass clean images through the channel that applies FFT compression.
The SVD C&W label denotes that we pass adversarial images found with the C&W attack through the channel that applies SVD compression.
We tune the channels for a given dataset across the attacks, and present the results in Table 4.
The accuracy of the channels on clean data gives us the upper bound for the accuracy on the adversarial examples.
Thus, we choose the channel parameter based on the highest accuracy on the adversarial images.
[Table]
C. N EIGHBORHOOD OF ADVERSARIAL EXAMPLES
Please, see Figure 9 and description in Section 4.1.
C. D ISTRIBUTIONS OF DELTAS BETWEEN INPUTS & OUTPUTS FOR CHANNELS
We plot the distribution of deltas for six imprecise channels in Figure 10.
We compute the deltas by subtracting an original image from the perturbed adversarial image and plot the histograms of differences.
We use an image from the ImageNet dataset.
For all the examples, the correct labels were recovered.
We use the C&W attack with 1000 iterations and the initial value c = 0 .
The CD channel resembles the Uniform distribution.
The FFT and SVD compression methods belong to double-sided exponential distributions, thus they are more related to the Laplace distribution than to the Gaussian distribution.
[Figure]
[Figure] [Figure]
bits
[Figure]
FC (10%
[Figure]
dist
[Figure] [Figure]
Laplace
[Figure] [Figure]
L
[Figure] [Figure]
C.
V ISUALIZATIONS OF ATTACKS AND IMPRECISE CHANNELS
Figure 11 presents a sample image from ImageNet for the Carlini-Wagner L2 attack.
Figure 12 shows the effect of the FC (frequency-based imprecise channel) in both spatial and Fourier domains.
C. M ULTIPLE T RIALS F OR S TOCHASTIC C HANNELS
In addition to the robustness to non-adaptive attacks, another compelling reason to use a stochastic channel (like Uniform noise) as a defense is that it can be run repeatedly in a number of random trials.
We show that doing so improves the efﬁcacy of the defense.
We randomly choose 1000 images from the CIFAR-10 test set.
For each of these images, we generate an adversarial attack.
We then pass each image through the same stochastic channel multiple times.
We take the most frequent prediction.
Figure 13 illustrates the results.
Only 16 trials are needed to get a relatively strong defense.
We argue that this result is signiﬁcant.
Randomized defenses are difﬁcult to attack.
The attacker cannot anticipate which particular perturbation to the model will happen.
The downside is a potential of erratic predictions.
We show that a relatively small number of trials can greatly reduce this noise.
Furthermore, the expense of running multiple trials of a randomized defense is small relative to the expense of synthesizing an attack in the ﬁrst place.
C. W HITE -B OX A DAPTIVE A TTACK
We consider the problem setting when the adversary knows the defense method (i.e., has full knowledge of C ( x ) ).
We use the strategy described in He et al.
(2017) to construct attacks for each case.
Not surprisingly, deterministic channels (CD, FC, and SVD) are easy for an adversary to fool when they are known.
However, such attacks incur higher distortion (distance to the original image) when compared to attacks against unprotected networks.
Intuitively, when the channel is deterministic and known, the adversary can account for the error introduced by the channel.
Table 5 illustrates the results.
[Table]
The stochastic channel is harder to attack with a gradient-based adaptive methods.
Unlike for deterministic compression, the adversary cannot anticipate the particular error pattern.
We build an adaptive attack against the additive uniform noise channel.
Our strategy is to send an output from the adversarial algorithm through the stochastic channel and the network at least as many times as set in the defense.
We mark the attack as successful if the most frequent output label is different from the ground truth.
The more passes through the noisy channel we optimize for, the stronger the attack.
Furthermore, we run many iterations of the attack to decrease the L 2 distortion.
An attack that always evades the noise injection defense is more difﬁcult to generate because of randomization.
The other randomized approach was introduced in dropout (Feinman et al., 2017).
The attacks against randomized defenses require optimization of complex loss functions, incur higher distortion, and the attacks are not fully successful (Carlini & Wagner, 2017).
In the Figure 14, we present the result of running attacks and defenses on CIFAR-10 data with single and many iterations.
The defense with many trials can be drawn to 0% accuracy, however, the defense not fully optimized by the adversary (single noise injection) can result in about 40% or higher accuracy.
C. H YBRID A PPROACHES
A natural thought is whether these defenses can be made more effective by combining them.
This is an approach that has been applied, for example, in random discretization (Zhang & Liang, 2019).
Our experiments contrast with the previous work and suggest that there is little beneﬁt to hybrid approaches.
We present an illustrative example for pair-wise combinations of Frequency Compression (FC), Color Depth (CD) compression, and Uniform (Unif) noise.
Other combinations are possible but for brevity, we exclude them from the manuscript; we found no to very small beneﬁt to combining for properly tuned channels.
We show the recovery rate, i.e. a fraction of original labels recovered.
Table
C.9 presents results for the L2 Carlini-Wagner attack on the MNIST, the whole test set of CIFAR-10, and the whole development set of ImageNet.
[Table]
MNIST
[Table]
CIFAR
[Table]
ImageNet
[Table]
We believe these results suggest that the noisy channels do not exploit anything inherent to the images.
Simply the addition of noise (through reconstruction error) is the mechanism for robustness.
Composing two different schemes usually increases this noise.
C. T RANSFERABILITY OF THE ADVERSARIAL IMAGES
We present more results on the transferability of adversarial examples between different channels in
Table 1 and Table 7).
[Table] [Table]
C. G RADIENT - BASED A NALYSIS
We run the experiment on the ImageNet dataset.
We analyze only the clean images that were classiﬁed correctly.
We present how the gradient of the loss w.r.t.
the input image changes for the correct class as we add the Gaussian noise to the original image in Figure 16.
C. H ESSIAN - BASED A NALYSIS
We present the Hessian spectrum in Figure 17 for top 20 eigenvalues and in Figure 18 the distribution of top eigenvalues of Hessians on ImageNet.
For the adversarial images, the eigenvalues are clearly higher in both cases, which indicates higher instability and proclivity to prediction changes.
C. D ETAILS ON THE EXPERIMENTAL SETUP
We use the foolbox library (Rauber et al., 2017) in our experiments.
We borrowed the nomenclature used for the attacks from the library.
For example, the name for the attack initially proposed in Szegedy et al.
and extended in Tabacof & Valle (2015) is LBFGS.
In most of our experiments, we also use the default foolbox parameters for the attacks.
For example, for PGD the initial limit on the perturbation size epsilon is set to 0.3, step size to 0.01, default number of iterations is 40.
For Carlini & Wagner, we set maximum number of iterations to 1000, learning rate to 0.005, initial value of the constant c to 0.01.
Note that for the Carlini & Wagner attack presented in 2, we use the c parameter as described in (Carlini & Wagner, 2017) and the code from Liu et al.
For the LBFGS attack, we use the epsilon parameter set to 0.00001 and up to 150 iterations.
For the FGSM attack, different epsilons starting from 50 and up
D A DDITIONAL E XPERIMENTS FOR BLACK - BOX ATTACKS
As a black box attack, we deﬁne an attack that does not need the knowledge about the gradient or the model.
D. D ECISION - BASED ATTACKS
The attacks require neither gradients nor probabilities.
They operate directly on the images.
D.1. R OBUSTNESS TO U NIFORM AND G AUSSIAN N OISE
We evaluate the robustness of band-limited CNNs.
Speciﬁcally, models trained with more compression discard part of the noise by removing the high frequency Fourier coefﬁcients (FC channel).
In
Figure 19, we show the test accuracy for input images perturbed with different levels of uniform and Gaussian noise, which is controlled systematically by the sigma parameter, fed into models trained with different compression levels (i.e., 0%, 50%, or 85%) and methods (i.e., band-limited vs.
RPA-based 3 ).
Our results demonstrate that models trained with higher compression are more robust to the inserted noise.
Interestingly, band-limited CNNs also outperform the RPA-based method and under-ﬁtted models (e.g., via early stopping), which do not exhibit the robustness to noise.
Input test images are perturbed with uniform or Gaussian noise, where the sigma parameter is changed from 0 to 1 or 0 to 2, respectively.
The more band-limited model, the more robust it is to the introduced noise.
D.1. C ONTRAST R EDUCTION A TTACK
This black-box attack gradually distorts all the pixels:
[Equation]
where min and max values are computed across all pixels of images in the dataset.
We can defend the attack with CD (Color Depth reduction) until certain value of epsilon, but then every pixel is perturbed in a smooth way so there are no high-frequency coefﬁcients increased in the
FFT domain of the image.
The contrast reduction attack becomes a low-frequency based attack when considered in the frequency domain.
Another way to defend the attack is to run a high-pass ﬁlter in the frequency domain instead of the low-pass ﬁlter.
We run the experiments for different models with CD and two band-limited models (the model with full spectra and no compression as well as model with 85% of compression - with FC layers).
The
CD does defend the attack to some extent and the fewer pixels per channel (the stronger the CD in a model), the more robust the model is against the contrast reduction attack.
Test accuracy as a function of the contrast reduction attack for ResNet-18 on CIFAR-10 (afterepochs) is plotted in Figure 19.
We control the strength of the attack with parameter epsilon that is changed systematically from 0.0 to 1.0.
We use the whole test set for CIFAR-10.
R denotes the number of values used per channel (e.g., R=32 means that we use 32 values instead of standard 256).
D.1. M ULTIPLE PIXELS ATTACK
The foolbox library supports a single pixel attack, where a given pixel is set to white or black 4 .
A certain number of pixels (e.g., 1000) is chosen and each of them is checked separately if it can lead to the misclassiﬁcation of the image.
The natural extension is to increase the number of pixels to be perturbed, in case where the single pixel attack does not succeed.
We present results for the multiple pixel attack in Figure 19.
D. S PATIAL - BASED ATTACKS
Spatial attacks apply adversarial rotations and translations that can be easily added to the data augmentation during training.
However, these attacks are defended neither by removing the high frequency coefﬁcients nor by quantization (CD) .
We separately apply rotation by changing its angle from 0 to 20 degrees and do the translations within a horizontal and vertical limit of shifted pixels (Figure 19).
D. S CORE - BASED ATTACKS
The score based attack require access to the model predictions and its probabilities (the inputs to the softmax) or the logits to estimate the gradients.
D.3. L OCAL S EARCH A TTACK
The local search attack estimates the sensitivity of individual pixels by applying extreme perturbations and observing the effect on the probability of the correct class.
Next, it perturbs the pixels to which the model is most sensitive.
The procedure is repeated until the image is misclassiﬁed, searching for additional critical pixels in the neighborhood of previously found ones.
We run the experiments for the attack on 100 test images from CIFAR-10, since the attack is relatively slow (Figure 19).
[Figure]
Original label: Loafer confidence: 0.
L2 distance: 0.
[Figure]
Adv. label: pencil box, pencil case confidence: 0.
L2 distance: 0.
[Figure] [Figure]
) label
[Figure]
Loafer confidence: 0.
L2 distance: 7.
[Figure]
FC (30) label:
Loafer confidence: 0.
L2 distance: 12.
[Figure]
Gauss (0.03) label:
Loafer confidence: 0.
L2 distance: 16.
[Figure]
Uniform (0.03) label:
Loafer confidence: 0.
L2 distance: 16.
[Figure] [Figure]
[Figure] [Figure] [Figure]
[Figure]
σ (strength of the Gaussian noise)
E x | | ∂ x  ( x , c o r g ) || [Figure] [Figure] [Figure]
[Figure]
σ (strength of the Gaussian noise)
[Figure]
( x [Figure] [Figure] [Figure]
i g
[Figure]
l u e
[Figure]
v
[Figure]
[Figure]
[Figure]