T HERMOMETER E NCODING : O NE H OT WAY TO RESIST A DVERSARIAL E XAMPLES
Anonymous authors
Paper under double-blind review
A BSTRACT
It is well known that it is possible to construct “adversarial examples” for neural networks: inputs which are misclassiﬁed by the network yet indistinguishable from true data.
We propose a simple modiﬁcation to standard neural network architectures, thermometer encoding , which signiﬁcantly increases the robustness of the network to adversarial examples.
We demonstrate this robustness with experiments on the MNIST, CIFAR-10, CIFAR-100, and SVHN datasets, and show that models with thermometer-encoded inputs consistently have higher accuracy on adversarial examples, without decreasing generalization.
State-of-the-art accuracy under the strongest known white-box attack was increased from 93.20% to 94.30% on MNIST and 50.00% to 79.16% on CIFAR-10.
We explore the properties of these networks, providing evidence that thermometer encodings help neural networks to ﬁnd more-non-linear decision boundaries.
I NTRODUCTION AND R ELATED W ORK
Adversarial examples are inputs to machine learning models that are intentionally designed to cause the model to produce an incorrect output.
The term was introduced by Szegedy et al.
(2014) in the context of neural networks for computer vision.
In the context of spam and malware detection, such inputs have been studied earlier under the name evasion attacks (Biggio et al., 2013).
Adversarial examples are interesting from a scientiﬁc perspective, because they demonstrate that even machine learning models that have superhuman performance on I.I.D. test sets fail catastrophically on inputs that are modiﬁed even slightly by an adversary.
Adversarial examples also raise concerns in the emerging ﬁeld of machine learning security because malicious attackers could use adversarial examples to cause undesired behavior (Papernot et al., 2016).
Unfortunately, there is not yet any known strong defense against adversarial examples.
Adversarial examples that fool one model often fool another model, even if the two models are trained on different training examples (corresponding to the same task) or have different architectures (Szegedy et al., 2014), so an attacker can fool a model without access to it.
Attackers can improve their success rate by sending inputs to a model, observing its output, and ﬁtting their own own copy of the model to the observed input-output pairs (Papernot et al., 2016).
Attackers can also improve their success rate by searching for adversarial examples that fool multiple different models—such adversarial examples are then much more likely to fool the unknown target model (Liu et al., 2016).
Szegedy et al.
(2014) proposed to defend the model using adversarial training (training on adversarial examples as well as regular examples) but it was not feasible to generate enough adversarial examples in the inner loop of the training process for the method to be effective at the time.
Szegedy et al.
(2014) used a large number of iterations of L-BFGS to produce their adversarial examples.
Goodfellow et al.
(2014) developed the fast gradient sign method (FGSM) of generating adversarial examples and demonstrated that adversarial training is effective for reducing the error rate on adversarial examples.
A major difﬁculty of adversarial training is that it tends to overﬁt to the method of adversarial example generation used at training time.
For example, models trained to resist FGSM adversarial examples usually fail to resist L-BFGS adversarial examples.
Kurakin et al.
(2016) introduced the basic iterative method (BIM) which lies between FGSM and L-BFGS on a curve trading speed for effectiveness (the BIM consists of running FGSM for a medium number of iterations).
Adversarial training using BIM still overﬁts to the BIM, unfortunately, and different iterative methods can still successfully attack the model.
Recently, Madry et al.
(2017) showed that adversarial training using adversarial examples created by adding random noise before running BIM results in a model that is highly robust against all known attacks on the MNIST dataset.
However, it is less effective on more complex datasets, such as CIFAR.
A strategy for training networks which are robust to adversarial attacks across all contexts is still unknown.
In this work, we demonstrate that thermometer code discretization and one-hot code discretization of real-valued inputs to a model signiﬁcantly improves its robustness to adversarial attack, advancing the state of the art in this ﬁeld.
I NPUT D ISCRETIZATION
We propose to break the linear extrapolation behavior of machine learning models by preprocessing the input with an extremely nonlinear function.
This function must still permit the machine learning model to function successfully on naturally occurring inputs.
The recent success of the PixelRNN model (Oord et al., 2016) has demonstrated that one-hot discrete codes for 256 possible values of color pixels are effective representations for input data.
Other extremely nonlinear functions may also defend against adversarial examples, but we focused attention on vector-valued discrete encoding as our nonlinear function because of the evidence from PixelRNN that it would support successful machine learning.
Images are often encoded as a 3D tensor of integers in the range [0, 255].
The tensor’s three dimensions correspond to the image’s height, width, and color channels (e.g. three for RGB, one for greyscale).
Each value represents an intensity value for a given color at a given horizontal/vertical position.
For classiﬁcation tasks, these values are typically normalized to ﬂoating-point approximations in the range (0, 1).
Input discretization refers to the process of separating these continuousvalued pixel inputs into a set of non-overlapping buckets , which are each mapped to a ﬁxed binary vector.
Past work, for example depth-color-squeezing (Xu et al., 2017), has explored what we will refer to as quantization of inputs as a potential defense against adversarial examples.
In that approach, each pixel value is mapped to the low-bit version of its original value, which is a ﬁxed scalar.
The key novel aspect of our approach is that rather than replacing a real number with a number of low bit depth, we replace each real number with a binary vector.
Different values of the real number activate different bits of the input vector.
Multiplying the input vector by the network’s weights thus performs an operation similar to an embedding lookup in a language model, so different input values actually use different parameters of the network.
To avoid confusion, we will consistently refer to scalar-toscalar precision reduction as quantization and scalar-to-vector encoding schemes as discretization throughout this work.
A comparison of these techniques can be seen in Table 1.
Note that, unlike depth-color-squeezing, discretization makes a meaningful change to the model, even when it is conﬁgured to use enough discretization levels to avoid losing any information from a traditionally formatted computer vision training set; discretizing each pixel to 256 levels will preserve all of the information contained in the original image.
Discretization defends against adversarial examples by changing which parameters of the model are used, and may also discard information if the number of discretization levels is low; quantization can only defend the model by discarding information.
[Table]
2. D ISCRETIZATION AS A DEFENSE
In Goodfellow et al.
, the authors provide evidence that several network architectures, including LSTMs (Hochreiter & Schmidhuber, 1997), sigmoid networks (Han & Moraga, 1995), and maxout networks (Goodfellow et al., 2013), are vulnerable to adversarial examples due to the empirical fact that, when trained, the loss function of these networks tends to be highly linear with respect to its inputs.
We brieﬂy recall the reasoning of Goodfellow et al.
Assume that we have a logistic regressor with weight matrix w .
Consider an image x ∈ R n which is perturbed into (cid:101) x
= x + η by some noise η such that (cid:107) η (cid:107) ∞ ≤ ε for some ε .
The probability that the model assigns to the true class is equal to:
[Equation]
If the perturbation η is adversarial, such as in the case where η i := ε · sign (cid:16)
∂ L ( x ) ∂x
i (cid:17) , then the input to the sigmoid is increased by ε · n .
If n is large, as is typically the case in images and other highdimensional spaces of interest, this linearity implies that even imperceptibly small values of ε can have a large effect on the model’s prediction, making the model vulnerable to adversarial attacks.
Though neural networks in principle have the capacity to represent highly nonlinear functions, networks trained via stochastic gradient descent on real-world datasets tend to converge to mostly-linear solutions.
This is illustrated in the empirical studies conducted by Goodfellow et al.
One hypothesis proposed to explain this phenomenon is that the nonlinearities typically used in networks are either piecewise linear, like ReLUs, or approximately linear in the parts of their domain in which training takes place, like the sigmoid function.
One potential solution to this problem is to use more non-linear activation functions, such as quadratic or RBF units.
Indeed, it was shown by Goodfellow et al.
(2014) that such units were more resistant to adversarial perturbations of their inputs.
However, these units are difﬁcult to train, and the resulting models do not generalize very well (Goodfellow et al., 2014), sacriﬁcing accuracy on clean examples.
As an alternative to introducing highly non-linear activation functions in the network, we propose applying a non-differentiable and non-linear transformation ( discretization ) to the input, before passing it into the model.
A comparison of the input to the model under various regimes can be seen in Figure 1, highlighting the strong non-linearity of discretization techniques.
[Figure]
2. T YPES OF DISCRETIZATION
In this work we consider two approaches to constructing discretized representations f ( x ) of the input image x .
Assume for the sake of simplicity that the entries of x take values in the continuous domain [0 , 1] .
We ﬁrst describe a quantization function b .
Choose 0 < b 1 < b 2 < · · · < b k = 1 in some fashion.
(In this work, we simply divide the domain evenly, i.e. b i = ik .)
For a real number θ ∈ [0 , 1] deﬁne b ( θ ) to be the largest index α ∈ { 1 , . . .
, k } such that θ ≤ b α .
2.2. O NE - HOT ENCODINGS
For an index j ∈ { 1 , . . .
, k } let χ ( j ) ∈
R k be the indicator or one-hot vector of j , i.e.,
[Equation]
The discretization function is deﬁned pixel-wise for a pixel i ∈ { 1 , . . . , n } as:
[Equation]
One-hot encodings are simple to compute and understand, and are often used when it is necessary to represent a categorical variable in a neural network.
However, one-hot encodings are not well suited for representing categorical variables with an interpretation of ordering between them.
Note that the ordering information between two pixels x i and x j is lost by applying the transformation f onehot ; for a pair of pixels i, j whenever b ( x i ) (cid:54) =
b ( x j )) , we see:
[Equation]
In the case of pixel values, this is not a good inductive bias, as there is a clear reason to believe that neighboring buckets are more similar to each other than distant buckets.
2.2. T HERMOMETER ENCODINGS
In order to discretize the input image x without losing the relative distance information, we propose thermometer encodings .
For an index j ∈ { 1 , . . . , k } , let τ ( j
) ∈
R k be the thermometer vector deﬁned as
[Equation]
1 if l ≥ j 0 otherwise .
Then the discretization function f is deﬁned pixel-wise for a pixel i ∈ { 1 , . . . , n } as:
[Equation]
where C is the cumulative sum function, C ( c ) l = l (cid:80) j =
[Equation]
c l
[Equation]
Note that the thermometer encoding preserves pairwise ordering information, i.e., for pixels i, j if b ( x j ) (cid:54)
=
b ( x k ) and x i < x j then
[Equation]
2. W HITE - BOX ATTACKS ON DISCRETIZED INPUTS
Discretizing the input makes it difﬁcult to attack the model with standard white-box attack algorithms, such as FGSM (Goodfellow et al., 2014) and PGD (Madry et al., 2017), since it is impossible to backpropagate through our discretization function to determine how to adversarially modify the model’s input.
In this section, we describe two novel iterative attacks which allow us to construct adversarial examples for networks trained on discretized inputs.
Constructing white-box attacks on discretized inputs serves two primary purposes.
First, it allows us to more completely evaluate whether the model is robust to all adversarial attacks, as white-box attacks are typically more powerful than their black-box counterparts.
Secondly, adversarial training is typically performed in a white-box fashion, and so in order to utilize and properly compare against the adversarial training techniques of Madry et al.
(2017), it is important to have strong white-box attacks.
For ease of presentation, we will describe the attacks assuming that f : R → R k discretizes inputs into thermometer encodings; in order to attack one-hot encodings, simply replace all instances of f therm with f onehot , τ with χ , and C with the identity function I .
We represent the adversarial image after t steps of the attack as z t , where the value of the i th pixel is z ti .
The ﬁrst attack, Discrete Gradient Ascent (DGA), follows the direction of the gradient of the loss with respect to f ( x ) , but is constrained at every step to be a discretized vector.
If we have discretized the input image into k -dimensional vectors using the one-hot encoding, this corresponds to moving to a vertex of the simplex (∆ k ) n at every step.
The second attack, Logit-Space Projected Gradient
Ascent (LS-PGA), relaxes this assumption, allowing intermediate iterates to be in the interior of the simplex.
The ﬁnal adversarial image is obtained by projecting the ﬁnal point back to the nearest vertex of the simplex.
Note that if the number of attack steps is 1, then the two attacks are equivalent; however, for larger numbers of attack steps, LS-PGA is a generalization of DGA.
2.3. D ISCRETE G RADIENT A SCENT (DGA)
Following PGD (Madry et al., 2017), we initialize DGA by placing each pixel into a random bucket that is within ε of the pixel’s true value.
At each step of the attack, we look at all buckets that are within ε of the true value, and select the bucket that is likely to do the most ‘harm’, as estimated by the gradient of setting that bucket’s indicator variable to 1, with respect to the model’s loss at the previous step.
[Equation]
Because the outcome of this optimization procedure will vary depending on the initial random perturbation, we suggest strengthening the attack by re-running it several times and using the perturbation with the greatest loss.
The pseudo-code for the DGA attack is given in Section B of the appendix.
2.3. L OGIT -S PACE P ROJECTED G RADIENT A SCENT (LS-PGA)
To perform LS-PGA, we soften the discrete encodings into continuous relaxations, and then perform standard Projected Gradient Ascent (PGA) on these relaxed values.
We represent the distribution over embeddings as a softmax over logits u , each corresponding to the unnormalized log-weight of a speciﬁc bucket’s embedding.
To improve the attack, we scale the logits with temperature T , allowing us to trade off between how closely our softmax approximates a true one-hot distribution as in the Gumbel-softmax trick (Jang et al., 2016; Maddison et al., 2016), and how much gradient signal the logits receive.
At each step of a multi-step attack, we anneal this value via exponential decay with rate δ .
[Equation]
We initialize each of the logits randomly with values sampled from a standard normal distribution.
At each step, we ensure that the model does not assign any probability to buckets which are not within ε of the true value by ﬁxing the logits to be −∞ .
The model’s loss is a continuous function of the logits, so we can simply utilize attacks designed for continuous-valued inputs, in this case PGA with step-size ξ .
[Equation]
−∞ otherwise .
[Equation] if ∃ ( − ε ≤ η ≤ ε ) s.t. b ( x i + η ) = l −∞ otherwise .
Because the outcome of this optimization procedure will vary depending on the initial perturbation, we suggest strengthening the attack by re-running it several times and using the perturbation with the greatest loss.
The pseudo-code for the LS-PGA attack is given in Section B of the appendix.
E XPERIMENTS
We compare models trained with input discretization to state-of-the-art adversarial defenses on a variety of datasets.
We match the experimental setup of the prior literature as closely as possible.
Rows labeled with “Vanilla (Madry)” give the numbers reported in Madry et al.
; other rows contain results of our own experiments, with “Vanilla” containing a direct replication.
For our
MNIST experiments, we use a convolutional network; for CIFAR-10, CIFAR-100, and SVHN we use a Wide ResNet (Zagoruyko & Komodakis, 2016).
We use a network of depth 30 for the CIFAR10 and CIFAR-100 datasets, while for SVHN we use a network of depth 15 .
The width factor of all the Wide ResNets is set to k = 4 .
Unless otherwise speciﬁed, all quantized and discretized models use 16 levels.
We found that in all cases, LS-PGA was strictly more powerful than DGA, so all attacks on discretized models use LS-PGA with ξ = 0 .
01 , δ = 1 . 2 , and 1 random restart.
To be consistent with
Madry et al.
(2017), we describe attacks in terms of the maximum (cid:96) ∞ -norm of the attack, ε .
All
MNIST experiments used ε = 0 .
3 and 40 steps for iterative attacks; experiments on CIFAR used ε = 0 .
031 and 7 steps for iterative attacks; experiments on SVHN used ε = 0 . 047 and 10 steps for iterative attacks.
These settings were used for adversarial training, white-box attacks, and blackbox attacks.
Figure 3 plots the effectiveness of the iterated PGD/LS-PGA attacks on vanilla and discretized models for MNIST and shows that increasing the number of iterations beyond 40 would have no effect on the performance of the model on (cid:96) ∞ -bounded adversarial examples for MNIST.
In Madry et al.
, adversarially-trained models are trained using exclusively adversarial inputs.
This led to a small but noticeable loss in accuracy on clean examples, dropping from 99.2% to 98.8% on MNIST and from 95.2% to 87.3% on CIFAR-10 in return for more robustness towards adversarial examples.
Past work has also sometimes performed adversarial training on batches composed of half clean examples and half adversarial examples (Goodfellow et al., 2014; Cisse et al., 2017).
To be consistent with Madry et al.
, we list experiments on models trained only on adversarial inputs in the main paper; additional experiments on a mix of clean and adversarial inputs can be found in the appendix.
We also run experiments exploring the model’s relationship with the number of distinct levels to which we quantize the input before discretizing it, and exploring various settings of hyperparameters for LS-PGA.
R ESULTS
Our adversarially-trained baseline models were able to approximately replicate the results of Madry et al.
On all datasets, discretizing the inputs of the network dramatically improves resistance to adversarial examples, while barely sacriﬁcing any accuracy on clean examples.
Quantized models also beat the baseline, but with lower accuracy on clean examples.
Discretization via thermometer encodings outperformed one-hot encodings in most settings.
See Tables 2,3,4 and 5 for results on
MNIST and CIFAR-10.
Additional results on CIFAR-100 and SVHN are included in the appendix.
In Figures 2 and 5 (located in appendix), we plot the test-set accuracy across training timesteps for various adversarially trained models on the SVHN and CIFAR-10 datasets, and observe that the discretized models become robust against adversarial examples more quickly.
D ISCUSSION
In Goodfellow et al.
(2014), the seeming linearity of deep neural networks was shown by visualizing the networks in several different ways.
To test our hypothesis that discretization breaks some of this
[Table] [Table]
[Table]
linearity, we replicate these visualizations and contrast them to visualizations of discretized models.
See Appendix G for an illustration of these properties.
For non-discretized, clean trained models, test-set examples always yield a linear boundary between correct and incorrect classiﬁcation; in contrast, non-adversarially-trained models have a more interesting parabolic shape (see Figure 9).
When discretizing the input, we introduce C w · C h · C o · c · ( k − 1) extra parameters, where c is the number of channels in the image, k is the number of levels of discretization
[Table]
and C w , C h , C
o are the width, height
[Table]
and output channels of the ﬁrst convolutional layer.
Discretizing usinglevels introduced 0 .
03% extra parameters for MNIST, 0 .
08% for CIFAR-10 and CIFAR-
[Table]
and 2 . 3% for SVHN.
This increase is negligible, so it is likely that the robustness comes from the input discretization, and is not merely a
byproduct of
having
a
slightly
higher
model
(cid:88)(cid:88)(cid:88)(cid:88)(cid:88)(cid:88)(cid:88)(cid:88)(cid:88) Target
Source
Clean
Adv. train
Vanilla
One-hot
Thermometer
Vanilla
One-hot
Thermometer
C l e an
Vanilla (Madry) 0.
Vanilla 3.
Quantized 70.
One-hot 83.
Thermometer 80.
A d v .
t r a i n
Vanilla (Madry) 85.
Vanilla 85.
Quantized 84.
One-hot 86.
Thermometer 88.
[Figure] [Figure]
C ONCLUSION
Our ﬁndings convincingly demonstrate that the use of thermometer encodings, in combination with adversarial training, can reduce the vulnerability of neural network models to adversarial attacks.
Our analysis reveals that the resulting networks are signiﬁcantly less linear with respect to their inputs, supporting the hypothesis of Goodfellow et al.
(2014) that many adversarial examples are caused by over-generalization in networks that are too linear.
A H YPERPARAMETERS
In this section, we describe the hyperparameters used in our experiments.
For CIFAR-10 and
CIFAR-100 we follow the standard data augmenting scheme as in (Lin et al., 2013;
He et al., 2016;
Huang et al., 2016; Zagoruyko & Komodakis, 2016): each training image is zero-padded withpixels on each side and randomly cropped to a new 32 × 32 image.
The resulting image is randomly ﬂipped with probability 0 .
5 , it’s brightness is adjusted with a delta chosen uniformly at random in the interval [ − 63 , 63) and it’s contrast is adjusted using a random contrast factor in the interval [0 . 2 , 1 . 8] .
For MNIST we use the Adam optimizer with a ﬁxed learning rate of 1e − 4 as in Madry et al.
For CIFAR-10 and CIFAR-100 we use the Momentum optimizer with momentum 0 .
9 , (cid:96) 2 weight decay of λ = 0 .
0005 and an initial learning rate of 0 . 1 which is annealed by a factor of 0 .
2 after epochs 60 , 120 and 160 respectively as in Zagoruyko & Komodakis (2016).
For SVHN we use the same optimizer with initial learning rate of 1e − 2 which is annealed by a factor of 0 .after epochs 80 and 120 respectively.
We also use a dropout of 0 .
3 for CIFAR-10, CIFAR-100 and
SVHN.
B P SEUDO - CODE
The DGA attack is described in Algorithm 2 and the LS-PGA attack is described in Algorithm 3.
Both these algorithms make use of a getMask () sub-routine which is described in Algorithm 1.
Input: Image x , parameter ε Output: ε -discretized mask around x 1 mask ← (0) n × k 2 low ← max { 0 , x − ε } 3 high ← min { 1 , x + ε } 4 for α ← 0 to 1 by 1 k do mask ← mask + f onehot ( α ∗ low
+ (1 − α ) ∗ high
7 return
/* Loop invariant: z ti is discretized for every pixel i */
← ∇ z t − 1 L ( θθθ, z t − 1 , y ) if f is one-hot then l ← t − 1 − χ l
else l
−
end t
13 end l
Input: Image x , label y , discretization function f , loss L ( θθθ, f ( x ) , y ) , l attack steps, parameters ε , δ
Output: Adversarial input to the network z (cid:48) 1 mask ← getMask ( x, ε ) 2 u 0 ← (cid:0) N ( 0 k ; 1 k ) (cid:1) n ∗ mask − (1 − mask) ∗ ∞ 3 T ←4 if f is one-hot then
F ← I 6 else
F ← C 8 end 9 z 0 ← F (cid:16) σ (cid:16) u 0 T (cid:17)(cid:17) 10 for t ← 1 to l do grad ← ∇ u t − 1 L ( θθθ, z t − 1 , y )
u t ← u t − 1 + ξ · sign (grad) z t ← F (cid:16) σ (cid:16) u t T (cid:17)(cid:17)
T ← T · δ 15 end 16 return z (cid:48) ← τ (arg max( u li ))
Algorithm 3: Logit-Space Projected Ascent (LS-PGA)
C A DDITIONAL E XPERIMENTS ON MNIST
In this section we list the additional experiments we performed using discretized models on MNIST.
The main hyperparameters of Algorithm 3 are the step size ξ used to perform the projected gradient ascent, and the annealing rate of δ .
We found that the choice of these hyperparameters was not critical to the robustness of the model.
In particular, we performed experiments with ξ = 1 . 0 and ξ = 0 . 001 , and both achieved similar accuracies as in Table 2 and Table 3.
Additionally, we found that without annealing, i.e., δ = 1 . 0 , the performance was only slightly worse than with δ = 1 . 2 . Model
[Table]
One
[Table] [Table]
[Table] [Table]
[Table] [Table]
One
[Table]
We also experimented with discretizing by using percentile information per color channel instead of using uniformly distributed buckets .
This did not result in any signiﬁcant changes in robustness or accuracy for the MNIST dataset.
Finally, we also trained on a mix of clean and adversarial examples: this resulted in signiﬁcantly higher accuracy on clean examples, but decreased accuracy on white-box and black-box attacks compared to Tables 2 and 3.
[Table]
[Table] [Table]
D A DDITIONAL E XPERIMENTS ON CIFAR-
In this section we list the additional experiments we performed on CIFAR-10.
Firstly
[Table]
we trained models on a mix of both clean and adversarial examples.
The results for mixed training are listed in
Tables 10 and 11; as expected it has lower accuracy on adversarial examples, but higher accuracy on clean examples
[Table]
compared to training on only adversarial examples (Tables 4 and 5).
[Table] [Table]
In order to explore whether the number of levels of discretization affected the performance of the model
[Table]
we trained several models which varied this number.
As expected
[Table]
we found that models with fewer levels had worse accuracy on clean examples
[Table]
likely because there was not enough information to correctly classify the image
[Table]
but greater robustness to adversarial examples, likely because larger buckets mean a greater chance that a given perturbation will not yield any change in input to the network (Xu et al., 2017).
Results can be seen in Tables
[Table]
and are visualized in Figure
[Table]
E E XPERIMENTS ON CIFAR-
We list the experimental results on CIFAR-100 in Table 13.
We choose ξ = 0 .
01 and δ = 1 .
2 for the
LS-PGA attack hyperparameters.
For the discretized models, we used 16 levels.
All adversarially trained models were trained on a mix of clean and adversarial examples.
[Table]
F E XPERIMENTS ON SVHN
We list the experimental results on SVHN in Table 14.
All adversarially trained models were trained only on adversarial examples.
[Table]
G S UPPLEMENTARY F IGURES
In Figure 4 we plot the effect of increasing the levels of discretization for the MNIST and CIFARdatasets.
[Figure]
In Figure 5 we plot the convergence rate of clean trained and adversarially trained models on the
CIFAR-10 dataset.
Note that thermometer encoded inputs converge much faster in accuracy on both clean and adversarial
Figure 6 plots the norm of the gradient as a function of the number of iterations of the attack on
MNIST.
Note that the gradient vanishes at around 40 iterations, which coincides with the loss stabilizing in Figure 3.
In Figure 7, we create a linear interpolation between a clean image and an adversarial example, and then continue to extrapolate along this line, evaluating probability of each class at each point.
In models trained on unquantized inputs, the class probabilities are all mostly piecewise linear in both the positive and negative directions.
In contrast, the discretized model has a much more jagged and irregular shape.
In Figure 8, we plot the error for different models on various values of ε .
The discretized models are extremely robust to all values less-than-or-equal-to the values that they have been exposed to during training.
However, beyond this threshold, discretized models collapse immediately, while real-valued models still maintain some semblance of robustness.
This exposes a weakness of the discretization approach; the same nonlinearity that helps it learn to become robust to all attacks it sees during training-time causes its behavior is unpredictable beyond that.
However, we believe that the fact that the performance of thermometer-encoded models degrades more quickly than that of vanilla models beyond the training epsilon is a signiﬁcant weakness in practice, but no worse than other defenses. The “standard setting” for the adversarial example problem (in which we constrain the L-inﬁnity norm of the perturbed image to an epsilon ball around the [Figure] [Figure]
original image) was designed to ensure that any adversarially-perturbed image is still recognizable as its original image by a human. However, this artiﬁcial constraint excludes many other potential attacks that also result in human-recognizable images.
State-of-the art defenses in the standard setting can still be easily defeated by non-standard attacks; for an example of this, see appendix A of ICLR submission “Adversarial Spheres”. A “larger epsilon” attack is just one special case of a “non-standard” attack. If we permit non-standard attacks, a fair comparison would show that all current approaches are easily breakable. There is nothing special about the “larger epsilon” attack that makes a vulnerability to this non-standard attack in particular more problematic than vulnerabilities to other non-standard attacks.
In Figures 9, 10 , 11, 12 , 13 and 14 we plot several examples of church-window plots for MNIST (Goodfellow et al., 2014). Each plot is crafted by taking several test-set images, calculating the vector corresponding to an adversarial attack on each image, and then choosing an additional random orthogonal direction.
In each plot, the clean image is at the center and corresponds to the color white , the x -axis corresponds to the magnitude of a perturbation in the adversarial direction, and the y -axis corresponds to the magnitude of a perturbation in the orthogonal direction.
Note that we use the same random seed to generate the test set examples and the adversarial directions across different church-window plots.
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure] [Figure]