A B AYES -O PTIMAL V IEW ON A DVERSARIAL E XAMPLES
Anonymous authors
Paper under double-blind review
Original ”male”
Adversarial ”female” Adversarial ”female”
Adversarial ”female”
Adversarial ”female” Asymmetric dataset
Symmetric dataset
Symmetric dataset
Symmetric dataset Bayes-Optimal
Bayes-Optimal
CNN
SVM (RBF)
[Figure]
d e
A BSTRACT
Adversarial attacks on CNN classiﬁers can make an imperceptible change to an input image and alter the classiﬁcation result.
The source of these failures is still poorly understood, and many explanations invoke the “unreasonably linear extrapolation” used by CNNs (Goodfellow et al., 2018) along with the geometry of high dimensions.
In this paper we show that similar attacks can be used against the Bayes-Optimal classiﬁer for certain class distributions, while for others the optimal classiﬁer is robust to such attacks.
We present analytical results showing conditions on the data distribution under which all points can be made arbitrarily close to the optimal decision boundary and show that this can happen even when the classes are easy to separate, when the ideal classiﬁer has a smooth decision surface and when the data lies in low dimensions.
We introduce new datasets of realistic images of faces and digits where the Bayes-Optimal classiﬁer can be calculated efﬁciently and show that for some of these datasets the optimal classiﬁer is robust and for others it is vulnerable to adversarial examples.
In systematic experiments with many such datasets, we ﬁnd that standard CNN training consistently ﬁnds a vulnerable classiﬁer even when the optimal classiﬁer is robust while large-margin methods often ﬁnd a robust classiﬁer with the exact same training data.
Our results suggest that adversarial vulnerability is not an unavoidable consequence of machine learning in high dimensions, and may be the result of the speciﬁc distributions of commonly used datasets or of suboptimal training methods used in current practice.
I NTRODUCTION
Perhaps the most intriguing property of modern machine learning methods is their susceptibility to adversarial examples (Szegedy et al., 2014; Goodfellow et al., 2018): for many powerful classiﬁers it is possible to change the input by an imperceptible amount and change the decision of the classiﬁer.
While adversarial examples were most famously reported for Neural Network classiﬁers (Szegedy et al., 2014), subsequent research has shown that other classiﬁers can also fall prey to similar attacks (Goodfellow et al., 2018).
Attempts to make classiﬁers robust to these attacks have generated a tremendous amount of interest (e.g. (Schott et al., 2019) and references within), although a recent survey argued that “There has been much work showing that basically all defenses suggested so far in the literature do not substantially increase robustness over undefended neural networks.”
(Schott et al., 2019)
As a ﬁrst step towards solving the problem, many authors have attempted to understand the source of the failure (Goodfellow et al., 2018; Szegedy et al., 2014; Tanay & Grifﬁn, 2016; Fawzi et al., 2018).
Intuitively the failure suggests that the decision surface learned by neural networks is “discontinuous to a signiﬁcant extent”, analogous to an attempt to discriminate the rational numbers from the rest of the real numbers (Szegedy et al., 2014).
The most prominent theory suggests that the problem is that neural network classiﬁers are “unreasonably linear” combined with the fact that they operate in high dimensions (Goodfellow et al., 2018; 2015).
In high dimensions the output of a random linear classiﬁer can be changed by making a small change in the (cid:96) ∞ norm of an example.
(Fawzi et al., 2018) show a connection between the error rate of linear and quadratic classiﬁers and the adversarial vulnerability and show that linear classiﬁers in high dimensions must be vulnerable for data that is not linearly separable.
(Ford et al., 2019) show that adversarial vulnerability is closely related to the lack of generalization to random perturbations and that in high dimensions even moderate failures to generalize to high amounts of noise imply the existence of adversarial examples.
Shamir et al.
(2019) have also focused on the geometry of high dimensions arguing that adversarial attacks may be a “natural consequence of the geometry of R n with the L 0 (Hamming) metric”.
Jacobsen et al.
(2018) connect the problem of adversarial examples with the excessive invariance of modern CNNs to large image transformations.
Zhang et al.
(2019) discuss the tradeoff between robustness and accuracy and show a very discontinuous toy distribution where the Bayes-Optimal classiﬁer is not robust.
Most recently, (Ilyas et al., 2019) have argued that adversarial examples are a feature, not a bug, and showed that one can in fact obtain information about the true decision boundary from adversarial examples. Their analysis suggests that vulnerability results from the presence of predictive features that are not robust.
They presented a synthetic dataset which was constructed to not contain such features, and showed that CNN training on that dataset was robust. .
In (Nakkiran, 2019), a similar dataset was presented without nonrobust features, but CNN training did lead to non robust classiﬁers when the dataset was noisy, leading the authors to conclude that “Adversarial Examples are Just Bugs, Too”.
This result is consistent with Tanay & Grifﬁn (2016) who show that adversarial vulnerability is related to overﬁtting in learning algorithms that are not sufﬁciently regularized.
Similarly, Lyu & Li (2019) show theoretically and experimantally that the the details of the training procedure can signiﬁcantly change the robustness of CNNs.
In this paper we give an alternative perspective on adversarial examples by focusing on BayesOptimal classiﬁers.
We ask: under what conditions on the data distribution will a large fraction of the points be close to the optimal decision boundary?
We show that it is easy to construct data distributions in which this happens (bottom of ﬁgure 2) and seemingly similar distributions in which this does not happen (top of ﬁgure 2).
Contrary to previous explanations, adversarial examples may succeed when the optimal decision surface is continuous and the data does not need to lie in high dimensions.
Rather we show that the susceptibility to adversarial examples is related to the presence of asymmetries in the predictive power of different features in the data distribution, and we show analytically that when such asymmetries exist, optimal classiﬁers will suffer from adversarial examples.
We introduce new datasets of realistic images for which the Bayes-Optimal classiﬁer can be calculated efﬁciently and show that for some of these datasets the optimal classiﬁer is robust to adversarial examples and in others it is not.
In systematic experiments with datasets where the optimal classiﬁer is robust, we show that standard CNN training methods consistently ﬁnd a non-robust classiﬁer even though the optimal classiﬁer is robust, while large-margin methods often ﬁnd a robust classiﬁer using exactly the same training data.
Our results suggest that adversarial vulnerability is not an unavoidable consequence of machine learning in high dimensions, and in many cases may be a result of suboptimal training methods.
Spherical Gaussians
Elliptical Gaussians
Discrete
Mixtures - -   10 -10 - -6 - -226 - -   10 -10 - -6 - -226 - -   10 -10 - -6 - -226 - -   10 -10 - -6 - -226 a b c d - -   10 -10 - -6 - -226 - -   10 -10 - -6 - -226 - -   10 -10 - -6 - -226 - -   10 -10 - -6 - -
e f g h  A NALYSIS : I S THE O PTIMAL C LASSIFIER N ECESSARILY R OBUST ? We focus on a two class classiﬁcation problem and denote by p 1 ( x ) the distribution of x when x is in the ﬁrst class, and similarly p 2 ( x ) for the second class. We assume that p 1 ( x ) , p 2 ( x ) are known and that the two classes have equal priors, so the Bayes-Optimal classiﬁer simply classiﬁes x as belonging to class 1 if p 1 ( x ) > p 2 ( x ) and to class 2 otherwise.
It is well known that this classiﬁcation rule is optimal and no other classiﬁcation rule can achieve higher accuracy (assuming of course that p 1 ( x ) , p 2 ( x ) are correct) (Duda et al., 1973).
A textbook example of Bayes-Optimal classiﬁcation is when both classes are generated with a Gaussian with the same spherical covariance matrix, in which case the optimal classiﬁer is a linear discriminant which is orthogonal to the difference between the two means (ﬁgure 2a).
In this case, if the distance between the two means is large relative to the covariance, then almost all points are far from the decision boundary and so an adversarial attack which only makes small changes to the input will typically fail.
But as shown in the bottom of ﬁgure 2 there are other examples where the decision boundary is close to many of the datapoints and an adversarial attack which only makes small changes to the input will often succeed.
What distinguishes these two cases?
To gain intuition, consider the images shown in the left side of ﬁgure 3.
Assume we have inﬁnite examples of images of Cindy Crawford and denote by p 1 ( x ) the probability of generating an image under this distribution.
Denote by p 2 ( x ) the probability of generating an image of a different person.
Under p 1 ( x ) all images have a mole, while under p 2 ( x ) this probability is very low.
Thus an optimal classiﬁer will assign an extremely high weight to the presence of a mole in an image.
This means that by making a tiny change in the image and erasing the mole (right side of ﬁgure 3) we can drastically change the output of the optimal classiﬁer (and in fact drive p 1 ( x ) down to zero).
Note that this is not the same as the standard deﬁnition of overﬁtting (i.e. a difference between performance on the training samples and test samples): even unseen images of Cindy Crawford that are generated from p 1 ( x ) will have a mole present, so giving this feature high weight will not hurt generalization, but will make the classiﬁer susceptible to adversarial attacks.
This intuition is similar to that used in the recent work of (Ilyas et al., 2019) who argue that adversarial vulnerability is a feature, not a bug, and results from the presence of predictive features that are not robust.
The way we formalize this intuition, below, is quite different from that of (Ilyas et al., 2019) (whose analysis focused on linear classiﬁers) and also leads to quite different conclusions.
We ﬁrst focus on the case where both class distributions are Gaussians, i.e. p 1 ( x ) , p 2 ( x ) are both
Gaussian with means µ 1 , µ 2 and covariances Σ 1 , Σ 2 .
Observation 1: Let d be a direction of minimal variance under Σ 1 : d = arg min d (cid:48) T Σ 1 d (cid:48) .
Let σbe the variance projecting x onto that direction when x comes from class 1 .
If σ 1 → 0 and Σ 2 is full rank then almost any point in class 1 is arbitrarily close to the optimal decision surface.
Proof: We denote by σ 2 =
d T Σ 2 d the variance of the data in direction d under the distribution of the second class p 2 .
Note that by the assumption that Σ 2 is full rank, this variance must be nonzero.
We write the vector x as ( t, s ) where t is the projection in direction d
and s is a vector of projections in directions orthogonal to d .
We denote by t i the projection of µ i in direction d .
The decision surface as a function of t is a solution to:
[Equation]
Now since d is a direction of minimal variance it must be an eigenvector of Σ 1 so that s and t are independent under p 1 and we can write log p 1 ( s | t )
= log p 1 ( s ) .
Using the standard equation for conditional Gaussians we see that p 2 ( s | t ) is a Gaussian with the following means and variances.
[Equation]
where Σ ss 2 , Σ st 2 are the appropriate submatrices of the covariance matrix Σ 2 .
As σ 1 → 0 then t → t 1 so we can approximate log p 2 ( s | t ) with a Gaussian with the following means and variances:
[Equation]
Note that now neither the mean nor the variances depend on t .
This means that the decision surface for t is a solution to the following equation:
[Equation]
where the right hand side is a constant that does not depend on t or σ 1 .
As σ 1 σ 2 approaches zero, the solutions of this equation approach t 1 .
And because σ 1 → 0 almost all samples from the ﬁrst
Gaussian will be close to t 1 so moving x by a tiny amount in direction d will change the optimal decision.
[Equation]
As can be seen from equation 6, we do not need σ 1 to be exactly equal to zero for the decision surface to be close to the datapoints.
It is enough for σ 1 to be signiﬁcantly smaller than the minimal variance of Σ 2 .
Figures 2a,b,e,f illustrate this dependence: when the minimal variances in both
Gaussians is similar then the decision boundary is far from most points (top), but when there is a strong asymmetry in the minimal variances then the decision boundary becomes close to the datapoints in one of the classes.
In (Tanay & Grifﬁn, 2016), the existence of directions of small variance was suggested as a possible explanation for adversarial examples by invoking overﬁtting: when such directions exist, a linear classiﬁer may fail to learn the optimal decision boundary.
Our analysis, on the other hand, is for the Bayes-Optimal classiﬁer which by deﬁnition does not overﬁt.
We now assume that the distribution in each class can be represented as a mixture model:
[Equation]
where π ik represent the prior probability of component k in p i and p ik is the probability distribution of that component.
We also assume that within each class, the components are well separated, i.e. that for each datapoint the assignment probabilities put all their mass on one of the components.
More formally, denote by g k ( x ) the assignment probability of a datapoint x to component k :
[Equation]
then we assume that for each x , g ik ( x ) =
δ ( k − k ∗ i ( x )) where k ∗ i ( x ) is the index of the component that is most likely to have generated x under probability p
i ( x ) .
Under this assumption the probability of generating a point x under p 1 is simply:
[Equation]
We will also assume that for well separated components, within each distribution, the assigned component does not change when we perturb x by a small amount k ∗ i ( x ) =
k ∗ i ( x + (cid:15) ) where (cid:15) is a small perturbation.
Observation 2: Assume that both classes are generated by well separated Gaussian mixtures.
Let Σ 1 k be the covariance matrix of the k th Gaussian in the mixture for class 1 .
If there exists a Gaussian j in class 2 such that Σ 1 k and Σ 2 j satisfy the asymmetry conditions of observation 1, then almost any point generated by that Gaussian is close to the optimal decision boundary.
Proof: We simply apply observation 1 to the two Gaussians, Σ 1 k in class 1 and Σ 2 j in class 2.
By observation 1, we can make all points generated by the Gaussian in class 1 more likely to have been generated by the Gaussian in class 2, by moving a tiny amount.
By the well separatedness asssumption, the original Gaussian in class 1 is still more probable than all the other Gaussians in the same class, so the point will now be classiﬁed as coming from class 2 by the Bayes-Optimal classiﬁer.
(cid:3)
Figures 2d,h illustrate the importance of asymmetry in data generated from a Gaussian Mixture
Model.
In the top ﬁgure, both classes are generated from a mixture of two Gaussians, whose minimal variance in the same.
In the bottom ﬁgure, each class data is generated by a mixture of three
Gaussians: the two original Gaussian plus a third isotropic Gaussian that has low mixing proportion but very large variance.
Once this third Gaussian is present in the other class, the Bayes-Optimal decision surface becomes very close to all points in both classes (except for those generated by the rare, broad Gaussian).
2. N ON G AUSSIAN DISTRIBUTIONS
A natural question following observation 1 is to what extent the result depends on the Gaussian distribution.
To address this, we now consider discrete distributions.
We assume that every instance x is described by quantized features that can take on a discrete number of values.
For example, the features can be wavelet coefﬁcients of an image that are discretized into 256 possible values.
This means that p 1 ( f ) , p 2 ( f ) are simply very large tables that give the probability of observing a particular discrete set of image features given each of the classes.
Of course learning such a large table is infeasible without additional assumptions, but recall that we are analyzing the BayesOptimal case, where we assume p 1 ( f ) , p 2 ( f ) are known.
Observation 3: Assume there exists a feature i and a quantization level k so that p 1 ( f i = k ) → 1 .
Assume also that for any feature vector f , p 2 ( f ) > 0 .
Then almost any point in class 1 is one quantization level away from the optimal decision boundary.
Proof: We again write f = ( s, t ) where s is the ith feature and t are all other features.
[Equation]
Now assume that p 1 ( s ) approaches 1 for s = k and 0 otherwise.
This means that for almost any point in class 1 the value of that feature is equal to k .
We now change the feature by one quantization level and obtain a new feature vector (˜ s, t ) and p 1 (˜ s, t ) =
p 1 (˜ s ) p 1 ( t | ˜ s )
= 0 .
On the other hand, by the assumption p 2 (˜ s, t ) > 0 so that this point would now be classiﬁed as belonging to class 2.
[Equation] As in the Gaussian case, we do not need p 1 ( s ) to be exactly equal to a delta function for the decision surface to be close to most points. It is enough that p 1 (˜ s ) p 1 (˜ s ) be much smaller than the minimal values of p 2 for the decision to be ﬂipped when we replace s with ˜ s . Figures 2g,c illustrate this dependence.
In both cases, the data is sampled from a discrete distribution where the features are simply discretization of the two spatial coordinates into 100 levels each. In other words, p 1 and p 2 are tables of size 10 , 000 and each entry in the table represents the probability of generating a point at one of the 10 , 000 possible locations. In the top example, the minimal value of the probability table is approximately the same in both classes, while in the bottom example, there is a strong asymmetry and the decision boundary becomes close to all points in one of the classes.
Observation 4: Assume that both classes are generated by well separated discrete feature distributions.
Let p 1 k ( f ) be be the k th table of feature distributions under class 1.
If there exists a j in class 2 such that p 1 k ( f ) , p 2 j ( f ) satisfy the asymmetry conditions of observation 2, then almost any point generated by that component is close to the optimal decision boundary.
The proof is identical to the proof of observation 2.
D ATASETS OF REALISTIC IMAGES WITH EFFICIENT CALCULATION OF THE B AYES -O PTIMAL C LASSIFIER
In order to connect the discussion of optimal classiﬁers with datsets that can be used with modern classiﬁers, we introduce new datasets of realistic images of faces and digits for which the BayesOptimal classiﬁer can be calculated efﬁciently.
We do this by leveraging recent advances in generative modeling of full images (Goodfellow, 2016; Richardson & Weiss, 2018).
To create these datasets, we start with a labeled training set (e.g. MNIST) with two classes.
We then train a separate Mixture of Factor Analyzer (MFA) model p 1 ( x ) , p 2 ( x ) on images from the two classes.
The MFA model is a standard density model: it models each density as a mixture of
Gaussians where the covariance of each Gaussian is the sum of a low rank matrix and a diagonal one.
We now create a new training set by sampling images from the two models, and similarly a new test set.
Since these datasets were created by sampling from a known model, we can calculate the Bayes-Optimal classiﬁer.
At the same time, the images are realistic and MFA models have been shown to capture much of the variability of the images in the original data (Richardson &
Weiss, 2018).
The top of ﬁgure 4 shows samples from the p 1 ( x ) , p 2 ( x ) when the MFA were learned separately from the ”male” and ”female” attributes in the CelebA dataset (Liu et al., 2015).
More samples from other datasets including digits and faces can be found in the appendix.
While these samples are typically somewhat blurred, it can be seen they are realistic and highly variable.
In fact, in many real world applications one needs to classify somewhat blurry images (e.g. analyzing faces in surveillance videos).
In order to investigate the importance of asymmetry in the variances, we created symmetric and asymmetric variants of these datasets.
In the symmetric version, we regularized the MFA so that all covariance matrices had the same minimal variance.
In the asymmetric version, we added to each MFA model one additional “outlier” component: it had a diagonal covariance with much larger minimal variance than all the other covariances and a mean which is close to the mean of the data.
The advantage of using a MFA model over other generative models such as VAEs or GANS (Kingma & Welling, 2014; Gulrajani et al., 2017) is that the log likelihood of any image can be calculated efﬁciently.
Speciﬁcally we follow (Richardson & Weiss, 2018) and ﬁnd the Gaussian component that is most likely to have generated the image, and then evaluate the likelihood of the image given that Gaussian.
We can then use this log likelihood to classify an image as belonging to one of the two classes (e.g. ”male” or ”female”).
Since the data were generated by the assumed distributions,
[Figure]
L
[Figure] [Figure] Unlike our theoretical results, deciding whether or not a real classiﬁer is robust or not requires an operational deﬁnition of what constitutes a ”tiny” or ”imperceptible” perturbation.
We follow the standard practice of calculating the mean L2 perturbation of an adversarial attack.
In other words, we allow the adversary an unlimited budget in attacking the classiﬁers, and we then measure how large (in terms of Euclidean norm) a perturbation was required to cross the decision boundary.
The mean is calculated only over successful attacks, when the original sample was correctly classiﬁed and the adversarial example was not.
Since this deﬁnition is sensitive to outliers and the particular choice of Euclidean norm, we also examined the histograms of changes made to each pixel in the adversarial attack.
Finally, we visually inspected the adversarial images.
In all 15 datasets, we found that these three methods of deﬁning robustness are consistent.
For the face images, when the mean L2 is less than 1 . 5 , then the adversarial images are almost indistinguishable from the original images, and the vast majority of the pixels in the adversarial images are within 5 / 255 intensity levels from their original value.
On the other hand, when the mean L2 is around 3 , then the adversarial images are perceptually quite different from the original ones, and many pixels differ by more than 5 / 255 from their original values.
Similar deﬁnitions of robustness were used in previous reports (Schott et al., 2019; Carlini et al., 2019).
We used a simple gradient attack in which we take a small step in the direction of the gradient of the MFA log likelihood (see appendix for details).
Similar results are achieved with a standard implementation (Papernot et al., 2016) of the CW-L2 attack (Carlini & Wagner, 2017).
As shown in ﬁgure 4 the difference between the symmetric datasets and asymmetric datasets is dramatic (see appendix for similar results on other classes).
When there exists a large asymmetry between the minimal variances of different Gaussians, the conditions of observation 3 hold, and a tiny imperceptible change is sufﬁcient to fool the Bayes-Optimal classiﬁer.
However, when all
Gaussians have the same minimal variance, the adversary needs to make much larger changes and the adversarial examples become perceptually meaningful.
E XPERIMENTS : W HY ARE CNN S SO BRITTLE ?
Given our analysis, the fact that modern machine learning methods are often susceptible to tiny adversarial attacks may be due to two very different reasons (ﬁgure 5).
One reason could be that the data distribution is asymmetric, so that the Bayes-Optimal classiﬁer is not robust, and hence it is not surprising that a CNN is also not robust.
A second possible reason is illustrated in ﬁgure 5b: here the data distribution is symmetric and the Bayes-Optimal classiﬁer is robust, yet SGD starting from a bad initial condition ﬁnds a brittle classiﬁer.
If this is the case, then the brittleness is not due to the data distribution but rather a failure of the learning method. Indeed (Tanay & Grifﬁn, [Figure]
2016; Nakkiran, 2019) have presented toy distributions where the optimal classiﬁer is robust but suboptimal learning fails to ﬁnd it.
In order to separate the contribution of the dataset from the estimation method in the vulnerability of machine learning methods, we trained a CNN on samples from all 15 ”symmetric” datasets described in section 3, and measured the vulnerability on the learned CNN.
We used the standard CNN implementation available as part of the CleverHans library.
We asked: will the CNN ﬁnd a brittle classiﬁer even though the optimal one is robust?
Results are shown in ﬁgure 6.
In all 15 cases, the CNN found a high accuracy classiﬁer that was vulnerable to small adversarial attacks, even though the optimal classiﬁer is robust.
The difference is most dramatic in the celebA tasks, where the CNN adversarial examples are almost indistinguishable from the original images (examples of the attacks on different datasets are shown in the appendix).
In the easier, MNIST tasks, the CNN adversarial perturbations are still signiﬁcantly smaller than those of the Bayes-Optimal classiﬁer and less perceptually meaningful.
While there are many possible architectures and optimization methods for CNNs, we did not ﬁnd any improvement in the CNN robustness in our attempts to change the number of ﬁlters, layers, training iterations etc.
In particular, (Schmidt et al., 2018) have argued that one needs more training examples to achieve robust classiﬁcation, so we systematically varied the amount of training images (generated dynamically at each SGD iteration), and found no signiﬁcant improvement in robustness as we increased the number of training examples up to 1 million examples (see appendix for details).
Even adversarial training of the CNN (using the method of (Zhang et al., 2019)) did not signiﬁcantly improve robustness.
Is it possible to learn a robust classiﬁer for these datasets from ﬁnite training data?
To answer that question, we then trained linear and RBF support vector machine classiﬁers on exactly the same datasets.
The linear SVM attempts to maximize the margin while maintaining high accuracy, but since the optimal classiﬁer is nonlinear it ends up learning a brittle classiﬁer.
More importantly, with an appropriate bandwidth parameter RBF SVMs often ﬁnd a robust classiﬁer when trained on exactly the same data (when the bandwidth parameter is too large, the RBF performs similarly to a linear SVM).
Returning to ﬁgure 5, our results strongly support the hypothesis that for these cases brittleness is due to suboptimal learning methods, even when the data distributions are symmetric and the optimal classiﬁer is robust.
D ISCUSSION
Since the initial discovery of adversarial examples for CNNs there has been much discussion whether they are a ”bug” that is speciﬁc to neural networks or a ”feature” of high dimensional geometry.
On the one hand, our results show that even the Bayes-Optimal classiﬁer may be susceptible to tiny adversarial perturbations, and this can happen in low dimensions and when the optimal classiﬁcation function is smooth.
Perhaps more signiﬁcantly, our analysis has also enabled us to construct realistic datasets for which the Bayes-Optimal classiﬁer is robust.
We ﬁnd that standard
CNN training consistently fail to ﬁnd a robust classiﬁer for these datasets, while large-margin methods can succeed when training on exactly the same data.
This suggests that in some situations, the presence of adversarial examples represents a failure of current, suboptimal, learning methods, rather than being an unavoidable property of learning in high dimensions.
[Figure] We are by no means advocating a return to using RBF SVMs rather than CNNs.
Rather, we believe that explicit regularization methods for CNNs may enable learning robust classiﬁers while maintaining the power of deep architectures.
Recent theoretical work on gradient descent methods suggests that they implicitly reward large margin classiﬁers in both shallow and deep architectures (Soudry et al., 2018; Poggio et al., 2017; Lyu & Li, 2019) although convergence to a large margin classiﬁer may require exponential time.
While our experiments have focused on the ”symmetric” datasets, where there is no tradeoff between accuracy and robustness, there is no reason to believe that real datasets will be symmetric rather than asymmetric.
As our analysis has shown, if the real datasets do have strong asymmetries, then the
Bayes-Optimal classiﬁer will not be robust.
Nevertheless, improved learning algorithms should allow us to trade off robustness and accuracy in a principled manner, as can currently be done in the training of shallow architectures such as SVMs.
In general, when trying to understand a complex effect, it is often useful to disentangle the different causes.
The Bayes-Optimal perspective on adversarial examples identiﬁes two possible causes: asymmmetries in the datasets and suboptimal learning.
Furthermore, it allows us to disentangle these two possible causes by creating synthetic datasets in which one of the two causes can be clearly implicated.
We are optimistic that such an approach will be of great use in developing new learning algorithms that are practical and robust.
A S YMMETRIC VS . A SYMMETRIC
Figure 7 presents additional examples comparing symmetric and asymmetric datasets and the relative robustness of their Bayes-Optimal classiﬁers to adversarial examples.
[-Smiling/NotSmiling Asymmetric: ( L 2 = 0 . 8 ) Smiling/NotSmiling Symmetric: ( L 2 = 3 . 1 ) 0/
 Asymmetric: ( L 2 = 0 . 5 ) 0/6 Symmetric: ( L 2 = 3 . 0 )-][Figure]
B M ODELS
In this section we provide additional information about the different classiﬁcation models – architecture, hyper-parameters and training procedure.
B. MFA
The Mixture of Factor Analyzers (MFA) model was trained using the code provided by Richardson & Weiss (2018).
The models are trained using Stochastic Gradient Descent.
The training data (CelebA, MNIST) is ﬁrst split by the desired binary attribute (e.g. Smiling / Not Smiling) and then a separate MFA model was trained independently for each subset of training samples.
Because of imbalance in the number of samples per class in CelebA, we set the number of components as the number of samples divided by 1000.
For MNIST we used a ﬁxed value of 25 components per class.
We chose an MFA latent dimension of 10 for CelebA and 6 for MNIST.
To allow attacking the MFA model with standard adversarial attacks such as CW-L2, we implemented it in TensorFlow as a standard CleverHans (Papernot et al., 2016) model.
B. B AYES -O PTIMAL
The MFA model is the Bayes-Optimal classiﬁer when the data is sampled from the model.
We modiﬁed the trained MFA models to deﬁne pairs of Bayes-optimal models.
A single Factor Analyzer component is deﬁned by a mean vector µ , a rectangular matrix A and per-pixel noise variance D (diagonal).
The component distribution is then x ∼ N ( µ, AA T + D ) .
For the symmetric models, we simply ﬁxed all noise variance values in D to a small value (std of 0 . 001 ).
For the asymmetric case we added two outlier components (one for each class) that are modiﬁcations from the dataset global mean in directions of low-variance: We performed PCA over the entire dataset and took the eigenvector for the 50th largest eigenvalue as this direction, where the mean of one outlier is in the positive direction and the mean of the other in the negative one.
We set
A = 0 and D = 0 .
5 for both outliers, making them spherical gaussians around the two means. [Equation] [Equation]
CNN
We used the reference fully-convolutional network implemented as part of the CleverHans library ( ModelAllConvolutional ).
The architecture consists of 2D convolution layers with a kernel size of 3 followed by Leaky ReLU activations and average pooling.
We set the basic number of ﬁlters (nb ﬁlters) to the minimal required number that achieves 100% classiﬁcation accuracy on our datasets.
All other hyper-parameters were left at their default values (calculated automatically by the library according to the image resolution).
Optimization method is Adam .
The reported results are not sensitive to the speciﬁc architecture (e.g. number of layers, channel depth) and training parameters.
B. L INEAR SVM
We used the standard 2-class linear SVC implementation provided by sklearn (no hyper-parameters).
Linear SVM is trained directly on the vectorized image samples.
The learned model consists of a weight vector W and a scalar bias b .
The decision for a sample x is simply sign( W T x + b ) .
B. RBF SVM
We used sklearn for the Radial Basis Function (RBF) kernel SVM as well.
A single hyper-parameter is required, γ = 12 σ 2 .
We chose the highest γ value that still provides a high classiﬁcation accuracy.
C A TTACKS C. CW-L
The Carlini & Wagner L2 attack (Carlini & Wagner, 2017) is a recommended strong attack that minimizes the perturbation L2 norm.
The attack minimizes a weighted combination of a classiﬁcation loss with the perturbation L2 size.
The relative weight is a parameter that is found using a binary-search.
We used the CleverHans implementation with the following hyper-parameters:iterations, 3 binary-searches and a learning rate of 0.01.
C. LD-G RAD
Since the MFA and SVM models provides a simple closed-form expression for the likelihood and its gradient, we implemented a simple and fast version of a gradient-attack for these models.
Our attack performs multiple (typically 3) steps in the direction of the gradient of the difference in loglikelihood between the source and target Gaussian components.
We compute the required step size to cross the gap in the log-likelihood between the source and the target components.
We repeated some of the experiments with the (much slower) CW-L2 attack and veriﬁed that the results are
Dataset / Attribute
[Table]
CelebA / Bangs
[Table] [Table]
[Table] [Table]
CelebA / Black Hair
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
CelebA / Brown Hair
[Table] [Table]
[Table] [Table]
[Table] [Table]
CelebA / Heavy Makeup
[Table] [Table]
[Table] [Table]
CelebA / High Cheekbones
[Table] [Table]
[Table] [Table]
CelebA / Male
[Table] [Table]
[Table] [Table]
CelebA / Mouth Slightly Open
[Table]
[Table] [Table]
[Table] [Table]
CelebA / No Beard
[Table] [Table]
[Table] [Table]
[Table] [Table]
CelebA / Smiling
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
CelebA / Wearing Earrings
[Table]
CelebA / Wearing Lipstick
[Table]
[Table] [Table]
CelebA / Eyeglasses
[Table]
[Table] [Table]
[Table] [Table]
MNIST /
[Table] [Table]
[Table] [Table]
MNIST /100%
[Table] [Table]
[Table] [Table] [Figure]
MNIST
[Table]
[Table]
[Figure]
[Table]
[Figure]
similar (i
[Figure]
e. models that are shown to be robust to the LD-Grad attack are also robust to the CW-L attack with similar perturbation magnitudes).
D A DDITIONAL R ESULTS
In this section we provide additional experimental results for the different datasets, attributes and models.
Table 1 lists the clean and adversarial classiﬁcation accuracy of all models for all datasets and attributes
. D. CNN – L ONGER T RAINING AND W IDER N ETWORKS
As shown in Figure 6, unlike the Bayes-optimal and the SVM RBF classiﬁers, CNNs trained on the symmetric datasets are vulnerable to adversarial examples (required perturbation size is small).
Here we investigate the effect of longer training with additional training samples and of wider networks.
We trained a CNN with 8 convolution layers and tested different values for the width (number of output channels) of the ﬁrst layer.
Each second layer has a stride of 2 and the width is doubled, so the width of the ﬁnal layer was 16 times that of the ﬁrst layer.
To test the effect of longer training with more samples, we modiﬁed the training procedure so that new samples from the symmetric dataset were generated dynamically.
We trained the networks for a total of 1 million samples, in mini-batches of 50.
The results are shown in Figure 8, indicating that wider networks and more data does not improve robustness.
[Figure] [Figure] D. I NCREASING THE D ATA N OISE V ARIANCE
As described in Appendix B.
[Figure]
in the main experiments we ﬁxed the noise standard deviation (the added diagonal part of the MFA covariance) of all Gaussians in the model that generated the symmetric dataset to σ = 0 .
Here we verify that our results are not sensitive to this value.
We increased the noise std to 50 times the original value.
At these levels (Figure 9, left) the noise is clearly visible in the data samples.
As can be seen (Figure 9, right)
[Figure]
the effect on the robustness of trained CNN models is minor.
As expected, the Bayes-Optimal model does not change its decision boundary and remains robust.
D. R OBUST (A DVERSARIAL ) CNN T RAINING
To test the effect of adversarial training on the CNN model for the symmetric dataset
[Figure]
we used the method proposed by Zhang et al.
(2019), which won the NeurIPS 2018 Adversarial Vision
Challenge (Robust Model Track).
Although slightly more robust, the robustiﬁed CNN is still far from the Bayes-optimal model (Figure 10).
D. R ESULTS ON R EAL D ATA
To estimate how close our symmetric datasets are to the real datasets, we tested the CNNs that were trained on the symmetric dataset on real test samples and compared the test accuracy to that of CNNs that were trained on the real training data.
As can be seen in Table 2, there is an average accuracy reduction of just 5%, indicating that the symmetric datasets are not that far from the original data. [Table]
D. A DDITIONAL R ESULTS FOR THE S YMMETRIC D ATASET
Figures 11-16 show original and adversarial samples and perturbations as well as histograms of the perturbations in pixel values for all models in different symmetric datasets (the number at the top of each histogram is the mean perturbation L2 over all test samples).
[Figure]
[Table]
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
Orig
[Figure]
[Figure]
Lin
[Figure]
[Figure] [Figure]
Orig
[Figure]
Lin
[Figure]