D ISENTANGLING A DVERSARIAL R OBUSTNESS IN D IRECTIONS OF THE D ATA M ANIFOLD
Anonymous authors
Paper under double-blind review
A BSTRACT
Using generative models (GAN or VAE) to craft adversarial examples, i.e. generative adversarial examples, has received increasing attention in recent years.
Previous studies showed that the generative adversarial examples work differently compared to that of the regular adversarial examples in many aspects, such as attack rates, perceptibility, and generalization.
But the reasons causing the differences between regular and generative adversarial examples are unclear.
In this work, we study the theoretical properties of the attacking mechanisms of the two kinds of adversarial examples in the Gaussian mixture data model case.
We prove that adversarial robustness can be disentangled in directions of the data manifold.
Speciﬁcally, we ﬁnd that: 1.
Regular adversarial examples attack in directions of small variance of the data manifold, while generative adversarial examples attack in directions of large variance.
Standard adversarial training increases model robustness by extending the data manifold boundary in directions of small variance, while on the contrary, adversarial training with generative adversarial examples increases model robustness by extending the data manifold boundary directions of large variance.
In experiments, we demonstrate that these phenomena also exist on real datasets.
Finally, we study the robustness trade-off between generative and regular adversarial examples.
We show that the conﬂict between regular and generative adversarial examples is much smaller than the conﬂict between regular adversarial examples in different norms.
I NTRODUCTION
In recent years, deep neural networks (DNNs) (Krizhevsky et al.
; Hochreiter and Schmidhuber (1997)) have become popular and successful in many machine learning tasks.
They have been used in different problems with great success.
But DNNs are shown to be vulnerable to adversarial examples (Szegedy et al.
; Goodfellow et al.
(2014a)).
A well-trained model can be easily attacked by adding a small perturbation to the image.
An effective way to solve this issue is to train the robust model using training data augmented with adversarial examples, i.e. adversarial training.
With the growing success of generative models, researchers have tried to use generative adversarial networks (GAN) (Goodfellow et al.
(2014b))
and variational autoencoder (VAE) (Kingma and Welling (2013)) to generate adversarial examples (Xiao et al.
; Zhao et al.
; Song et al.
(2018a);
Kos et al.
; Song et al.
(2018b))
to fool the classiﬁcation model with great success.
They found that standard adversarial training cannot defend these new attacks.
Unlike the regular adversarial examples, these new adversarial examples are perceptible by humans but they preserve the semantic information of the original data.
A good DNN should be robust to such semantic attacks.
Since the
GAN and VAE are approximations of the true data distribution, these adversarial examples will stay in the data manifold.
Hence they are called on-manifold adversarial examples by (Stutz et al.
On the other hand, experimental evidences support that regular adversarial examples leave the data manifold (Song et al.
We call the regular adversarial examples as off-manifold adversarial examples.
The concepts of on-manifold and off-manifold adversarial examples are important.
Because they can help us to understand the issue of conﬂict between adversarial robustness and generalization (Stutz et al.
; Raghunathan et al.
(2019)), which is still an open problem.
In this paper, we study the attacking mechanisms of these two types of examples, as well as the corresponding adversarial training methods.
This study, as far as we know, have not been done before.
Speciﬁcally, we consider a generative attack method that adds a small perturbation in the latent space of the generative models.
Since standard adversarial training cannot defend this attack, we consider the training methods that use training data augmented with these on-manifold adversarial examples, which we call latent space adversarial training.
Then we compare it to standard adversarial training (training with off-manifold adversarial examples).
Contributions: We study the theoretical properties of latent space adversarial training and standard adversarial training in Gaussian mixtures model with linear generator.
We give the excess risk analysis and saddle point analysis in this model.
Based on this case study, we claim that:
• Regular adversarial examples attack in directions of small variance of the data manifold and leave the data manifold.
• Standard adversarial training increases the model robustness by amplifying the small variance.
Hence, it extends the boundary of the data manifold in directions of small variance.
• Generative adversarial examples attack in directions of large variance of the data manifold and stay in the data manifold.
• Latent space adversarial training increases the model robustness by amplifying the large variance.
Hence, it extends the boundary of the data manifold in directions of large variance.
We provide experiments on MNIST and CIFAR-10 and show that the above phenomena also exist in real datasets.
It gives us a new perspective to understand the behavior of on-manifold and off-manifold adversarial examples.
Finally, we study the robustness trade-off between generative and regular adversarial examples.
On MNIST, robustness trade-off is unavoidable, but the conﬂict between generative adversarial examples and regular adversarial examples are much smaller than the conﬂict between regular adversarial examples in different norms.
On CIFAR-10, there is nearly no robustness trade-off between generative and regular adversarial examples.
R ELATED WORK
Our work is related to attack and defense methods.
Speciﬁcally, we care about attacks and defenses with generative models.
Attack Adversarial examples for deep neural networks were ﬁrst intruduced in (Szegedy et al.
However, adversarial machine learning or robust machine learning has been studied for a long time (Biggio and Roli (2018)).
In the setting of white box attack (Kurakin et al.
Papernot et al.
(2016); Moosavi-Dezfooli et al.
; Carlini and Wagner (2017)), the attackers have fully access to the model (weights, gradients, etc.).
In black box attack (Chen et al.
; Su et al.
; Ilyas et al.
, the attackers have limited access to the model.
First order optimization methods, which use the gradient information to craft adversarial examples, such as PGD (Madry et al.
(2017)), are widely used for white box attack.
Zeroth order optimization methods (Chen et al.
(2017)) are used in black box setting.
Li et al.
(2019) improved the query efﬁciency in black-box attack.
HopSkipJumpAttack
(Chen et al.
(2020)) is another query-efﬁcient attack.
Generative adversarial examples
Recently, generative models have been used to craft adversarial examples (Xiao et al.
; Song et al.
(2018b)
; Kos et al.
; Schott et al.
The adversarial examples are more natural (Zhao et al.
These adversarial examples lie in the data manifold, and they are called on-manifold adversarial examples.
Defense Training algorithms against adversarial attacks can be subdivided into the following categories.
Adversarial training: The training data is augmented with adversarial examples to make the models more robust (Madry et al.
(2017); Szegedy et al.
; Tramèr et al.
Preprocessing:
Inputs or hidden layers are quantized, projected onto different sets or other preprocessing methods (Buckman et al.
(2018); (Guo et al.
(2017); Kabilan et al.
Stochasticity: Inputs or hidden activations are randomized (Prakash et al.
; Dhillon et al.
; Xie et al. (2017)).
However, some of them are shown to be false defenses given by obfuscated gradients (Athalye et al.
Adaptive attack Tramer et al.
(2020) is used for evaluating defenses to adversarial examples.
Defense with generative model Using generative models to design defense algorithms have been studied extensively.
Using GAN, we can project the adversarial examples back to the data manifold (Jalal et al.
; Samangouei et al.
VAE is also used to train robust model (Schott et al.
P ROBLEM DESCRIPTION
Original space adversarial training: Consider the classiﬁcation problem of training a classifer f θ to map the data points x ∈ X ⊂ R d to the labels y ∈ Y , where X and Y are the input data space and the label space.
The classiﬁer f θ is parameterized by θ .
We assume that the data pairs ( x, y ) are sampled from the distribution P ( X, Y ) over X × Y .
Standard training is to ﬁnd the solution of min θ E ( x,y ) ∼ P (cid:96) ( f θ ( x ) , y ) , where (cid:96) ( · , · ) is the loss function.
The goal of dversarial training is to solve the minimax problem
[Equation]
where ε is the threshold of perturbation.
Here we can use (cid:96) 1 , (cid:96) 2 or (cid:96) ∞
-norm (Madry et al.
The inner maximization problem is to ﬁnd the adversarial examples x (cid:48) to attack the given classiﬁer f θ .
The outer minimization problem is to train the classiﬁer to defend the given adversarial examples x (cid:48) .
we refer to these attacks as regular attacks.
We refer to these minimax problems as standard adversarial training or original space adversarial training.
Latent space adversarial training: We assume that the data lie in a low dimensional manifold of R d .
Furthermore, we assume the true distribution D is a pushforward from a prior Guassian distribution z ∼ N (0 , I ) using G ( z ) ,
where G : Z → X is a mapping from the latent space Z to the original space X .
This is a basic assumption of GAN or VAE.
Let I : X → Z be the inverse mapping of G ( z ) .
The goal of latent space adversarial training is to solve the following minimax problem
[Equation]
Unlike the regular attacks, the distance between the original examples and adversarial examples can be large.
To preserve the label of the data, we use the conditional generative models (e.g. C-GAN (Mirza and Osindero (2014)) and C-VAE (Sohn et al.
i.e. the generator G y
( z ) and inverse mapping I y ( x ) are conditioned on the label y , for adversarial training.
We refer to these attacks as generative attack, and these adversarial training as latent space adversarial training.
Regular attack algorithms Two widely used gradient-based attack algorithms for the inner maximization problem in equation (1) are fast gradient sign method (FGSM)
(Goodfellow et al.
(2014a))
and projected gradient descend (PGD) (Madry et al.
Using FGSM, the adversarial examples are calculated by
[Equation]
where ∇ x denotes the gradient with respect to x .
PGD attempts to ﬁnd a near optimal adversarial example for the inner maximization problem (1) in multiple steps.
In the t th step,
[Equation]
where α is the step size, Π x + S [ · ] is the projection operator to project the given vector to the constraint x + S
= { x (cid:48)
[Equation]
In the whole paper, we refer to these as FGSM-attack and PGD-attack, and the corresponding original space adversarial training as FGSM-adv and PGD-adv.
FGSM-attack is a weak attack and PGD-attack is a stronger attack.
In section 5, we use them to show that a strong original space adversarial training,
PGD-adv, does not work well against a weak attack, FGSM-attack in the latent space.
Conversely, latent space adversarial training cannot defend a simple FGSM-attack in the original space.
Generative attack algorithm
In our experiments, we use FGSM in the latent space for the inner maximization problem in equation (2)
[Equation]
Because of the mode collapse issue of GAN (Salimans et al.
; Gulrajani et al.
, adding a small perturbation in the latent space of GAN may output the same images.
Thus we use VAE in our experiments.
we refer to this generative attack and latent space adversarial training as VAE-attack and VAE-adv.
T HEORETICAL ANALYSIS
In this section, we study the difference between adversarial training in the latent space and in the original space.
We study the simple binary classiﬁcation setting proposed by Ilyas et al.
The main reason for using this model is that we can ﬁnd the optimal closed-form solution, which gives us insights to understand adversarial training.
For a more complex model, we can only solve it numerically.
We leave all the proofs of the lemmas and theorems in appendix A.
4. T HEORETICAL MODEL SETUP
Gaussian mixture (GM) model Assume that data points ( x, y ) are sampled according to y ∼ {− 1 , 1 } unifomly and x ∼ N ( y µ ∗ , Σ ∗ ) , where µ ∗ and Σ ∗ denote the true mean and covariance matrix of the data distribution.
For the data in the class y = − 1 , we replace x by − x , then we can view the whole dataset as sampled from D = N ( µ ∗ , Σ ∗ ) .
Classiﬁer The goal of standard training is to learn the parameters Θ = ( µ , Σ ) such that
[Equation]
where (cid:96) ( · ) represents the negative log-likelihood function.
The goal of adversarial training is to ﬁnd
[Equation]
We use L and L r to denote the standard loss and adversarial loss.
After training, we classify a new data point x to the class sgn ( µ T Σ − 1 x ) .
Generative model
In our theoretical study, we use a linear generative model, that is, probabilistic principle components analysis (P-PCA)
(Tipping and Bishop (1999)).
P-PCA can be viewed as linear
GAN (Feizi et al.
; Feizi et al.
and linear VAE (Dai et al.
Given dataset { x i } ni =1 ⊂ R d , let µ and S be the sample mean and sample covariance matrix.
The eigenvalue decomposition of S is S =
U Λ U T , then using the ﬁrst q eigenvactors, we can project the data to a low dimensional space.
P-PCA is to assume that the data are generated by
[Equation]
where z ∼ N (0 , I ) , (cid:15) ∼ N (0 , σ 2 I ) , z ∈ R q and W ∈ R d × q .
Then we have x ∼ N ( µ , W W T + σ 2 I ) , x | z ∼ N ( W z + µ , σ 2 I ) and z | x ∼ N ( P − 1 W T ( x − µ ) , σ 2 P − 1 ) where P = W T W + σ 2 I .
The maximum likelihood estimator of W and σ 2 are
[Equation]
where U q is the matrix of the ﬁrst q columns of U , Λ q is the matrix of the ﬁrst q eigenvalues of Λ .
In the following study, we assume that n is large enough such that we can learn the true µ ∗ and Σ ∗ .
Thus we have S = Σ ∗ ,
U q = U q ∗ , Λ q = Λ q ∗ for the generative model.
4. M INIMAX PROBLEM OF LATENT SPACE ADVERSARIAL TRAINING
To perturb the data in the latent space, Data will go through the encode-decode process x
→ z → ∆ z + z → x (cid:48) .
Based on the probablistic model, we may choose z with the highest probabiliy or on just sample it from the distribution we learned.
Hence, we could have different strategies.
Below we list 3 strategies.
Strategy 1 is used in practice and the other two are alternative choices.
In lemmawe show that these strategies are equivalent under the low dimensional assumption.
Hence, we do not need to worry about the effect of sampling strageties.
Strategy 1: Sample x ∼ D , encode z = arg max q ( z | x ) =
P − 1 W T ( x − µ ∗ ) , add a perturbation ∆ z , and ﬁnally, decode x adv = arg max p ( x | z + ∆ z ) =
W ( z + ∆ z ) + µ ∗ .
Strategy 2: Sample x ∼ D , then sample z ∼ q ( z | x ) , add a perturbation ∆ z , and ﬁnally, sample x adv ∼ p ( x | z + ∆ z ) .
Strategy 3: Sample z ∼ N (0 , I ) , add a perturbation ∆ z , and then sample x adv ∼ p ( x | z + ∆ z ) .
In this strategy, x adv can be viewed as the adversarial example of x = arg max x q ( z | x ) .
The following lemma shows that the adversarial examples can be uniﬁed in one formula.
Hence sampling strategies will not affect our analysis.
Lemma 1 (Adversarial examples perturbed in the latent space) .
Using these 3 strategies, the adversarial examples can be uniﬁed as
[Equation]
where
[Equation]
If the data lie in a q dimensional subspace, i.e. the covariance matrix Σ ∗ is rank q , we have Λ (1) =
Λ (2) =
Λ (3) =
Λ ∗ .
Then D (cid:48)
=
D .
In general, the adversarial example can be decomposed into 2 parts, the change of distribution x (cid:48) ∼ D (cid:48) and the small perturbation W ∆ z .
Therefore the adversarial expected risk can be written as the following minimax problem
[Equation]
We aim to analyze the different properties between the minimax problems in equations (4) and (6).
We give the excess risk and optimal saddle point analysis in the following subsections.
4. E XCESS RISK ANALYSIS
We consider the difference between L ls and L given the true Θ ∗ , i.e. L ls ( Θ ∗ ; D (cid:48) j ) − L ( Θ ∗ ; D ) .
It characterizes the excess risk incured by the optimal perturbation.
To derive the expression of excess risk, we decompose it into two parts
[Equation]
change
[Table]
distribution
[Equation]
To simplify the notation, we consider the Lagrange penalty form of the inner maximization problem in equation (6), i.e. max (cid:96) ( x (cid:48)
+ W ∆ z, µ , Σ ) − L (cid:107)
∆ z (cid:107) 2 / 2 , where L is the Lagrange multiplier.
The following theorem gives the solution in the general case.
Theorem 2 (Excess risk) .
Let L ls and L be the loss with and without perturbation in latent space (equations (6) and (3) respectively), given the non-robustly learned Θ ∗ = ( µ ∗ , Σ ∗ ) , thus the excess risk caused by perturbation is
[Equation]
and the excess risk caused by the changed of distribution is
[Equation]
It is hard to see which part dominates the excess risk.
If we further assume that the data lie in a q dimensional manifold, the excess risk caused by the change of distribution becomes 0 by Lemma 1.
We have the following corollary.
Corollary 3 (Excess risk) .
Let L ls and L be the loss with or without perturbation in latent space (equation (6) and (3) respectively), given the non-robustly learned Θ ∗ = ( µ ∗ , Σ ∗ ) , and rank ( Σ ∗ ) =
q .
The excess risk
[Equation]
The optimal perturbation in the latent space will incur an excess risk in O ( qL − 2 ) .
The adversarial vulnerability depends on the dimension q and the Lagrange multiplier L .
It does not depend on the shape of the data manifold.
This is because the perturbation constraint (the black block) aligns with the shape of the data manifold (the ellipse) as we demonstrate in Figure 1 (c).
Thus generative attacks will focus on the directions of the largest q variance.
Then, we analyze the excess risk of original space adversarial training.
Since the perturbation thresholds ε are on different scales in the original space attack and latent space attack, the corresponding
Lagrange multipliers L are different.
We use L (cid:48) for original space adversarial training in the following
Theorem.
Theorem 4 (Excess risk of original space adversarial training) .
Let L r and L be the loss with or without perturbation in original space (equations (4) and (3) respectively), given the non-robustly learned Θ ∗ = ( µ ∗ , Σ ∗ ) .
Denote λ min be the smallest eigenvalue of Σ ∗ .
The excess risk is
[Equation]
If the data lie in a low dimensional manifold, i.e. λ min = 0 , the excess risk equals to + ∞ .
Optimal perturbation in the original space will incur an excess risk in O ( d ( λ min L (cid:48) )
− 2 ) .
The adversarial vulnerability depends on the smallest eigenvalues λ min , the dimension d , and the Lagrange multiplier L (cid:48) .
The λ min comes from the misalignment between the perturbation constraint (the black block) and the shape of the data manifold (the ellipse) as we demonstrate in Figure 1 (a).
Notice that λ min also exists in the lower bound.
Hence, the excess risk equals to + ∞ when λ min = 0 .
Thus regular attacks focus on the directions of small variance.
Speciﬁcally, when λ min = 0 , regular adversarial examples leave the data manifold.
4. S ADDLE POINT ANALYSIS
In this subsection we study the optimal solution of optimization problem (6).
Since it is not a standard minimax problem, we consider a modiﬁed problem:
[Equation]
We will explain more about the connection between optimization problems in equation (6) and (8).
See appendix A.
The following theorem is our main result.
It gives the optimal solution of latent space adversarial training.
Theorem 5 (Main result: Optimal Saddle point) .
The optimal solution of the modiﬁed problem in equation (8) is
[Equation]
where
[Equation]
for i > q, and j = 1 , 2 , 3 corresponding to strategies 1,2 and 3.
We assume that the data lie in a q -dimensional manifold again.
Then we have λ lsi /λ
i = 1 / 2 + 1 /L + (cid:112) 1 / 4 + 1 /L ≥ 1 for i ≤ q and λ lsi /λ i =
for i > q .
Latent space adversarial training increases the model robustness by amplifying large eigenvalues of the data manifold.
The illustration of the two dimensional case is in Figure 1, (c) and (d).
In the same setting, the optimal solution of standard adversarial training (Problem (4)) on the original space is in Theorem 6 (which is Theorem 2 in Ilyas et al.
Theorem 6 (Optimal saddle point Ilyas et al.
The optimal solution of the problem in equation (4) is
[Equation]
Theorem 6 is for the problem that the covariance matrix is restricted to be diagonal.
Consider the ratio
[Equation]
For small true eigenvalue λ i , the ratio is large.
Standard adversarial training increases the robustness by amplifying the small eigenvalues of the data manifold.
The illustration of the two dimensional case in Figure 1, (a) and (b).
E XPERIMENTS
In this section we report our experimental results on training LeNet on MNIST (LeCun et al.
and ResNet (He et al.
on CIFAR-10 (Krizhevsky et al.
(2009)) to conﬁrm our theoretical ﬁndings.
Both of the two datasets contain 50000 training samples and 10000 test samples.
Details of the hyperparameters setting are in appendix B.
5. E IGENVALUES OF MNIST WITH AND WITHOUT ADVERSARIAL TRAINING
In this subsection, we show that the properties of the eigenvalues are also valid on the real dataset.
We use MNIST as an example.
Firstly, for each image in the test set D test of MNIST, we vectorize it into a 784 dimension vector.
Then we calculate the EVD (Eigenvalue decomposition) of the covariance matrix of each class.
We plot the 784 eigenvalues of class 0 and 1 in Figure 2 in blue line as examples (the other 8 classes are shown in Appendix B.3).
Secondly, we adversarially train a classiﬁer f r ( x ) in original space.
For each point x in D test , we construct x r by PGD-attack with ∇ x f r ( x ) .
Then we obtain a robust test set D rtest .
We plot the intra class eigenvalues of the covariance matrix in Figurein orange line.
Lastly, we adversarially train a classiﬁer f ls ( x ) in latent space.
Then we construct latent space robust dataset D lstest by generative attack against f ls ( x ) on D test .
The eigenvalues of the covariance matrix are shown in Figure 2 in green line.
We show all the eigenvalues in the ﬁrst column in Figure 2.
After adding a small perturbation in the original space or in the latent space, the distribution of D rtest and D lstest are close to D test .
We show the details of large eigenvalues (the ﬁrst 30 eigenvalues) in the second column and small eigenvalues (last 754 eigenvalues) in the last column.
Original space adversarial training focuses on the small variance direction We adversarially train a robust classiﬁer f r ( x ) in the original space.
The robust test set D rtest against f r ( x ) ampliﬁed the small eigenvalues a lot.
In the third column, the orange line is signiﬁcantly larger that the otherlines.
While in the second column, the orange line is below the other two lines.
The experiments give us an understanding of how the adversarial examples leave the data manifolds.
The original dataset
D test lies in a low dimension afﬁne plane in R 784 .
After adversarial training, the data move towards the small variance directions.
Finally D r test stay in a full dimension subspace.
Latent space adversarial training focuses on the large variance direction We adversarially train a robust classiﬁer f ls ( x ) in the latent space.
The latent space robust test set D lstest against f ls ( x ) ampliﬁed the large eigenvalues.
In the second column, we can see that the green line is above the other two lines.
And in the last column, the green line shows that adversarial training in the latent space will not affect the small eigenvalues.
The adversarial examples in D lstest move along the large variance direction.
Therefore, D ls test stay in the low dimension afﬁne space.
5. R OBUST TEST ACCURACIES
In this subsection, we compare the test accuracy of different attacks (FGSM, PGD, VAE) versus different defense (adversarial training with FGSM, PGD, and VAE) to help explain our theoretical results.
We explain the results of MNIST in Table 1 as an example.
On-manifold and off-manifold adversarial examples The test accuracies of the standard training model on PGD and VAE-attack data are 3.9% and 42.4% respectively.
Firstly, They show that on-manifold adversarial examples exist.
Then, we can see that on-manifold adversarial examples are harder to ﬁnd.
Attack versus defense Our theory tells us that original or latent space adversarial training increase the model robustness by amplifying the small or large eigenvalues, which is the variance of the distribution, in the data manifold respectively.
In experiments, the test accuracy of VAE-adv vs
PGD-attack is 1.23%, which shows that extending the manifold boundary in directions of large variance gives no contribution to defending attacks in directions of small variance.
Similarly, the test accuracy of PGD-adv vs VAE-attack is 52.18%, which shows that amplifying the boundary in the directions of small variance does not work well in defending attacks in directions of high variance.
Further more, we can see that latent space adversarial training does not work well in a simple FGSM attack, and vice versa.
We see that PGD-adv can increase the test accuracies on VAE-attack from 42% to 52% on MNIST and from 19.40% to 26.31% on CIFAR-10.
A possible reason is that original space adversarial training can amplify the small eigenvalues of the ﬁrst q dimension.
In Figure 2, both original space and latent space adversarial training will increase the eigenvalues which are small but not equal to zero (around the 100th eigenvalue) of the covariance matrix.
5. R OBUSTNESS TRADE - OFF
Robustness trade-off are common in practice (Su et al.
; Tramèr and Boneh (2019)).
For example, (cid:96) 1 and (cid:96) ∞ adversarial examples cannot be defended simultaneously.
The experiments on robustness trade-off are shown in Table 2.
On MNIST, the jointly-trained model using both (cid:96) 1 and (cid:96) ∞ adversarial examples will decrease the robust test accuracy by 20% comparing to the single trained model.
On CIFAR-10, the jointly-trained model will decrease the test accuracies by 6% on (cid:96) 1 -attack and by 10% on (cid:96) ∞ -attack.
A possible reason is that they are all original space attacks and focus on the same directions of small variance.
Hence they conﬂict with each other.
We study the robustness trade-off between regular and generative adversarial examples in this subsection.
[Table] [Table]
Our theory suggests that adversarial robustness can be disentangled in different directions.
Hence adversarial robustness against attacks on directions of low variance and large variance can be guaranteed simultaneously.
On MNIST, the jointly-trained model decreases the test accuracies by 6% comparing to the single trained model.
The conﬂict between on-manifold and off-manifold adversarial examples is much small than the conﬂict between off-manifold adversarial examples in different norms.
On CIFAR-10, the jointly-trained model gets test accuracies 74% on PGD-attack and 42% on VAE-attack, which exhibit nearly no robustness trade-off.
Therefore, if our goal is to defend mixture regular and generative attacks, The jointly-train model will perform well.
Discussion Under the q -dimensional manifold assumption, there is an overlap between the directions of the q largest variance and the directions of small variance.
This is supported by Theorem 5,
Theorem 6 and the ﬁrst experiment.
A possible reason for the robustness trade-off between regular and generative attacks is that they conﬂict with each other in the overlap directions.
The conﬂct is much smaller because they mainly focus on different directions.
C ONCLUSION
In this paper, we show that adversarial robustness can be disentangled in directions of small variance and large variance of the data manifold.
Theoretically, we study the excess risk and optimal saddle point of the minimax problem of latent space adversarial training.
Experimentally, we show that these phenomena also exist in real datasets.
Future works In future works, we can design defense algorithms based on this property.
We can generate adversarial examples based on the directions of the dataset without access to the model architecture.
In while-box settings, we can use them for data augmentation to accelerate adversarial training or even increase the model robustness.
But we need to carefully compare the computational cost between calculating the EVD of the datasets and the cost of standard adversarial training.
In black-box attacks, we can use them to query the target model.
But the data manifolds should be transferred from some known datasets (since we have no information about the training dataset of the target model).
Theoretically, if we use a nonlinear generative model, whether we can ﬁnd the closed-form solution is not clear.
How to analyze the nonlinear model is an open problem.
Based on the linear model, we can further analyze the conﬂict between robustness and generalization.
O VERVIEW
In appendix A, we provide the proof of the Theorems.
In appendix B, we show the settings about the experiments.
Appendix C is a further discussion of data augmentation using generative models.
It is not closely related to our main paper.
A P ROOF OF THE T HEOREMS A. P ROBLEM DESCRIPTION
Lemma 1 (Adversarial examples perturbed in the latent space) .
Using these 3 strategies, the adversarial examples can be uniﬁed as
[Equation]
and
[Equation]
where
[Equation]
If the data lie in a q dimensional subspace, i.e. the covariance matrix Σ ∗ is rank q , we have Λ (1) =
Λ (2) =
Λ (3) =
Λ ∗ .
Then D (cid:48)
=
D .
Proof:
Recall that x ∼ N ( µ , W W T + σ 2 I ) , x | z ∼ N ( W z + µ , σ 2 I ) and z | x ∼ N ( P − 1 W T ( x − µ ) , σ 2 P − 1 ) where P = W T W + σ 2 I .
The maximum likelihood estimator of W and σ 2 are
W ML =
U q ( Λ q − σ 2 I ) 1 /and σ 2 ML =
d − q d (cid:88) i =
q
+ λ i .
Strategy
[Table]
[Equation]
, encode
[Equation]
arg max q ( z | x ) =
P − 1 W T ( x − µ ∗ ) , add a perturbation ∆ z , and ﬁnally, decode x adv = arg max p ( x | z + ∆ z ) =
W ( z + ∆ z ) + µ ∗ .
Then x adv = W ( P − 1 W T ( x − µ ∗ ) + ∆ z ) + µ ∗ =
W P − 1 W T ( x − µ ∗ ) + µ ∗ + W ∆ z = x (cid:48)
+ W ∆ z.
Since x ∼ ( µ ∗ , Σ ∗ ) , we have x − µ ∗ ∼ (0 , Σ ∗ ) , Then x (cid:48) ∼ N ( µ ∗ , W P − 1 W T Σ ∗ ( W P − 1 W T ) T ) ,
With
W P − 1 W T Σ ∗ ( W P − 1 W T ) T
= U ∗ (cid:20) Λ q − σ   (cid:21) 1 / 2 (cid:20) Λ q  σ
I (cid:21) − 1 (cid:20) Λ q − σ   (cid:21) 1 /Λ ∗ (cid:20) Λ q − σ   (cid:21) 1 / 2 (cid:20) Λ q  σ
I (cid:21) − 1 (cid:20) Λ q − σ   (cid:21) 1 /
U T ∗ =
U ∗ (cid:20) ( Λ q − σ 2 I ) 2 Λ − 1 q   (cid:21)
U T ∗ =
U ∗ Λ ( j ) U T ∗ , j = 1 .
Strategy 2 Sample x ∼ D , then sample z ∼ q ( z | x ) , add a perturbation ∆ z , and ﬁnally, sample x adv ∼ p ( x | z + ∆ z ) .
Then
[Equation]
and
[Equation]
With
[Equation]
Strategy 3 Sample z ∼ N (0 , I ) , add a perturbation ∆ z , and then sample x adv ∼ p ( x | z + ∆ z ) .
In this strategy, x adv can be viewed as the adversarial example of x = arg max x q ( z | x ) .
[Equation]
With
[Equation]
In these 3 strageties, the adversarial examples can be summerized as
[Equation]
and x (cid:48)
[Equation]
D
[Equation]
1 , 2 , 3 , where j = 1 , 2 , 3 corresponding to strategy 1,2 and 3.
If the data lie in a low dimensional space, i.e. the covariance matrix Σ ∗ is rank q .
Then the maximum likelihood of σ 2 ML = (cid:80) di
= q
+1 λ
i / ( d − q ) = 0 .
Then
[Equation]
There is no difference among these 3 strategies and there is no change of distribution, i.e. D (cid:48)
=
D .
A.1. E XCESS RISK ANALYSIS
Before we prove Theorem 2.
We need to prove the following lemma ﬁrst.
Lemma 2 (optimal perturbation) .
Given Θ = ( µ , Σ ) the optimal solution of the inner max problem in equation 6 is
[Equation]
where L is the lagrange multiplier satisfying (cid:107) ∆ z ∗ (cid:107) = ε .
Proof: Consider problem
[Equation]
The Lagrangian function is
[Equation]
Notice that this quadratic objective function is concave when L is larger than the largest eigenvalue of W T Σ − 1 W .
Calculate the partial derivative with respect to ∆ z and set it to be zero, we have
[Equation]
The last equation comes from the Woodbury matrix inversion Lemma.
We can obtain L by solving the equation (cid:107) ∆ z ∗ (cid:107) = ε .
We don’t have a closed form solution of L but we can solve it numerically.
L → ∞ as ε → 0 .
We only need to know L is a constant in our whole theory.
Theorem 2 (Excess risk) .
Let L ls and L be the loss with or without perturbation in latent space (equation 6 and 3 respectively), given the non-robustly learned Θ ∗ = ( µ ∗ , Σ ∗ ) , The excess risk caused by perturbation is
[Equation]
and the excess risk caused by changed of distribution is
[Equation]
Proof: Since x
[Equation]
Denote
[Equation]
And we have
[Equation]
The excess risk caused by perturbation is
[Equation]
I
[Equation]
=
[Equation]
and the excess risk caused by changed of distribution is
[Equation]
It is hard to see which part dominates the excess risk.
If we further assume that the data lie in a q dimension manifold.
The excess risk caused by the change of distribution becomes 0 .
We have the following corollary.
Corollary 3 (Excess risk) .
Let L ls and L be the loss with or without perturbation in latent space (equation (6) and (3) respectively), given the non-robustly learned Θ ∗ = ( µ ∗ , Σ ∗ ) , and rank ( Σ ∗ ) =
q .
The excess risk
[Equation]
Proof: By Lemma 1, we have σ 2 = 0 .
λ ( j ) i
[Equation]
and D (cid:48)
j = D .
Hence the excess risk caused by changed of distribution
[Equation]
The excess risk caused by perturbation is
[Equation]
Theorem 4 (Excess risk of orginal space adversarial training) .
Let L r and L be the loss with or without perturbation in original space (equations (4) and (3) respectively), given the non-robustly learned Θ ∗ = ( µ ∗ , Σ ∗ ) .
Denote λ min be the smallest eigenvalue of Σ ∗ .
The excess risk
[Equation]
Theorem 4 can be viewed as a corollary of Theorem 1 in Ilyas et al.
We give the prove here.
Proof: Consider the Lagrange multiplier form of the inner maximization problem in equation 4.
[Equation]
The Lagrangian function is
[Equation]
Notice that this quadratic objective function is concave when L is larger than the largest eigenvalue of Σ − 1 .
Calculate the partial derivative with respect to ∆ x and set it to be zero, we have
[Equation]
The excess risk is
[Equation]
i
[Equation]
On the one hand,
[Equation]
On the other hand,
[Equation]
A.1. S ADDLE POINT ANALYSIS
Theorem 5 (Main result: Optimal Saddle point) .
The optimal solution of the modiﬁed problem in equation (8) is
[Equation]
and Σ ls =
U ∗ Λ ls U T ∗ , where
[Equation]
i
[Equation]
q and λ lsi =
λ ( j ) i for i > q. j = 1 , 2 , 3 corresponding to strategies 1,2 and 3.
Problem 6 is not a standard minimax problem, consider the modiﬁed problem
[Equation]
By lemma
[Equation]
the optimal perturbation ∆ z ∗ is a matrix M times x − µ .
Consider the problem
[Equation]
Lemma 6 (optimal perturbation) .
Given Θ = ( µ , Σ ) the optimal solution of the inner max problem of 10 is
[Equation]
Proof: Consider the problem
[Equation]
The lagrangian function is
[Equation]
Let x (cid:48) − µ = v
[Equation]
Take the gradient with respect to M and set it to be zero
[Equation]
we have
[Equation]
Then we have
[Equation]
The last equality is the Woodbury matrix inversion Lemma.
Notice that lemma 3 and Lemma 6 have the same form of solution.
This is why we can use Problem 10 to approximate Problem 6.
To solve the problem 10, we need to introduce Danskin’s Theorem.
Theorem 7 (Danskin’s Theorem) .
Suppose φ ( x, z ) :
X × Z
→ R is a continuous function of two arguments, where Z ⊂ R m is compact.
Deﬁne f ( x ) = max z ∈Z φ ( x, z ) .
Then, if for every z ∈ Z , φ ( x, z ) is convex and differentiable in x , and ∂φ/∂x is continuous:
The subdifferential of f ( x ) is given by
[Equation]
where conv ( · ) is the convex hull, and Z 0 ( x ) is
[Equation]
If the outer minimization problem is convex and differentiable, we can use any maximizer for the inner maximization problem to ﬁnd the saddle point.
But the outer problem of problem 4 is not convex, we need to modify the problem again.
Assume that we have already obtained the eigenvector
U ∗ from the ML estimator.
The optimization variables of the outer minimization problem are µ and Λ .
Then we have Σ
= U ∗ Λ U T ∗ in problem 10.
Another reason to make this assumption is that we only need to consider the eigenvalue problem to compare with standard adversarial training (Theorem 2 of Andrew Ilyas et al., 2019).
Proof of Theorem 5:
By Lemma 6, we have
[Equation]
Which is a diagonal matrix Λ M times U T ∗ .
Let T = Λ − 1 , m =
Λ
−
1 U T ∗ µ and x (cid:48)(cid:48) =
U T ∗ x (cid:48) .
The optimization problem becomes
[Equation]
Obviously, the inner constraint is compact (by Heine-Borel theorem), we only need to prove the convexity of the outer problem to use Danskin’s Theorem.
For any x (cid:48) and Λ M ,
[Equation]
Let u
= U T ∗ ( x (cid:48) − µ )
[Equation]
and A = ( I + (cid:20) Λ q − σ
[Equation]
Λ M )
[Equation]
consider the third term
[Equation]
we have
[Equation]
By Daskalakis et al.
, 2018Daskalakis et al.
, The hessian matrix is
[Equation]
Therefore, this is a convex problem.
By the same calculation in Lemma 6, a maximizer of the inner problem is
[Equation]
Then
[Equation]
Then the ﬁrst order derivative (by Daskalakis et al.
(2018)) is
[Equation]
From the second equation
[Equation]
we directly have µ ls = µ ∗ .
From the ﬁrst equation, for i > q , we have (1 +
[Equation]
For i ≤ q , we have
[Equation]
It equivalents to a second order equation of (cid:112) λ lsi
[Equation]
Solving this equation, we obtained λ
[Equation]
[Equation]
(cid:105) for i = 1 ≤ q and λ lsi = λ ( j ) i , for i > q.
B
E XPERIMENTS SETTINGS B. MNIST
For Mnist, we use LeNet5 for the classiﬁer and 2 layers MLP (with hidden size 256 and 784) for the encoder and decoder of conditional VAE.
For standard training of the classiﬁer, we use 30 epochs, batch size 128, learning rate 10 − 3 , and weight decay 5 × 10 − 4 .
For the CVAE, we use 20 epochs, learning rate 10 − 3 , batch size 64, and latent size 10.
For standard adversarial training, we use ε = 0 .
25 for FGSM and PGD.
in PGD, we use 40 steps for the inner part.
Adversarial training start after 10 epochs standard training.
For generative adversarial training, we use ε = 1 in the latent space with FGSM.
Adversarial training start after 10 epoches standard training.
In the attack part, we use ε = 0 .
2 for norm-based attack and ε = 1 for generative attack on the test set.
B. CIFAR
For CIFAR10, we use ResNet32 for the classiﬁer and 4 layers CNN for the encoder and decoder of conditional VAE.
For standard training of the classiﬁer, we use 200 epochs, batch size 128, learning rate 10 − 3 , and weight decay 5 × 10 − 4 .
For the CVAE, we use 100 epochs, learning rate 10 − 3 , batch size 64, and latent size 128.
For standard adversarial training, we use ε = 4 / 255 for FGSM and PGD.
in PGD, we use 10 steps for the inner part.
Adversarial training start after 100 epochs standard training.
For generative adversarial training, we use ε = 0 .
1 in the latent space with FGSM.
Adversarial training start after 100 epoches standard training.
Since we see that the modeling power of VAE in
CIFAR10 is not good enough.
For each of the image, the encode variance is very small.
When we add a small perturbation to the encode mean value, the output image are blured.
Hence we only use a small ε = 0 .
In the attack part, we use ε = 4 / 255 for norm-based attacks and ε = 0 .
1 for generative attack on the test set.
The test accuracy of VAE-adv against VAE attack is 40 .
18% in our experiments.
It is not good enough because of the modeling power of VAE.
But the results on standard adversarial training versus VAE-attacks are worse, and vice versa.
The experiments support our ﬁndings.
B. E IGENVALUES OF COVARIANCE MATRIX OF MNIST
We plot the eigenvalues of all the classes in this section, see Figure 3.
C F INITE SAMPLES CASE : D ATA AUGMENTATION OR ADVERSARIAL TRAINING
In this section we discuss the question that can we use the generative model to generate more examples for training in our theoretical framework.
Because it is not closely related to our main results, we only discuss it in appendix.
Let us focus on the case that the number of samples are not enough.
Generative model cannot help data augmentation Given dataset { x i } ni =1 ⊂ R d , let ˆ µ and ˆ S
=
ˆ U
ˆΛ
ˆ U be the sample mean and sample covariance matrix.
The generative model learned by this dataset is x = ˆ U q ( ˆΛ q − σ 2 I ) 1 / 2 z
+ ˆ µ + (cid:15) .
The distribution of the data sample from the model is
[Equation]
If we use n samples from the original dataset and m samples from the generative model, the loss function is
[Equation]
Theorem 8 (Data augmentation by generative model) .
Given MLE ˆ µ and ˆ S , The optimal solution training with n true samples and m generated samples is µ da = ˆ µ and Σ da
= ˆ U Λ da ˆ U T , where λ dai
=
ˆ λ i for i ≤ q and λ dai = n
[Equation]
ˆ λ
[Equation]
k
[Equation]
ˆ λ k for i > q.
Proof: Consider the derivative of Problem 13.
[Equation]
Then we can obtain µ da = ˆ µ .
[Equation]
Then we have
[Equation]
For i ≤ q ,
[Equation]
For i > q ,
[Equation]
The optimal solution is a little bit destoryed.
In this perspective, generative models give no help to data augumentation.
Generative model can help robust training
Theorem 9 (Adversarial learned parameters) .
The Adversarial learned features of problem 10 under ﬁnite samples is µ ls = ˆ µ and Σ ls
= ˆ U Λ ls ˆ U T where Λ ls is of the same form in Theorem 5 except replacing λ
i by ˆ λ i under strategy 3.
Proof: Considering strategy 3.
In this case
[Equation]
and
[Equation]
Since
[Equation]
In words, the eigenvectors of the dsitribution ˆ U is the same as the one in perturbation W ∆ z .
The optimization problem is the same as the one we use in the proof of Theorem 5 if we replace λ i by ˆ λ i .
If the data lie in a q dimensional subspace, we do not neet to make the assumption on strategy 3.