I MPLICIT B IAS OF A DVERARIAL T RAINING FOR D EEP N EURAL N ETWORKS
A BSTRACT
We provide theoretical understandings of the implicit bias imposed by adversarial training for homogeneous deep neural networks without any explicit regularization.
In particular, for deep linear networks adversarially trained by gradient descent on a linearly separable dataset, we prove that the direction of the product of weight matrices converges to the direction of the max-margin solution of the original dataset.
Furthermore, we generalize this result to the case of adversarial training for non-linear homogeneous deep neural networks without the linear separability of the dataset.
We show that, when the neural network is adversarially trained with (cid:96) 2 or (cid:96) ∞ FGSM, FGM and PGD perturbations, the direction of the limit point of normalized parameters of the network along the trajectory of the gradient ﬂow converges to a KKT point of a constrained optimization problem that aims to maximize the margin for adversarial examples.
Our results theoretically justify the longstanding conjecture that adversarial training modiﬁes the decision boundary by utilizing adversarial examples to improve robustness, and potentially provides insights for designing new robust training strategies.
I NTRODUCTION
Deep neural networks (DNNs) have achieved great success in many ﬁelds such as computer vision (Krizhevsky et al., 2012;
He et al., 2015) and natural language processing (Collobert & Weston, 2008) and other applications.
These breakthroughs also lead to the importance of research on the robustness and security issues of DNNs, among which those about adversarial examples are especially prevalent.
Adversarial examples are obtained by adding craftily designed imperceptible perturbations to the original examples, which can sharply change the predictions of the DNNs with high conﬁdence (Szegedy et al., 2014; Nguyen et al., 2015).
Such vulnerability of DNNs renders security concerns about deploying them in security-critical systems including vision for autonomous cars and face recognition.
It is therefore crucial to develop defense mechanisms against these adversarial examples for deep learning.
To this end, many defense strategies have been proposed to make DNNs resistant to adversarial examples, such as adding a randomization layer before the input to the classiﬁer (Xie et al., 2018), input transformations (Guo et al., 2018), adversarial training (Madry et al., 2018), etc.
However,
Athalye et al.
(2018) pointed out that most of the defense techniques are ineffective and give a false sense of security due to obfuscated gradients except for adversarial training—a method which has not been comprehensively attacked yet.
Given a C -class training dataset { ( x i , y i ) } ni =1 with natural example x i ∈ R d and corresponding label y i ∈ { 1 , ..., C } , adversarial training is formulated as solving the following minimax optimization problem
[Equation]
where f is the DNN function, (cid:96) is the loss function and δ i ( W ) is the adversarial perturbation generated by some adversary A ( x i , y i ; W ) typically depending on model parameters W , within the set B (0 , (cid:15) )
= { δ : (cid:107) δ (cid:107) p ≤ (cid:15) } around the natural example x i .
Commonly, adversarial training conducts a two-player game at each iteration: the inner maximization is to attack the model and ﬁnd the corresponding perturbation of the original example that maximizes the classiﬁcation loss (cid:96) ; the outer minimization, on the other hand, is to update the model parameters W with the gradient descent method such that the loss (cid:96) is minimized on adversarial examples generated by the inner maximization (see Algorithm 1 for details).
To have a better theoretical understanding of the fact that adversarial training empirically improves the robustness of DNNs against adversarial examples, Zhang et al.
(2019) decomposed the prediction error for adversarial examples and identiﬁed a trade-off between robustness and accuracy while Li et al.
(2020) studied the inductive bias of gradient descent based adversarial training for logistic regression on linearly separable data.
Faghri et al.
(2021) studied adversarial robustness of linear neural networks by exploring the optimization bias of different methods.
Yu et al.
(2021) studied adversarial training through the bias-variance decomposition and showed that its generalization error on clean examples mainly comes from the bias.
However, even for the simplest DNN—the deep linear network, we notice that there exists no work to theoretically understand such robustness achieved by adversarial training through exploring its implicit bias.
On the other hand, for the standard training, recent works extensively explored the implicit bias imposed by gradient descent or its variants for DNNs in different settings.
For a simple setting of linear logistic regression on linearly separable data, Soudry et al.
; Ji & Telgarsky (2018);
Nacson et al.
(2019b) derived that the direction of the model parameter converges to that of the max- (cid:96) 2 -margin with divergent norm.
Shamir (2021) concluded that gradient methods never overﬁt when training linear predictors over separable dataset.
Viewing the above model as a single-layer network, a natural but more complicated extension is that for the deep linear network on linearly separable data: Ji & Telgarsky (2019) proved that the gradient descent aligns the weight matrices across layers and the product of weight matrices also converges to the direction of the max- (cid:96) 2 -margin solution; Gunasekar et al.
(2018) showed the implicit bias of gradient descent on linear convolution networks.
Nacson et al.
(2019a)
; Lyu & Li (2020) further promoted the study of implicit bias of gradient descent for standard training to the case of more general homogeneous non-linear neural networks and proved that the limit point of the optimization path is along the direction of a KKT point of the max-margin problem.
Wei et al.
(2019) studied the regularization path for homogeneous
DNNs and also proved the convergence to the max-margin direction in this setting.
Banburski et al.
(2019) showed that gradient descent induces a dynamics of normalized weights converging to an equilibrium which corresponds to a minimal norm solution.
It is therefore our goal in this paper to theoretically understand the resistance to adversarial examples for adversarially trained DNNs, linear and non-linear ones, through the lens of implicit bias imposed by adversarial training.
Due to the inner maximization, adversarial training differs a lot from standard training and one should be careful to analyze the perturbed training dynamics.
For the adversarial training objective Eq.
(1), various approaches have been proposed to solve the inner maximization, such as fast gradient sign method (FGSM (Goodfellow et al., 2015)) and its stronger version projected gradient descent (PGD (Madry et al., 2018)).
The widely used adversarial training adopts PGD to attack the model, while recent work (Wong et al., 2020) also suggested that a weaker adversary can also surprisingly yield a model with satisfying robustness.
Thus to conduct a comprehensive study about the implicit bias of adversarial training for DNNs, we will use (cid:96) 2 -fast gradient method (FGM (Miyato et al., 2016)), (cid:96) ∞ FGSM, (cid:96) 2 -PGD and (cid:96) ∞ -PGD to solve the inner maximization of the adversarial training objective.
1. O UR C ONTRIBUTION
In this paper, we devote to answering two questions.
First, is there any implicit bias imposed by adversarial training for DNNs without explicit regularization?
Second, if there exists such an implicit bias, what are the convergence properties of the model parameters along the adversarial training trajectory?
To this end, we ﬁrst investigate the adversarial training with (cid:96) 2 adversarial perturbations for deep linear networks on linear separable data where the allowed Euclidean distances from the adversarial examples to their corresponding original examples (cid:107) x (cid:48)
i − x i (cid:107) 2 are less than the max- (cid:96) 2 -margin of the original dataset.
Despite the simplicity of this setting, this problem is meaningful due to its non-convexity and the introduction of adversarial examples, which heavily depend on the model parameters, during the training process.
We prove that gradient descent for adversarial training implicitly aligns weight matrices across layers and the direction of the product of weight matrices also surprisingly converges to that of the max- (cid:96) 2 -margin solution of the original dataset—similar to that of standard training for deep linear network (Ji & Telgarsky, 2019).
Our results signiﬁcantly generalize those in Li et al.
(2020) for adversarial training of logistic regression.
This simple yet insightful case positively answers our ﬁrst question but partially answers the second one because it still remains unclear why such convergence property can improve robustness considering its similarity to that for the standard training.
In fact, adversarial training differs from standard training in the way how they impose such convergence property for parameters: the ﬁrst one is to maximize the margin of adversarial examples while the latter is maximizing that of the original dataset, and these two optimization problems happen to possess solutions along the same direction.
We then move forward to explore a more general situation, adversarial training for homogeneous non-linear DNNs without the linear separability of the dataset.
We study the limit point of the normalized model parameters along the adversarial training trajectory and show that
Theorem 1 (Informal) .
When the deep neural network is adversarially trained with one of the (cid:96) 2 -FGM, FGSM, (cid:96) 2 -PGD and (cid:96) ∞ -PGD perturbations, the limit point of the normalized model parameters is along the direction of a KKT point of a constrained optimization problem which aims to maximize the margin of adversarial examples.
This indicates that adversarial training is implicitly maximizing the margin of adversarial examples rather than that of original dataset.
Thus Theorem 1 provides another view for the high bias error on clean examples of adversarial training in Yu et al.
(2021) since distributions of adversarial and clean examples are different.
To the best of knowledge, these results are the ﬁrst attempt to analyze the implicit bias of adversarial training for DNNs.
We believe our results provide a theoretical understanding on the effectiveness of adversarial training for improving robustness against adversarial examples.
It could potentially shed light on how to enhance the robustness of adversarially trained models or even further inspire more effective defense mechanisms.
Organization.
This paper is organized as follows.
Section 2 is about notations and settings.
Section 3 presents our main results on the implicit bias of adversarial training for DNNs.
Section 4 provides numerical experiments to support our claims.
We conclude this work in Section 5 and discuss future directions.
Some technical proofs are deferred to supplementary materials.
Algorithm
Input: Training set S
= { ( x i , y i ) } ni =1 , Adversary A to solve the inner maximization, learning rate η , initialization W k for k ∈ { 1 , . . .
, L } for t = 0 to T − 1 do
S (cid:48) ( t ) = ∅ for i = 1 to n do x (cid:48) i ( t )
=
A ( x i , y i , W ( t ))
S (cid:48) ( t ) = S
[Equation]
t ) (cid:83) ( x
[Equation]
( t ) , y i ) end for for k = 1 to L do
[Equation]
end for end for
P RELIMINARIES
Notations.
For any matrix A ∈ R m × n , we denote its i -th row j -th column entry by A ij .
Let A T denote the transpose of A .
(cid:107)·(cid:107) F represents the Frobenius norm and (cid:107)·(cid:107) p is the (cid:96) p norm.
The training set is { ( x i , y i ) } ni =1 where x
i ∈ R d , (cid:107) x i (cid:107) 2 ≤ 1 and y i ∈ { 1 , − 1 } .
For a scalar function f : R d (cid:55)→ R , we denote its gradient by ∇ f .
Furthermore, tr ( A ) =
(cid:80) i
A ii denotes the trace for the matrix A .
We study the adversarial training for the L -layer positively homogeneous deep neural network
[Equation]
where W k is the k -th layer weight matrix and ϕ k is the activation function of the k -th layer 1 .
The multi- c -homogeneity of the network is deﬁned by
[Equation]
for any positive constants a k ’s and c ≥ 1 , where W =
( W 1 , · · · , W L ) is the collection of the parameters of the network.
For example, deep ReLU networks are multi-1-homogeneous.
For convenience, we also adopt the following notations for a multi- c -homogeneous DNN:
[Equation]
and
[Equation]
where (cid:99) W k
=
W
k
(cid:107) W k (cid:107) F with (cid:107) (cid:99) W k (cid:107) F = 1 for k ∈ { 1 , . . .
, L } .
We use δ i ( W ) to represent the adversarial perturbation of the original example x i within the perturbation set B (0 , (cid:15) ) : { δ : (cid:107) δ (cid:107) p ≤ (cid:15) } around the original example x i for f ( x ; W ) .
Furthermore, we use the scale invariant adversarial perturbations deﬁned as follows for adversarial training in this paper.
Deﬁnition 1 (Scale invariant adversarial perturbation) .
An adversarial perturbation is said to be a scale invariant adversarial perturbation for f ( x i ; W 1 , . . .
, W L ) and loss function (cid:96) if it satisﬁes
[Equation]
for any positive constants a k ’s.
We will show in Section 3.2 that FGSM, (cid:96) 2 -FGM, (cid:96) 2 -PGD and (cid:96) ∞ -PGD perturbations for homogeneous DNNs are all scale invariant perturbations, which are important for analyzing different types of perturbation in a uniﬁed manner.
The empirical adversarial training loss with the perturbation δ i ( W ) is given by
[Equation]
For ease of notation, we denote ˜ (cid:96) i ( W ) =
˜ (cid:96) ( y i , x i ; W ) .
The loss function (cid:96) is continuously differentiable and satisﬁes
Assumption
[Equation]
Many widely used loss functions satisfy the above assumption such as (cid:96) ( x ) =
e − x and the logistic loss (cid:96) ( x ) =
ln(1 + e − x ) .
Furthermore, we make the following common assumptions about the smoothness of f ( x ; W ) and the adversarial perturbation δ ( W ) :
Assumption 2 (Smoothness) .
With respect to W , yf ( x ; W ) is locally Lipschitz for any ﬁxed x ; y i f ( x i ; W ) further have locally Lipschitz gradients and δ i ( W ) are locally Lipschitz for all training examples x i .
Remark.
Our results can also be generalized to non-smooth homogeneous neural networks straightforwardly (Appendix B.4).
Assuming Lipschitzness about perturbations is because we focus on popular perturbations such as (cid:96) 2 -FGM and PGD perturbations which have explicit forms and depend on gradients of the network, whose Lipschitzness assumptions are quite common.
M AIN R ESULTS
We present our main theoretical results on the implicit bias of the gradient ﬂow/gradient descent for adversarial training in this section.
For the deep linear neural network, a special kind of homogeneous models, in Section 3.1 we further restrict the dataset to be linearly separable and focus on (cid:96) 2 adversarial perturbations.
We prove that the singular vectors corresponding to the largest singular values of weight matrices get aligned across layers with the progress of adversarial training 2 .
Based on this key result, the product of weight matrices converges to the direction of the maximum margin solution
This phenomenon for standard training has been studied by Ji & Telgarsky (2019)
ﬁrst.
under the original data.
Furthermore, we study a much more general scenario in Section 3.2: without the assumption of linearly separability of data and norm constraints on adversarial perturbations, we show that the gradient ﬂow of adversarial training with scale invariant adversarial perturbations for the homogeneous nonlinear neural networks implicitly performs margin maximization under the adversarial data .
Due to the space limit, some technical proofs of Section 3.1 are deferred to
Appendix A, and we present proofs of Section 3.2 in Appendix B.
3. A DVERSARIAL T RAINING FOR D EEP L INEAR N ETWORK
In this section we restrict the dataset { ( x i , y i ) } ni =1 to be linearly separable in the sense that there exists a unit vector u such that ∀ i ∈ { 1 , . . . , n } , y i (cid:104) u, x i (cid:105) >
and we explore the adversarial training dynamics of gradient descent for deep linear networks, i.e., with identity activation function ϕ k ( x ) =
x for all layers,
[Equation]
Now let
[Equation]
be the max-margin solution given by the hard-margin SVM and γ m = max (cid:107) u (cid:107)
=
1 y m (cid:104) x m , u (cid:105) be the max-margin, where m = arg min i ∈{ 1 ,...,n } y i (cid:104) x
i , u (cid:105) .
We only consider the (cid:96) 2 -norm adversarial perturbations in this section, i.e., B (0 , (cid:15) )
= { δ : (cid:107) δ (cid:107) 2 ≤ (cid:15) } .
The allowed Euclidean distance from an adversarial example x (cid:48) to its original example x is assumed to satisfy
[Equation]
We denote the product of all weight matrices by W Π = W L · · · W 1 , which is simply a vector.
Our model is adversarially trained by gradient descent
[Equation]
The inner maximization of the adversarial training objective Eq.
(1) can be solved exactly in this case, where the adversarial perturbations are taken as
[Equation]
for the t -th iteration of adversarial training.
For any layer k , the weight matrix can be decomposed as
[Equation]
where U k and V k are the left and right singular matrices, respectively.
Furthermore, we use u k and v k to represent the left and right singular vectors corresponding to the largest singular value, respectively.
We emphasize that the training dynamics abruptly changes due to this perturbation and one should be careful to analyze its effects.
It can be easily seen that Eq.
(10) is a scale invariant adversarial perturbations by our deﬁnition.
Note that with this kind of perturbation, the adversarial data during the training are still linearly separable, since we have required that the perturbation is smaller than the max-margin of the original dataset described in Eq.
We further assume that (cid:96) satisﬁes
Assumption 3.
The loss function (cid:96) is β -smooth and | (cid:96) (cid:48) | ≤ α .
such as logistic loss.
We are now ready to explore the implicit bias of the gradient descent for adversarial training.
Inspired by the idea of studying the smoothness of loss function from Ji &
Telgarsky (2019) for standard training, here we ﬁrst examine whether our adversarial training loss
Eq.
(1) possesses smoothness.
Let the set S ( r ) denote the set of weights with bounded norm
[Equation]
where r ≥ 1 and ¯ r is a small constant to avoid trivial solution.
To simplify the notation, we let
[Equation]
The following lemma provides us a ﬁrst view of the overall trends of gradient descent for adversarial training in deep linear network on linear separable dataset.
Lemma 1 (Overall trends of adversarial training for deep linear network) .
With adversarial perturbation Eq.
(10) , under Assumption 1, Assumption 3 and the requirement Eq.
(8) , for the deep linear network adversarially trained by gradient descent on linear separable data, the adversarial training objective (1) is β ( r, ¯ r, (cid:15) )
-smooth within the given set S ( r ) for a constant r .
However, with constant step size for gradient descent, the weight matrices will eventually leave the set S ( r ) if r remains unchanged during the training
[Equation]
for some t
[Equation] [Equation]
[Equation]
The ﬁrst part of this lemma implies the smoothness of the adversarial training loss Eq.
(1) while also leading the weight matrices to leave the set S ( r ) .
In fact, this is saying that max k ∈{ 1 ,...,L } (cid:107) W k (cid:107) F is divergent along the adversarial training trajectory since max k ∈{ 1 ,...,L } (cid:107) W k (cid:107) F will be larger than r for any given r .
We can keep W ( t + 1) inside S ( r ( t )) at every step t by delicately adjusting the step size of gradient descent and treating r as a function of the step t .
Lemma 2 (Smoothness of the adversarial training loss) .
If the learning rate is taken as η ( t ) =
min { 1 , 1 /β ( r, ¯ r, (cid:15) ) } , then the adversarial training loss ˜
L ( W ) is β ( r, ¯ r, (cid:15) )
-smooth where r ( t + 1) is taken as
[Equation]
during the training, where µ = r L − 1 (1 + (cid:15) )
α/β ( r, ¯ r, (cid:15) ) .
We show the divergence of max k ∈{ 1 ,...,L } (cid:107) W k (cid:107) F along the adversarial training trajectory in Lemma 1, and now we can further derive that the Frobenius norms of weight matrices for all layers are divergent with the smoothness of the adversarial training loss in hand, considering that the differences between any two layers during the adversarial training are always bounded (Lemma 6).
Due to this divergence of weight norms, only the direction of the product of weight matrices is important for the model to make predictions.
Now we present our main theorem to provide insights on the theoretical understanding of the implicit bias of gradient descent for adversarial training of deep linear network.
Theorem 2 (Convergence to the direction of the max-margin solution) .
With adversarial perturbation
Eq.
(10) , under Assumption 1, Assumption 3 and requirement (8) , for gradient descent of the adversarial training objective Eq.
(6) with logistic loss, the singular vectors of the largest singular values of adjacent layers get aligned if the learning rate is taken as that in Lemma 2:
[Equation]
(14) as t → ∞ .
As a result, the direction of the product of weight matrices converges to that of the max-margin solution
[Equation]
Remark.
The alignment phenomenon for standard training has been showed in the previous work
Ji & Telgarsky (2019), here we further extend this result to the case of adversarial training, where the training objective is different from that of the standard training due to the introduction of adversarial perturbation dependent on the parameters of the network.
Our work is also a signiﬁcant generalization of Li et al.
(2020) which studied the convergence to max-margin solution of adversarial training for logistic regression, a single
-layer linear network.
Furthermore, although the direction of the solution Eq.
(15) given by adversarial training does not differ from that of standard training, i.e., the direction of ¯ u , these two solutions are not same: the ﬁrst is maximizing the margin of adversarial examples while the latter is maximizing that of original dataset—which happens to be the same one in this setting.
The similarity of these two solutions comes from the fact that the adversarial data are also linearly separable under the requirement Eq.
(8), whose max-margin solution has the same direction as that of ¯ u .
The above reasoning will be more clear in the next section where we show the solution of adversarial training as a max-margin solution of adversarial examples more explicitly.
3. A DVERSARIAL T RAINING FOR H OMOGENEOUS D EEP N EURAL N ETWORK
Additional deﬁnitions For simplicity, we will use exponential loss (cid:96) ( x ) =
e − x in this section.
For an original example x i , the margin for its adversarial example x i + δ i ( W ) is deﬁned as ˜ γ i
= y
i f ( x i + δ i ( W ); W ) while for the whole dataset { ( x i , y i ) }
ni =1 , the margin for their corresponding adversarial examples is denoted by ˜ γ m where m =
arg min m ∈{ 1 ,...,n } y i f
( x i + δ i ( W ); W ) .
We introduce the normalized margin as ˆ γ i =
y
i f ( x i + δ i ( (cid:99) W ); (cid:99) W ) to explore the convergence properties, where (cid:99) W k
=
W
k
(cid:107) W k (cid:107) F such that (cid:107) (cid:99) W k (cid:107) F = 1 for any layer k .
Furthermore, the margin for the original example x i is deﬁned to be γ i
= y
i f ( x i ; W ) ≥ ˜ γ
i .
To relieve the headaches of picking suitable learning rates, we consider gradient ﬂow, gradient descent with inﬁnitesimal step, of the training loss Eq.
(6), which leads to
[Equation]
(16) for k ∈ { 1 , . . .
, L } .
Note that the introduction of adversarial perturbations (relying on the network parameters) abruptly perturbs the training dynamics.
In this section, to conduct a comprehensive study of adversarial training, we do not restrict the perturbation used for solving the inner maximization of the adversarial training objective in advance while only require it to be a scale invariant adversarial perturbation.
By noting the following lemma, we can easily generalize our results to the commonly used adversarial attacks:
Lemma 3.
Under Assumption 1, (cid:96) 2 -FGM, FGSM, (cid:96) 2 -PGD and (cid:96) ∞ -PGD perturbations for homogeneous DNNs are all scale invariant adversarial perturbations.
The RHS of Eq.
(16) is different from that of standard training due to the inner maximization.
Our key observation is the following property of adversarial training which can drastically simplify our analysis despite of the complexity brought by the adversarial examples.
Lemma 4.
For the exponential loss, along the gradient ﬂow trajectory of adversarial training with scale invariant adversarial perturbations for homogeneous DNN f ( x ; W ) , we have
[Equation]
In the case of deep linear network, we adopt linearly separable dataset with the requirement Eq.
(8), yielding the fact that the adversarial data are also linearly separable.
In this section we do not impose such conditions but only apply a weaker assumption in a more general scenario.
Assumption 4.
There exists a separability of adversarial examples of the dataset 3 : there exists a time t 0 such that y i f
( x i + δ i ( W ( t 0 )); W ( t 0 )) > 0 for all i ∈ { 1 , . . . , n } .
Theoretically, Zhang et al.
; Gao et al.
(2019) proved the convergence to low robust training loss for heavily overparametrized nets.
Moreover, adversarial training can typically achieve this separability in practice, i.e., the model can ﬁt adversarial examples of the training dataset, which makes the above assumption a reasonable one.
In Section 3.1, we obtain divergent weight norms for adversarial training.
Thanks to Lemma 4, we can further generalize it to the case of homogeneous network without linearly separability of dataset and the requirement Eq.
(8) for adversarial examples.
This divergence of model parameters is necessary for the convergence of the adversarial training loss.
To begin with, recalling our deﬁnition of ρ = (cid:81)
Lk =1 ρ ck = (cid:81)
Lk =1 (cid:107) W k (cid:107) cF , we state our ﬁrst main theorem regarding the overall trend for gradient ﬂow of adversarial training as follows.
Theorem 3 (Divergence of weight norms) .
For the adversarial training objective (6) of the binary classiﬁcation task with exponential loss and scale invariant adversarial perturbations, under Assumption 2 and Assumption 4, the product of the Frobenius norms of weight matrices diverges to inﬁnity along the trajectory of the gradient ﬂow
[Equation]
and, as a result, the training loss ˜ L converges to zero as t → ∞ .
Remark.
The divergence of weight norms for standard training of homogeneous DNNs has been studied by, for example, Lyu & Li (2020);
Wei et al.
The above theorem considers adversarial training , which implies that the margin for an adversarial example, ˜ γ i for any i ∈ { 1 , . .
. , n } , also goes to inﬁnity along the adversarial training trajectory.
This makes the normalized margin ˆ γ i necessary to understand the convergence properties of the adversarial training solution as t → ∞ .
Built upon this result, we ﬁrst shed some lights on the implicit bias of the gradient ﬂow for the adversarial training through the following theorem, which discusses the training dynamics of the normalized margins for the adversarial examples.
Theorem 4 (Non-decreasing adversarial margin) .
For adversarial training with scale invariant adversarial perturbations, when the separability of adversarial examples is achieved at t 0 , the weighted sum of the changing rates of normalized margins for the adversarial examples are nonnegative for all t ≥ t 0 :
[Equation]
there exists a sufﬁciently large time t 1 > t 0 such that d ˆ γ m dt ≥ 0 if ∀ i : ˆ γ i − ˆ γ m = ω
( 1 ρ ) for all t ≥ t 1 .
The above theorem reveals a relation for the normalized margins for adversarial examples .
According to the above lemma, since ρ → ∞ as t → ∞ , the term with weight e − ρ ˆ γ m , roughly speaking, dominates the LHS of (18) after some time t 1 ≥ t 0 .
Then d ˆ γ m /dt 4 will never be negative for ∀ t ≥ t 1 .
This directly implies that the gradient ﬂow is actually maximizing the margin for the adversarial examples of the original dataset.
Applying Lemma 4 and Theorem 3, in the following theorem, we further strengthen the above observation for adversarial training for homogeneous DNNs.
Theorem 5 (Convergence to the direction of a KKT point) .
Under Assumption 2 and 4, for exponential loss and multi-c-homogeneous DNNs, the limit point of normalized weight parameters { W/ (cid:107) W (cid:107) 2 : t ≥ 0 } of the gradient ﬂow for the adversarial training objective Eq.
(6) with scale invariant adversarial perturbations is along the direction of a KKT point of the constrained normminimization problem
[Equation]
Remark 1.
According to Lemma 3, the above theorem can be directly applied to (cid:96) 2 -FGM, FGSM, (cid:96) 2 and (cid:96) ∞ -PGD perturbations.
The requirement Eq.
(8) in Section 3.1 is not required since the dataset in this section is no longer linearly separable.
It is well-known that the norm-minimization problem is closely related to the margin-maximization problem in the sense that
Optimization
[Equation]
⇐⇒
[Equation]
Remark 2.
Theorem 5 is the key theorem of this section.
It shows that, for adversarial training, the gradient ﬂow is implicitly maximizing the margin of adversarial examples .
Therefore, for perturbations with norm constraints other than (cid:96) 2 -norm, (19) is no longer an (cid:96) 2 -norm optimization problem but a mixed-norm one due to the perturbation δ ( W ) in ˜ γ m , which was ﬁrst observed by
Li et al.
(2020) for adversarial training of one-layer neural network while our results are for more general deep non-linear homogeneous neural networks.
This is one of the major differences between adversarial training and standard training—they are formulated from solving different optimization problem.
Theorem 5 provides theoretical understandings of the folklore that adversarial training can modify the decision boundary against adversarial examples to improve robustness.
Remark 3.
In the context of standard training, Lyu & Li (2020) studied the properties of the normalized margin through its approximation, the smooth-normalized margin.
One can try to generalize the analysis method in Lyu & Li (2020) to adversarial training with various kinds of adversarial perturbations by further assuming the local smoothness regarding gradients when considering nonsmooth analysis.
Generalizinig our results to non-smooth case can be found in Appendix B.4.
To easily adapt to adversarial training, our new strategy in this work is to directly analyze the normalized margin itself for adversarial examples instead and utilizes the alignment phenomena observed by
Ji & Telgarsky (2020).
Most importantly, what we seek to push forward in the current work are the theoretical understandings of the adversarial training for DNNs through its implicit bias.
[Figure]
Trade-off between robustness and accuracy.
More signiﬁcantly, Theorem 5 also implies that adversarial training leads to the trade-off between robustness and accuracy as in Zhang et al.
This can be seen as follows.
Let f ( · , W ) denote the classiﬁer obtained from adversarial training.
Then for a clean training example ( x, y ) with y = 1 for simplicity, we have yf ( x + δ ( W ); W ) =
f ( x + δ ( W ); W ) > 0 since f ( x ; W ) is a max-margin classiﬁer over adversarial data.
This means that, for any clean test example ( x (cid:48) , y (cid:48) )
around x in the sense that x (cid:48)
=
x + τ with (cid:107) τ (cid:107) p ≤ (cid:15) , we will have
[Equation]
due to the deﬁnition of δ ( W ) when y = 1 : δ ( W ) =
max v ∈B ( x,(cid:15) )
(cid:96) ( yf ( x + v ; W )) =
min v ∈B ( x,(cid:15) )
f ( x + v ; W ) for (cid:96) (cid:48) < 0 .
Therefore, as a result of forcing our model to correctly classify the adversarial example x + δ ( W ) , it will classify any such x (cid:48)
=
x + τ as having labelseven there may be some of them having true labels -1, which leads to the drop of accuracy at the cost of increasing robustness.
Besides, since adversarial training maximizes the margin of the speciﬁc adversarial perturbation used for training, the trained model performs worse on other perturbations.
N UMERICAL E XPERIMENTS
In this section, we conduct numerical experiments on MNIST dataset to support our claims.
We adversarially trained a 3-layer neural network using SGD with constant learning rate and batch-size 80.
The model has the architecture of input layer- 1024 -ReLU- 64 -ReLU-output layer.
We present results for adversarial training with: (1) FGSM perturbations with (cid:15)
= 16 / 255 ; (2) (cid:96) ∞ -PGD perturbations, where the PGD is ran for 5 steps with step size 6 / 255 and (cid:15)
= 16 / 255 .
As a comparison, we also standardly trained a model with the same architecture to evaluate the normalized margin for adversarial examples by attacking it with FGSM and PGD during its training process.
Fig.
1(a) shows that the normalized margin for FGSM adversarial examples keeps increasing during the process of adversarial training with FGSM perturbations.
Meanwhile, the standardly trained model has lower normalized margins for FGSM adversarial examples.
Similar results exist for (cid:96) ∞ -PGD adversarial examples as showed by Fig. 1(b).
To see that adversarial normalized keeps increasing more clearly, we also plot the adversarial training with FGSM perturbations afterepochs in Fig. 1(c).
More detailed experiments are in Appendix C. These empirical ﬁndings support our claims that adversarial training implicitly maximizes the margin of adversarial examples.
C ONCLUSION
In this paper, we have studied the implicit bias of adversarial training for the deep linear networks and, more generally, the homogeneous DNNs.
We proved that adversarial training with scale invariant adversarial perturbations implicitly performs margin-maximization for adversarial data during its training process.
Intuitively, such implicit bias strongly implies that adversarial training encourages the neural networks to utilize adversarial examples more to improve its robustness.
It will be an interesting future direction to improve the effect of adversarial training by promoting it to enlarge the margins of adversarial examples more effectively.
It is also possible to design new defense mechanisms by explicitly requiring the standard training to have the bias of maximizing the margin of adversarial examples.
We will leave these potential beneﬁts of our theoretical analysis as future work.
A CKNOWLEDGEMENT
This project is supported by Beijing Nova Program (No. 202072) from Beijing Municipal Science
Technology Commission.
A P ROOFS OF S ECTION 3.
In Section A. we discuss the smoothness of the adversarial training loss.
In Section A. we prove the alignment phenomenon of adversarial training for deep linear networks.
We make the common assumption that ˜ L ( W (0)) ≤ ˜ L (0) =
(cid:96) (0) and ∇ ˜ L ( W (0))
(cid:54) = 0 to start adversarial training.
A. S MOOTHNESS OF THE ADVERSARIAL TRAINING LOSS A.1. P ROOF OF L EMMA
Proof.
We ﬁrst prove that the adversarial training loss for the deep linear network is smooth within
S ( r ) .
For any W, V ∈ S ( r )
[Equation]
let δ W and δ V denote the adversarial perturbations of the network f ( x ; W ) and f ( x ; V )
[Equation]
respectively.
Then we have
[Equation]
by taking derivatives of L w
[Equation]
r
[Equation]
t W and using that (cid:107) A (cid:107) F + (cid:107) B (cid:107) F ≥ (cid:107) A − B
[Equation]
F
[Equation]
[Table]
[Equation]
where the term (b) abruptly changes the training dynamics.
For simplicity
[Equation]
we use x
[Equation]
y and ˜ (cid:96) (cid:48) without the subscript i because the analysis can be done similarly for any i ∈ {
[Equation]
. , n } .
We begin with the term (a) of Eq.
[Equation]
where c follows from (cid:107) x (cid:107) ≤ 1 , (cid:107) W k (cid:107) ≤ r
and (cid:96) (cid:48) ≤ α ; repeating the procedure of c until k
=
L gives us d ; e is because (cid:107) W k − V k (cid:107) ≤ (cid:107) W − V (cid:107) for any k and
[Equation] [Equation]
considering that (cid:96) is a β -smooth function.
It now remains us to bound the term (b) of Eq.
Note that
[Equation]
then
[Table]
[Equation]
) where c follows from (cid:107) W k (cid:107) ≤ r and ˜ (cid:96) (cid:48) ≤ α ; repeating c for 2 L − 1 times gives d ; e follows from (cid:107)(cid:107) W Π (cid:107) − (cid:107) V Π (cid:107)(cid:107) ≤ (cid:107) W Π − V Π (cid:107) ; invoking Eq.
(23) gives us f .
Now we can bound Eq.
(21) by combing the above results
[Equation] [Equation]
where the last inequality is because we choose r ≥ 1 .
The above analysis is for the ﬁrst layer.
Now we can apply the same technique to other layers, which ﬁnally gives us
[Equation] [Equation]
thus ˜ L is β ( r, ¯ r, (cid:15) )
-smooth inside S ( r ) where
[Equation]
We now prove the second statement of Lemma 1, which says that W will leave S ( r ) with constant r and learning rate η .
Let W ( t ) , W ( t + 1) ∈ S ( r ) and η ( t ) =
min { 1 , 1 /β ( r, ¯ r, (cid:15) ) } .
Lemma 5 (Adapted from Lemma 6 in Ji & Telgarsky (2019)) .
Under Assumption 1 and Assumption 3, suppose gradient descent for adversarial training is run with a constant step size 1 /β ( r, (cid:15), ¯ r ) .
Then there exists a time t when max k (cid:107) W k ( t ) (cid:107) F > r .
Eq.
(27) states that ˜ L is β ( r, ¯ r, (cid:15) )
-smooth, therefore
[Equation]
29) which also implies that ˜ L ( W ( t )) never increases during the training
[Equation]
By repeatedly applying Lemma 5 when the learning rate is constant, we know W will eventually leave S ( r ) , which also implies that max k (cid:107) W k (cid:107) F is unbounded in this case.
A.1. P ROOF OF L EMMA
Proof.
For any step t with the learning rate in the Lemma 2, suppose that W ( t ) ∈ S ( r ( t ) − µ ( t )) , we then have
[Equation]
where the last inequality follows from
[Equation]
Therefore (cid:107) W k ( t + 1) (cid:107) F will always stay in S ( r ( t )) and the smoothness is preserved.
If (cid:107) W k ( t + 1) (cid:107) F is still less than r ( t ) − µ ( t ) , then we keep r ( t + 1) =
r ( t ) while we increase r ( t + 1) to
[Equation]
if r ( t ) − µ ( t ) ≤ (cid:107) W k
( t + 1) (cid:107) F ≤ r ( t )
such that (cid:107) W k
( t + 1) (cid:107) F ≤ r ( t ) ≤ r ( t + 1) − µ ( t + 1) , which means that W ( t + 2) will stay inside S ( r ( t + 1)) .
The overall smoothness can then be proved by induction.
A. A LIGNMENT PHENOMENON
We start with the following lemma for the adversarial training to prove the alignment phenomenon 5 :
Lemma 6 (Bounded differences between weight norms) .
The difference of Frobenius norms for any two layers is bounded during the adversarial training process
[Equation]
Proof.
If there is not adversarial perturbation, then we can easily obtain
[Equation] [Equation]
by conducting some algebra, where L stands for the empirical loss without adversarial perturbation.
Then invoking Eq.
(24) for k by noting that tr ( A ) = tr (cid:0) A T (cid:1) and tr ( ABC ) = tr ( CAB ) , we have
[Equation]
Combining Eq.
[Equation]
for the adversarial empirical loss
[Equation]
we have
[Equation]
which will give us the following relation
[Equation]
which can be easily seen by writing out the gradient descent update explicitly.
One can derive a similar relation for W k ( t + 1) W Tk ( t + 1) .
Combing Eq.
(32), we have
[Equation]
Summing up the above equation from 0 to t will give us
[Equation]
where symmetric matrices A k and B k are deﬁned by
[Equation]
and
[Equation]
Also notice that
[Equation]
37) where
[Equation]
where a is because η < 1 ; b is due to Eq.
(29); summing up from τ = 0 to t − 1 and noting that ˜
L > 0 give us c .
For any two layers k < j , taking trace of both sides of (36) and summing from k to j , we have that the differences between the Frobenius norms for any two layers
[Equation]
are always bounded.
Since max 1 ≤ k ≤ L (cid:107) W k (cid:107) F → ∞ , the above result implies that (cid:107) W k (cid:107) F → ∞ for all k .
A.2. P ROOF OF T HEOREM
Intuitively, W k can be decomposed as
[Equation]
For W k , denote the ﬁrst singular value, the corresponding left singular vector and right singular vector by σ k , u k and v k .
Since σ k → ∞ , the initialization becomes negligible as t → ∞ .
According to Eq.
(36), this will lead to
[Equation]
as t → ∞ .
On the other hand, since W L W TL has rank 1 for W L being a row vector, all layers have rank 1 because they get aligned with W L and
[Equation]
To further elaborate the above reasoning, we provide a more exact proof starting with the deﬁnition of the alignment phenomenon.
Deﬁnition 2 (Alignment phenomenon for deep linear networks.) .
For deep linear network f ( x ; W ) =
W L · · · W 1 x, the alignment phenomenon is deﬁned as
[Equation]
and W k / (cid:107) W k (cid:107) F → u k v Tk as t → ∞ along the training trajectory, where u k , v k are the left and right singular vectors correspond to the largest singular value of W k .
The proof for the alignment phenomenon of adversarially trained deep linear networks in our setting is as follows 6 .
Proof.
Note that
[Equation]
we have
[Equation]
by utilizing the deﬁnition of matrix norm and
[Equation]
According to Eq.
(36), let
[Equation]
and replace W Tk +1 W k +1 with Γ k + W k W Tk in Eq.
(41), we have
[Equation]
To prove the alignment phenomenon, we now bound the RHS of Eq.
(44) with 1 − α where α →when t → ∞ .
We are going to bound 3 terms, namely u T k Γ k u k , (cid:107) W k (cid:107) 2 2 − (cid:107) W k (cid:107) 2 F and σ 2 k /σ 2 k
+1 .
Bound
u Tk
[Equation] [Equation]
where we used Eq.
(38) in the third inequality.
Bound (cid:107) W k (cid:107) 22 − (cid:107) W k (cid:107) 2 F .
According to the deﬁnition of singular values, we have
[Equation]
where ∆ W k (0)
=
W k (0) W k (0) T − W Tk +1 (0) W k +1 (0) and we used that tr ( B k ) ≥ (cid:107) B k
(cid:107)in the second inequality.
On the other hand, by taking trace of both sides of Eq.
(36), we have
[Equation] [Equation]
Combined with Eq.
(46), we have
[Equation]
) where a follows from Eq.
(37); b follows from summing a from k to L − 1 and that (cid:107) W L (cid:107) 2 = (cid:107) W L (cid:107) F ; c is because Eq.
(38) and that
[Equation]
is ﬁnite.
Bound
σ 2 k /σ
k +
Eq.
(46) gives us that
[Equation]
where v Tk +1 ∆ W k (0) v k +1 + 2 ˜ L ( W (0)) is ﬁnite.
Putting the bounds obtained from the above 1, 2 and 3 back to Eq.
(44) will give us (cid:104)
[Equation]
Considering that σ k → ∞ and the numerator is ﬁnite, we thus conclude that the RHS of Eq.
(51) will converge to 1 .
Therefore, we have that
[Equation]
as t → ∞ .
Furthermore, since (cid:107) W k +1 (cid:107) 2 F − (cid:107) W k +1 (cid:107) 22 is ﬁnite (see Eq.
(48)) and σ k → ∞ (i.e., (cid:107) W k (cid:107) 22 / (cid:107) W k (cid:107) 2 F → 1 ), we have
[Equation]
because other singular values are negligible.
We now have the alignment phenomenon.
Furthermore, we have
[Equation] [Equation]
We now move forward to prove the alignment phenomenon for the ﬁrst layer.
We assume that the support vectors span the data space R d and denote the orthogonal complement of span (¯ u ) by ¯ u ⊥ .
Let P ⊥ denote the projection onto ¯ u ⊥ .
Under this assumption, Ji & Telgarsky (2019) showed the following lemma
Lemma 7 (Lemma 3 in Ji & Telgarsky (2019)) .
Let S denote the set of indices of support vectors , then with probability 1,
[Equation]
if the data is sampled from absolutely continuous distribution.
To begin with
[Equation]
we ﬁrst introduce the following lemma.
Lemma 8.
For the logistic loss function (cid:96) =
[Equation] [Equation]
and
[Equation]
ϕ , then
[Equation]
Proof.
We ﬁrst decompose Eq.
(55) as two parts:
[Equation]
where
[Equation]
Since ˜ (cid:96) (cid:48) < 0 , we have (cid:42)
P
[Equation] [Equation]
n
[Equation]
(cid:29)
[Equation]
n n (cid:88)
[Equation]
=
[Equation]
(cid:68) P
[Equation]
x
[Equation]
˜ (cid:96) (cid:48) ( W
[Equation]
(cid:69) .
Let x j y j ∈ arg max i ∈ S (cid:104)−P ⊥ W Π , x i y i (cid:105) which means that (cid:104)−P
[Equation]
W Π , x j y j (cid:105) ≥ κ (cid:107)P ⊥ W Π (cid:107) .
Then we have
[Equation]
=n
[Equation]
=n
[Equation]
≥n
[Equation] [Equation]
where C k = { k : (cid:104)−P ⊥ W Π , P ⊥ x k y k (cid:105) ≤ 0 } .
Considering that x j y j is a support vector
[Equation]
the ﬁrst term of (58) can be bounded as follows:
[Equation] [Equation]
where we use that (cid:15) (cid:107) W Π (cid:107) , (cid:104) W Π , γ ¯ u (cid:105) , (cid:104)P ⊥ W Π , P ⊥ x j y j (cid:105) > 0 .
Each term of the second part of (58) can be bounded by
[Equation] [Equation]
where the second inequality follows from
[Equation]
while the last inequality comes from f ( x ) = − xe − x ≥ − 1 /e for x ≥ 0 .
Finally, we have
[Equation]
Therefore,
[Equation]
P ⊥
[Equation]
if
[Equation]
We are now ready to prove the alignment phenomenon for the ﬁrst layer.
Let P ⊥ W 1 denote the projection of rows of W 1 onto ¯ u ⊥ .
We start from exploring the asymptotic behavior of (cid:107)P ⊥ W
[Equation]
F
[Equation]
W
[Equation]
F when t → ∞ .
The update of P ⊥ W 1 ( t + 1) can be written explicitly as
[Equation]
where the last inequality follows from Eq.
We deﬁne a large enough step t 0 as follows: for any t ≥ t 0 , we have
[Equation]
where ”large enough t 0 ” means that
[Equation]
65) as (cid:107) W 1 (cid:107) F → ∞ .
Suppose that there exists a t 1 ≥ t 0 such that
[Equation] [Equation]
which is to say (recalling Lemma 8)
[Equation]
for t 0 ≤ t 1 ≤ t ≤ t 2 and t 2 = ∞ if we never have (cid:107)P ⊥ W Π ( t ) (cid:107) F < ϕ after t 1 .
If there does not exist such a t 1 , then we directly conclude that (cid:107)P
[Equation] [Equation]
F
[Equation]
F → 0 since (cid:107) W 1 (cid:107) F → ∞ .
On the other hand, for any t 0 ≤ t 1 ≤ t ≤ t
[Equation]
where the last inequality follows from Eq.
(31) and Φ ( ϕ, t 1 ) is deﬁned by
[Equation]
for any t (cid:48) ≥ t 1 because the loss and µ never increase.
Therefore we can conclude that
[Equation]
for any t ≥ t 1 .
As a result,
[Equation]
B P ROOFS OF S ECTION 3.
We present some useful properties and proofs of Lemma 3 and Lemma 4 in Section B.1.
Section
B.2 focuses on the divergences of weight norms.
Section B.3 is about the convergence to KKT points(Theorem 5).
Additional notations We use W ij ; k to represent the i -th row j -th column entry of the k -th layer weight martrix.
B. U SEFUL PROPERTIES AND PROOFS OF L EMMA 3 AND L EMMA
For any m × n matrix A , we denote v ( A ) , an mn -dimensional vector, as the vectorized version of it:
[Equation]
(72) then the trace operator can be represented by
[Equation]
We ﬁrst introduce the Euler’s theorem on homogeneous functions.
Lemma 9 (Euler’s theorem on homogeneous functions) .
If f ( W 1 , . . .
, W L ) is a positive multi- c khomogeneous function
[Equation]
for positive constants ρ k ’s and c k ≥ 1 , then we have:
[Equation]
Proof.
Taking derivatives of ρ k on both sides of Eq.
(74) for any given k ∈ { 1 , . . .
, L } , we have
[Equation]
Since ρ k is arbitrary, we let ρ L = · · · = ρ k = · · · = ρ 1 = 1 , then the above equation becomes
[Equation]
for any k ∈ { 1 , . . .
, L } .
Furthermore, we have
[Equation]
where K = (cid:80)
Lk
=1 c k .
For the deep neural networks deﬁned in Eq.
(2) with homogeneous property, we can then apply the above lemma to them.
We examine whether Assumption 2 can still be applied to adversarial margin under Assumption 1.
Lemma 10 (Assumption 2 for adversarial training) .
For any ﬁxed x ,
• if yf ( x ; W ) is locally Lipschitz, then so does yf ( x + δ ( W ); W ) with perturbation δ ( W ) .
Proof.
For ﬁxed x , suppose yf ( W ) is locally Lipschitz on Y , then for each W ⊂ Y there is a Z W containing W such that yf ( W ) is Lipschitz on Z W :
[Equation]
For the adversarial perturbation, by deﬁnition δ ( W ) and δ ( V ) are solutions of the inner maximization for the adversarial training, in other words,
[Equation]
because then we will have (cid:96) ( x + δ ( W ); W ) > (cid:96) ( x + δ ( V ); W ) since (cid:96) is non-increasing under Assumption 1.
As a result,
[Equation]
for yf ( x + δ ( W ); W ) > yf ( x + δ ( V ); V ) .
The result for yf ( x + δ ( W ); W ) < yf ( x + δ ( V ); V ) is similar
if ∀ i : y i f ( x i ; W ) have locally Lipschitz gradients, then so do y i f
( x i + δ i ( W ); W ) .
Proof.
For ﬁxed x , suppose y ∇ f ( W ) is locally Lipschitz on Y , then for each W ⊂ Y there is a Z W containing W such that y ∇ f ( W ) , δ ( W ) are Lipschitz on Z W , for W, V ⊂ Z W :
[Equation] [Equation]
Then
[Equation]
B.1. P ROOF OF L EMMA
Proof.
The proof can be done for multi- c k -homogeneous networks as deﬁned in Lemma 9.
We ﬁrst note that
[Equation]
by taking derivatives with respect to x on both sides of Eq.
Therefore ∇ x f ( x ; W 1 , . . .
, W L ) is also positive homogeneous.
Under Assumption 1, we prove Lemma 3 in the following cases: 1.
(cid:96) 2 -FGM perturbation.
This perturbations is taken as
[Equation]
because ˜ (cid:96) (cid:48) < 0 .
Invoking Eq.
(86), we know that
[Equation]
2. FGSM perturbation.
[Equation]
where sgn() denotes the sign function.
Then we have
[Equation]
since ρ k > 0 for any k thus will not affect the sign operation.
(cid:96) and (cid:96)
∞ -PGD
perturbation.
We only prove the case for (cid:96)
PGD and the case for (cid:96) ∞ -PGD is similar.
[Equation]
where P B (0 ,(cid:15) ) denotes the projection operator, B (0 , (cid:15) ) is the perturbation set, j is the step indices of PGD and ξ is the step size.
We prove by induction.
For j = 0 , we have
[Equation]
If we have δ j PGD ( ρ 1 W 1 , . .
, ρ L W L ) =
δ j PGD ( W 1 , . . .
, W L )
[Equation]
then for j +
[Equation]
we have
[Equation]
Therefore, these four adversarial perturbations are all scale invariant adversarial perturbations by our deﬁnition.
B.1. P ROOF OF L EMMA
Proof.
This can be easily proved by noting that, for any multi- c k -homogeneous functions,
[Equation]
because δ ( W ) is a scale invariant adversarial perturbations.
Then f ( x + δ ( W ); W ) is still a homogeneous function and Lemma 9 can be applied to f ( x + δ ( W ); W 1 , . .
, W L ) and we have
[Equation]
Taking c 1 = · · · = c L gives us Lemma 4.
B. D IVERGENCES OF F ROBENIUS NORMS OF WEIGHT MATRICES B.2. D IVERGENCES OF ALL LAYERS
Lemma 11.
The Frobenius norms of all layers grow at approximately the same rate along the trajectory of gradient ﬂow for adversarial training with scale invariant adversarial perturbations for multi- c k -homogeneous DNNs
[Equation]
Remark.
Note that a similar conclusion as that of Lemma 11 exists for gradient ﬂow of the standard training for multi-1-homogeneous networks Du et al.
We generalize this property to the adversarial training of multi- c k -homogeneous neural networks.
When c 1 = · · · = c L , we have that all layer grow at the same rate.
Proof.
For any W k ,
[Equation]
) where we use dWdt = − (cid:16) ∂
˜ L ∂W k (cid:17) T and Lemma 4.
One can then immediately notice that the above equation does not depend on any speciﬁc k
[Equation]
thus we have
[Equation]
B.2. P ROOF OF T HEOREM
We prove this theorem by exploring the time evolution of ρ .
Proof.
We start from a multi- c k -homogeneous function then take c 1 = · · · = c L .
Recalling f
( x ; W ) =
ρf ( x ; (cid:99) W ) where (cid:99) W k
=
W
k
(cid:107) W k (cid:107) F and ρ =
L (cid:89)
[Equation] [Equation]
(cid:89) k
[Equation] [Equation]
(cid:107)
[Equation]
(99) the adversarially trained predictor with scale invariant adversarial perturbation is still homogeneous (see Eq.
[Equation]
[Equation]
For ρ k = (cid:107) W k (cid:107) F , we have
[Equation]
by invoking Eq.
The dynamical evolution of ρ will then be dρ
[Equation]
[Equation]
Let t 0 denote the time such that all worst case adversarial examples are correctly classiﬁed.
We study the trends for ρ after t 0 .
On one hand, the empirical adversarial loss does not increase:
[Equation]
On the other hand, recalling our deﬁnition of normalized adversarial margin, let
[Equation]
arg
[Equation]
and note the deﬁnition of t 0 , we have
[Equation]
because ˜ L does not increase and ˜ L ( t ) ≤ ˜ L ( t 0 ) < 1 .
This also implies that ˜ γ m , thus ˜ γ i for all i ∈ { 1 , . . . , n } , can not be arbitrarily close to 0.
Otherwise one would conclude that ˜ L ( t ) may be arbitrarily close to 1 which is a contradiction.
Therefore we can immediately conclude that dρ/dt > 0 in Eq.
(102), which implies that ρ may diverge as t → ∞ .
To have a clearer view about the convergence property of ρ , we study the following relation regarding the time evolution of e ρy i f
( x i + δ i ( (cid:99) W ))
after t 0 :
[Equation]
The term (a) can be computed by invoking Eq.
[Equation]
where the ﬁrst inequality follows from x + 1 /x ≥ 2 for x ≥
and we denote the minimum of (cid:80) i ˆ γ
i for t ≥ t 0 as ζ > 0 .
The existence of such ζ has been discussed below Eq.
The term (b) needs more analysis.
For any ﬁxed adversarial example x i + δ
( (cid:99) W ) , we have
[Equation]
where d (cid:99) W k /dt can be computed as follows
[Equation]
On the other hand, we note that
[Equation]
W k (cid:33) T (110) because ∂f ( W ) ∂W k = ρρ k ∂f ( (cid:99) W )
∂ (cid:99) W k .
Combing Eq.
(109) and Eq.
(110), we have d (cid:99) W k
[Equation]
(cid:99)
[Equation]
; (cid:99)
[Equation] [Equation]
(cid:99)
[Equation]
(cid:33)
[Equation]
(cid:99)
[Equation]
x
[Equation]
( (cid:99) W ); (cid:99) W )   .
Therefore
[Equation]
(cid:33) , (112) where
[Equation]
can be seen as a projector operator.
Putting Eq.
(112) into Eq.
(108) will give us the expression
[Equation]
Now the term (b) in Eq.
(106) is, by putting Eq.
(114) back into it,
[Equation]
k (cid:19) (cid:104) e ρ (ˆ γ i − ˆ γ j ) +
e − ρ (ˆ γ i − ˆ γ j ) (cid:105) (cid:124) (cid:123)(cid:122) (cid:125) (c) .
Rearranging e ρ (ˆ γ i − ˆ γ j ) +
e − ρ (ˆ γ i − ˆ γ j ) as follows e ρ (
[Equation]
γ i − ˆ γ j ) +
[Equation]
γ
[Equation]
γ i ) will allow us to rewrite (c) as (c)
= (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:88) i e ρ ˆ γ
i
A k v (cid:18) ∂ ˆ γ i ∂ (cid:99) W k
(cid:19)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)  + (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:88) i e − ρ ˆ γ i
A k v (cid:18) ∂ ˆ γ i ∂ (cid:99) W k
(cid:19)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)  − (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:88)
i (cid:0) e ρ ˆ γ i − e − ρ ˆ γ i (cid:1)
A k v (cid:18) ∂ ˆ γ i ∂ (cid:99) W k
(cid:19)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)  = (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:88)
[Equation]
(cid:19)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
≥ 0 .
Combing Eq.
(107) and Eq.
(116) with Eq.
(106), we will have
[Equation]
According to Eq.
(101), we know that ∀ k :
dρ k /dt > 0 after t 0 , therefore ∀ k : ρ 2 /ρ 2 k is lower bounded by its value at t 0 for t ∈ [ t 0 , ∞ ) because d dt (cid:18) ρρ 2 k
[Equation] [Equation]
where ˙
ρ k denotes
dρ k /dt > 0 for any k
due to Eq.
Integrating both sides of Eq.
(117) from t 0 to t gives us
[Equation]
where the last inequality follows from Eq.
Let t → ∞ in the above equation, we will have
[Equation]
since y i f ( x i + δ i ( (cid:99) W ); (cid:99) W ) is upper bounded, which can be easily deduced considering that all weights have norm 1, and ζ is lower bounded as discussed before.
Taking c 1 = · · · = c L gives us the conclusion.
Note that ρ = ρ c 1 1 · · · ρ c L L and ρ k for any k grows at the same rate for multi- c -homogeneous functions (Lemma 11), we immediately have
Corollary 1.
∀ k ∈ { 1 , . . .
, L } : ρ k → ∞ as t → ∞ .
B. C ONVERGENCE TO KKT POINT B.3. P ROOF OF T HEOREM
Proof.
Most calculations needed for the proof of this theorem have been done in Section B.2.2.
According to Eq.
(114), we have (cid:88)
[Equation]
[Equation]
B.3. P ROOF OF T HEOREM
The KKT conditions for the optimization problem Eq.
(19) are
[Equation]
Following Dutta et al.
; Lyu & Li (2020), we deﬁne the approximate KKT point in our settings
Deﬁnition 3 (Adapted from Deﬁnition C.4 in Lyu & Li (2020)) .
The approximate KKT points of the optimization problem (19) are those feasible points which satisfy the following condtions:
[Equation]
∀ i ∈ { 1 , . .
. , n } : λ i (˜ γ i − 1) ≤ ξ ,
where χ, ξ > 0 and λ i ≥ 0 .
These points are said to be ( χ, ξ ) -approximate KKT points.
We now present the proof of Theorem 5.
Proof.
Let
[Equation]
where K = (cid:80)
Lk =1 c = cL .
Since
[Equation] [Equation]
according to the deﬁnition of the multi- c -homogeneous functions and
[Equation]
we have
[Equation]
Speciﬁcally, in the following, we will show that V is a ( χ, ξ ) -approximate KKT points of the optimization problem (19) with χ, ξ → 0 along the training trajectory.
Because the homogeneous property of the network trained with scale invariant adversarial perturbations, it satisﬁes the MFCQ condition by simply noting that
[Equation]
This makes a point being a KKT point a necessary condition for it to be the optimal solution.
We now show that the limit point of V along the adversarial training trajectory is an ( χ, ξ ) -approximate
KKT point with χ, ξ → 0 starting with the ﬁrst condition.
Let
[Equation]
then we have
[Equation]
V )
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) W
[Equation]
(cid:88)
i
[Equation]
i
[Equation]
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]

[Equation]
(cid:68)
[Equation]
L
[Equation]
(cid:69) (cid:107)
[Equation]
(cid:107) (cid:13)(cid:13)(cid:13) ∂
˜ L ∂W (cid:13)(cid:13)(cid:13) 
[Equation]
(cid:48)
[Equation]
(cid:48)(cid:48)
[Equation] [Equation]
(cid:68)
[Equation]
, ∂ ˜ L ∂W (cid:69) (cid:107)
[Equation]
(cid:107) (cid:13)(cid:13)(cid:13)
[Equation]
∂W (cid:13)(cid:13)(cid:13)
[Equation]
[Equation]
) (131) where a follows from (128); k (cid:48) =
arg max k ρ k and k (cid:48)(cid:48) =
arg min k ρ k .
This is the ﬁrst condition for the limit point of V being an approximate KKT point.
We can further show that χ → 0 as t →by
noting the following alignment phenomenon which was originally observed by Ji & Telgarsky (2020) and intended for standard training with ﬁxed training examples:
Lemma 12 (Adapted from Theorem 4.1 in Ji & Telgarsky (2020)) .
Under Assumption 2 and
Assumption 4 for exponential loss and homogeneous deep neural networks, along the trajectory of adversarial training with scale invariant adversarial perturbations, we have
[Equation]
The extension of Theorem 4.1 in Ji & Telgarsky (2020), which was intended for ﬁxed training examples, to our settings is because, by our construction for adversarial training with scale invariant adversarial perturbations, the adversarial training margin are locally Lipschitz with locally Lipschitz gradients and the prediction function f ( x + δ ( W ); W ) is positively (multi)homogeneous with respect to W (see Lemma 10 and Eq.
Invoking Lemma 12 and considering that ˆ γ m can not be arbitrarily close to 0 8 and ∀ k : ρ k → ∞ at the same rate according to Lemma 11 thus ρ k
[Equation]
ρ k
[Equation]
can not be inﬁnite, we have,
[Equation]
fact some
Theorem
On the other hand, the second condition for the limit point of V being an approximate KKT point is
[Equation]
where a follows from (cid:68) W,
[Equation]
∂ ˜ L ∂W (cid:13)(cid:13)(cid:13) ; b is due to Lemma 4 and ∀ i : ˜ γ i ≥ ˜
γ m ; c uses ∀ i : ˜ γ i ≥ ˜ γ m again; d is due to e − x x is upper bounded by its value at x = 1 for x ≥ 0 .
Since ∀ k : lim t →∞ ρ k ( t ) → ∞ at the same rate, we conclude that lim t →∞ ρ 2 k (cid:48) ρ 2 k (cid:48)(cid:48) can not be inﬁnite thus lim
t →∞ ξ ( t ) → 0 because ˆ γ m can not be arbitrarily close to 0 and lim t →∞ ρ → ∞ .
Therefore, the limit point of V is an ( χ, ξ ) -approximate
KKT point of the mini-norm problem along the trajectory of adversarial training with scale invariant adversarial perturbations where lim t →∞ χ ( t ) , ξ ( t ) → 0 .
Restating the theorem in Dutta et al.
(2013) regarding the relation between ( χ, ξ ) -approximate KKT point and KKT point in our setting:
Theorem 6 (Theorem 3.6 in Dutta et al.
and Theorem C.4 in Lyu & Li (2020)) .
Let { x t : t ∈ N } be a sequence of feasible point of the optimization problem 19.
x t is an ( χ t , ξ t ) -approximate
KKT poiint for all t with two sequences { χ t > 0 : t ∈ N } and { ξ t > 0 : t ∈ N} and χ t , ξ t → 0 .
If x t → x as t → ∞ and MFCQ holds at x , then x is a KKT point of this optimization problem.
We then immediately conclude that the limit point of { W ( t ) / (cid:107) W (cid:107) : t ≥ 0 } of gradient ﬂow for the adversarial training objective Eq.
(6) is along the direction of a KKT point of the optimization problem (19).
B. G ENERALIZATION TO NON - SMOOTH CASE
It is not hard to generalize the current analysis to non-smooth case, which will include the deep ReLU network, because, in our main steps, there are counterparts of our conclusions for non-smooth case for that sub-differential (Clarke, 1975) is a generalization of gradient.
The non-smooth analysis can be done by ﬁrst replacing the gradient ﬂow equation Eq.
(16) with its generalization,
[Equation]
where ∂ ˜ L ( W ( t )) is the sub-differential.
Then one can follow similar procedures as our approach to make the generalization.
Proof.
For simplicity, we only consider multi-1-homogeneous networks here, i.e., c 1 = · · · = c L = 1 , which include the deep ReLU neural networks.
Lemma 9 can be generalized to non-smooth case according to Lemma C.1 in Ji & Telgarsky (2020) or Theorem B.2 in Lyu & Li (2020):
Lemma 13 (Lemma C.1 in Ji & Telgarsky (2020)) .
Suppose f : R n → R is locally Lipschitz and
L -positively homogeneous, then for any W ∈ R n and any W ∗ ∈ ∂f ( W ) ,
[Equation]
The starting point of the proof for Lemma 3 is the homogeneity of f ( x ; W ) and can be easily promoted to handle the non-smooth case.
With generalizations of Lemma 3 and Euler’s theorem on homogeneous functions for non-smooth case, the proof of Lemma 4 can also be generalized to non-smooth case accordingly because it is based on the Euler’s theorem on homogeneous functions and Lemma 3 for adversarial training.
Speciﬁcally, it will become
[Equation]
The proof of Theorem 3 adopts Lemma 4 and can be generalized to non-smooth case by combining the chain rule (Clarke, 1983) for gradient ﬂow to ensure Eq.
[Equation]
with similar approach as in Lemma B.9 in Ji & Telgarsky (2020) or Lemma 5.2 in Davis et al.
Then generalizing the rest of the proof of Theorem 3 is straightforward.
Finally, the generalization of Theorem 5 to non-smooth case can also be done because it adopts
Theorem 3, Lemma 4, the chain rule, and the gradient alignments Lemma 12 which holds for non-smooth case.
B. D EEP LINEAR NETWORKS ADVERSARIALLY TRAINED WITH (cid:96) ∞ PERTURBATIONS
Following the settings of Section 3.1, the (cid:96) ∞ -perturbation for deep linear networks f ( x ; W ) =
W L · · · W 1 x can be given exactly as
[Equation]
[Equation]
According to Theorem 5, the adversarial training solution is formulated from solving the following optimization problem:
[Equation]
[Equation]
It can be seen that this optimization problem, a mixed-norm optimization problem, will have different solutions when compared to the margin-maximization problem of the original data, which will lead to a different decision boundary distinguishing adversarial training methods from the standard ones.
Besides, the constraint on (cid:15) (Eq.
(8) for (cid:96) 2 perturbation) should also be changed accordingly such that the adversarial training examples are still linear separable:
[Equation] [Equation]
where γ m, 1 is the max- (cid:96) 1 -norm margin:
[Equation]
[Equation]
C S UPPLEMENTED E XPERIMENTS C. A DVERSARIAL TRAINING WITH DIFFERENT LOSS FUNCTIONS , ARCHITECTURES AND VARYING PERTURBATION SIZES
We present the results for adversarial training with different loss functions, architectures and varying perturbation sizes in this section to further verify our theorems and assumptions.
We use two different models: one is a 3-layer neural network with the same architecture as that in the Section 4; the other is a CNN with architecture Input-Conv-ReLU-Pooling-Conv-ReLU-Pooling-FC-FC, where Conv stands for convolution layer, FC stands for fully connected layer and Pooling is max-pooling layer.
[Figure]
a
[Figure]
v e r s a r i a l n o r m a li
[Figure]
d m a r g i n
[Figure]
r i
[Figure]
l
[Figure]
a i n i
[Figure]
cc
[Figure]
For the 3-layer network, we conduct adversarial training using
∞ -PGD perturbations with perturbation sizes (cid:15) = 8 / 255 , 12 / 255 and 24 / 255 to perform binary classiﬁcation where we choose all examples with labels ”3” and ”8” from MNIST to be our dataset.
The loss function is logistic loss.
Fig.
2(a) shows that for both FGSM and (cid:96) ∞ -PGD perturbations with varying perturbation sizes, the adversarial normalized margins during training keep increasing after some step.
Fig.
2(b) reveals that the adversarial training accuracy can achieve 100% (after ∼ 200 epochs), which supports
Assumption 4.
For the multi-classiﬁcation task, let f ( x ; W )
[ j ] denote the j -th output.
Then the margin for an adversarial example x
i + δ i corresponding to the original example x i is deﬁned by
[Equation]
[Figure]
a d v e r s a r i a l n o r m a li z e d m a r g i n
[Figure]
a
[Figure]
n
[Figure]
n
[Figure]
and the margin for the adversarial data is deﬁned to be
[Equation]
Then for the CNN, we conduct adversarial training using both FGSM and (cid:96) ∞ -PGD perturbations with perturbation sizes (cid:15)
= 16 / 255 and 32 / 255 to perform multi-classiﬁcation on all training examples of
MNIST to verify our claims and assumptions when the perturbation sizes are large.
The loss function is cross-entropy loss.
It can be seen from Fig. 3(a) that the adversarial normalized margins keep increasing after about 250 epochs, i.e., after the model ﬁts all adversarial training examples, for all adversarial perturbations.
Fig.
3(b) clearly shows that the adversarial training accuracy can reach 100% for all perturbation types and sizes, even the large size (cid:15) = 32 / 255 .
These experiments show that adversarial normalized margins are different whenever perturbation types or sizes are different.
All models adversarially trained with all perturbations and loss functions can achieve 100% adversarial training accuracy, which veriﬁes Assumption 4.
Moreover, the trends that adversarial normalized margins all keep increasing are clear, even long after the separation of adversarial training examples.
These results well support the claims of Theorem 5.
C. E XPERIMENTS ABOUT (cid:96) 2 -FGM PERTURBATIONS FOR BINARY CLASSIFICATION
[Figure]
a l t r a i n i n g a cc u r a c y
[Figure]
In the MNIST dataset, we adversarially trained a 3-layer neural network with the same architecture as that in Section 4 using SGD with constant learning rate and batch-size 64.
To have a clear view about the difference between the implicit bias for standard training and adversarial training, the adversarial perturbation used for training is (cid:96) 2 -FGM perturbation with (cid:15) = 8 since even the standard model can ﬁt (cid:96) 2 -FGM adversarial examples easily in the MNIST dataset when the perturbation size (cid:15) is not large.
As a comparison, we also standardly trained a model with the same architecture to evaluate the normalized margin for adversarial examples by attacking it with (cid:96) 2 -FGM during its training process.
As showed in Fig.4(b), the adversarilly training accuracy reaches 100% very quickly, which supports
Assumption 4.
It can be seen in Fig.4(a) that the adversarial normalized margin keeps increasing during the adversarial training process while the standardly trained model maintains a lower adversarial normalized margin.
C.2. E XPERIMENTS ABOUT EXPONENTIAL LOSS
For completeness, we also conduct experiment about the binary classiﬁcation where the model is adversarially trained with exponential loss (Fig.5) for (cid:96) 2 -FGM perturbations.
We select the examples with label ”2” and ”8” from MNIST and adversarially trained a 2-layer neural network using SGD with constant learning rate 10 − 5 .
The hidden layer is of size 10000 and the activation function is
ReLU.
The normalized margin for adversarial data is deﬁned exactly as in Section 3.2.
The adversarial
[Figure] [Figure]
perturbations during the adversarial training are given by (cid:96) 2 -FGM with (cid:15) = 3 .
As showed in Fig.5, although extremely slowly, the normalized margin for adversarial data gradually increases during the adversarial training.
C. D IVERGENCE OF WEIGHT NORMS FOR DEEP LINEAR NETWORKS DURING ADVERSARIAL TRAINING
We conduct adversarial training of deep linear networks with (cid:96) 2 perturbation on a linearly separable dataset.
The loss function is logistic loss.
Speciﬁcally, we use deep linear networks with layers 2,and 4, respectively.
Since the divergence is slow ( ∼ ln t ), it is hard to observe that (cid:107) W k (cid:107) F → ∞ .
However, as showed in Fig. 6(a), Fig. 6(b) and Fig. 6(c), the trends that (cid:107) W k (cid:107) F for all layers of deep linear networks with different layers keep increasing are clear.
Furthermore, we can see that the ratios (cid:107) W k (cid:107) 2 / (cid:107) W k (cid:107) F for all layers of deep linear networks with different layers also keep increasing, as showed by Fig. 7(a), Fig. 7(b) and Fig. 7(c).
[Figure]