MMA T RAINING : D IRECT I NPUT S PACE M ARGIN M AXIMIZATION THROUGH A DVERSARIAL T RAINING
A BSTRACT
We study adversarial robustness of neural networks from a margin maximization perspective, where margins are deﬁned as the distances from inputs to a classiﬁer’s decision boundary.
Our study shows that maximizing margins can be achieved by minimizing the adversarial loss on the decision boundary at the “shortest successful perturbation”, demonstrating a close connection between adversarial losses and the margins.
We propose Max-Margin Adversarial (MMA) training to directly maximize the margins to achieve adversarial robustness.
Instead of adversarial training with a ﬁxed (cid:15) , MMA offers an improvement by enabling adaptive selection of the “correct” (cid:15) as the margin individually for each data point.
In addition, we rigorously analyze adversarial training with the perspective of margin maximization, and provide an alternative interpretation for adversarial training, maximizing either a lower or an upper bound of the margins.
Our experiments empirically conﬁrm our theory and demonstrate MMA training’s efﬁcacy on the
MNIST and CIFAR10 datasets w.r.t.
(cid:96) ∞ and (cid:96) 2 robustness.
Code and models are available at https://github.com/BorealisAI/mma_training .
I NTRODUCTION
Despite their impressive performance on various learning tasks neural networks have been shown to be vulnerable to adversarial perturbations (Szegedy et al., 2013; Biggio et al., 2013).
An artiﬁcially constructed imperceptible perturbation can cause a signiﬁcant drop in the prediction accuracy of an otherwise accurate network.
The level of distortion is measured by the magnitude of the perturbations (e.g. in (cid:96) ∞ or (cid:96) 2 norms), i.e. the distance from the original input to the perturbed input.
Figure 1 shows an example, where the classiﬁer changes its prediction from panda to bucket when the input is perturbed from the blue sample point to the red one.
Figure 1 also shows the natural connection between adversarial robustness and the margins of the data points, where the margin is deﬁned as the distance from a data point to the classiﬁer’s decision boundary.
Intuitively, the margin of a data point is the minimum distance that x has to be perturbed to change the classiﬁer’s prediction.
Thus, the larger the margin is, the farther the distance from the input to the decision boundary is, the more robust the classiﬁer is w.r.t.
this input.
Although naturally connected to adversarial robustness, “directly” maximizing margins has not yet been thoroughly studied in the adversarial robustness literature.
Instead, the method of minimax adversarial training (Huang et al., 2015; Madry et al., 2017) is arguably the most common defense to adversarial perturbations due to its effectiveness and simplicity.
Adversarial training attempts to minimize the maximum loss within a ﬁxed sized neighborhood about the training data using projected gradient descent (PGD).
Despite advancements made in recent years (Hendrycks et al., 2019; Zhang et al., 2019a; Shafahi et al., 2019; Zhang et al., 2019b; Stanforth et al., 2019; Carmon et al., 2019), adversarial training still suffers from a fundamental problem, the perturbation length (cid:15) has to be set and is ﬁxed throughout the training process.
In general, the setting of (cid:15) is arbitrary, based on assumptions on whether perturbations within the deﬁned ball are “imperceptible” or not.
Recent work (Guo et al., 2018; Sharma et al., 2019) has demonstrated that these assumptions do not consistently hold true, commonly used (cid:15) settings assumed to only allow imperceptible perturbations in fact do not.
If (cid:15) is set too small, the resulting models lack robustness, if too large, the resulting models lack in accuracy.
Moreover, individual data points may have different intrinsic robustness, the variation in ambiguity in collected data is highly diverse, and ﬁxing one (cid:15) for all data points across the whole training procedure is likely suboptimal.
Instead of improving adversarial training with a ﬁxed perturbation magnitude, we revisit adversarial robustness from the margin perspective, and propose Max-Margin Adversarial (MMA) training for “direct” input margin maximization.
By directly maximizing margins calculated for each data point, MMA training allows for optimizing the “ current robustness” of the data, the “ correct ” (cid:15) at this point in training for each sample individually , instead of robustness w.r.t.
a predeﬁned magnitude.
While it is intuitive that one can achieve the greatest possible robustness by maximizing the margin of a classiﬁer, this maximization has technical difﬁculties.
In Section 2, we overcome these difﬁculties and show that margin maximization can be achieved by minimizing a classiﬁcation loss w.r.t.
model parameters, at the “shortest successful perturbation”.
This makes gradient descent viable for margin maximization, despite the fact that model parameters are entangled in the constraints.
We further analyze adversarial training (Huang et al., 2015; Madry et al., 2017) from the perspective of margin maximization in Section 3.
We show that, for each training example, adversarial training with ﬁxed perturbation length (cid:15) is maximizing a lower (or upper) bound of the margin, if (cid:15) is smaller (or larger) than the margin of that training point.
As such, MMA training improves adversarial training, in the sense that it selects the “correct” (cid:15) , the margin value for each example.
Finally in Section 4, we test and compare MMA training with adversarial training on MNIST and
CIFAR10 w.r.t.
(cid:96) ∞ and (cid:96) 2 robustness.
Our method achieves higher robustness accuracies on average under a variety of perturbation magnitudes, which echoes its goal of maximizing the average margin.
Moreover, MMA training automatically balances accuracy vs robustness while being insensitive to its hyperparameter setting, which contrasts sharply with the sensitivity of standard adversarial training to its ﬁxed perturbation magnitude.
MMA trained models not only match the performance of the best adversarially trained models with carefully chosen training (cid:15) under different scenarios, it also matches the performance of ensembles of adversarially trained models.
In this paper, we focus our theoretical efforts on the formulation for directly maximizing the input space margin, and understanding the standard adversarial training method from a margin maximization perspective.
We focus our empirical efforts on thoroughly examining our MMA training algorithm, comparing with adversarial training with a ﬁxed perturbation magnitude.
1. R ELATED W ORKS
Although not often explicitly stated, many defense methods are related to increasing the margin.
One class uses regularization to constrain the model’s Lipschitz constant (Cisse et al., 2017; Ross & Doshi-Velez, 2017; Hein & Andriushchenko, 2017; Sokolic et al., 2017; Tsuzuku et al., 2018), thus samples with small loss would have large margin since the loss cannot increase too fast.
If the Lipschitz constant is merely regularized at the data points, it is often too local and not accurate in a neighborhood.
When globally enforced, the Lipschitz constraint on the model is often too strong that it harms accuracy.
So far, such methods have not achieved strong robustness.
There are also efforts using ﬁrst-order approximation to estimate and maximize input space margin (Matyasko & Chau, 2017; Elsayed et al., 2018; Yan et al., 2019).
Similar to local Lipschitz regularization, the reliance on local information often does not provide accurate margin estimation and efﬁcient maximization.
Such approaches have also not achieved strong robustness so far. Croce et al.
(2018) aim to enlarge the linear region around a input example, such that the nearest point, to the input and on the decision boundary, is inside the linear region.
Here, the margin can be calculated analytically and hence maximized.
However, the analysis only works on ReLU networks, and the implementation so far only works on improving robustness under small perturbations.
We defer some detailed discussions on related works to Appendix B, including a comparison between MMA training and SVM.
1. N OTATIONS AND D EFINITIONS
We focus on K -class classiﬁcation problems.
Denote S =
{ x i , y i } as the training set of inputlabel data pairs sampled from data distribution D .
We consider the classiﬁer as a score function f θ ( x ) = (cid:0)
f 1 θ ( x ) , . . .
, f Kθ ( x ) (cid:1) , parametrized by θ , which assigns score f iθ ( x ) to the i -th class.
The predicted label of x is then decided by ˆ y = arg max i f iθ ( x ) .
Let L 01 θ ( x, y )
=
I (ˆ y (cid:54) =
y ) be the 0-1 loss indicating classiﬁcation error, where I ( · ) is the indicator function.
For an input ( x, y ) , we deﬁne its margin w.r.t.
the classiﬁer f θ ( · ) as:
[Equation]
where δ ∗ = arg min L 01 θ ( x + δ,y )
=1 (cid:107) δ (cid:107) is the “shortest successful perturbation”.
We give an equivalent deﬁnition of margin with the “logit margin loss” L LM θ ( x, y ) =
max j (cid:54) =
y f jθ ( x ) − f yθ ( x ) .
The level set { x : L LM θ ( x, y ) = 0 } corresponds to the decision boundary of class y .
Also, when L LM θ ( x, y ) < 0 , the classiﬁcation is correct, and when L LM θ ( x, y ) ≥ 0 , the classiﬁcation is wrong.
Therefore, we can deﬁne the margin in Eq.
(1) in an equivalent way by L LM θ ( · ) as:
[Equation]
where δ ∗ = arg min
L LM θ ( x + δ,y ) ≥ 0 (cid:107) δ (cid:107) is again the “shortest successful perturbation”.
For the rest of the paper, we use the term “margin” to denote d θ ( x, y ) in Eq.
For other notions of margin, we will use speciﬁc phrases, e.g. “SLM-margin” or “logit margin.”
M AX -M ARGIN A DVERSARIAL T RAINING
We propose to improve adversarial robustness by maximizing the average margin of the data distribution D , called Max-Margin Adversarial (MMA) training, by optimizing the following objective:
[Equation]
where S + θ = { i : L LM θ ( x i , y i ) < 0 } is the set of correctly classiﬁed examples, S − θ
= { i :
L LM θ ( x i , y i ) ≥ 0 } is the set of wrongly classiﬁed examples, J θ ( · ) is a regular classiﬁcation loss function, e.g. cross entropy loss, d θ ( x i , y i ) is the margin for correctly classiﬁed samples, and β is the coefﬁcient for balancing correct classiﬁcation and margin maximization.
Note that the margin d θ ( x i , y i ) is inside the hinge loss with threshold d max (a hyperparameter), which forces the learning to focus on the margins that are smaller than d max .
Intuitively, MMA training simultaneously minimizes classiﬁcation loss on wrongly classiﬁed points in S − θ and maximizes the margins of correctly classiﬁed points in d θ ( x i , y i ) until it reaches d max .
Note that we do not maximize margins on wrongly classiﬁed examples.
Minimizing the objective in Eq.
(3) turns out to be a technical challenge.
While ∇ θ J θ ( x j , y j ) can be easily computed by standard back-propagation, computing the gradient of d θ ( x i , y i ) needs some technical developments.
In the next section, we show that margin maximization can still be achieved by minimizing a classiﬁcation loss w.r.t.
model parameters, at the “shortest successful perturbation”.
For smooth functions, a stronger result exists: the gradient of the margin w.r.t.
model parameters can be analytically calculated, as a scaled gradient of the loss.
Such results make gradient descent viable for margin maximization, despite the fact that model parameters are entangled in the constraints.
2. M ARGIN M AXIMIZATION
Recall that
[Equation]
Note that the constraint of the above optimization problem depends on model parameters, thus margin maximization is a max-min nested optimization problem with a parameter-dependent constraint in its inner minimization.
Computing such gradients for a linear model is easy due to the existence of its closed-form solution, e.g. SVM, but it is not so for general functions such as neural networks.
The next theorem provides a viable way to increase d θ ( x, y ) .
Theorem 2.1.
Gradient descent on L LM θ ( x + δ ∗ , y ) w.r.t.
θ with a proper step size increases d θ ( x, y ) , where δ ∗ =
arg min
L LM θ ( x + δ,y ) ≥ 0 (cid:107) δ (cid:107) is the shortest successful perturbation given the current θ .
Theorem 2.1 summarizes the theoretical results, where we show separately later
1) how to calculate the gradient of the margin under some smoothness assumptions; 2) without smoothness, margin maximization can still be achieved by minimizing the loss at the shortest successful perturbation.
Calculating gradients of margins for smooth loss and norm: Denote L LM θ ( x + δ, y ) by L ( δ, θ ) for brevity.
It is easy to see that for a wrongly classiﬁed example ( x, y ) , δ ∗ is achieved at 0 and thus ∇ θ d θ ( x, y )
= 0 .
Therefore we focus on correctly classiﬁed examples.
Denote the Lagrangian as
L θ ( δ, λ ) = (cid:107) δ (cid:107) + λL ( δ, θ ) .
For a ﬁxed θ , denote the optimizers of L θ ( δ, λ ) by δ ∗ and λ ∗ .
The following theorem shows how to compute ∇ θ d θ ( x, y ) .
Proposition 2.1.
Let (cid:15) ( δ ) =
(cid:107) δ (cid:107) .
Given a ﬁxed θ , assume that δ ∗ is unique, (cid:15) ( δ ) and L ( δ, θ ) are
C 2 functions in a neighborhood of ( δ ∗ , θ ) , and the matrix
[Equation]
is full rank, then
[Equation]
, where C ( θ, x, y ) =
[Equation]
is a scalar.
Remark 2.1.
By Proposition 2.1, the margin’s gradient w.r.t.
to the model parameter θ is proportional to the loss’ gradient w.r.t.
θ at δ ∗ , the shortest successful perturbation.
Therefore to perform gradient ascent on the margin, we just need to ﬁnd δ ∗ and perform gradient descent on the loss.
Margin maximization for non-smooth loss and norm: Proposition 2.1 requires the loss function and the norm to be C 2 at δ ∗ .
This might not be the case for many functions used in practice, e.g.
ReLU networks and the (cid:96) ∞ norm.
Our next result shows that under a weaker condition of directional differentiability (instead of C 2 ), learning θ to maximize the margin can still be done by decreasing
L ( δ ∗ , θ ) w.r.t.
θ , at θ = θ 0 .
Due to space limitations, we only present an informal statement here.
Rigorous statements can be found in Appendix A.2.
Proposition 2.2.
Let δ ∗ be unique and L ( δ, θ ) be the loss of a deep ReLU network.
There exists some direction (cid:126)v in the parameter space, such that the loss L ( δ, θ ) | δ
= δ ∗ can be reduced in the direction of (cid:126)v .
Furthermore, by reducing L ( δ, θ ) | δ = δ ∗ , the margin is also guaranteed to be increased.
Figure 2 illustrates the relationship between the margin and the adversarial loss with an imaginary example.
Consider a 1-D example in Figure 2 (a), where the input example x is a scalar.
We perturb x in the positive direction with perturbation δ .
As we ﬁx ( x, y ) , we overload L ( δ, θ ) =
L LM θ ( x + δ, y ) , which is monotonically increasing on δ , namely larger perturbation results in higher loss.
Let L ( · , θ 0 ) (the dashed curve) denote the original function before an update step, and δ ∗ 0 =
arg min L ( δ,θ 0 ) ≥ 0 (cid:107) δ (cid:107) denote the corresponding margin (same as shortest successful perturbation in 1D).
As shown in Figure 2 (b), as the parameter is updated to θ 1 such that L ( δ ∗ 0 , θ 1 ) is reduced, the new margin δ ∗
1 = arg min L ( δ,θ 1 ) ≥ 0 (cid:107) δ (cid:107) is enlarged.
Intuitively, a reduced value of the loss at the shortest successful perturbation leads to an increase in margin.
2. S TABILIZING THE L EARNING WITH C ROSS E NTROPY S URROGATE L OSS
In practice, we ﬁnd the gradients of the “logit margin loss” L LM θ to be unstable.
The piecewise nature of the L LM θ loss can lead to discontinuity of its gradient, causing large ﬂuctuations on the boundary between the pieces.
It also does not fully utilize information provided by all the logits.
In our MMA algorithm, we instead use the “soft logit margin loss” (SLM)
[Equation]
which serves as a surrogate loss to the “logit margin loss” L LM θ ( x, y ) by replacing the the max function by the LogSumExp (sometimes also called softmax) function
One immediate property is that the SLM loss is smooth and convex (w.r.t. logits).
The next proposition shows that SLM loss is a good approximation to the LM loss.
Proposition 2.3.
[Equation]
where K denote the number of classes.
Remark 2.2.
By using the soft logit margin loss, MMA maximizes a lower bound of the margin, the
SLM-margin, d SLMθ ( x, y ) :
[Equation]
x + δ, y ) ≥ 0 . To see that, note by Proposition 2.3, L SLM θ ( x, y ) upper bounds L LM θ ( x, y ) .
So we have { δ : L LM θ ( x + δ, y ) ≤ 0 } ⊆
{ δ : L SLM θ ( x + δ, y ) ≤ 0 } .
Therefore, d SLM θ ( x, y ) ≤ d θ ( x, y ) , i.e. the SLM-margin is a lower bound of the margin.
We next show the gradient of the SLM loss is proportional to the gradient of the cross entropy loss, L CE θ ( x, y ) = log (cid:80) j exp( f jθ ( x )) − f yθ ( x ) , thus minimizing L CE θ ( x + δ ∗ , y )
w.r.t.
θ “is” minimizing L SLM θ ( x + δ ∗ , y ) .
Proposition 2.4.
For a ﬁxed ( x, y ) and θ , [Equation] [Equation]
[Equation]
where the scalar r ( θ, x, y ) = [Equation]
i (cid:54) [Equation]
exp [Equation]
[Equation]
Therefore, to simplify the learning algorithm, we perform gradient descent on model parameters using L CE θ ( x + δ ∗ , y ) .
As such, we use L CE θ on both clean and adversarial examples, which in practice stabilizes training:
[Equation]
( S ) , where L
[Equation]
where δ ∗ = arg min
L SLM θ ( x + δ,y ) ≥ 0 (cid:107) δ (cid:107) is found with the SLM loss, and H θ
= { i : d θ ( x i , y i ) <
d max } is the set of examples that have margins smaller than the hinge threshold.
2. F INDING THE OPTIMAL PERTURBATION δ ∗
To implement MMA, we still need to ﬁnd the δ ∗ , which is intractable in general settings.
We propose an adaptation of the projected gradient descent (PGD) (Madry et al., 2017) attack to give an approximate solution of δ ∗ , the Adaptive Norm Projective Gradient Descent Attack (AN-PGD).
In AN-PGD, we apply PGD on an initial perturbation magnitude (cid:15) init to ﬁnd a norm-constrained perturbation δ 1 , then we search along the direction of δ 1 to ﬁnd a scaled perturbation that gives
L = 0 , we then use this scaled perturbation to approximate (cid:15) ∗ .
Note that AN-PGD here only serves as an algorithm to give an approximate solution of δ ∗ , and it can be decoupled from the remaining parts of MMA training.
Other attacks that can serve a similar purpose can also ﬁt into our MMA training framework, e.g. the Decoupled Direction and Norm (DDN) attack (Rony et al., 2018).
Algorithm 1 describes the Adaptive Norm PGD Attack (AN-PGD) algorithm.
Inputs : ( x, y ) is the data example.
(cid:15) init is the initial norm constraint used in the ﬁrst PGD attack.
Outputs : δ ∗ , approximate shortest successful perturbation.
Parameters : (cid:15) max is the maximum perturbation length.
PGD( x, y, (cid:15) ) represents PGD perturbation δ with magnitude (cid:15) . [Figure]
Remark 2.3. Finding the δ ∗ in Propositions 2.1 and 2.2 requires solving a non-convex optimization problem, where the optimality cannot be guaranteed in practice.
Previous adversarial training methods, e.g. Madry et al.
(2017), suffer the same problem. Nevertheless, as we show later in
Figure 4, our proposed MMA training algorithm does achieve the desired behavior of maximizing the margin of each individual example in practice. 2. A DDITIONAL C LEAN L OSS DURING T RAINING
In practice, we observe that when the model is only trained with the objective function in Eq.
(7), the input space loss landscape is very ﬂat, which makes PGD less efﬁcient in ﬁnding δ ∗ for training, as shown in Figure 3.
Here we choose 50 examples from both the training and test sets respectively, then perform the PGD attack with (cid:15)
= 8 / 255 and keep those failed perturbations.
For each example, we linearly interpolate 9 more points between the original example and the perturbed example, and plot their logit margin losses.
In each sub-ﬁgure, the horizontal axis is the relative position of the interpolated example: e.g. 0.0 represents the original example, 1.0 represents the perturbed example with (cid:15)
= 8 / 255 , 0.5 represents the average of them.
The vertical axis is the logit margin loss.
Recall that when L LM θ ( x + δ, y ) < 0 , the perturbation δ fails.
OMMA-32 in Figure 3a represents model trained with only L MMA θ in Eq.
(7) with d max = 8 .
PGDFigure 3b represents model trained with PGD training (Madry et al., 2017) with (cid:15) = 8 .
As one can see, OMMA-32 has “ﬂatter” loss curves compared to PGD-8.
This could potentially weaken the adversary during training, which leads to poor approximation of δ ∗ and hampers training.
To alleviate this issue, we add an additional clean loss term to the MMA objective in Eq.
(7) to lower the loss on clean examples, such that the input space loss landscape is steeper.
Speciﬁcally, we use the following combined loss
[Equation]
The model trained with this combined loss and d max = 32 is the MMA-32 shown in Figure 3c.
Adding the clean loss is indeed effective.
Most of the loss curves are more tilted, and the losses of perturbed examples are lower.
We use L CB θ for MMA training in the rest of the paper due to its higher performance.
A more detailed comparison between The training the trained model Parameters perturbation lengths of training data min minimum perturbation length maximum perturbation length A x y
(cid:15) init the approximate shortest successful perturbation returned by an algorithm A (e.g. AN-PGD) on the data example ( x, y ) and at the initial norm (cid:15) init
{+B 
 L θ ( j y j + 13 (cid:80) j ∈ B L CE j y j+}
+ 23 (cid:80) j ∈ B adv 1 L CE θ ( x j , y j bined w θ
Algorithm 2 summarizes our practical MMA training algorithm.
During training for each minibatch, we 1) separate it into 2 batches based on if the current prediction matches the label; 2) ﬁnd δ ∗ for each example in the “correct batch”; 3) calculate the gradient of θ based on Eqs.
(7) and (8).
U NDERSTANDING A DVERSARIAL T RAINING W . R . T . M ARGIN
Through our development of MMA training in the last section, we have shown that margin maximization is closely related to adversarial training with the optimal perturbation length (cid:107) δ ∗ (cid:107) .
In this section, we further investigate the behavior of adversarial training in the perspective of margin maximization.
Adversarial training (Huang et al., 2015; Madry et al., 2017) minimizes the “worst-case” loss under a ﬁxed perturbation magnitude (cid:15) , as follows.
[Equation]
Looking again at Figure 2, we can see that an adversarial training update step does not necessarily increase the margin.
In particular, as we perform an update to reduce the value of loss at the ﬁxed perturbation (cid:15) , the parameter is updated from θ 0 to θ 1 .
After this update, we can imagine two different scenarios for the updated loss functions L θ 1 ( · ) (the solid curve) in Figure 2 (c) and (d).
In both (c) and (d), L θ 1 ( (cid:15) ) is decreased by the same amount.
However, the margin is increased in (c) with δ ∗ 1 > δ ∗ 0 , but decreased in (d) with δ ∗ 1 < δ ∗ 0 .
Formalizing the intuitive analysis, we present two theorems connecting adversarial training and margin maximization.
For brevity, ﬁxing ( x, y ) , let
L ( δ, θ ) =
L LM θ ( x + δ, y ) , d θ = d θ ( x, y ) , and (cid:15) ∗ θ ( ρ ) =
min
δ : L ( δ,θ ) ≥ ρ (cid:107) δ (cid:107) .
Theorem 3.1.
Assuming an update from adversarial training changes θ 0 to θ 1 , such that ρ ∗ = max (cid:107) δ (cid:107)≤ (cid:15) L ( δ, θ 0 ) >
max (cid:107) δ (cid:107)≤ (cid:15) L ( δ, θ 1 ) , then
1) if (cid:15) =
d θ 0 , then ρ ∗ = 0 , (cid:15) ∗ θ 1 ( ρ ∗ )
=
d θ 1 ≥
d
θ
=
(cid:15) ∗ θ
( ρ ∗ ) ; 2) if (cid:15) <
d θ 0 , then ρ ∗ ≤ 0 , (cid:15) ∗ θ
( ρ ∗ ) ≤ d θ 0 , (cid:15) ∗ θ 1 ( ρ ∗ )
≤ d θ 1 , and (cid:15) ∗ θ 1 ( ρ ∗ ) ≥ (cid:15) ∗ θ 0 ( ρ ∗ ) ; 3) if (cid:15) >
d θ 0 , then ρ ∗ ≥ 0 , (cid:15) ∗ θ
( ρ ∗ ) ≥
d θ 0 , (cid:15) ∗ θ 1 ( ρ ∗ ) ≥ d θ 1 , and (cid:15) ∗ θ 1 ( ρ ∗ ) ≥ (cid:15) ∗ θ 0 ( ρ ∗ ) .
Remark 3.1.
In other words, adversarial training, with the logit margin loss and a ﬁxed (cid:15)
1) exactly maximizes the margin, if (cid:15) is equal to the margin; 2) maximizes a lower bound of the margin, if (cid:15) is less than the margin; 3) maximizes an upper bound of the margin, if (cid:15) is greater than the margin.
Next we look at adversarial training with the cross entropy loss through the connection between cross entropy and the soft logit margin loss from Proposition 2.4.
We ﬁrst look at adversarial training on the SLM loss.
Fixing ( x, y ) , let d SLM θ =
d SLM θ ( x, y ) , and (cid:15) ∗ SLM ,θ ( ρ ) =
min L SLM θ ( x + δ,y ) ≥ ρ (cid:107) δ (cid:107) .
Corollary 3.1.
Assuming an update from adversarial training changes θ 0 to θ 1 , such that max (cid:107) δ
(cid:107)≤ (cid:15)
L SLM θ 0 ( x + δ, y ) > max (cid:107) δ (cid:107)≤ (cid:15)
L SLM θ 1 ( x + δ, y ) , if (cid:15) ≤ d SLM θ 0 , then ρ ∗ = max (cid:107) δ (cid:107)≤ (cid:15) L SLM θ 0 ( x + δ, y ) ≤ 0 , (cid:15) ∗ SLM ,θ 0 ( ρ ∗ ) ≤ d θ 0 , (cid:15) ∗ SLM ,θ 1 ( ρ ∗ ) ≤ d θ 1 , and (cid:15) ∗ SLM ,θ 1 ( ρ ∗ ) ≥ (cid:15) ∗ SLM ,θ 0 ( ρ ∗ ) .
Remark 3.2.
In other words, if (cid:15) is less than or equal to the SLM-margin, adversarial training, with the SLM loss and a ﬁxed perturbation length (cid:15) , maximizes a lower bound of the SLM-margin, thus a lower bound of the margin.
Recall Proposition 2.4 shows that L CE θ and L SLM θ have the same gradient direction w.r.t.
both the model parameter and the input.
In adversarial training (Madry et al., 2017), the PGD attack only uses the gradient direction w.r.t.
the input, but not the gradient magnitude.
Therefore, in the inner maximization loop, using the SLM and CE loss will result in the same approximate δ ∗ .
Furthermore, ∇ θ L CE θ ( x + δ ∗ , y ) and ∇ θ L SLM θ ( x + δ ∗ , y ) have the same direction.
If the step size is chosen appropriately, then a gradient update that reduces L CE θ ( x + δ ∗ , y ) will also reduce L SLM θ ( x + δ ∗ , y ) .
Combined with Remark 3.2, these suggest:
Adversarial training with cross entropy loss (Madry et al., 2017) approximately maximizes a lower bound of the margin, if (cid:15) is smaller than or equal to the SLM-margin.
On the other hand, when (cid:15) is larger then the margin, such a relation no longer exists.
When (cid:15) is too large, adversarial training is maximizing an upper bound of the margin, which might not necessarily increase the margin.
This suggests that for adversarial training with a large (cid:15) , starting with a smaller (cid:15) then gradually increasing it could help, since the lower bound of the margin is maximized at the start of training.
Results in Sections 4.1 and 4.2 corroborate exactly with this theoretical prediction.
E XPERIMENTS
We empirically examine several hypotheses and compare MMA training with different adversarial training algorithms on the MNIST and CIFAR10 datasets under (cid:96) ∞ / (cid:96) 2 -norm constrained perturbations.
Due to space limitations, we mainly present results on CIFAR10- (cid:96) ∞ for representative models in Table 1.
Full results are in Table 2 to 15 in Appendix F. Implementation details are also left to the appendix, including neural network architecture, training and attack hyperparameters.
Our results conﬁrm our theory and show that MMA training is stable to its hyperparameter d max , and balances better among various attack lengths compared to adversarial training with ﬁxed perturbation magnitude.
This suggests that MMA training is a better choice for defense when the perturbation length is unknown, which is often the case in practice.
Measuring Adversarial Robustness: We use the robust accuracy under multiple projected gradient descent (PGD) attacks (Madry et al., 2017) as the robustness measure.
Speciﬁcally, given an example, each model is attacked by both repeated randomly initialized whitebox PGD attacks and numerous transfer attacks, generated from whitebox PGD attacking other models.
If any one of these attacks succeed, then the model is considered “not robust under attack” on this example.
For each dataset-norm setting and for each example, under a particular magnitude (cid:15) , we ﬁrst perform N randomly initialized whitebox PGD attacks on each individual model, then use N · ( m − 1) PGD attacks from all the other models to perform transfer attacks, where m is the total number of models considered under each setting (e.g. CIFAR10- (cid:96) ∞ ).
In our experiments, we use N = 10 for models trained on CIFAR10, thus the total number of the “combined” (whitebox and transfer) set of attacks is 320 for CIFAR10- (cid:96) ∞ ( m = 32 ).
We use ClnAcc for clean accuracy, AvgAcc for the average over both clean accuracy and robust accuracies at different (cid:15) ’s, AvgRobAcc for the average over only robust accuracies under attack. Correspondence between d max and (cid:15) :
In MMA training, we maximize each example’s margin until the margin reaches d max .
In standard adversarial training, each example is trained to be robust at (cid:15) .
Therefore in MMA training, we usually set d max to be larger than (cid:15) in standard adversarial training.
It is difﬁcult to know the “correct” value of d max , therefore we test different d max values, and compare a group of MMA trained models to a group of PGD trained models.
4. E FFECTIVENESS OF M ARGIN M AXIMIZATION DURING T RAINING
As discussed in Section 3, MMA training enlarges margins of all training points,while PGD training, by minimizing the adversarial loss with a ﬁxed (cid:15) , might fail to enlarge margins for points with initial margins smaller than (cid:15) .
This is because when d θ ( x, y ) < (cid:15) , PGD training is maximizing an upper bound of d θ ( x, y ) , which may not necessarily increase the margin.
To verify this, we track how the
[Table]
margin distribution changes during training processes in two models under the CIFAR10- (cid:96) 2 4 case,
MMA-3.0 vs PGD-2.5.
We use MMA- d max to denote the MMA trained model with the combined loss in Eq.
(8) and hinge threshold d max , and PGD- (cid:15) to represent the PGD trained (Madry et al., 2017) model with ﬁxed perturbation magnitude (cid:15) .
Speciﬁcally, we randomly select 500 training points, and measure their margins after each training epoch.
We use the norm of the 1000-step DDN attack (Rony et al., 2018) to approximate the margin.
The results are shown in Figure 4, where each subplot is a histogram (rotated by 90 ◦ ) of margin values.
For the convenience of comparing across epochs, we use the vertical axis to indicate margin value, and the horizontal axis for counts in the histogram.
The number below each subplot is the corresponding training epoch.
Margins mostly concentrate near 0 for both models at the beginning.
As training progresses, both enlarge margins on average.
However, in PGD training, a portion of the margins stay close to 0 throughout the training process, at the same time, also pushing some margins to be even higher than 2.5, likely because PGD training continues to maximize lower bounds of this subset of the total margins, as we discussed in Section 3, the (cid:15) value that the PGD-2.5 model is trained for.
MMA training, on the other hand, does not “give up” on those data points with small margins.
At the end of training, 37.8% of the data points for PGD-2.5 have margins smaller than 0.05, while only 20.4% for MMA.
As such, PGD training enlarges the margins of “easy data” which are already robust enough, but “gives up” on “hard data” with small margins.
Instead, MMA training pushes the margin of every data point, by ﬁnding the proper (cid:15) .
In general, when the attack magnitude is unknown, MMA training is more capable in achieving a better balance between small and large margins, and thus a better balance among adversarial attacks with various (cid:15) ’s as a whole.
4. G RADUALLY I NCREASING (cid:15) H ELPS PGD T RAINING WHEN (cid:15) IS L ARGE
Our previous analysis in Section 3 suggests that when the ﬁxed perturbation magnitude (cid:15) is small,
PGD training increases the lower bound of the margin.
On the other hand, when (cid:15) is larger than the margin, PGD training does not necessarily increase the margin.
This is indeed conﬁrmed by our experiments.
PGD training fails at larger (cid:15) , in particular (cid:15)
= 24 / 255 for the CIFAR10- (cid:96) ∞ as shown in Table 1.
We can see that PGD-24’s accuracies at all test (cid:15) ’s are around 10%.
Aiming to improve PGD training, we propose a variant of PGD training, named PGD with Linear
Scaling (PGDLS), in which we grow the perturbation magnitude from 0 to the ﬁxed magnitude linearly in 50 epochs.
According to our theory, gradually increasing the perturbation magnitude could avoid picking a (cid:15) that is larger than the margin, thus managing to maximizing the lower bound of the margin rather than its upper bound, which is more sensible.
It can also be seen as a “global magnitude scheduling” shared by all data points, which is to be contrasted to MMA training that gives magnitude scheduling for each individual example.
We use PGDLS- (cid:15) to represent these models and show their performances also in Table 1.
We can see that PGDLS-24 is trained successfully, whereas PGD-24 fails.
At (cid:15) = 8 or 16 , PGDLS also performs similar or better than PGD training, conﬁrming the beneﬁt of training with small perturbation at the beginning.
4. C OMPARING MMA T RAINING WITH PGD T RAINING
From the ﬁrst three columns in Table 1, we can see that MMA training is very stable with respect to its hyperparameter, the hinge threshold d max .
When d max is set to smaller values (e.g. 12 and 20), MMA models attain good robustness across different attacking magnitudes, with the best clean accuracies in the comparison set.
When d max is large, MMA training can still learn a reasonable model that is both accurate and robust.
For MMA-32, although d max is set to a “impossible-tobe-robust” level at 32/255, it still achieves 84.36% clean accuracy and 47.18% robust accuracy at 8 / 255 , thus automatically “ignoring” the demand to be robust at larger (cid:15) ’s, including 20, 24, 28 and 32, recognizing its infeasibility due to the intrinsic difﬁculty of the problem.
In contrast, PGD trained models are more sensitive to their speciﬁed ﬁxed perturbation magnitude.
In terms of the overall performance, we notice that MMA training with a large d max , e.g. 20 or 32, achieves high AvgAcc values, e.g. 28.86% or 29.39%.
However, for PGD training to achieve a similar performance, (cid:15) needs to be carefully picked (PGD-16 and PGDLS-16), and their clean accuracies suffer a signiﬁcant drop.
We also compare MMA models with ensembles of PGD trained models.
PGD-ens/PGDLS-ens represents the ensemble of PGD/PGDLS trained models with different (cid:15) ’ s.
The ensemble produces a prediction by performing a majority vote on label predictions, and using the softmax scores as the tie breaker.
MMA training achieves similar performance compared to the ensembled PGD models.
PGD-ens maintains a good clean accuracy, but it is still marginally outperformed by MMA-32 w.r.t.
robustness at various (cid:15) ’s. Further note that 1) the ensembling requires signiﬁcantly higher computation costs both at training and test times; 2) Unlike attacking individual models, attacking ensembles is still relatively unexplored in the literature, thus our whitebox PGD attacks on the ensembles may not be sufﬁciently effective; 5 and 3) as shown in Appendix F, for MNIST- (cid:96) ∞ / (cid:96) 2 , MMA trained models signiﬁcantly outperform the PGD ensemble models. Comparison with TRADES: TRADES (Zhang et al., 2019b) improves PGD training by decoupling the minimax training objective into an accuracy term and an robustness regularization term. We compare MMA-32 with TRADES 6 in Table 1 as their clean accuracies are similar. We see that
TRADES outperforms MMA on the attacks of lengths that are less than (cid:15) = 12 / 255 , but it sacriﬁces the robustness under larger attacks.
We note that MMA training and TRADES’ idea of optimizing a calibration loss are progresses in orthogonal directions and could potentially be combined.
Testing on gradient free attacks: As a sanity check for gradient obfuscation (Athalye et al., 2018), we also performed the gradient-free SPSA attack (Uesato et al., 2018), to all our (cid:96) ∞ -MMA trained models on the ﬁrst 100 test examples.
We ﬁnd that, in all cases, SPSA does not compute adversarial examples successfully when gradient-based PGD did not.
C ONCLUSIONS
In this paper, we proposed to directly maximize the margins to improve adversarial robustness.
We developed the MMA training algorithm that optimizes the margins via adversarial training with perturbation magnitude adapted both throughout training and individually for the distinct datapoints in the training dataset.
Furthermore, we rigorously analyzed the relation between adversarial training and margin maximization.
Our experiments on CIFAR10 and MNIST empirically conﬁrmed our theory and demonstrate that MMA training outperforms adversarial training in terms of sensitivity to hyperparameter setting and robustness to variable attack lengths, suggesting MMA is a better choice for defense when the adversary is unknown, which is often the case in practice. A CKNOWLEDGEMENT
We thank Marcus Brubaker, Bo Xin, Zhanxing Zhu, Joey Bose and Hamidreza Saghir for proofreading earlier drafts and useful feedbacks, also Jin Sung Kang for helping with computation resources.
Appendix A P ROOFS A. P ROOF OF PROPOSITION 2.
Proof.
Recall (cid:15) ( δ ) = (cid:107) δ (cid:107) .
Here we compute the gradient for d θ ( x, y ) in its general form.
Consider the following optimization problem:
[Equation]
where ∆( θ ) =
{ δ : L θ ( x + δ, y ) = 0 } , (cid:15) and L ( δ, θ ) are both C 2 functions 7 .
Denotes its Lagrangian by L ( δ, λ ) , where
[Equation]
For a ﬁxed θ , the optimizer δ ∗ and λ ∗ must satisfy the ﬁrst-order conditions (FOC)
[Equation]
Put the FOC equations in vector form,
[Equation]
Note that G is C 1 continuously differentiable since (cid:15) and L ( δ, θ ) are C 2 functions.
Furthermore, the
Jacobian matrix of G w.r.t ( δ, λ ) is
[Equation]
which by assumption is full rank.
Therefore, by the implicit function theorem, δ ∗ and λ ∗ can be expressed as a function of θ , denoted by δ ∗ ( θ ) and λ ∗ ( θ ) .
To further compute ∇ θ d θ ( x, y ) , note that d θ ( x, y )
= (cid:15) ( δ ∗ ( θ )) .
Thus,
[Equation]
where the second equality is by Eq.
The implicit function theorem also provides a way of computing ∂δ ∗ ( θ ) ∂θ which is complicated involving taking inverse of the matrix ∇ ( δ,λ ) G (( δ ∗ , λ ∗ ) , θ ) .
Here we present a relatively simple way to compute this gradient.
Note that by the deﬁnition of δ ∗ ( θ ) ,
L
[Equation]
and δ ∗ ( θ ) is a differentiable implicit function of θ restricted to this level set.
Differentiate with w.r.t.
θ on both sides:
[Equation]
Combining Eq.
(11) and Eq.
[Equation]
[Equation]
Lastly, note that
[Equation]
= 0 .
Therefore, one way to calculate λ ∗ ( θ ) is by
[Equation]
A. P ROOF OF P ROPOSITION 2.
We provide more detailed and formal statements of Proposition 2.2.
For brevity, consider a K -layers fully-connected ReLU network, f ( θ ; x ) =
f θ ( x ) as a function of θ .
[Equation]
where the D k are diagonal matrices dependent on ReLU’s activation pattern over the layers, and
W k ’s
and V are the weights (i.e. θ ).
Note that f ( θ ; x ) is a piecewise polynomial functions of θ with ﬁnitely many pieces.
We further deﬁne the directional derivative of a function g , along the direction of (cid:126)v , to be:
[Equation]
Note that for every direction (cid:126)v , there exists α > 0 such that f ( θ ; x ) is a polynomial restricted to a line segment [ θ, θ + α(cid:126)v ] .
Thus the above limit exists and the directional derivative is well deﬁned.
We ﬁrst show the existence of (cid:126)v and t for l ( θ 0 + t(cid:126)v ) given any (cid:15) .
Let l θ
[Equation]
=
[Equation]
θ 0 + t(cid:126)v ) .
Proposition A.1.
For (cid:15) > 0 , t ∈ [0 , 1] , and θ 0 ∈ Θ , there exists a direction (cid:126)v ∈ Θ , such that the derivative of l θ 0 ,(cid:126)v,(cid:15) ( t ) exists and is negative.
Moreover, it is given by
[Equation]
Proof.
[Proof sketch]
Since θ 0 is not a local minimum, there exists a direction d , such that
[Equation]
The Danskin theorem provides a way to compute the directional gradient along this direction (cid:126)v .
We basically apply a version of Danskin theorem for directional absolutely continuous maps and semicontinuous maps (Yu, 2012).
the constraint set { δ : (cid:107) δ (cid:107) ≤ (cid:15) } is compact; 2.
L ( θ 0 + t(cid:126)v ; x + δ, y ) is piecewise Lipschitz and hence absolutely continuous (an induction argument on the integral representation over the ﬁnite pieces).
L ( θ 0 + t(cid:126)v ; x + δ, y ) is continuous on both δ and along the direction (cid:126)v and hence upper semi continuous.
Hence we can apply Theorem 1 in Yu (2012).
Therefore, for any (cid:15) > 0 , if θ 0 is not a local minimum, then there exits a direction d , such that for θ 1 = θ 0 + t(cid:126)v for a proper t ,
[Equation]
Our next proposition provides an alternative way to increase the margin of f θ .
Proposition A.2.
Assume f
θ 0 has a margin (cid:15) 0 , and θ 1 such that l θ 0 ,(cid:126)v,(cid:15)
( t ) ≤ l θ 1 ,(cid:126)v,(cid:15)
(0) , then f θ 1 has a larger margin than (cid:15) 0 .
Proof.
Since f θ 0 has a margin (cid:15) 0 , thus
[Equation]
Further by l θ 0 ,(cid:126)v,(cid:15) 0 ( t ) ≤ l θ 0 ,
[Equation]
[Equation] [Equation] [Equation]
[Equation]
To see the equality (constraint not binding), we use the following argument.
The envolope function’s continuity is passed from the continuity of L ( θ 0 ; x + δ, y ) .
The inverse image of a closed set under continuous function is closed.
If δ ∗ lies in the interior of max (cid:107) δ (cid:107)≤ (cid:15)
L (cid:126)v,(cid:15) ( θ 0 ; x + δ, y ) ≥ 0 , we would have a contradiction.
Therefore the constraint is not binding, due to the continuity of the envolope function.
By Eq.
(15), max (cid:107) δ (cid:107)≤ (cid:15)
L ( θ 1 ; x + δ, y ) < 0 .
So for the parameter θ 1 , f θ 1 has a margin (cid:15) 1 > (cid:15) 0 .
Therefore, the update θ 0 → θ 1 = θ
+ t(cid:126)v increases the margin of f θ .
A. P ROOF OF P ROPOSITION 2.
Proof.
[Equation]
Therefore,
[Equation]
A. A LEMMA FOR LATER PROOFS
The following lemma helps relate the objective of adversarial training with that of our MMA training.
Here, we denote L θ ( x + δ, y ) as L ( δ, θ ) for brevity.
Lemma A.1.
Given ( x, y ) and θ , assume that L ( δ, θ ) is continuous in δ , then for (cid:15) ≥ 0 , and ρ ≥ L (
, θ ) ∈
Range ( L ( δ, θ )) , it holds that
[Equation]
[Equation] [Equation]
Proof.
Eq.
We prove this by contradiction.
Suppose max (cid:107) δ (cid:107)≤ (cid:15) L ( δ, θ ) > ρ .
When (cid:15) = 0 , this violates our asssumption ρ ≥ L (0 , θ ) in the theorem.
So assume (cid:15) > 0 .
Since L ( δ, θ ) is a continuous function deﬁned on a compact set, the maximum is attained by ¯ δ such that (cid:107) ¯ δ (cid:107) ≤ (cid:15) and L (¯ δ, θ ) > ρ .
Note that L ( δ, θ )) is continuous and ρ ≥ L (0 , θ ) , then there exists ˜ δ ∈ (cid:104) 0 , ¯ δ (cid:105) i.e. the line segment connecting 0 and ¯ δ , such that (cid:107) ˜
δ (cid:107) < (cid:15) and L (˜ δ, θ )
= ρ .
This follows from the intermediate value theorem by restricting L ( δ, θ ) onto (cid:104) 0 , ¯ δ (cid:105) .
This contradicts min L ( δ,θ ) ≥ ρ (cid:107) δ (cid:107) =
(cid:15) .
If max (cid:107) δ (cid:107)≤ (cid:15) L ( δ, θ ) < ρ , then { δ : (cid:107) δ (cid:107) ≤ (cid:15) } ⊂ { δ : L ( δ, θ ) < ρ } .
Every point p ∈ { δ : (cid:107) δ (cid:107) ≤ (cid:15) } is in the open set { δ : L ( δ, θ ) <
ρ } , there exists an open ball with some radius r p centered at p such that B r p ⊂ { δ : L ( δ, θ ) < ρ } .
This forms an open cover for { δ : (cid:107) δ (cid:107) ≤ (cid:15) } .
Since { δ : (cid:107) δ (cid:107) ≤ (cid:15) } is compact, there is an open ﬁnite subcover U (cid:15) such that: { δ : (cid:107) δ (cid:107) ≤ (cid:15) } ⊂ U (cid:15) ⊂ { δ : L ( δ, θ ) < ρ } .
Since U (cid:15) is ﬁnite, there exists h > 0 such that { δ : (cid:107) δ (cid:107) ≤ (cid:15) + h } ⊂ { δ : L ( δ, θ ) < ρ } .
Thus { δ : L ( δ, θ ) ≥ ρ } ⊂ { δ : (cid:107) δ (cid:107) > (cid:15) + h } , contradicting min L ( δ,θ ) ≥ ρ (cid:107) δ (cid:107) =
(cid:15) again.
Eq.
Assume that min L ( δ,θ ) ≥ ρ (cid:107) δ (cid:107) > (cid:15) , then { δ : L ( δ, θ ) ≥ ρ } ⊂ { δ : (cid:107) δ (cid:107) > (cid:15) } .
Taking complementary set of both sides, { δ : (cid:107) δ (cid:107) ≤ (cid:15) } ⊂ { δ : L ( δ, θ ) < ρ } .
Therefore, by the compactness of { δ : (cid:107) δ (cid:107) ≤ (cid:15) } , max (cid:107) δ (cid:107)≤ (cid:15) L ( δ, θ )
< ρ , contradiction.
A. P ROOF OF T HEOREM 3.
Proof.
Recall that L ( δ, θ )
=
L LM θ ( x + δ, y ) , d θ = d θ ( x, y ) , (cid:15) ∗ θ ( ρ ) =
min
δ : L ( δ,θ ) ≥ ρ (cid:107) δ (cid:107) , and ρ ∗ = max (cid:107) δ (cid:107)≤ (cid:15)
We ﬁrst prove that ∀ (cid:15) , (cid:15) ∗ θ 1 ( ρ ∗ ) ≥ (cid:15) ∗ θ
( ρ ∗ ) by contradiction.
We assume (cid:15) ∗ θ 1 ( ρ ∗ ) < (cid:15) ∗ θ 0 ( ρ ∗ ) .
Let δ ∗ θ ( ρ ) =
arg min δ : L ( δ,θ ) ≥ ρ (cid:107) δ (cid:107) , which is (cid:107) δ ∗ θ 1 ( ρ ∗ ) (cid:107) <
(cid:107) δ ∗ θ 0 ( ρ ∗ ) (cid:107) .
By Eq.
(24), we have (cid:107) δ ∗ θ 0 ( ρ ∗ ) (cid:107) ≤ (cid:15) .
Therefore, (cid:107) δ ∗ θ 1 ( ρ ∗ ) (cid:107) <
(cid:15) .
Then there exist a δ # ∈ { δ : (cid:107) δ (cid:107) ≤ (cid:15) } such that L ( δ # , θ 1 ) ≥ ρ ∗ .
This contradicts max (cid:107) δ (cid:107)≤ (
cid:15) L ( δ, θ 1 )
< ρ ∗ .
Therefore (cid:15) ∗ θ 1 ( ρ ∗ ) ≥ (cid:15) ∗ θ
( ρ ∗ ) .
For 1), (cid:15) =
d θ 0 .
By deﬁnition of margin in Eq.
(1), we have ρ ∗ = max (cid:107) δ (cid:107)≤ d θ 0 L ( δ, θ 0 )
= 0 .
Also by deﬁnition of (cid:15) ∗ θ ( ρ ) , (cid:15) ∗ θ 0 (0) =
d θ 0 and (cid:15) ∗ θ 1 (0) =
d θ 1 .
For 2), (cid:15) <
d θ 0 .
We have ρ ∗ = max (cid:107) δ
(cid:107)≤ (cid:15) L ( δ, θ 0 ) ≤ max (cid:107) δ (cid:107)≤ d θ 0 L ( δ, θ 0 ) = 0 .
Therefore (cid:15) ∗ θ
( ρ ∗ ) ≤ (cid:15) ∗ θ 0 (0) =
d θ 0 and (cid:15) ∗ θ 1 ( ρ ∗ ) ≤ (cid:15) ∗ θ 1 (0)
=
d θ 1 .
For 3), (cid:15) >
d θ 0 .
We have ρ ∗ = max (cid:107) δ
(cid:107)≤ (cid:15) L ( δ, θ 0 ) ≥ max (cid:107) δ (
cid:107)≤ d θ 0 L ( δ, θ 0 ) = 0 .
Therefore (cid:15) ∗ θ
( ρ ∗ ) ≥ (cid:15) ∗ θ 0 (0) =
d θ 0 and (cid:15) ∗ θ 1 ( ρ ∗ ) ≥ (cid:15) ∗ θ 1 (0)
= d θ 1 .
B M ORE R ELATED W ORKS
We next discuss a few related works in details.
First-order Large Margin: Previous works (Matyasko & Chau, 2017; Elsayed et al., 2018; Yan et al., 2019) have attempted to use ﬁrst-order approximation to estimate the input space margin.
For ﬁrst-order methods, the margin will be accurately estimated when the classiﬁcation function is linear.
MMA’s margin estimation is exact when the shortest successful perturbation δ ∗ can be solved, which is not only satisﬁed by linear models, but also by a broader range of models, e.g. models that are convex w.r.t.
input x .
This relaxed condition could potentially enable more accurate margin estimation which improves MMA training’s performance.
(Cross-)Lipschitz Regularization: Sokolic et al. ; Tsuzuku et al.
(2018) enlarges their margin by controlling the global Lipschitz constant, which in return places a strong constraint on the model and harms its learning capabilities.
Instead, our method, alike adversarial training, uses adversarial attacks to estimate the margin to the decision boundary.
With a strong method, our estimate is much more precise in the neighborhood around the data point, while being much more ﬂexible due to not relying on a global Lipschitz constraint.
Hard-Margin SVM (Vapnik, 2013) in the separable case: Assuming that all the training examples are correctly classiﬁed and using our notations on general classiﬁers, the hard-margin SVM objective can be written as:
[Equation]
On the other hand, under the same “separable and correct” assumptions, MMA formulation in Eq.
(3) can be written as
[Equation]
which is maximizing the average margin rather than the minimum margin in SVM.
Note that the theorem on gradient calculation of the margin in Section 2.1 also applies to the SVM formulation of differentiable functions.
Because of this, we can also use SGD to solve the following “SVM-style” formulation:
[Equation]
As our focus is using MMA to improve adversarial robustness which involves maximizing the average margin, we delay the maximization of minimum margin to future work.
Maximizing Robust Radius for Randomized Smoothing: Randomized smoothing (Cohen et al., 2019) is an effective technique for certifying the (cid:96) 2 robust radius of a “smoothed classiﬁer” with high probability bound.
Here the certiﬁed robust radius is a lower bound of the input space margin of the “smoothed classiﬁer”.
Zhai et al.
(2020) introduce a method to directly maximizing this certiﬁed robust radius. Bayesian Adversarial Learning: Ye & Zhu (2018) introduce a full Bayesian treatment of adversarial training. In practice, the model is trained to be robust under adversarial perturbations with different magnitudes. Although both their and our methods adversarially train the model with different perturbation magnitudes, the motivations are completely different.
Dynamic Adversarial Training: Wang et al.
(2019) improve adversarial training’s convergence by dynamically adjusting the strength (the number of iterations) of PGD attacks, based on its proposed
First-Order Stationary Condition.
In a sense, MMA training is also dynamically adjusting the attack strength.
However, our aim is margin maximization and the strength here represents the magnitude of the perturbation.
B. D ETAILED C OMPARISON WITH A DVERSARIAL T RAINING WITH DDN
For (cid:96) 2 robustness, we also compare to models adversarially trained on the “Decoupled Direction and
Norm” (DDN) attack (Rony et al., 2018), which is concurrent to our work.
The DDN attack aims to achieve successful perturbation with minimal (cid:96) 2 norm, thus, DDN could be used as a drop-in replacement for the AN-PGD attack for MMA training.
We performed evaluations on the downloaded
DDN trained models.
The DDN MNIST model is a larger ConvNet with similar structure to our LeNet5, and the CIFAR model is wideresnet-28-10, which is similar but larger than the wideresnet-28-4 that we use.
DDN training, “training on adversarial examples generated by the DDN attack”, differs from MMA in the following ways.
When the DDN attack does not ﬁnd a successful adversarial example, it returns the clean image, and the model will use it for training.
In MMA, when a successful adversarial example cannot be found, it is treated as a perturbation with very large magnitude, which will be ignored by the hinge loss when we calculate the gradient for this example.
Also, in DDN training, there exists a maximum norm of the perturbation.
This maximum norm constraint does not exist for MMA training.
When a perturbation is larger than the hinge threshold, it will be ignored by the hinge loss.
There also are differences in training hyperparameters, which we refer the reader to
Rony et al.
(2018) for details.
Despite these differences, in our experiments MMA training achieves similar performances under the (cid:96) 2 cases.
While DDN attack and training only focus on (cid:96) 2 cases, we also show that the MMA training framework provides signiﬁcant improvements over PGD training in the (cid:96) ∞ case.
C D ETAILED S ETTINGS FOR T RAINING
With respect to the weights on L CE and L MMA in Eq.
(8), we tested 3 pairs of weights, (1/3, 2/3), (1/2, 1/2) and (2/3, 1/3), in our initial CIFAR10 (cid:96) ∞ experiments.
We observed that (1/3, 2/3), namely (1/3 for L CE and 2/3 for L MMA ) gives better performance.
We then ﬁxed it and use the same values for all the other experiments in the paper, including the MNIST experiments and (cid:96) 2 experiments.
We train LeNet5 models for the MNIST experiments and use wide residual networks (Zagoruyko & Komodakis, 2016) with depth 28 and widen factor 4 for all the CIFAR10 experiments.
For all the experiments, we monitor the average margin from AN-PGD on the validation set and choose the model with largest average margin from the sequence of checkpoints during training.
The validation set contains ﬁrst 5000 images of training set.
It is only used to monitor training progress and not used in training.
Here all the models are trained and tested under the same type of norm constraints, namely if trained on (cid:96) ∞ , then tested on (cid:96) ∞ ; if trained on (cid:96) 2 , then tested on (cid:96) 2 .
The LeNet5 is composed of 32-channel conv ﬁlter + ReLU + size 2 max pooling + 64-channel conv ﬁlter + ReLU + size 2 max pooling + fc layer with 1024 units + ReLU + fc layer with 10 output classes.
We do not preprocess MNIST images before feeding into the model.
For training LeNet5 on all MNIST experiments, for both PGD and MMA training, we use the Adam optimizer with an initial learning rate of 0.0001 and train for 100000 steps with batch size 50.
In our initial experiments, we tested different initial learning rate at 0.0001, 0.001, 0.01, and 0.1 and do not ﬁnd noticeable differences.
We use the WideResNet-28-4 as described in Zagoruyko & Komodakis (2016) for our experiments, where 28 is the depth and 4 is the widen factor.
We use “per image standardization” 9 to preprocess
CIFAR10 images, following Madry et al.
For training WideResNet on CIFAR10 variants, we use stochastic gradient descent with momentum 0.9 and weight decay 0.0002.
We train 50000 steps in total with batch size 128.
The learning rate is set to 0.3 at step 0, 0.09 at step 20000, 0.03 at step 30000, and 0.009 at step 40000.
This setting is the same for PGD and MMA training.
In our initial experiments, we tested different learning rate at 0.03, 0.1, 0.3, and 0.6, and kept using 0.3 for all our later experiments.
We also tested a longer training schedule, following Madry et al.
(2017), where we train 80000 steps with different learning rate schedules.
We did not observe improvement with this longer training, therefore kept using the 50000 steps training.
For models trained on MNIST, we use 40-step PGD attack with the soft logit margin (SLM) loss deﬁned in Section 3, for CIFAR10 we use 10 step-PGD, also with the SLM loss.
For both MNIST and CIFAR10, the step size of PGD attack at training time is
[Table]
· In AN-PGD, we always perform 10 step bisection search after PGD, with the SLM loss.
For AN-PGD, the maximum perturbation length is always 1.05 times the hinge threshold: (cid:15) max = 1 .
05 d max .
The initial perturbation length at the ﬁrst epoch, (cid:15) init , have different values under different settings.
(cid:15) init = 0 .
5 for MNIST (cid:96) 2 , (cid:15) init = 0 .
1 for MNIST (cid:96) ∞ , (cid:15) init = 0 .
5 for CIFAR10 (cid:96) 2 , (cid:15) init = 0 .
for CIFAR10 (cid:96) 2 .
In epochs after the ﬁrst, (cid:15) init will be set to the margin of the same example from last epoch.
Trained models : Various PGD/PGDLS models are trained with different perturbation magnitude (cid:15) , denoted by PGD- (cid:15) or PGDLS- (cid:15) .
PGD-ens/PGDLS-ens represents the ensemble of PGD/PGDLS trained models with different (cid:15) ’
s.
The ensemble makes prediction by majority voting on label predictions, and uses softmax scores as the tie breaker.
We perform MMA training with different hinge thresholds d max , also with/without the additional clean loss (see next section for details).
We use
OMMA to represent training with only L MMA θ in Eq.
(7), and MMA to represent training with the combined loss in Eq.
When train For each d max value, we train two models with different random seeds, which serves two purposes: 1) conﬁrming the performance of MMA trained models are not signiﬁcantly affected by random initialization; 2) to provide transfer attacks from an “identical” model.
As such, MMA trained models are named as OMMA/MMA- d max -seed.
Models shown in the main body correspond to those with seed “sd0”.
For MNIST- (cid:96) ∞ , we train PGD/PGDLS models with (cid:15) = 0 . We also train
OMMA/MMA models with d max = 0 . 45 , each with two random seeds.
We also consider a standard model trained on clean images, PGD-ens, PGDLS-ens, and the PGD trained model from Madry et al.
Therefore, we have in total m = 2 × 5 + 2 × 2 + 4 = 18 models under MNIST- (cid:96) ∞ .
For MNIST- (cid:96) 2 , we train PGD/PGDLS models with (cid:15) = 1 . 0 , 2 . 0 , 3 . 0 , 4 . 0 . We also train
OMMA/MMA models with d max = 2 . 0 , 4 . 0 , 6 . 0 , each with two random seeds.
We also consider a standard model trained on clean images, PGD-ens, PGDLS-ens, and the DDN trained model from
Rony et al.
Therefore, we have in total m = 2 × 4 + 2 × 3 × 2 + 4 = 24 models under
MNIST- (cid:96) 2 . For CIFAR10- (cid:96) ∞ , we train PGD/PGDLS models with (cid:15) = 4 , 8 , 12 , 16 , 20 , 24 , 28 , 32 . We also train OMMA/MMA models with d max = 12 , 20 , 32 , each with two random seeds.
We also consider a standard model trained on clean images, PGD-ens, PGDLS-ens, and the PGD trained model from
Madry et al.
Therefore, we have in total m = 2 × 8 + 2 × 3 × 2 + 4 = 32 models under
CIFAR10- (cid:96) ∞ . For CIFAR10- (cid:96) 2 , we train PGD/PGDLS models with (cid:15) = 0 .
We also train
OMMA/MMA models with d max = 1 . 0 , 2 . 0 , 3 . 0 , each with two random seeds. We also consider a standard model trained on clean images, PGD-ens, PGDLS-ens, and the DDN trained model from Rony et al. Therefore, we have in total m = 2 × 5 + 2 × 3 × 2 + 4 = 26 models under
MNIST- (cid:96) 2 . With regard to ensemble models, for MNIST- (cid:96) 2 PGD/PGDLS-ens, CIFAR10- (cid:96) 2 PGD/PGDLS-ens,
MNIST- (cid:96) ∞
PGDLS-ens, and CIFAR10- (cid:96)
∞ PGDLS-ens, they all use the PGD (or PGDLS) models trained at all testing (attacking) (cid:15) ’
s. For CIFAR10- (cid:96)
∞
PGD-ens, PGD-24,28,32 are excluded for the same reason.
D D ETAILED S ETTINGS OF A TTACKS
For both (cid:96) ∞ and (cid:96) 2 PGD attacks, we use the implementation from the AdverTorch toolbox (Ding et al., 2019b).
Regarding the loss function of PGD, we use both the cross entropy (CE) loss and the
Carlini & Wagner (CW) loss.
As previously stated, each model will have N whitebox PGD attacks on them, N/ 2 of them are
CE-PGD attacks, and the other N/ 2 are CW-PGD attacks.
Recall that N = 50 for MNIST and
N = 10 for CIFAR10.
At test time, all the PGD attack run 100 iterations.
We manually tune the step size parameter on a few MMA and PGD models and then ﬁx them thereafter.
The step size for
MNIST- (cid:96) ∞ when (cid:15) = 0 .
3 is 0 .
0075 , the step size for CIFAR10- (cid:96) ∞ when (cid:15) = 8 / 255 is 2 / 255 , the step size for MNIST- (cid:96) 2 when (cid:15)
= 1 . 0 is 0 .
25 , the step size for CIFAR10- (cid:96) 2 when (cid:15) = 1 . 0 is 0 .
For other (cid:15) values, the step size is linearly scaled accordingly.
The ensemble model we considered uses the majority vote for prediction, and uses softmax score as the tie breaker.
So it is not obvious how to perform CW-PGD and CE-PGD directly on them.
Here we take 2 strategies.
The ﬁrst one is a naive strategy, where we minimize the sum of losses of all the models used in the ensemble.
Here, similar to attacking single models, we CW and CE loss here and perform the same number attacks.
The second strategy is still a PGD attack with a customized loss towards attacking ensemble models.
For the group of classiﬁers in the ensemble, at each PGD step, if less than half of the classiﬁers give wrong classiﬁcation, we sum up the CW losses from correct classiﬁers as the loss for the PGD attack.
If more than half of the classiﬁers give wrong classiﬁcation, then we ﬁnd the wrong prediction that appeared most frequently among classiﬁers, and denote it as label0, with its corresponding logit, logit0.
For each classiﬁer, we then ﬁnd the largest logit that is not logit0, denoted as logit1.
The loss we maximize, in the PGD attack, is the sum of “logit1 - logit0” from each classiﬁer.
Using this strategy, we perform additional (compared to attacking single models) whitebox PGD attacks on ensemble models.
For MNIST, we perform 50 repeated attacks, for CIFAR10 we perform 10.
These are also 100-step PGD attacks.
We expect more carefully designed attacks could work better on ensembles, but we delay it to future work.
For the SPSA attack (Uesato et al., 2018), we run the attack for 100 iterations with perturbation size 0.01 (for gradient estimation), Adam learning rate 0.01, stopping threshold -5.0 and 2048 samples for each gradient estimate.
For CIFAR10- (cid:96) ∞ , we use (cid:15) = 8 / 255 .
For MNIST- (cid:96) ∞ , we use (cid:15) = 0 . 3 .
E E FFECTS OF A DDING C LEAN L OSS IN A DDITION TO THE MMA L OSS
We further examine the effectiveness of adding a clean loss term to the MMA loss.
We represent
MMA trained models with the MMA loss in Eq.
(7) as MMA- d max .
In Section 2.4, we introduced
MMAC- d max models to resolve MMA- d max model’s problem of having ﬂat input space loss landscape and showed its effectiveness qualitatively.
Here we demonstrate the quantitative beneﬁt of adding the clean loss.
We observe that models trained with the MMA loss in Eq.
(7) have certain degrees of TransferGaps.
The term TransferGaps represents the difference between robust accuracy under “combined (whitebox+transfer) attacks” and under “only whitebox PGD attacks”.
In other words, it is the additional attack success rate that transfer attacks bring.
For example, OMMA-32 achieves 53.70% under whitebox PGD attacks, but achieves a lower robust accuracy at 46.31% under combined (whitebox+transfer) attacks, therefore it has a TransferGap of 7.39% (See Appendix F for full results.).
After adding the clean loss, MMA-32 reduces its TransferGap at (cid:15) = 8 / 255 to 3.02%.
This corresponds to our observation in Section 2.4 that adding clean loss makes the loss landscape more tilted, such that whitebox PGD attacks can succeed more easily.
Recall that MMA trained models are robust to gradient free attacks, as described in Section 4.3.
Therefore, robustness of MMA trained models and the TransferGaps are likely not due to gradient masking.
We also note that TransferGaps for both MNIST- (cid:96) ∞ and (cid:96) 2 cases are almost zero for the MMA trained models, indicating that TransferGaps, observed on CIFAR10 cases, are not solely due to the
MMA algorithm, data distributions (MNIST vs CIFAR10) also play an important role.
Another interesting observation is that, for MMA trained models trained on CIFAR10, adding additional clean loss results in a decrease in clean accuracy and an increase in the average robust accuracy, e.g. OMMA-32 has ClnAcc 86.11%, and AvgRobAcc 28.36%, whereas MMA-32 has
ClnAcc 84.36%, and AvgRobAcc 29.39%.
The fact that “adding additional clean loss results in a model with lower accuracy and more robustness” seems counter-intuitive.
However, it actually conﬁrms our motivation and reasoning of the additional clean loss: it makes the input space loss landscape steeper, which leads to stronger adversaries at training time, which in turn poses more emphasis on “robustness training”, instead of clean accuracy training.
F F ULL R ESULTS AND T ABLES
We present all the empirical results in Table 4 to 15.
Speciﬁcally, we show model performances under combined (whitebox+transfer) attacks in Tables 4 to 7.
This is our proxy for true robustness measure.
We show model performances under only whitebox PGD attacks in Tables 8 to 11.
We show TransferGaps in Tables 12 to 15.
In these tables, PGD-Madry et al.
models are the “secret” models downloaded from https:// github.com/MadryLab/mnist_challenge and https://github.com/MadryLab/ cifar10_challenge/ .
DDN-Rony et al.
models are downloaded from https://github.
com/jeromerony/fast_adversarial/ .
TRADES models are downloaded from https: //github.com/yaodongyu/TRADES .
For MNIST PGD-Madry
et al. models, our whitebox attacks brings the robust accuracy at (cid:15) =
0 .down to 89.79%, which is at the same level with the reported 89.62% on the website, also withrepeated random initialized PGD attacks.
For CIFAR10 PGD-Madry
et al. models, our whitebox attacks brings the robust accuracy at (cid:15) = 8 / 255 down to 44.70%, which is stronger than the reported 45.21% on the website, with 10 repeated random initialized 20-step PGD attacks.
As our PGD attacks are 100-step, this is not surprising.
We also compared TRADES with MMA trained models under (cid:96) ∞ attacks.
On MNIST, overall
TRADES outperforms MMA except that TRADES fails completely at large perturbation length of 0.4.
This may be because TRADES is trained with attacking length (cid:15)
= 0 .
3 , and therefore cannot defend larger attacks. On CIFAR10, we compare MMA-32 with TRADES as their clean accuracies are similar. Similar to the results on MNIST, we see that TRADES outperforms MMA on the attacks of lengths that are less than (cid:15) = 12 / 255 ,
but it sacriﬁces the robustness under larger attacks.
We also note that MMA training and TRADES’ idea of optimizing a calibration loss are progresses in orthogonal directions and could potentially be combined.
As we mentioned previously, DDN training can be seen as a speciﬁc instantiation of the general
MMA training idea, and the DDN-Rony et al.
models indeed performs very similar to MMA trained models when d max is set relatively low.
Therefore, we do not discuss the performance of DDN-Rony et al.
separately.
In Section 4, we have mainly discussed different phenomena under the case of CIFAR10- (cid:96) ∞ .
For
CIFAR10- (cid:96) 2 , we see very similar patterns in Tables 7, 11 and 15.
These include
• MMA training is fairly stable to d max , and achieves good robustness-accuracy trade-offs.
On the other hand, to achieve good AvgRobAcc, PGD/PGDLS trained models need to have large sacriﬁces on clean accuracies.
• Adding additional clean loss increases the robustness of the model, reduce TransferGap, at a cost of slightly reducing clean accuracy.
As a simpler datasets, different adversarial training algorithms, including MMA training, have very different behaviors on MNIST as compared to CIFAR10.
We ﬁrst look at MNIST- (cid:96) ∞ .
Similar to CIFAR10 cases, PGD training is incompetent on large (cid:15) ’s, e.g. PGD-0.4 has signiﬁcant drop on clean accuracy (to 96.64%) and PGD-0.45 fails to train.
PGDLS training, on the other hand, is able to handle large (cid:15) ’s training very well on MNIST- (cid:96) ∞ , and MMA training does not bring extra beneﬁt on top of PGDLS.
We suspect that this is due to the “easiness” of this speciﬁc task on MNIST, where ﬁnding proper (cid:15) for each individual example is not necessary, and a global scheduling of (cid:15) is enough.
We note that this phenomenon conﬁrms our understanding of adversarial training from the margin maximization perspective in Section 3.
Under the case of MNIST- (cid:96) 2 , we notice that MMA training almost does not need to sacriﬁce clean accuracy in order to get higher robustness.
All the models with d max ≥ 4 .
behaves similarly w.r.t.
both clean and robust accuracies.
Achieving 40% robust accuracy at (cid:15) = 3 . 0 seems to be the robustness limit of MMA trained models.
On the other hand, PGD/PGDLS models are able to get higher robustness at (cid:15) = 3 . 0 with robust accuracy of 44.5%, although with some sacriﬁces to clean accuracy.
This is similar to what we have observed in the case of CIFAR10.
We notice that on both MNIST- (cid:96) ∞ and MNIST- (cid:96) 2 , unlike CIFAR10 cases, PGD(LS)-ens model performs poorly in terms of robustness.
This is likely due to that PGD trained models on MNIST usually have a very sharp robustness drop when the (cid:15) used for attacking is larger than the (cid:15) used for training.
Another signiﬁcant differences between MNIST cases and CIFAR10 cases is that TransferGaps are very small for OMMA/MMA trained models on MNIST cases.
This again is likely due to that
MNIST is an “easier” dataset.
It also indicates that the TransferGap is not purely due to the MMA training algorithm, it is also largely affected by the property of datasets.
Although previous literature (Ding et al., 2019a; Zhang et al., 2019c) also discusses related topics on the difference between
MNIST and CIFAR10 w.r.t.
adversarial robustness, they do not directly explain the observed phenomena here.
We delay a thorough understanding of this topic to future work.
F. A DDITIONAL EVALUATION WITH THE CW- (cid:96) 2 ATTACK
We run the CW- (cid:96) 2 attack (Carlini & Wagner, 2017) on the ﬁrst 1000 test examples for models trained with (cid:96) 2 attacks. This gives the minimum (cid:96) 2 norm a perturbation needs to change the prediction. For each model, we show the clean accuracy, also mean, median, 25th percentile and 75th percentile of these minimum distances in Tables 2 and 3 below. Looking at CW- (cid:96) 2 results, we have very similar observations as compared to the observations we made based on robust accuracies (under PGD attacks) at different (cid:15) ’s
and the average robust accuracy.
On MNIST, for MMA trained models as d max increases from 2.0 to 6.0, the mean minimum distance goes from 2.15 to 2.76, with the clean accuracy only drops to 97.7% from 99.3%.
On the contrary, when we perform PGD/PGDLS with a larger (cid:15) , e.g. (cid:15) = 4 . 0 , the clean accuracy drops signiﬁcantly to 91.8%, with the mean minimum distance increased to 2.53.
Therefore MMA training achieves better performance on both clean accuracy and robustness under the MNIST- (cid:96) 2 case.
On CIFAR10, we observed that MMA training is fairly stable to d max , and achieves good robustnessaccuracy trade-offs.
With carefully chosen (cid:15) value, PGD-1.0/PGDLS-1.0 models can also achieve similar robustness-accuracy tradeoffs as compared to MMA training.
However, when we perform
PGD training with a larger (cid:15) , the clean accuracy drops signiﬁcantly.
[Table] [Table] [Table] [Table]
0.22 PGD77.
0.70 PGD68.
1.54 PGD61.
2.42 PGD10.
8.68 PGD10.
10.00 PGD10.
PGDLS89.
0.05 PGDLS85.
0.21 PGDLS79.
0.65 PGDLS70.
1.31 PGDLS-
[Table] [Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
OMMA-
[Table]
-sd 88.
0.13 OMMA-12-sd 87.
0.14 OMMA-20-sd 87.
0.23 OMMA-20-sd 87.
0.25 OMMA-32-sd 86.
0.45 OMMA-32-sd 86.
PGD-ens 87. 0.39 PGDLS-ens 76. PGD-Madry et al.
TRADES 84.
[Table]
[Table] [Table]
[Table]
PGD77.
0.72 PGD68.
1.57 PGD61.
2.73 PGD10.
8.81 PGD10.
10.00 PGD10.
PGDLS89.
0.05 PGDLS85.
0.21 PGDLS79.
0.67 PGDLS70.
1.34 PGDLS65.
[Table]
MMA-
[Table]
OMMA-
[Table]
[Table]
[Table]
1.03 OMMA-12-sd 87.
1.29 OMMA-20-sd 87.
[Table]
5.60 OMMA-20-sd 87.
[Table]
[Table]
.76 OMMA-32-sd 86.
[Table]
[Table] 76 OMMA-32-sd 86. [Table]
[Table]
[Table]
PGD-ens 87.
[Table]
[Table]
[Table]
[Table]
[Table] [Table] 52 PGDLS-ens 76. [Table]
[Table]
[Table]
[Table]
PGD-Madry et al.
[Table]
TRADES 84.
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
DDN
[Table]
Rony et al.
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
0.02 PGD- - 0.
0.03 PGD- - 0.
0.31 PGD- - 0.
0.13 PGD- - 0.
0.00 PGD- - 0.
PGDLS- - 0.
0.00 PGDLS- - 0. 0.00 PGDLS- - 0.
0.02 PGDLS- - 0.
0.03 PGDLS- - 0.
0.13 PGDLS- - 0.
0.65 PGDLS- - 1.
1.92 PGDLS- - 2.
MMA-12-sd0.
0.35 MMA-12-sd0.
0.06 MMA-20-sd2.
1.68 MMA-20-sd4.
3.58 MMA-32-sd7.
8.26 MMA-32-sd4.
OMMA-12-sd3.
0.90 OMMA-12-sd4.
1.15 OMMA-20-sd8.
5.37 OMMA-20-sd6.
3.51 OMMA-32-sd10.
9.31 OMMA-32-sd10.
PGD-ens0.
0.13 PGDLS-ens1.
PGD-Madry et al.
TRADES 0.
[Table] [Table]
.07 MMA-2.0-sd0.
[Table]
[Table]
[Table]
36 MMA- [Table] 0-sd - [Table]
[Table]
[Table]
[Table] [Table] [Table] 79 MMA-3.0-sd2.
[Table]
3.77 MMA-3.0-sd2.
[Table]
OMMA-1.0-sd0.
[Table]
[Table]
[Table]
[Table]
[Table]
21 OMMA [Table] [Table] sd -
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] 10 OMMA [Table] sd -
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
69 OMMA- [Table] [Table] sd -
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
56 OMMA- [Table] -sd - [Table]
[Table]
[Table]
[Table]
11.05 OMMA-3.0-sd4.
PGD-ens0.
1.37 PGDLS-ens1.
DDN-Rony et al.