S ENSIBLE A DVERSARIAL L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
The trade-off between robustness and standard accuracy has been consistently reported in the machine learning literature.
Although the problem has been widely studied to understand and explain this trade-off, the problem seems to remain as an open problem.
In this paper, motivated by the fact that the high dimensional distribution is poorly represented by limited data samples, we introduce sensible adversarial learning and demonstrate the synergistic effect between pursuits of natural accuracy and robustness.
Speciﬁcally, we deﬁne a sensible adversary which is useful for learning a defense model and keeping a high natural accuracy simultaneously.
We theoretically establish that the Bayes rule is the most robust multi-class classiﬁer with the 0-1 loss under sensible adversarial learning.
We propose a novel and efﬁcient algorithm that trains a robust model with sensible adversarial examples, without a signiﬁcant drop in natural accuracy.
Our model on CIFAR10 yields state-of-the-art results against various attacks with perturbations restricted to (cid:96) ∞ with (cid:15) = 8 / 255 , e.g., the robust accuracy 57.23% against
PGD attacks as well as the natural accuracy 91.51%.
I NTRODUCTION
With many impressive successes of deep learning, there are a multitude of applications of deep neural networks (DNNs) that permeate in our everyday life.
As DNNs are applied in securitycritical systems such as malware detection, face identiﬁcation, and autonomous driving, robustness of DNNs against adversarial attacks, i.e., the intently perturbed inputs to fool the system, has become an important research topic (Szegedy et al., 2013; Papernot et al., 2016; Biggio et al., 2013).
One of the most widely studied classes of adversarial perturbations is (cid:96) p -norm constrained adversarial perturbations (Szegedy et al., 2013).
Madry et al.
(2017) formalize the adversarial learning against this class of perturbations as a minimization problem of adversarial risk deﬁned in a following way.
Let ( X, y ) ∈
X
× Y be from some unknown distribution P X,Y .
Given a loss function (cid:96) :
Y × Y → R and a constraint constant (cid:15) > 0 , the adversarial robust risk is
[Equation]
Many adversarial learning methods can be interpreted as empirical minimization of (1) (Goodfellow et al., 2014; Kurakin et al., 2016b; Ruitong Huang & Szepesvari, 2015; Madry et al., 2017).
For this optimization problem, Madry et al.
(2017) propose to train a robust model with the augmented data generated by the projected gradient descent method (PGD).
On this adversarial training, they make two important observations.
First, it costs natural accuracy.
A network trained with adversarial examples tends to have a lower natural accuracy than a naturally trained network.
This trade-off is observed even with a small training (cid:15) .
Second, the adversarial training requires a larger model capacity than the natural training does.
If the model capacity is only sufﬁcient for the natural learning, the adversarial training can converge to a constant function.
For a large (cid:15) , the optimization problem of (1) itself may pose the trade-off.
For instance, Tsipras et al.
(2018) show an example of an inherent tension between pursuits of accuracy and robustness when (cid:15) is large enough to change the true class.
For a smaller (cid:15) , however, the formulation in (1) does not explicitly pose any conﬂict between the pursuit of robustness and accuracy.
Note that R rob ( f ) is an upper bound of the standard risk of f .
A perfectly robust model f
with R rob ( f ) = 0 is also perfectly accurate for natural learning.
If the perfect classiﬁer exists in a given model class, the trade-off may be caused by the large sample complexity of adversarially robust generalization
(Schmidt et al., 2018;
Yin et al., 2018; Stutz et al., 2019).
Without sufﬁciently large amount of data, the empirical minimization of (1) may result in a large standard risk by converging to a model of a poor robust risk.
On the other hand, if robust learning converges to a constant function, it cannot achieve natural accuracy.
In this sense, to resolve the trade-off problem, we may need to deal with the increased requirement on the model capacity.
In this paper, we propose a novel framework, called sensible adversary , in order to overcome the trade-off between natural accuracy and robustness.
In particular, we restrict adversarial perturbations not to cross the Bayes decision boundary besides the (cid:15) -ball constraint, so that the perturbation ball is adaptively modiﬁed for every single data point.
Our main contributions are:
• Under the framework of sensible adversary , the pursuit of robustness and accuracy given an enough model capacity can align with each other, i e., there is no trade-off.
We theoretically establish the Bayes rule is most robust against the sensible adversary.
If the Bayes decision boundary can be far from data manifolds at least by (cid:15) , our pursuit of sensible robustness does not cost any adversarial robust risk.
• We propose an efﬁcient algorithm for sensible adversarial training , which utilizes sensible adversaries in the absence of the true Bayes rule.
This sensible adversarial training enjoys robustness without a signiﬁcant drop of natural accuracy.
Furthermore, the algorithm is not sensitive to the model capacity.
When insufﬁcient model capacity is given, our algorithm does not collapse to a constant function.
Instead, it trains a model as robust as possible.
• We experimentally demonstrate that sensible adversarial training enables to stably learn a robust and accurate model.
In particular, on CIFAR10, we achieve 91.51% natural test accuracy and 57.23% robust test accuracy against (cid:96) ∞ PGD attacks constrained to (cid:15) = 8 / 255 .
To the best of our knowledge, there is no approaches known to achieve natural accuracy more
than 90% , while achieving more than 55% of robust accuracy against PGD attacks of (cid:15) = 8 / 255 .
Moreover, no previous approaches pursuing robustness against this attack achieved the natural accuracy more than .
1. R ELATED WORK
Madry et al.
(2017) formalize adversarial learning as a mini-max problem given perturbation restriction, and theoretically and empirically established the feasibility of the optimization.
Our sensible adversary redeﬁnes the set of perturbation in the inner maximization problem on which their theoretical result is directly applicable.
Tsipras et al.
(2018) investigate the possible source of robust trade-off.
The key idea is that when there are features that are useful for natural classiﬁcation but vulnerable to adversarial perturbations, a robust model would abandon these features because otherwise all of these features can adversarially move to promote incorrect prediction.
Our work explores the possibility of learning a robust model while not allowing such collective adversarial migration.
While a class change by adversarial examples typically has been prevented by using a small (cid:15) , Suggala et al.
(2018) explicitly ignore an adversarial perturbation that crosses the decision boundary of a Bayes rule.
Zhang et al.
(2019) also investigate the Bayes decision boundary to resolve the trade-off problem.
They search for a model f having a small weighted sum of the natural risk and a probability that an adversarial example can cross the decision boundary of f .
Gilmer et al.
(2018) show that in high dimensional setting, even small test error can imply the existence of adversarial examples for most of data points.
Our effort to prioritize natural accuracy to ﬁnd a robust model is consistent to the view in Gilmer et al.
More related work will be presented in Appendix A.
1. N OTATION
Let F denote the class of functions represented by DNNs with a ﬁxed architecture.
An optimal robust model w.r.t.
P X,Y is denoted by f rob =
arg min f ∈F R rob ( f ) .
Denote the standard risk w.r.t P X,Y by R std ( f ) =
E X,Y [ (cid:96) ( f ( X ) , y )] and its optimal natural model by f std = arg min f ∈F
R std ( f ) .
Let ˜ P X,Y =
P X,Y | ˜
X×Y denote a restricted distribution of P X,Y on a subset ˜ X
× Y ⊂ X
× Y .
Likewise, denote the standard risk w.r.t ˜ P X,Y by ˜ R std ( f ) and the robust risk w.r.t ˜ P X,Y by ˜ R rob ( f ) .
For a set A , the (cid:15) -neighborhood in (cid:96) p -norm is deﬁned as
B ( A, (cid:15) ) =
{ y |(cid:107)
y − x (cid:107) p ≤ (cid:15), x ∈ A } , and the interior is denoted by int ( A ) .
Denote the (cid:15) - neighborhood of the decision boundary of f by DB ( f, (cid:15) )
= { x (cid:12)(cid:12) ∃ x (cid:48)
∈ B ( x, (cid:15) )
s.t.
f ( x ) (cid:54) =
f ( x (cid:48) ) } .
Let ˆ p f,y ( x ) denote the predicted probability of the label of x being y by a model f .
A DVERSARIAL LEARNING MAY HELP STANDARD LEARNING
In this section, we use a toy example to investigate the synergistic effect between the pursuit of robustness and natural accuracy.
Example: We can easily ﬁnd an example of trade-off wherever an (cid:15) -ball attack can cross the manifold boundary between classes.
For example, consider a two-dimensional random vector
X =
( X 1 , X 2 ) on X = (0 , 1) ×
(0 , 1) with a binary class Y ∼ Ber ( p ) ,
where p > 0 .
Let the conditional distribution be ( X 1 , X 2 ) | Y = 0 ∼ Unif ((0 , 12 ) × (0 , 1)) and ( X 1 , X 2 ) | Y = 1 ∼
Unif (( 12 , 1) × (0 , 1)) .
Then the Bayes Rule is f B ( x ) = sign ( x 1 − 0 . 5) , and it is a perfect classiﬁer, in that R std ( f B ) = 0 with (cid:96) as the 0-1 loss.
If p > 0 . 5 , the robust classiﬁer against (cid:15) -ball attacks is f rob ( x ) = sign ( x 1 − (0 .
5 − (cid:15) )) .
Its decision boundary is deviated by − (cid:15) from that of the Bayes rule, and this deviation costs natural accuracy by R std ( f rob ) − R std ( f ) =
(1 − p ) (cid:15) .
However, when the data points reside in a high-dimensional space, the samples are too sparse to represent the underlying true distribution.
To take this phenomenon into account, consider a distribution ˜ P X,Y
= P X,Y | ˜ X×Y on a subset
˜ X
× Y ⊂ X
× Y .
Assume that we only observe data generated from ˜ P X,Y , and the training and test sets do not provide any information on ˜ X
c × Y .
Now by applying this support restriction to the example above, we demonstrate how adversarial learning dramatically changes from harming to beneﬁting natural accuracy.
The Cheese hole distribution: For the above example, we restrict the support on ˜ X
= ∪ 3 i =1 ∪ 3 j =
H ij , where H ij =
(( α 2 , 3 α 2 )+2 α
( i − 1))
× (( α 2 , 3 α 2 )
+2 α
( j − 1)) , α = 16 , and j = 1 , 2 , 3 .
Therefore the sampling support ˜ X comprises of nine small squares equally spaced by α .
This is illustrated in
Figure 1 (a).
Among all classiﬁers which predict the same as f B on ˜ X , the worst one, denoted by ˜ f B ∗ , predicts exact opposite on x ∈ ˜
X c as illustrated in Figure 1 (b).
For this worst case, the true standard risk w.r.t P X,Y is 34 , although the standard risk w.r.t.
˜ P X,Y is zero.
We argue that pursuing robustness can mitigate this discrepancy.
Among minimizers of ˜ R rob , let ˜ f ∗ rob be the worst classiﬁer in that it predicts incorrectly outside ˜ X as depicted in Figure 1 (c).
On
B ( ˜ X , (cid:15) ) , ˜ f ∗ rob should correctly classify, except on A (cid:15) = { ( x 1 , x 2 ) | 0 .
5 − (cid:15) ≤ x 1 ≤ 0 .
5 } ∩ B ( ˜ X , (cid:15) ) .
In this situation, the inaccuracy of ˜ f ∗ rob is compensated by its increased accuracy on B ( ˜ X , (cid:15) )
\ A (cid:15) .
Moreover, the composition of R rob ( ˜ f B ∗ ) is particularly interesting, and it is easy to show that
[Equation]
where 3/4 is from the standard inaccuracy on ˜ X c .
The major part of R rob ( ˜ f B ∗ ) comes from the inaccuracy of ˜ f B ∗ outside of the sampling support ˜ X .
Therefore by simply reducing this inaccuracy on ˜ X c , we can lower the robust risk by 3 / 4 .
Another interesting observation is that
B (cid:15) = { ( x 1 , x 2 ) | 0 .
5 − (cid:15) < x 1 < 0 .
5 + (cid:15) } is the only area where the Bayes rule f B has its robust risk.
Note that R rob ( f B ) =
P ( X ∈ B (cid:15) , f B ( X ) = Y ) = 2 (cid:15) which is greater than the optimal robust risk.
The reason why f B is not most robust is simply because it is accurate on B (cid:15) .
This prevents f B from being a robust function under the current adversarial robustness framework.
These observations have two implications.
First, the robust pursuit can help to increase natural accuracy outside of ˜ X where the samples are poorly representative.
Second, by accurately correcting the model outside of ˜ X in a standard sense, the robust risk can be signiﬁcantly decreased.
This is where the pursuit of robustness and accuracy coincides.
Toward respecting the original data structure: What if we regard the adversarial robust risk near the class border as reasonable gullibility ?
This corresponds to pursuit of robustness only against adversarial examples which do not cross between the class manifolds, i.e., as long as this does not harm the natural accuracy on ˜ X .
We call this robustness as sensible robustness , which respects the structure represented by data.
In this example, the sensible robustness can increase both robustness and natural accuracy.
Let ˜ f
s ∗ rob denote the worst case sensibly robust classiﬁer w.r.t ˜ P X,Y .
The decision boundary of ˜ f
s ∗ should not deviate to the left on B ( ˜ X , (cid:15) )
no matter how large (cid:15) is, as depicted in Figure 1 (d).
Note that when (cid:15) ≥ α/ 2 , ˜ f ∗ rob is the unique minimizer of ˜ R rob ( f ) as
B ( ˜ X , (cid:15) ) covers X .
S ENSIBLE ADVERSARIAL ROBUSTNESS
In this section, we introduce a sensible adversary framework.
Consider a general multi-class case with the 0-1 loss, where Y =
[ K ] .
Assume the model capacity is enough so that the Bayes rule f B ∈ F .
We consider (cid:96) p -norm constrained adversarial attacks, where p ∈ { 0 , 1 , ..., ∞} .
Deﬁnition 1.
(sensible adversarial example)
For a classiﬁer f , let S x,(cid:15)
( f ) = { z ∈ X|(cid:107)
z − x (cid:107) p ≤ (cid:15), f ( z ) = y } .
Then the sensible adversarial example of ( x, y ) w.r.t f is deﬁned as
[Equation]
Deﬁnition 2.
(sensible robustness)
Let the sensible adversarial loss be (cid:96) srob,(cid:15)
[Equation] where ˜ x is a sensible adversarial example as deﬁned above.
Then the sensible robust risk of a model f is deﬁned by
[Equation]
We call its minimizer as a sensibly robust model w.r.t P X,Y and denote by f srob , i.e.,
[Equation]
Remark 1.
Intuitively, a sensible adversarial example is an adversarial example restricted not to cross the decision boundary of the Bayes rule.
In addition, a sensible adversary does not perturb a data point that the Bayes rule incorrectly classiﬁes.
Therefore, it is natural to expect that pursuing sensible robustness would not cost natural accuracy, and the following theorem conﬁrms it.
Theorem 1.
Let R ∗ std denote the minimum standard risk which is R std ( f B ) .
Then we have
R srob ( f B ) =
R ∗ std .
Furthermore, f B is the unique minimizer of R srob ( f ) among f ∈ F .
Theorem 2.
Let S X,(cid:15) be an (cid:15) -ball centered at X .
Then for any f ∈ F and for any set A ⊂
X \ DB ( f B , (cid:15) ) ,
[Equation]
Remark 2.
According to the Theorem 1, a sensibly robust model w.r.t P X,Y is the Bayes rule, i.e., f s rob = f B .
This sensible robustness costs adversarial robustness since f B may have a larger [Table] adversarial robust risk compared with f rob , a direct minimizer of (1).
However, Theorem 2 shows f B is equally or even more robust than f rob except on a certain area.
In particular, Theoremshows that f B is the most robust model except on DB ( f B , (cid:15) ) .
Therefore, f B is most robust almost everywhere if the decision boundary of f B can lie outside of B ( X , (cid:15) ) , so that X \ DB ( f B , (cid:15) )
=
X .
For example, this happens when each class has its own support apart from each other by at least 2 (cid:15) .
The next theorem shows that even when we only have data from ˜ P X,Y restricted on
˜ X
× Y , we can ﬁnd f srob , the optimal function w.r.t.
P X,Y .
Theorem 3.
Let A (cid:15) =
(cid:8) f ∈ F (cid:12)(cid:12)
˜ P (cid:0) f ( x ) =
f B ( x ) , ∀ x ∈ S X,(cid:15)
( f B ) (cid:1) = 1 (cid:9) and ˜ R srob ( f ) =
E ˜ P X,Y [ (cid:96) srob,(cid:15) ( f, X, Y )] .
Then, for any (cid:15) > 0 , ˜ R srob ( f ) is only minimized by any f ∈ A (cid:15) .
Furthermore, if B ( ˜ X , (cid:15) )
⊃ X , f B is the unique minimizer of ˜ R srob ( f ) . Remark 3.
It is interesting to compare our work with that of Suggala et al.
We notice there are some critical differences between these two works.
Given Y ∈ {− 1 , 1 } , Suggala et al. (2018) deﬁne an adversarial risk as [Equation] Unlike our deﬁnition of sensible adversary, if g is a Bayes rule f B , their adversary tries to increase the loss w.r.t.
not y but a deterministic function of x .
Therefore, for making f B being an optimal robust model, their objective should always have an additional term, e.g., R nat ( f ) + λR adv ( f ) for 0 < λ < ∞ because an f s.t. f
( x ) (cid:54) =
g ( x ) w.p. 1 can minimize R
adv ( f ) .
Unlike their approach, our sensible adversarial risk (3) can alone be optimized making the Bayes rule as the optimal model.
A LGORITHM
A transition from theory to algorithm poses two main challenges.
First, the 0-1 loss function in the theory is hard to optimize.
Therefore, as a common practice, we use the cross-entropy loss.
Second, f B on the entire space is practically unavailable.
We note that a model that performs well on natural data can be a nice approximation of f B on the restricted support
˜ X
× Y .
Therefore, we generalize sensible adversary in (2) to utilize a general loss function and a reference model that substitutes f B .
Deﬁnition 3.
(generalized sensible adversarial example) Consider a loss function (cid:96) .
For a classiﬁer f and c ∈ (0 , 1] , let S x,(cid:15)
( f ) = { z ∈ X|(cid:107)
z − x (cid:107) p ≤ (cid:15), (cid:96) ( f ( z ) , y ) ≤ log 1 c } ∪ { x } .
Then given a reference model f r , the sensible adversarial example of f for ( x, y ) is deﬁned as
[Equation]
otherwise [Equation] If (cid:96) is the cross-entropy loss, the condition (cid:96) ( f r ( x ) , y ) ≤ log 1 c is equivalent to ˆ p f r ,y ( x ) ≥ c .
In binary case with c = 0 . 5 , this requires the perturbed examples not to cross the decision boundary of f r , and for general c , not to reach to a vicinity of the boundary. [Table] [Table]
In our algorithm we set f r as a naturally trained model or a current model.
Given a f r , the implementation of sensible adversarial attacks is straightforward.
For a correctly classiﬁed natural example by f r , we add perturbations in the similar way to the PGD method (Madry et al., 2017).
The difference is that during the K-step of PGD iterations, once the loss of a currently generated example exceeds log 1 c , we reverse it back to the previous step and break the iteration.
This requires no additional forward- or backward-propagation, compared with the PGD method.
The proposed algorithm is in
Algorithm 1 for the (cid:96) ∞ -norm and in Algorithm 2 in Appendix C for the other (cid:96) p -norms.
The deﬁnition of the sensible adversary in (4) divides the data into three subsets: A f r = { x | f r ( x ) (cid:54) =
y or S x,(cid:15) ( f r ) = { x }} , B f r = { x | S x,(cid:15) ( f r ) (cid:54) =
S x,(cid:15) , S x,(cid:15) ( f r ) (cid:54) = { x }} , and C f r = X \ ( A f ∪ B f ) .
Therefore, our algorithms generate a sensible adversarial example ˜ x s | f r = f in three different ways: (1) ˜ x s = x for x ∈
A f , (2) ˜ x s (cid:54) =
˜ x p for x ∈ B f , and (3) ˜ x s = ˜ x p for x ∈ C f , where ˜ x p is a full K-step PGD attack.
Therefore, sensible adversarial training inherently involves a set selection mechanism.
The sensible adversarial loss (cid:96) s ( f, x, y ) can be written as
[Equation]
When c ≥ 0 .
5 , this optimization problem has a strong motivation to increase | C f | , i.e., the number of sensible adversarial attacks that are identical to full PGD attacks.
Because A f = { x | (cid:96) ( f ( x ) , y ) > log 1 c } for c ≥ 0 .
5 , the loss on the natural stage
A f is always greater than the loss of the sensibly reversed stage B f and the full PGD stage C f .
Furthermore, the loss for x ∈ B f is always approximately log 1 c by adaptive sensible perturbations.
Therefore, paradoxically, the smallest loss is only achievable by full PGD examples.
In other words, sensible adversarial training penalizes when x (cid:54)∈ C f .
We note that the optimization problem has a smooth landscape; Although the data points may jump around between A f , B f , and C f , there is no obvious discontinuity in the loss.
Therefore, during the training, (cid:96) s ( f, x, y ) smoothly slides down, making both (cid:96) ( f, x, y ) and (cid:96) ( f, ˜ x p , y ) smaller than log 1 c .
This enables to learn a naturally and adversarially accurate model.
More discussion on the stability of our algorithms are presented in Appendix D.
E XPERIMENTS 5. E XPERIMENT 1: R OBUSTNESS AGAINST (cid:15) - BALL ATTACKS
{+CIFAR10 We train WideResNet-34-
 (He et al., 2016) with sensible adversarial examples of (cid:15) = 8 / 255 and c = 0 . 7 with the step number K = 10 and size η 1 = (cid:15)/ 5 .+}
We attack our model with various white-box attacks with perturbations restricted to (cid:96) ∞ with (cid:15) = 8 / 255 .
We compare the performance with TRADE using the same architecture (Zhang et al., 2019), which is know to be robust and accurate.
The result is in Table 1.
Our model achieves 91.51% natural accuracy.
This is 3.7% drop in natural accuracy from 95.2%, which is an accuracy that a naturally trained model can achieve (Madry et al., 2017).
With this architecture, Madry et al.
(2017) achieve 47.04% robust
[Table]
accuracy against PGD20 attacks and 87.3% natural accuracy.
As a black-box attack, we try transfer attacks between the TRADE and SENSE model by the (cid:96) ∞ based PGD and MIFGSM, which are known to be effective for transfer attack ( (cid:15) = 8 / 255) .
We obtain adversarial examples by applying
PGD and MIFGSM on a generating model, and then use the examples to attack a defense model.
The result is in Table 2.
Overall we observe that our model outperforms both the TRADE and Madry model.
In particular, sensible adversary achieves more than 55% of robust accuracy against PGD attacks of (cid:15)
= 8 / 255 .
This performance is consistent to the test margin distribution in Figure 8, and this is discussed in detail in Appendix E.
MNIST We consider a CNN model with three convolutional layers followed by a fully connected linear layer, which is the same architecture in (Zhang et al., 2019).
We train an MNIST model with sensible adversarial examples of (cid:15) = 0 .
3 and c = 0 .
5 with the step number K = 10 and size η 1 = 0 .
The robust test result in Table 3 shows the comparable performance of our model.
The step size of the PGD500 attack is 0.01, and the serenity check on the PGD attack is in Appendix H.
5. E XPERIMENT 2: S ENSITIVITY ANALYSIS
MNIST:
We perform the sensibility analysis to understand the effect of c and the model capacity for two reasons.
First, the sensible training prevents full PGD perturbations on an example until the loss on it becomes less than log 1 c , which could hardly happen for a model with a small capacity.
Second, as Madry et al.
(2017) point out, when the model capacity is insufﬁcient for adversarial learning, the model collapses to a constant function.
We are interested in a range of c that keeps sensible learning from collapsing.
Therefore, we consider a sequence of CNNs of the increasing number of kernels similar to Madry et al.
(2017), where we denote the capacity by q ∈ { 1 , 2 , 3 , 4 , 5 } .
The details of the model architectures are presented in Appendix H. When we train them with natural examples, the networks of capacity 1 and 2 achieve about 95% and 97% accuracy, whereas the networks of the other capacities achieve more than 99%.
When trained with regular PGD examples, the networks with capacity 1,2, and 3 collapse.
Therefore, capacity 3 is enough only for natural learning, and capacities 1 and 2 are possibly insufﬁcient even for natural learning.
For each capacity q , we train the models with sensible adversarial examples on MNIST with the hyperparameters c ∈ { 0 , 0 . 1 , · · · , 0 . 9 , 1 } .
Note that sensible adversarial training is identical to natural training when c = 1 , and to adversarial learning without any perturbations for incorrectly classiﬁed natural examples when c = 0 .
Figure 2 shows the natural and robust accuracy against
PGD-40 attacks with varying c for each capacity.
In general, the natural accuracy tends to increase as c increases while robustness decreases.
For capacities 3, 4 and 5, the accuracies are almost insensitive to varying c .
On the other hand, the tendency is most distinct in capacity 1.
In this case, the network obtains best robustness when c
= 0 .
5 , which is the least loss bound among c ≥ 0 .
5 , the range of c with stable learning property.
Even for capacity 1, however, the sensible learning does not collapse except when c = 0 .
We also see that c = 0 .
5 is least sensitive to the model capacity.
Note that capacities 2 and 3 do not collapse even with c = 0 .
When c = 0 , the only difference of the sensible adversarial learning from the regular PGD training is that the sensible learning requires robustness only for the correctly classiﬁed natural examples.
When c = 0 , at the convergence of the models of capacities 1 and 2, about 5% of the data points are allowed to be free from the perturbation.
By paying only the 5% of robust training accuracy, the sensible learning avoids collapsing and obtains about 80% of robust accuracy and 90% of natural accuracy.
Intuitively, an insufﬁcient model capacity or locally close class manifolds can make a virtual decision boundary that is inevitable to keep a nice natural performance.
In the algorithm, the sensible reversion prevents adversarial examples from crossing this boundary.
This effectively reduces the requirement of the model capacity posed by the regular adversarial learning.
The sensible reversion also allows a robust model to have larger margins than the PGD trained model for the majority of the dataset as in Figure 3.
The PGD trained model has majority adversarial margins as positive.
Instead, it has much smaller natural margins than the naturally trained model.
The SENSE model with c = 0 .
5 has comparably large adversarial and natural margins.
Instead, the number of data points with negative adversarial margins is larger than that of the PGD trained model.
As c increases to 0 .
9 , this model has much smaller and more negative adversarial margins.
On the other hand, its two types of margins are generally even larger.
This phenomenon is consistent to the decreasing robustness in Figure 2 for capacity 5.
In general, for a ﬁxed capacity, increasing c increases the natural and adversarial margins of the majority of the data, while it also increases the portion of data of negative adversarial margins.
The cost for this highly conﬁdent prediction is the robustness near the decision boundary at convergence, i.e., the portion of the data points in the natural and sensibly reversed stage.
When we consider the margin on the training set, we ﬁnd that there is a linear relationship between the portion of the data points having negative adversarial margins and the test accuracy.
In practice, such a portion can be an indicator about the robustness of the model or the sufﬁciency of the model capacity, without directly testing the model.
CIFAR: We train [-WideResNet-34-
-]train WideResNet-34-10 with several different c values.
Except that for c = 0 .we stopped learning at 120 epoch, others all have trained for 300 epochs. We report the adversarial accuracy against and PGD100 attacks with a step size η 1 = 2 / 255 for the ﬁrst 2000 test examples. For random 20 restarts, we count only the worst case.
The step number 100 and step size 2/255 are justiﬁed by the plot in Figure 11 in Appendix H. The results are in Table 4.
[Table]
[Table]
Interestingly, while the natural accuracy positively correlated to the c value, the robustness does not show clear negative correlation to c .
Rather, as c become closer to 0.7, more robust result the model shows.
As we see that c = 0 . 8 has better robustness than c ≤ 0 .
5 , in CIFAR, the main reason of the observed trade-off could attribute to the inﬂuential adversarial perturbations.
As for the MNIST case, we also try to train a robust model with a small capacity. We intentionally choose the CNNs model that is used in Experiment 1 for MNIST. This model can be not enough for adversarial training on CIFAR.
We report the results in Table 5.
Given the serious lack of model capacity, the decrease in natural accuracy of SENSE models compared with the naturally trained model is not serious, while achieving better robust accuracy numbers.
Note that none of our models collapse, and therefore the possibility of sensible adversarial training is not sensitive to the choice of c even in the lack of model capacity.
C ONCLUDING REMARKS
In this paper, we proposed a sensible adversary which is useful for learning a defense model, keeping a high natural accuracy simultaneously.
We theoretically establish that the Bayes rule is most robust under the framework of sensible adversarial learning.
Our learning algorithm is efﬁcient and stable, and not sensitive to the choice of the main hyperparameter c .
Also, c has a clear meaning as the lowest prediction-probability bound.
Our empirical experiments yield state-of-the-art results of adversarial learning on the CIFAR10 and MNIST datasets.
In addition, we showed that the sensible approach can effectively deal with the lack of model capacity.
This is because by paying a robust accuracy on a certain area, the algorithm protects the model from being collapsed by inﬂuential adversarial examples.
Furthermore, the sensible adversarial learning trains a model to have high prediction margins on both natural and adversarial examples.
We now mention several future directions for research on sensible adversary.
One remaining theoretical problem is to develop generalization error bounds for sensible adversary learning, so that we can theoretically justify our empirical performance.
In fact, in this work, we did not tackle the lack of sample size.
In particular, as our algorithm tends to produce a model with large natural and adversarial prediction margins, in the lack of sample size, it is not clear if this large margins are always beneﬁcial.
Therefore, there is much remaining work to be done to theoretically understand the high margin tendency of the models trained with sensible adversarial examples with relation to generalization.
Appendices A A DDITIONAL RELATED WORK
Bubeck et al.
(2018) conjecture that learning a robust model is information theoretically possible but computa- tionally intractable.
They introduce an example which is not robustly learnable in polynomial time.
Our view is consistent with Bubeck et al.
(2018) in that a robust model exists, but we search for an efﬁcient algorithm to estimate a robust model in a reasonable time.
Su et al.
compared various naturally trained models on ImageNet, and found that the trade-off varies among different model architectures.
Also, they empirically discovered that more accurate models tend to be less robust when the models are trained with natural examples.
Stutz et al.
(2019) demonstrate that given a large training set, adversarial training can produce a robust model that is as accurate as a naturally trained model.
Our work to maximize the synergistic effect between natural and adversarial accuracy is consistent to their demonstration.
Also, they show that most of PGD attacks are off- manifold of the original data, and by on-manifold adversarial training, the natural accuracy can be improved.
If we see the restricted space ˜ X as the data manifold, and ˜ X c as off-manifold, our sensible framework aligns with their view, given the data manifolds of different classes are separated by at least 2 (cid:15) .
Kurakin et al.
(2016b) suggest adversarial learning that trains with data randomly divided into two parts, a natural and adversarial set.
We divide data in a data adaptive way into three parts including a sensibly reversed adversarial set.
The relationship between their approach and our algorithm is discussed more in Appendix F.
B P ROOFS OF T HEOREMS
To save the space, we use η ( k | x ) to denote P ( Y = k
| X = x ) .
Theorem 1.
(Restated)
Let R ∗ std denote the minimum standard risk which is R std ( f B ) .
Then we have
R srob ( f B ) =
R ∗ std .
Furthermore, f B is the unique minimizer of R srob ( f ) among f ∈ F .
Proof.
[Equation]
B ( ˜ X ) (cid:54) =
Y )
=
P ( f B ( X ) (cid:54) =
Y )
+ P ( f B ( X )
=
Y, and ∃ x (cid:48) ∈ S X,(cid:15) ( f B ) s.t.
f B ( x (cid:48) )
(cid:54) =
Y )
=
P ( f B ( X ) (cid:54) =
Y ) + 0 , by the deﬁnition of S X,(cid:15)
( f B ) =
R std
( f B ) =
R ∗ std .
It is obvious that f B is a minimizer of R srob ( f ) because R ∗ std is a lower bound of R srob ( f ) for any f ∈ F .
Now we show f B is the unique minimizer of R srob ( f ) .
R srob
[Equation]
[Equation] [Equation]
) =
[Equation] [Equation]
X
[Equation]
(cid:15)
[Equation]
f
[Equation]
.t.
[Equation]
( x (cid:48)
[Equation]
) =
[Equation]
k
[Equation] [Equation] [Equation]
[Equation]
The last equality is by 1 f B ( x ) (cid:54) =
k,f
[Equation]
x )
[Equation]
= k =
[Equation]
1 − 1 f B ( x )= k )1 f
[Equation]
x
[Equation]
(cid:54) = k .
The ﬁrst term R std ( f ) is uniquely minimized by the Bayes rule f B .
The second term is always non-negative, and is zero when f = f B .
Therefore, R srob ( f ) is uniquely minimized by f B .
Theorem 2.
(Restated)
Let S X,(cid:15) be an (cid:15) -ball centered at X .
Then for any f ∈ F and for any set A ⊂
X \ DB ( f B , (cid:15) ) ,
[Equation]
Proof.
For any set A ⊂ X \ DB ( f B , (cid:15) ) , P ( ∃ x (cid:48) ∈ S X,(cid:15)
s.t.
f B ( x (cid:48) )
(cid:54) =
Y, X ∈ A )
=
P ( f B ( X ) (cid:54) =
Y, X ∈ A ) .
Note that on any subset B ⊂ X , the Bayes rule has the least error probability.
Therefore, P ( f B ( X ) (cid:54) =
Y, X ∈
A ) ≤ P ( f ( X ) (cid:54) =
Y, X ∈ A ) ≤ P ( ∃ x (cid:48)
∈ S X,(cid:15)
s.t.
f ( x (cid:48) ) (cid:54) =
Y, X ∈ A ) .
The last inequality is trivial because if f ( X ) (cid:54) =
Y ⇒ ∃
x (cid:48) ∈ S X,(cid:15)
s.t.
f ( x (cid:48) ) (cid:54) =
Y .
Theorem 3.
(Restated)
Let A (cid:15)
[Equation]
f
[Equation]
[Equation] [Equation]
( f
[Equation]
) (cid:1)
[Equation]
1 (cid:9) and ˜ R srob ( f ) =
E ˜ P X,Y [ (cid:96) srob,(cid:15) ( f, X, Y )] .
Then, for any (cid:15) > 0 , ˜ R srob ( f ) is only minimized by any f ∈ A (cid:15) .
Furthermore, if
B ( ˜ X , (cid:15) )
⊃ X , f B is the unique minimizer of ˜ R srob ( f ) .
Proof.
The sensible risk of f w.r.t.
the restricted distribution corresponding to (6) is
[Equation]
because ˜ P ( Y = k | X = x ) =
P ( Y = k
| X = x ) for x ∈ ˜ X .
The minimum of ˜ R srob ( f ) is achieved by f B with the minimum value ˜ R ∗ =
˜ R std ( f B ) + 0 .
Therefore, any function f that ˜ R std ( f ) >
˜ R ∗ cannot achieve the minimum of ˜ R srob ( f ) because the term (1 ∃ x (cid:48) ∈ S x,(cid:15) ( f B ) s.t. f
( x (cid:48) ) (cid:54) =
f B ( x ) − 1 f ( x ) (cid:54) =
f B ( x ) ) in (8) is always non-negative.
Therefore, only functions in A = (cid:8) f ∈ F| ˜ P (cid:0) f
( X ) =
f B ( X ) (cid:1) =
1 (cid:9) need to be considered as possible minimizers of ˜ R srob ( f ) .
Note that A (cid:15) ⊂
A .
By the deﬁnition, we know that f ∈
A \ A (cid:15) if and only if
[Equation]
Therefore, for f ∈ A\A (cid:15) , ∃ A ⊂ ˜ X s.t.
˜ P (cid:0)
X ∈ A (cid:1) > 0 and ∃ x (cid:48) ∈ S x,(cid:15)
( f B ) for ∀ x ∈
A s.t. f
( x (cid:48) ) (cid:54) =
f B ( x (cid:48) ) .
For this f , the equation in (8) can be written as ˜ R srob ( f ) =
˜ R ∗ + α for some α ≥ 0 .
Now we show that α > 0 .
Note that by the deﬁnition of the Bayes rule, P ( y = f B ( x ) | X = x ) ≥ 1 K .
Otherwise, 1 = (cid:80) Kk =1 P ( y = k
| X = x ) ≤ K P ( y = f
B ( x ) | X = x ) < 1 , which is contradict.
Then, for the f ∈ A \ A (cid:15) and A ⊂ ˜ X that are described above,
[Equation]
Therefore, α > 0 .
Note that for any f ∈ A (cid:15) , the second term in (8) is zero by the deﬁnition of A (cid:15) .
Therefore, ﬁrst result of the theorem is proved.
Furthermore, for (cid:15) such that B ( ˜ X , (cid:15) )
⊃ X , A (cid:15) =
{ f B } .
Therefore, when
B ( ˜ X , (cid:15) )
⊃ X , f B is the unique minimizer of ˜ R srob ( f ) .
Theorem 4.
Let A = (cid:8) f ∈ F| ˜ P (cid:0) f
( X ) =
f B ( X ) (cid:1) = 1 (cid:9) , and take a reference function f r ∈
A
Theorem 4 says that for f r ∈ A , which behaves as the Bayes rule f B on the restricted support, the corresponding sensible adversarial risk E ˜ P X,
Y
[ (cid:96) ( f ( ˜ X | f r , Y )] is minimized only by f r .
This implies that if we do not have any information about the Bayes rule on ˜ X c , the sensibly optimal model w.r.t.
˜ P X,Y can be arbitrary on ˜ X c although this optimal model is the Bayes rule on ˜ X .
Our algorithm deals with this arbitrariness by searching for a better reference function in each iteration.
As a current model is used as a reference function, i.e., the estimation of the defense model and the reference model is identical, the algorithm essentially pursues sensibleness on ˜ X and robustness on ˜ X c of the trained model.
Note that on ˜ X , sensibleness a sufﬁcient condition for natural accuracy.
Proof.
By using the same way to derive (6) and (8) and noting that f r ( x ) = f B ( x ) on ˜ X , we get
[Equation]
Because f r ( x ) =
f B ( x ) on ˜ X , ˜ R std ( f r ) =
˜ R std ( f B ) .
This implies f r minimizes ˜ R srob ( f | f r ) because ˜
R std ( f B ) is the minimum of ˜ R std ( f ) for all f ∈ F and the second term in (9) is non-negative.
This implies any function f s.t.
˜ R std ( f ) >
˜ R std ( f B ) cannot achieve the minimum of ˜ R srob ( f | f r ) .
Therefore, as a minimizer of ˜ R srob ( f | f r ) , we only need to consider f ∈ A .
Note that 1 f ( x ) (cid:54) =
f B ( x ) = 0 for f ∈ A on ˜ X .
Therefore, by letting ˜ R ∗ std be ˜ R std ( f B ) , the equation in (9) can be written as
[Equation]
Note that η ( f B ( x ) | x ) is positive on ˜ X .
Therefore, for f ∈ A to minimize (10), it must satisfy that ˜ P (cid:0) ∃ x (cid:48) ∈
S X,(cid:15)
( f r ) s.t. f
( x (cid:48) ) (cid:54)
=
f r ( x (cid:48) ) (cid:1) = 0 .
This essentially says f should be f r .
C A LGORITHM
(0 , 1) , step number and sizes K, η 1 , η 2 , data X (0) adv
= X 2: repeat for i = 1 , ..., m , s.t. f
( x (0) i,adv ) =
y i 4: for k = 1 , ..., K
[Equation] : the projection operator 6: [Equation] 8: break
[Equation]
10: until training converged
Algorithm 2 is a straightforward extension of Algorithm 1 from (cid:96) ∞ norm to (cid:96) p norm. In addition, we present another way to realize the sensible reversion. After stepping back when the loss exceeds the threshold, we can add a random noise as the next step instead of just breaking the iteration as in Algorithm 1 and 2. Considering the nature of (mini-) batch leaning, this random noise does not add much computational cost because until every loss exceeds the threshold, the for loop would keep calculating the forward and backward loop with ﬁxed perturbation for the reversed examples.
This algorithm for (cid:96) ∞ norm is presented in Algorithm 3, of which the objective function is also (5). Algorithm 3 also reverses the adversarial example when the loss is greater than logc . The only difference is that instead of breaking the iteration after the reversion, it considers a random noise as the next step. This can lead to more effective search for the local loss maximizer in the ball x,(cid:15)
Input: Initialized f = f θ , c ∈ (0 , 1) , step number K , step sizes η 1 , η 2 , data X (0)
adv
= X repeat 3: for i = 1 , ..., m , s.t. f ( x (0) i,adv ) =
y i for k = 1 , ..., K ˜ x
i ← Π B ( x i ,(cid:15) ) ( step ( k ) i + x ( k − 1) i,adv ) , Π : the projection operator 6: if (cid:96) ( f, ˜ x i , y i ) ≤ log 1 c x ( k ) i,adv = ˜
x i step ( k +1)
i =
η 1 sign ( ∇ x (cid:96) ( f ( x ( k ) i,adv ) , y i )) 9: else x ( k ) i,adv = x
( k − 1) i,adv step ( k +1) i = random noise 12: θ ←
[Table]
(cid:80)
[Table]
=
[Table]
(cid:96) (
[Table]
[Table]
i,adv , y i ) /m until training converged
D T HE LANDSCAPE FOR S ENSIBLE ROBUST OPTIMIZATION
Note that when the original examples are incorrectly classiﬁed, full PGD examples can be very inﬂuential in regular adversarial training.
If we do not add any perturbations on these potentially inﬂuential examples and add full PGD perturbations on the other examples, the resultant examples are equivalent to the sensible adversarial examples when c = 0 .
Experiment 1 shows this small change keeps the models from collapsing, demonstrating how inﬂuential the PGD perturbations on the incorrectly classiﬁed natural examples.
However, an adversarial training with the sensible adversarial examples with c = 0 , the empirical loss can largely ﬂuctuate during the training; Once an incorrectly classiﬁed natural example becomes correctly classiﬁed, its full PGD attack can pose a sudden large gradient for the model update.
Then, when it is incorrectly classiﬁed again, the loss on it suddenly reduces to the natural loss that is distinctly smaller than the large loss value on its full PGD attack.
Therefore, whenever an example changes its state between a correctly and incorrectly classiﬁed example, i.e., the full PGD and natural stage, the corresponding loss can ﬂuctuate making the learning unstable.
However, if we add sensible reversion step, particularly if c ≥ 0 . 5 , this ﬂuctuation does not occur.
Between the two stages, the sensibly reversed stage provides a kind of cushion between the natural and full PGD stage, preventing the sudden change in the loss value as described.
Until S x,(cid:15) ( f ) = S x,(cid:15) , the sensibly adversarial perturbation for x is adapted to make the loss of the current function approximately equal to log 1 c .
As the abstraction in Figure 4, it is like to have a virtually extended area at ˆ p f,y ( x ) =
c on the loss function that is locally ﬂat on { ˜ x s | S x,(cid:15)
( f ) = S x,(cid:15) } , i.e., on a set of ˜
x s that are in the sensibly reversed stage.
However, in spite of the existence of such a ﬂat loss area, the model still can learn with ˜ x s in sensibly reversed stage.
This is obvious because the cross-entropy loss has its non-zero gradient when it is log 1 c .
Interestingly, when the model is updated in a way to decrease the loss of the previous ˜ x s , the new sensible perturbation is again adapted to have the loss approximately equal to log 1 c .
Therefore, the course of training directs the model to have sensible adversarial examples in the full PGD stage.
Furthermore, as long as c > 0 , even when ˜ x s is in a sensibly reversed stage, it still has perturbation in an adapted (cid:15) -ball.
This helps to obtain a robustness although it is not on a full (cid:15) ball.
Note that for c ≥ 0 .
5 , the learning is very stable because only natural examples can overpower the training;
The large loss values are only achievable by sensible attack in the natural stage.
Any single full PGD attack cannot dominate the next update of the training because the loss function has relatively small gradients on the full PGD stage.
Therefore, our algorithms not only effectively ignore any inﬂuential full PGD attacks that may overpower the next update but also train a model that allows as many sensible attacks to be full PGD attacks.
In other words, our algorithms can stably learn a robust model.
E H IGH MARGIN PROPERTY OF S ENSIBLE ADVERSARIAL TRAINING
In sensible adversarial learning, the natural accuracy clearly takes priority over the adversarial accuracy.
The perturbed example is not allowed to cross the decision boundary of the Bayes rule in (2) or to reach the vicinity of the decision boundary of a reference function in (4).
As mentioned, when the cross-entropy loss is used, ˆ p f,y ( x ) ≥ c is a prerequisite for adding any adversarial perturbation on x .
We note that this condition provides a lower bound of the following natural margin as
[Equation]
Therefore, the priority of natural accuracy hints that the learning will prevent the natural margin from being sacriﬁced for the sake of adversarial robustness.
We note that the natural margin of x is an upper bound of its adversarial margin.
Therefore, if a model cannot conﬁdently predict a natural example x , neither can the model conﬁdently predict any adversarial examples of x .
Experiment 1 We investigate the margins of the models trained in Experiment 2, to understand the effect of c .
In Figure 5 we draw density plots of the margins on the test set for varying c for the ﬁxed capacities.
Overall, we see that a larger c results in a larger adversarial margins and natural margins.
In Figure 6, we also draw the margins but for varying model capacity.
In general, for a each c , a smaller capacity has more data points of negative margins.
However, for naturally trained models, i.e., the models with c = 1 . 0 , a larger model has smaller adversarial margins.
This is consistent to the observation of Su et al.
(2018) that accurate models tend to be less robust when the models are trained with natural examples.
On the other hand, although not displayed, the plots corresponding to c = 0 .
9 are essentially similar to the plots in the second low.
This implies even for large c , our method is not like natural learning; The models trained with c = 0 .
9 still have larger natural and adversarial margins.
Note that in Figure 3, the regular PGD model of capacity 5 has only few negative adversarial margins.
Instead, their natural margins signiﬁcantly smaller than those of the natural models.
In contrast, our model has negative adversarial margins for a few more data points.
Note that the majority of both natural and adversarial margins of the our models are signiﬁcantly larger than that of the regular PGD models.
For capacity 1,2 and 3, the regular PGD models collapse having small ”mean” of adversarial losses.
On the contrary, our models deal with the lack of model capacity by letting more portion of examples to have negative margins.
As demonstrated by Figure 5, in our methods the mean adversarial loss can be arbitrarily large.
Instead, it maintains a large portion of points having relatively large adversarial margins, i.e., being far from decision boundaries.
The ﬂip side of the advantage of high margin is the possibility of over ﬁtting.
For c ≥ 0 .
5 in Figure 7, the best robust accuracies are not achieved when capacity is 5 but when capacity is 3.
On the contrary, the PGD method achieves better robustness as the capacity increases.
Therefore, there is a possibility of over-ﬁtting problem that arises form the high margin property of sensible adversarial learning.
Experiment 2 In Figure 8, we draw the plot of natural and adversarial margins on the testset of CIFAR10.
Compared with the distribution of the TRADE model, the natural margin of the SENSE model is large.
For the adversarial margins, the SENSE model has two clearly separated clusters; one is of negative margins and the other is of positive margins.
Instead, the positive margins are distributed on the larger values.
We remark that this phenomenon is consistent to the sensible idea to allow to be fooled near the decision boundary of the Bayes rule.
When the capacity is not enough, the concept of the decision boundary is not of the Bayes rule.
Its is projected to an inevitable boundary of a model with nice natural performance, caused by the lack of model capacity.
The portion of negative adversarial margin is a cost for sensibility and with this cost, the model can obtain robustness as much as possible given a model capacity.
We see the portion of adversarial margins of the SENSE model in the negative area in Figure 8 is not small.
This may imply the current model capacity and the sample size for SENSE are not enough.
F C OMPARISON WITH OTHER METHODS
The sensible selection and reversion in our approach distinguish sensible learning from other approaches that balance between the natural and adversarial accuracy.
The objective function of our algorithm can be rewritten as follows.
For f ∈ F ,
[Equation]
(11) where (cid:96) s ( f, x, y ) is deﬁned in (5).
Kurakin et al.
(2016b) suggest adversarial training that randomly divides data into two parts, a natural and adversarial set.
The objective function can be written as the following.
For random index sets A and C s.t.
| A | + | C | = n and A ∩ C = φ ,
[Equation]
where ˜ x
i is an adversarial example of x i .
Our approach is similar in that we also divide the data for different usage.
However, we divide the data by not a random but an adaptive way, so that (cid:96) s ( f, x i , y i ) ≥ (cid:96) s ( f, x j , y j ) for x i ∈
A f and x j ∈ C f .
Also, we have an additional set other than a natural and fully adversarial set.
This additional set, a sensibly reversed adversarial set, plays an important role in allowing a data point smoothly changes its identity between a natural example and a full adversarial example.
Note that we do not have any weight controllers like λ in (12).
The hyperparameter c itself controls the importance of the natural accuracy in comparison to the adversarial accuracy.
Zhang et al.
(2019) investigate the Bayes decision boundary to resolve the trade-off problem.
They propose TRADE, of which the objective function is for β > 0 ,
[Equation]
This formulation shows several key differences between TRADE and SENSE.
First, (13) uniformly restricts the perturbation norm to (cid:15) for all data points, whereas (11) selects the sets for A f , B f and C f and restricts [Table] Defence model [Table]
the perturbation in different ways. Second, the main parameter β in (13) controls the importance of the natural accuracy in comparison to the smoothness of the model.
However, the main parameter c in (11) directly controls the lower bound of the natural and adversarial loss of the individual data.
This is the lower bound on the prediction probability c if (cid:96) is the cross entropy loss.
Third, the term (cid:96) ( f ( X (cid:48) ) , f ( X )) in (13) leads the model to be smooth to all directions in the input space, but (cid:96) s ( f, x, y ) in (11) leads to be close to one.
Therefore, intuitively, TRADE achieves robustness by obtaining smoothness of the model, whereas SENSE achieves robustness by reformulating (1) in a way to promote high conﬁdency of the robust prediction.
This may provide an intuition on the plot of the margin density in Figure 8.
By this intuition, we apply PGD attacks with larger perturbation of the training (cid:15) .
We apply the attacks on our trained models and the TRADE model by Zhang et al.
for MNIST and CIFAR10.
The results are in Table 6 and Table 7.
G A DDITIONAL INFORMATION ABOUT C HEESE HOLE DISTRIBUTION
Figure 9 (e) compares the worst-case standard risk R std ( ˜ f ∗ rob ) and R std ( ˜ f B ∗ ) .
Although R std ( ˜ f ∗ rob ) does not consistently decrease as (cid:15) increases, it is always much smaller than R std ( ˜ f B ∗ ) .
Therefore, pursuit of robustness leads to a more naturally accurate classiﬁer.
Figure 9 (e) shows that, as (cid:15) increases from (cid:15) =
α/ 2 to 1, the standard risk of the robustly optimal model gradually increases whereas the sensibly robust model keeps zero risk.
Figure 9 (f) demonstrates the robustness against (cid:15) -ball attacks of ˜ f B ∗ nat (black),
˜
f ∗ rob (blue), and ˜ f
s ∗ rob (red).
Although sensibly robust models have large adversarial robustness increasing to 1 as (cid:15) increases, this is because more and more adversarial examples can cross the border, while the model keeps its decision boundary consistent to the class border line.
On the other hand, adversarially robust functions have constant robust risk for (cid:15) > 1 / 4 .
This is because the robust functions predict as y = 1 for every x ∈ X .
The sketch of the proof on the standard and adversarial robust risks in (e) and (f) in Figure
We ﬁrst calculate the three classes of functions which minimize natural, adversarial robust, and sensibly robust risk respectively, w.r.t ˜ P X,Y .
Then for each class, we consider the worst case function from each class, in that the function maximizes the standard risk w.r.t P X,Y .
The corresponding standard risks are in Figure 9 (e).
Likewise, for each class, we consider the worst case function from each class, in that the function maximizes the adversarial robust risk w.r.t P X,Y .
The corresponding adversarial robust risks are in Figure 1 (f).
First, the minimizers of each risk w.r.t.
˜ P X,Y are as following.
1) Let ˜ F B be a set of naturally optimal functions w.r.t
˜ P X ,Y :
[Equation]
2) Let ˜ F srob be a set of sensibly optimal functions w.
[Equation]
P X
[Equation]
Y :
[Equation]
3) Let ˜ F rob be a set of robustly optimal functions w.r.t ˜ P X,Y :
[Equation]
[Equation] [Equation]
where
[Equation]
Second, we consider the worst case standard risk w.r.t.
P X,Y for each class above.
1) max f ∈ ˜ F B R std ( f ) :
Although p (cid:54) = 0 .
5 , due to the symmetry of the shape of ˜ X , max f ∈ ˜ F B R std ( f ) is the area on X \ ˜ X , the area outside the small nine squares.
2) max f ∈ ˜ F srob R std ( f ) :
For the same reason above, max f ∈ ˜ F srob R std ( f ) is the area on X \ B ( ˜ X , (cid:15) ) , the area outside the small nine squares extended by (cid:15) .
3) max f ∈ ˜ F rob R std ( f ) : When (cid:15) < α/ 2 or (cid:15) ≥ 3 α/ 4 , we consider the deviated line on B ( ˜ X , (cid:15) ) , and regard the model outside B ( ˜ X , (cid:15) ) as incorrect.
The risk is calculated easily by using the fact that the risk of any f ∈ ˜ F rob on is B ( ˜ X , (cid:15) ) is 3 × min( a + 2 (cid:15), 2 a ) × (1 − p ) × min(2 (cid:15), 0 . 5) / 0 .
When α/ 2 ≤ (cid:15) < 3 α/ 4 , since B ( ˜ X , (cid:15) ) covers ˜ X , the worst case functions are in a form of f ( x ) = sign ( x 1 − 0 .
5 + c (cid:15) ) for some c on the entire ˜ X .
For each (cid:15) s.t. 3 α/ 2 + (cid:15) < x 1 < 0 .
5 − (cid:15) , it is easy to ﬁnd the corresponding c (cid:15) .
Last, for the worst case adversarial robust risk w.r
t. P X,Y , we can calculate the risks
in a similar way to above
H A DDITIONAL INFORMATION ABOUT E XPERIMENTS H. E XPERIMENT
We consider the MNIST and CIFAR-10 dataset (LeCun et al., 2010; Krizhevsky & Hinton, 2009). Training For each dataset, we initialize our model by a naturally trained model. Then, we train the initialized model with sensible adversarial examples with the speciﬁcations in Table 8. [Table]
Testing with white-box attacks For white-box attacks, we consider PGD (Madry et al., 2017), C&W (Ding et al., 2019a), DeepFool (Moosavi-Dezfooli et al., 2016), FGSM (Kurakin et al., 2016a), LBFGS (Taba- cof & Valle, 2016), and MIFGSM (Dong et al., 2018).
In Experiment 1, we consider adversarial perturbations with (cid:96) ∞ -norm less than (cid:15) , where (cid:15) = 0 .
3 for the MNIST dataset and (cid:15) = 8 / 255 for the CIFAR10 dataset.
We attack our models with the white-box attacks.
We use the attacks implemented in Foolbox (Rauber et al., 2017), Advertorch (Ding et al., 2019b), and Adversarial Robustness 360 Toolbox (ART) (Nicolae et al., 2018).
The attack speciﬁcations are in Table 9.
The options that are not listed in the table are kept as default of the attack generating functions.
[Table]
We set the step size for MIFGSM as slightly smaller than 8255 × 140 in order to keep the adversarial example from having the perturbation norm greater than (cid:15) = 8 / 255 .
For the performance of TRADE on C&W40, we apply the C&W40 attack with the same speciﬁcations in Table 9.
For the other results of TRADE in Table 1, we refer to Zhang et al.
For PGD attacks, we draw accuracy and loss plots for increasing step numbers in Figure 10 and Figure 11.
They show that for the chosen step sizes, the chosen step numbers are enough to generate proper local maximizer of the PGD objective function.
Particularly, for CIFAR, we use the step size 2/255 rather than (cid:15) × 2 / 255 as it is more efﬁcient.
PGD attack serenity check We test with 20 random restarts, and for the step size 0 .
01 , the step number 500 and random restart 20 seem enough by Figure 10.
[Figure]
MNIST with PGD for the first 100 examples
Testing with black-box attacks We attack our models with PGD40 and MIFGSM swith the speciﬁcations in Table 9.
As the Foolbox implementation for MIGSM only returns the successful attacks on the generating model, we only apply these attacks on the defense model.
We note that the argument return early of MIFGSM is set to False as in Table 9.
For TRADE, we use the models by Zhang et al.
for both MNIST and CIFAR10.
H. E XPERIMENT
Model Architecture We conduct Experiment 2 on the MNIST dataset (LeCun et al., 2010).
We consider a sequence of CNNs with the increasing number of kernels.
A network of capacity q has two convolutional layers with 2 ( d − 1) and 2 d ﬁlters respectively, followed by a fully connected linear layer of 2 ( d +4) units.
Each layer is activated by ReLU.
Each convolutional layer is followed by 2 × 2 a max-pooling layer.
The size of all convolutional ﬁlters is 5 × 5 .
With a similar sequence of CNNs, Madry et al.
(2017) investigate the model behavior when the capacity in- creases.
They have capacity scale 1,2,4,8 and 16.
In their experiment, capacity scale 1 and 2 collapse.
our capacities 2 and 3 are comparable to the capacity scale 1 and 2 by Madry et al.
Likewise, our capacities 4 and 5 are comparable to their capacity scale 4 and 8 1 .
Therefore, our result, which shows the PGD models of capacity 1,2 and 3 collapse, is consistent to the result by Madry et al.
Training We train the sequence of MNIST models with sensibly adversarial example with (cid:15) = 0 .
3 , η 1 = 0 .
05 and K =
for varying c ∈ { 0 . 0 , 0 . 1 , · · · , 0 . 9 } .
The initial learning rate η 2 is 0 .
01 , and we train forepochs.
When training the PGD models, we use the same hyperparameters except c .
Testing The MNIST models are tested with (cid:96) ∞ PGD attacks of (cid:15) = 0 .
3 with the step number K = 40 and step size η 1 = 0 .
We generate the attacks by using a Python package Advertorch by Ding et al.
(2019b).
H. A DDITIONAL E XPERIMENT FOR T ABLE 6 AND T ABLE
We conduct our additional experiment on the MNIST and CIFAR-10 dataset (LeCun et al., 2010; Krizhevsky & Hinton, 2009).
For each dataset, we consider the SENSE model trained in Experiment 1 and the TRADE model by Zhang et al.
We note that for each dataset, the TRADE and SENSE model share the same architecture.
On theses models, we apply the PGD attacks with perturbations larger than the training (cid:15) .
We generate the PGD attacks, using a Python package Advertorch by Ding et al.
(2019b) with the following attack speciﬁcations.
MNIST Let (cid:15)
= 0 . 3 , which is the training (cid:15) for each model.
For δ ∈ { 1 . 1 , 1 . 2 , 1 . 3 } , we generate the (cid:96) ∞ PGD attacks of (cid:15) = δ(cid:15)
with the step number K = 40 × δ and step size η 1 = 0 .
[Table] 10 Let [-(cid:15) 
-][Table]
= [Table] [Table] is the training (cid:15)
[-For δ ∈ { 108 , 128 , 148 , 168 } , we generate the (cid:96) ∞ PGD attacks of (cid:15) = δ(cid:15) 
-][Table] model [Table]
[Table] [Table] we generate [Table]