C ONFIDENCE -C ALIBRATED A DVERSARIAL T RAINING and Detection : M ORE R OBUST M ODELS G ENERALIZING B EYOND THE A TTACK U SED D URING T RAINING
Anonymous authors
Paper under double-blind review
A BSTRACT
Adversarial training is the standard to train models robust against adversarial examples.
However, especially for complex datasets, adversarial training incurs a signiﬁcant loss in accuracy and is known to generalize poorly to stronger attacks, e g., larger perturbations or other threat models.
In this paper, we introduce conﬁdence-calibrated adversarial training (CCAT) where the key idea is to enforce that the conﬁdence on adversarial examples decays with their distance to the attacked examples.
We show that CCAT preserves better the accuracy of normal training while robustness against adversarial examples is achieved via conﬁdence thresholding, i e., detecting adversarial examples based on their conﬁdence .
Most importantly, in strong contrast to adversarial training, the robustness of CCAT generalizes to larger perturbations and other threat models, not encountered during training.
For evaluation, we extend the commonly used robust test error to our detection setting, present an adaptive attack with backtracking and allow the attacker to select, per test example, the worst-case adversarial example from multiple black- and white-box attacks.
We present experimental results using L ∞ , L 2 ,
L 1 and L 0 attacks on MNIST, SVHN and Cifar10.
I NTRODUCTION
Deep neural networks have shown tremendous improvements in various learning tasks including applications in computer vision, natural language processing or text processing.
However, the discovery of adversarial examples, i.e., nearly imperceptibly perturbed inputs that cause mis-classiﬁcation, has revealed severe security threats, as demonstrated by attacking popular computer vision services such as Google Cloud Vision (Ilyas et al., 2018a) or Clarifai (Liu et al., 2016;
Bhagoji et al., 2017b).
As the number of safety- and privacy-critical applications is increasing, e.g., autonomous driving or medical imaging, this problem becomes even more important.
In practice, adversarial training and its variants (Szegedy et al., 2013; Goodfellow et al., 2014; Madry et al., 2018) , i.e., training on adversarial examples, can be regarded as the state-of-the-art to obtain models robust against adversarial examples.
In contrast to many other defenses, and to the best of our knowledge, adversarial training has not been broken so far.
However, adversarial training is known to increase test error signiﬁcantly.
Only on simple datasets such as MNIST (LeCun et al., 1998), adversarial training is able to preserve accuracy.
This observation is typically described as a trade-off between robustness and accuracy (Tsipras et al., 2018; Stutz et al., 2019; Raghunathan et al., 2019; Zhang et al., 2019) .
Furthermore, while adversarial training leads to robust models for the employed threat model used during training, the obtained robustness does not translate to other threat models, e.g., other L p -balls (Sharma & Chen, 2017; Song et al., 2018; Madry et al., 2018;
Tram`er & Boneh, 2019; Li et al., 2019a; Kang et al., 2019), larger perturbations than the ones used during training or distal adversarial examples (Hein et al., 2019).
Contributions: We address both problems of adversarial training: the drop in accuracy on datasets such as Cifar10 (Krizhevsky, 2009) and the poor “generalization” of robustness to larger perturbations or unseen threat models .
To this end, we introduce conﬁdence-calibrated adversarial training (CCAT) based on the idea that the predicted conﬁdence for an adversarial example should decrease with its distance to the attacked example.
Speciﬁcally, we bias the network to predict a convex combination of uniform and (correct) one-hot distribution for adversarial examples, which becomes uniform as the distance to the attacked test example increases.
Thus, CCAT implicitly biases the network to predict the uniform distribution over classes beyond the adversarial examples seen during training.
In contrast, standard adversarial training forces the network to classify adversarial examples correctly with maximal conﬁdence but provides no guidance how to extrapolate beyond adversarial examples seen during training.
Overall, CCAT allows to detect previously unseen adversarial examples, e.g., large perturbations, different L p attacks or distal adversarial examples, by conﬁdence thresholding, i.e., detection.
We also discuss a worst-case evaluation methodology using an adaptive attack to compare our detection-based approach with adversarial training.
First, we generalize the commonly used robust test error (Madry et al., 2018; Schott et al., 2019) to our detection setting.
Second, for each test example, the worst-case adversarial example, i.e., with highest conﬁdence, is selected from multiple white- and black-box attacks.
Using an adapted attack based on (Madry et al., 2018; Dong et al., 2018), augmented with a novel backtracking scheme, we show that CCAT produces models with improved accuracy on Cifar10 and robustness to various unseen adversarial examples: larger perturbations, L 2 , L 1 and L 0 attacks, distal adversarial examples and corrupted examples on MNIST(-C), (LeCun et al., 1998; Mu & Gilmer, 2019) SVHN (Netzer et al., 2011), Cifar10(-C) (Hendrycks &
Dietterich, 2019).
We will make our code for CCAT and evaluation publicly available.
Outline: After reviewing related work in Sec. 1.1, we discuss adversarial training (Madry et al., 2018) and introduce CCAT in Sec. 2.
In Sec.
3 we present our conﬁdence-thresholded robust test error and present experimental results in Sec. 4.
Finally, we conclude in Sec. 5.
1. R ELATED W ORK
Adversarial Examples: Adversarial examples can roughly be divided into white-box attacks, i.e., with access to the models, its weights and gradients, e.g. (Goodfellow et al., 2014;
Madry et al., 2017; Carlini & Wagner, 2017b), and black-box attacks, i.e., only with access to the output of the model, e.g. (Chen et al., 2017; Brendel & Bethge, 2017; Su et al., 2017; Ilyas et al., 2018b;
Sarkar et al., 2017; Narodytska & Kasiviswanathan, 2017).
Adversarial examples were also found to be transferable between similar models (Liu et al., 2016; Xie et al., 2018).
While these attacks are intended to be (nearly) imperceptible, visible adversarial transformations, e.g., rotations or rotations (Engstrom et al., 2017; 2019; Dumont et al., 2018; Alaifari et al., 2018; Xiao et al., 2018), or patches (Liu et al., 2018; Lee & Kolter, 2019; Brown et al., 2017; Karmon et al., 2018; Zajac et al., 2019) have also been proposed and transferred to the physical world (Athalye et al., 2018b;
Li et al., 2019b; Kurakin et al., 2016).
We refer to recent surveys (Barreno et al., 2006; Yuan et al., 2017; Akhtar & Mian, 2018; Biggio & Roli, 2018a) for more details. .
Recently, white-box attacks utilizing projected gradient ascent to maximize cross-entropy loss or surrogate objectives, e.g., (Madry et al., 2017; Dong et al., 2018; Carlini & Wagner, 2017b), have become standard.
Instead, we directly maximize the conﬁdence in any but the true class, similar to (Hein et al., 2019;
Goodfellow et al., 2019), to attack our proposed training procedure, CCAT.
Adversarial Training: Many defenses against adversarial attacks have been proposed, see, e.g., (Yuan et al., 2017; Akhtar & Mian, 2018; Biggio & Roli, 2018b), of which some have been shown to be ineffective, e.g., in (Athalye et al., 2018a; Athalye & Carlini, 2018).
Currently, adversarial training, i.e., training on adversarial examples, is the de-facto standard to obtain robust models.
While adversarial training was proposed in different variants (Szegedy et al., 2013; Zantedeschi et al., 2017; Miyato et al., 2016; Huang et al., 2015; Shaham et al., 2018; Sinha et al., 2018; Madry et al., 2017; Wang et al., 2019), the formulation by Madry et al.
(2017) received considerable attention and has been extended in various ways (Lee et al., 2017; Ye & Zhu, 2018; Liu et al., 2019).
For example, in (Shafahi et al., 2018; P´erolat et al., 2018), adversarial training is applied to universal adversarial example, in (Cai et al., 2018), curriculum learning is used, and in (Tram`er et al., 2017;
Grefenstette et al., 2018) ensemble adversarial training is proposed.
Adversarial training is also used in provable/certiﬁed defenses (Kolter & Wong, 2017; Zhang & Evans, 2018).
The increased sample complexity of adversarial training (Schmidt et al., 2018) has been addressed in (Lamb et al., 2019) by training on interpolated examples or in (Carmon et al., 2019; Uesato et al., 2019) using unlabeled examples.
The dependence on the threat model used during training was addressed in (Tram`er & Boneh, 2019) by using multiple threat models during training.
Finally, the frequently observed trade-off between accuracy and robustness has been studied both theoretically and empir- ically (Tsipras et al., 2018; Stutz et al., 2019; Zhang et al., 2019; Raghunathan et al., 2019).
CCAT differs from standard adversarial training in the used attack objective and the imposed distribution over the labels, which tends towards a uniform distribution for large perturbations.
Detection: Instead of correctly classifying adversarial examples, as intended in adversarial training, several works (Gong et al., 2017; Rouhani et al., 2018;
Grosse et al., 2017; Feinman et al., 2017;
Liao et al., 2018; Ma et al., 2018;
Amsaleg et al., 2017; Metzen et al., 2017;
Bhagoji et al., 2017a; Hendrycks & Gimpel, 2017; Li & Li, 2017; Pang et al., 2018) try instead to detect adversarial examples.
However, several detectors have been shown to be ineffective against adaptive attacks, i.e., adversaries aware of the used detection mechanism (Carlini & Wagner, 2017a).
Recently, the detection of adversarial examples based on their conﬁdence, similar to our approach with
CCAT, has also been discussed (Pang et al., 2018); however, the authors build on (Feinman et al., 2017) which has been shown to be ineffective (Carlini & Wagner, 2017a), as well.
Goodfellow et al.
(2019), instead, focus on evaluating conﬁdence-based detection methods using adaptive, targeted attacks maximizing conﬁdence.
Our attack, introduced in Sec. 2.2, is similar in spirit, however, untargeted by nature and, thus, suitable for usage with CCAT.
C ONFIDENCE C ALIBRATION OF A DVERSARIAL E XAMPLES
Adversarial training, as described by Madry et al.
(2018), has become standard to train robust models.
However, accuracy on challenging datasets such as Cifar10 is reduced and robustness does not generalize to larger perturbations, e.g., with respect to the used L ∞ constraint, or different threat models in other L p norms.
With CCAT, we intend to address both shortcomings: Instead of training models to correctly classify adversarial examples, we want to detect adversarial examples based on their conﬁdence.
In the following, we ﬁrst review adversarial training, before discussing the modiﬁcations needed for our CCAT, thereby turning robustness against adversarial examples into a detection problem.
Notation: We consider a classiﬁer f : R d → R K where K is the number of classes and f k denotes the conﬁdence for class k .
While we assume that the cross-entropy loss L is used for training, our approach can also be used with other losses.
Given x ∈ R d , classiﬁed correctly as y = argmax k f k ( x ) , an adversarial example x + δ is deﬁned as a “small” perturbation δ such that argmax k f k ( x + δ ) (cid:54) =
y , i.e., the classiﬁer changes its decision.
The strength of the change δ is measured by some L p -norm with p ∈ { 0 , 1 , 2 , ∞} ; p = ∞ is a popular choice in the literature as this leads to the smallest perturbation per feature/pixel.
2. A DVERSARIAL T RAINING
Standard adversarial training, as formulated by Madry et al.
(2018), is given as the following minmax problem:
[Equation] [Equation]
with w being the classiﬁer’s parameters and L being the cross-entropy loss.
During mini-batch training the inner maximization problem,
[Equation]
(2) is approximately solved.
In addition to the L ∞ -constraint, a box constraint, i.e., ˜ x i = ( x + δ ) i ∈ [0 , 1] , is enforced for images.
Note that maximizing the cross-entropy loss is equivalent to ﬁnding the adversarial example with minimal conﬁdence in the true class.
For neural networks, this is generally a non-convex optimization problem.
In (Madry et al., 2018)
the problem is tackled using projected gradient descent (PGD), which is typically initialized using a random δ with (cid:107) δ (cid:107) ∞ ≤ (cid:15) .
During adversarial training, PGD is applied to the batch of every iteration in order to update the classiﬁer’s parameters on the obtained adversarial examples.
At test time one uses the best out of several random restarts to assess robustness.
In contrast to adversarial training as proposed in (Madry et al., 2018), which computes adversarial examples for the full batch in each iteration, others compute adversarial examples only for half the
[Figure]
the probabilities for all adversarial directions.
Adversarial examples were computed using our L ∞ -PGD-Conf attack, cf.
Sec. using projected gradient descent (Madry et al., 2018)
the conﬁdence of the adversarial examples is maximized for T = 1000 and L ∞ -constraint (cid:15)
= 0 .
The robustness of AT does not generalize beyond the (cid:15) = 0 .
03 -ball
as high conﬁdence adversarial examples can be found for larger perturbations, whereas CCAT predicts close to uniform conﬁdence after some transition phase allowing to easily detect adversarial examples.
examples of each batch (Szegedy et al., 2013), i.e., instead of training only on adversarial examples, each batch is divided 50% into clean and 50% into adversarial examples .
Compared to Eq.
(1), 50% / 50% adversarial training effectively minimizes
[Equation]
This improves test accuracy on clean examples compared to 100% adversarial training , i.e., Eq.
(1), but typically leads to worse robustness.
Intuitively, by balancing both terms in Eq.
(3) , the trade-off between accuracy and robustness can already be optimized to some extent (Stutz et al., 2019).
There are two problems with adversarial training: First, the (cid:15) -ball around training examples might include examples from other classes.
Then, the attack, i.e., the inner maximization in Eq.
(1), will focus on these regions such that adversarial training for these examples gets “stuck”.
This case is illustrated in our theoretical toy dataset in Sec.
and has also been considered in related work (Jacobsen et al., 2019b;a).
Here, both 100% and 50% / 50% adversarial training, cf.
Eq.
(1) and (3), are not able to ﬁnd the Bayes optimal classiﬁer in a fully deterministic problem, i.e., zero Bayes error.
This might contribute to the observed drop in accuracy for adversarial training on, e.g., Cifar (Krizhevsky, 2009).
Second, and more importantly, adversarial training as in Eq.
(1) does not guide the classiﬁer how to extrapolate beyond the used (cid:15) -ball during training.
Even worse, it enforces high conﬁdence predictions within the (cid:15) -ball, which clearly cannot be extrapolated to arbitrary regions.
It is not surprising that adversarial examples can often be found right beyond the (cid:15) -ball, i.e., using larger perturbations than at training time or other threat models, e.g., other L p norm balls.
2. C ONFIDENCE -C ALIBRATED A DVERSARIAL T RAINING
Addressing these problems requires only few but effective modiﬁcations as outlined in the description of our proposed conﬁdence-calibrated adversarial training (CCAT) in Alg.
1: during training, we bias the network to predict a uniform distribution over the classes on adversarial examples that are sufﬁciently far away from the original training examples.
Subsequently, during testing, adversarial examples can be detected by conﬁdence thresholding – adversarial examples receive near-uniform conﬁdence while test examples receive high-conﬁdence.
Thus, given an example x , the ideal attacker during training maximizes the conﬁdence in any arbitrary other label k (cid:54) =
y instead of minimizing the conﬁdence in the true label y , as in Eq.
[Equation]
(4) where f k denotes the probability f assigns to class k (i.e., in practice, f is the output of the softmax layer).
A similar objective but for targeted attacks has been used in (Goodfellow et al., 2019), whereas our goal is an untargeted attack and thus our objective is the maximal conﬁdence over all
1: while true do 2: choose random batch ( x 1 , y 1 ) , . . .
, ( x B , y B ) .
3: for b = 1 , . . .
, B / 2 do 4: { maximize conﬁdence in other classes than true one for adversarial example ˜ x b , Eq.
(4): } 5: δ b := argmax (cid:107) δ (cid:107) ∞ ≤ (cid:15) max k (cid:54) =
y b f k ( x b + δ ) 6: ˜ x b := x
b + δ b 7: { probability over classes of ˜ x b becomes more uniform as (cid:107) δ b (cid:107) ∞ increases, Eq.
(6): } 8: λ :=
e − ρ (cid:107) δ b (cid:107) ∞ or λ :=
(1 − min(1 , (cid:107) δ (cid:107) ∞
(cid:15) ))
ρ 9: { ˜ y b is convex combination of one hot and uniform distribution over the classes, Eq.
(5): } 10: ˜
y b := λ one hot ( y b ) + (1 − λ )
K11: end for 12: { corresponds to 50% / 50% adversarial training, Eq.
(3) : } 13: update parameters using (cid:80) B /
2 b =1 L ( f (˜ x b ) , ˜ y b ) + (cid:80) B b = B / 2 L ( f ( x b ) , y b ) 14: end while other classes.
Then, during training, CCAT biases the classiﬁer towards predicting uniform distributions on adversarial examples by using the following distribution as target in the cross-entropy loss:
[Equation]
Here, p y ( k ) is the original “one-hot” distribution, i.e., p y ( k )
=
1 iff k
=
y and p y ( k )
=
otherwise with y being the true label, and u ( k ) = 1 / K is the uniform distribution.
Thus, we enforce a convex combination of the original label distribution and the uniform distribution which is controlled by the parameter λ .
We choose λ to decrease with the distance (cid:107) δ (cid:107) ∞ of the adversarial example to the attacked example x with the intention to enforce uniform predictions when (cid:107) δ (cid:107) ∞ =
δ .
Then, the network is encouraged to extrapolate this uniform distribution beyond the used (cid:15) -ball.
Even if extrapolation does not work perfectly, the uniform distribution is much more meaningful for extrapolation in the region between classes compared to high-conﬁdence predictions as encouraged in standard adversarial training.
We consider two variants of transitions, i.e., controlling the trade-off λ between one-hot and uniform distribution:
[Equation]
This ensures that for δ =
we impose the original (one-hot) label.
For growing δ , however, the inﬂuence of the original label decays proportional to (cid:107) δ (cid:107) ∞ .
The speed of decay is controlled by the parameter ρ .
For the exponential transition, we always have a bias towards the true label as even for large ρ , λ will be non-zero.
In case of the power transition, λ = 0 for (cid:107) δ (cid:107) ∞ ≥ (cid:15) , meaning a pure uniform distribution is enforced.
It is important to note that in CCAT in Alg.
1 we train on 50% clean and 50% adversarial examples in each batch, as in Eq.
Training only on adversarial examples will not work as we the network has no incentive to predict correct labels.
2. C ONFIDENCE -C ALIBRATED A DVERSARIAL T RAINING Y IELDS A CCURATE M ODELS
Proposition 1 analyzes 100% adversarial training and its 50% / 50% variant as well as our conﬁdencecalibrated variant, CCAT, to show that there exist problems where both 100% and 50% / 50% adversarial training are unable to reconcile robustness and accuracy, as recently discussed (Tsipras et al., 2018; Stutz et al., 2019; Raghunathan et al., 2019; Zhang et al., 2019).
However, our CCAT is able to obtain both robustness and accuracy given that λ in Eq.
(6) is chosen appropriately.
Proposition 1.
We consider a classiﬁcation problem with two points x = 0 and x = (cid:15) in R with deterministic labels, that is p ( y =
| x = 0)
= 1 and p ( y =
| x = (cid:15) )
=
and the problem is fully determined by the probability p
= p ( x = 0) as p ( x = (cid:15) )
= 1 − p 0 .
The Bayes error of this classiﬁcation problem is zero.
The predicted probability distribution over the classes is ˜ p ( y | x ) =
e gy ( x ) e g 1( x ) + e g 2( x ) , where g : R d → R 2 and we assume that the function λ : R + → [0 , 1] is monotonically decreasing and λ (0)
= 1 .
The Bayes optimal classiﬁer for the cross-entropy loss of
• adversarial training on 100% adversarial examples, cf.
Eq.
(1) , yields an error of min { p 0 , 1 − p 0 } .
• adversarial training with 50% adversarial and 50% clean examples per batch, cf.
Eq.
(3) , yields an error of min { p 0 , 1 − p 0 } .
• CCAT on 50% clean and 50% adversarial examples, cf.
Alg. 1, yields zero error if λ ( (cid:15) ) < min (cid:110) p
1 − p 0 , 1 − p
p 0 (cid:111) .
C ONFIDENCE -T HRESHOLDED R OBUST T EST E RROR (RE RR )
CCAT runs in a two-stage process: First, examples are rejected by conﬁdence thresholding.
Ideally, all adversarial examples are rejected as CCAT encourages low-conﬁdence adversarial examples.
Second, we evaluate robustness and accuracy on the non-rejected (i.e., correctly or incorrectly classiﬁed) examples.
In the ﬁrst stage (i.e., conﬁdence thresholding), we consider successful adversarial examples as negatives and correctly classiﬁed test examples as positives.
Then, we report the area under the ROC curve, i.e., ROC AUC, which shows how well we can discriminate adversarial examples from correctly classiﬁed clean examples.
In the second stage (i.e., after detection by conﬁdence thresholding), we ﬁx the threshold τ at a true positive rate (TPR) of 99% for correctly classiﬁed test samples, i.e., the network is allowed to reject at most 1% of correctly classiﬁed test examples.
We note that this also improves accuracy as errors on test examples typically have lower conﬁdence.
Then, we intend to use test error (Err) and robust test error (RErr), as also used in (Madry et al., 2018), to evaluate adversarial training.
However, extending RErr to take into account conﬁdencethresholding as in our detection setting is non-trivial: for example, correctly classiﬁed examples can can have lower conﬁdence than their corresponding adversarial examples.
Thus, given conﬁdence threshold τ , we deﬁne the conﬁdence-thresholded robust test error RErr as follows:
[Equation]
Here, τ is the conﬁdence-threshold ﬁxed on a held-out validation set, { ( x n , y n ) }
Nn =1 are test examples, c ( x n ) :
= max k f k ( x n ) denotes the classiﬁer’s conﬁdence on x n , f ( x n ) :
= argmax k f k ( x n ) denotes the classiﬁer’s decision , and ˜ x n are adversarial examples.
The numerator counts the number of incorrectly classiﬁed test examples x n with c ( x n ) ≥ τ (ﬁrst term) and the number of successful adversarial examples ˜ x n on correctly classiﬁed test examples with c (˜ x n ) ≥ τ (second term).
The denominator counts test examples x n with c ( x n ) ≥ τ (ﬁrst term) and the number of successful adversarial examples ˜ x n with c (˜ x n ) ≥ τ but where the corresponding test example x n has c ( x n ) <
τ (second term).
The latter takes care of the special case where adversarial examples have higher conﬁdence than their corresponding test examples, as mentioned above and is encouraged by the objective of our attack in Eq.
In total this yields a correct fraction within [0 , 1] and for τ = 0 ,
Eq.
(7) reduces to the “standard” RErr.
The conﬁdence-thresholded Err ( τ ) corresponds to taking only the ﬁrst terms in both numerator and denominator.
E XPERIMENTS
We compare normal training, adversarial training (AT) and our CCAT on MNIST, (LeCun et al., 1998) SVHN (Netzer et al., 2011) and Cifar10 (Krizhevsky, 2009).
We use ResNet-20 (He et al., 2016), implemented in PyTorch (Paszke et al., 2017), initialized following (He et al., 2015) and trained using stochastic gradient descent with batch size of 100 for 100 or 200 epochs (MNIST and
SVHN/Cifar10, respectively).
As detailed in Sec. 3, we report ROC AUC (higher is better) as well as our conﬁdence-thresholded test error (Err; lower is better) and
robust test error (RErr; lower is better) ; Err is reported on the full test sets, while ROC AUC and RErr are reported on the ﬁrstattacked test examples.
More details and experimental results can be found in Appendix B.
[Figure]
4. A TTACKS
White-box and adaptive attacks: We follow (Madry et al., 2018) and use projected gradient descent (PGD) to minimize the negatives of Eq.
(2) and (4); we denote them as PGD-CE and PGD-Conf.
The perturbation δ is initialized uniformly over direction and distance; for PGD-Conf, we additionally use δ = 0 as initialization.
Different from (Madry et al., 2018), we run exactly T iterations (no early stopping) and take the perturbation corresponding to the best objective of the T iterations.
In addition to momentum, as in (Dong et al., 2018), we propose to use an adaptive learning rate in combination with a backtracking scheme to improve the attacks: after each iteration, the computed update is only applied if it improves the objective; otherwise the learning rate is reduced.
We also implemented PGD for L 2 , L 1 and L 0 attacks; note that for each L p , p ∈ {∞ , 2 , 1 , 0 } , PGD-Conf has been explicitly designed to attack our CCAT. We train on L ∞ attacks using T = 40 iterations with (cid:15) = 0 . 3 (MNIST) or (cid:15) = 0 . 03 (SVHN/Cifar10).
For evaluation, we use T = 2000 iterations and 10 random retries for PGD-Conf; T = 200 with 50 random retries for PGD-CE.
For L 2 , L 1 and Lattacks, we set (cid:15) to 3 , 10 , 15 (MNIST) or 1 , 7 . 85 , 10 (SVHN/Cifar10), respectively, partly following (Tram`er & Boneh, 2019).
Learning rates have been tuned for each L p and objective (i.e., CE or Conf) independently.
Black-box attacks: For the L ∞ case, we additionally use random sampling, the attack by Ilyas et al.
(2018a), adapted with momentum and backtracking optimizing Eq.
(4) for T = 2000 iterations with 10 attempts, a variant of (Narodytska & Kasiviswanathan, 2017) with T = 2000 iterations and the “cube” attack (Andriushchenko, 2019) with T = 5000 iterations.
We also consider the cube attack in the L 2 setting and the attack of (Croce & Hein, 2019) in the L 0 setting.
The black-box attacks ensure that our defense avoids, e.g., gradient masking as described in (Athalye et al., 2018a).
4. W ORST -C ASE E VALUATION M ETHODOLOGY
Instead of reporting robustness against individual attacks, as commonly done in the literature, we use a worst-case evaluation scheme: For each individual test example, all adversarial examples from multiple attempts (i.e., multiple random restarts) and across all (white- and black-box) attacks are accumulated.
Subsequently, per text example, only the adversarial example with highest conﬁdence is kept; these are the worst-case adversarial examples for each test example.
In practice, this scheme allows to accumulate results from different attacks and thus corresponds to a much stronger robustness evaluation than usual. [Table] [Table] momentum+
[Table]
mom –
[Table]
zero rand zero zero
[Table]
AT 38.
[Table]
[Figure] [Figure]
τ .
For evaluation, we choose τ in order to obtain 99% TPR.
As described in the text, RErr subsumes both Err and FPR.
Curves based on worstcase examples across all tested L ∞ attacks.
4. A BLATION S TUDY
Momentum and Backtracking: Fig.
2 illustrates the advantage of momentum and the proposed backtracking scheme for PGD-Conf with T = 40 iterations on 10 test examples of SVHN.
As shown in Fig.
2 and Tab. 1, better objective values can be achieved within fewer iterations and avoiding oscillation which is important at training time.
However, also at test time, Tab. 1 shows that attacking our CCAT model effectively requires up to T = 2000 iterations and zero initialization so that RErr for τ @ 99% TPR stagnates.
In contrast, PGD-Conf performs better against AT even for smaller T and without momentum or backtracking.
Thus, ﬁnding high-conﬁdence adversarial examples against CCAT is more difﬁcult than for AT.
Overall, this illustrates our immense effort put into attacking our proposed defense with an adapted attack, novel optimization techniques, and large number of iterations.
Evaluation Metrics: Fig.
4 shows ROC and RErr curves on SVHN, considering AT and CCAT with ρ pow = 10 , i.e., using the power transition from Eq.
The ROC curves, and the corresponding
AUC value, quantify how well (successful) adversarial examples can be detected, i.e., distinguished from (correctly classiﬁed) test examples.
Note our conservative choice to use the conﬁdence threshold τ at 99% TPR, wrongly rejecting at most 1% correctly classiﬁed examples.
We note that τ depends only on correctly classiﬁed test examples, not adversarial examples, and RErr implicitly includes FPR as well as Err.
We focus on the conﬁdence-thresholded variants of RErr and Err as conﬁdence thresholding will naturally also improve results for AT, resulting in a fair comparison, i.e., AT is also allowed to beneﬁt from our two-stage detection setting; for Err this can be seen in
Tab. 2.
On SVHN and Cifar10, we found the power transition with ρ pow = 10 from Eq.
(6) to work best.
Up to ρ pow = 10 , performance regarding RErr for τ @ 99% TPR continuously improves and after ρ pow = 10 performance stagnates, as shown in detail in the appendix.
On MNIST, interestingly, exponential transition with ρ exp = 7 performs best against L ∞ attacks ; we assume that the
[Table]
[Table] [Table]
[Table] [Table]
99% TPR.
CCAT is competitive with AT on the L ∞ attacks seen during training, but generalizes signiﬁcantly better to previously unseen attacks.
slight bias towards the true label preserved in the exponential transition helps.
However, even on
MNIST, the power transition improves robustness against previously unseen attacks, e.g., L 2 , L 1 or
L 0 attacks, such that we include results for both.
4. R ESULTS
In Tab. 3, we report the main results of our paper, namely robustness across all evaluated L ∞ , L 2 , Land L 0 attacks; for L ∞ we include results for the (cid:15) used during training and an increased (cid:15) .
While on MNIST, CCAT incurs a drop of roughly 6% in RErr, and on Cifar10 a drop of roughly 5% , both against L ∞ with the same (cid:15) as during training, it signiﬁcantly outperforms AT on SVHN, by more than 16% .
On SVHN and Cifar10, Err is additionally improved – on Cifar10, the improvement is particularly signiﬁcant with roughly 6% .
For L ∞ attacks with larger (cid:15) , robustness of AT degrades signiﬁcantly, while CCAT is able to preserve robustness to some extent, especially on SVHN.
Only on Cifar10, RErr degrades similarly to AT.
In terms of generalization to previously unseen threat models, i.e., L 2 , L 1 and L 0 attacks, CCAT outperforms AT signiﬁcantly.
We note that on all datasets, the considered (cid:15) -balls for L 2 , L 1 and L 0 are not contained in the L ∞ -ball used during training.
Overall, robustness of AT degrades signiﬁcantly under threat models not seen during training, while CCAT generalizes to new threat models much better.
In Tab. 4 we also report results for detecting uniform noise and adversarial uniform noise (i.e., distal adversarial examples) based on their conﬁdence.
For the latter, we sample uniform noise and subsequently use PGD-Conf to maximize the conﬁdence (without considering any true label in Eq.
(4)) in the L ∞ -ball around the noise point.
Although ROC AUC values are high on uniform noise, an FPR of 40% or higher shows that normal training and AT assigns high conﬁdence to uniform noise.
When
[Table]
Robustness against uniform noise (“Rand”) and distal adversarial examples (“Dist”), i.e., highconﬁdence adversarial computed on uniform noise using our L ∞ PGD-Conf attack by considering
Eq.
(4) without any true label; we use (cid:15) = 0 .
3 on MNIST, (cid:15) = 0 .
03 on SVHN/Cifar10.
We report
ROC AUC and FPR for a conﬁdence threshold of τ @ 99% TPR.
In contrast to normal training and
AT, CCAT is able to reliably detect all (adversarial) noise examples.
maximizing conﬁdence on uniform noise, FPR approaches 100% on all datasets.
In contrast CCAT allows to separate these attacks perfectly from test examples.
Furthermore, Tab. 5 presents results on MNIST-C and Cifar10-C (Mu & Gilmer, 2019; Hendrycks & Dietterich, 2019), containing the
MNIST and Cifar10 test examples with various corruptions (e.g., spatial transformations, brightness/contrast changes, noise, etc.)
Again, we consider our two-stage detection approach, reporting
ROC AUC for detecting corrupted examples based on conﬁdence and conﬁdence-thresholded Err on the corrupted examples .
The results are averaged over all available corruptions ( 15 on MNIST, 19 on Cifar10).
As can be seen, allowing conﬁdence thresholding is beneﬁcial in this scenario for
AT and CCAT.
However, CCAT, outperforms normal training and AT signiﬁcantly.
C ONCLUSION
We proposed conﬁdence-calibrated adversarial training (CCAT) which addresses two problems of standard adversarial training: an apparent accuracy-robustness problem, i.e., adversarial training tends to worsen accuracy on difﬁcult datasets ; and, more importantly, the lack of “generalizable” robustness, i.e., obtaining robust models against threat models not encountered during training.
CCAT achieves comparable or better robustness against the threat model at training time, i.e., L ∞ - constrained adversarial examples, with better accuracy on Cifar10.
However, in strong contrast to adversarial training, CCAT generalizes to stronger L ∞ attacks as well as L 2 , L 1 , L 0 attacks and distal adversarial examples by allowing detection based on conﬁdence-thresholding.
For our experiments, we further introduced a generalized robust test error adapted to our detection-setting and used a worst-case evaluation methodology, both of independent interest to the community.
[Table]
A P ROOF OF P ROPOSITION
Proposition 2.
We consider a classiﬁcation problem with two points x = 0 and x = (cid:15) in R with deterministic labels, that is p ( y =
| x = 0)
= 1 and p ( y =
| x = (cid:15) )
=
and the problem is fully determined by the probability p
= p ( x = 0) as p ( x = (cid:15) )
= 1 − p 0 .
The Bayes error of this classiﬁcation problem is zero.
The predicted probability distribution over the classes is ˜ p ( y | x ) =
e gy ( x ) e g 1( x ) + e g 2( x ) , where g : R d → R 2 and we assume that the function λ : R + → [0 , 1] is monotonically decreasing and λ (0)
= 1 .
The Bayes optimal classiﬁer for the cross-entropy loss of
• adversarial training on 100% adversarial examples, cf.
Eq.
(1) , yields an error of min { p 0 , 1 − p 0 } .
• adversarial training with 50% adversarial and 50% clean examples per batch, cf.
Eq.
(3) , yields an error of min { p 0 , 1 − p 0 } .
• CCAT on 50% clean and 50% adversarial examples, cf.
Alg. 1, yields zero error if λ ( (cid:15) ) < min (cid:110) p
1 − p 0 , 1 − p
p 0 (cid:111) .
Proof.
First, we stress that we are dealing with three different probability distributions over the labels: the true one p ( y | x ) , the imposed one during training ˆ p ( y | x ) and the predicted one ˜ p ( y | x ) . We also note that ˆ p depends on λ through Eq.
(5) , and λ itself is a function of the norm (cid:107) δ (cid:107) through
Eq. (6) ; here, this dependence is made explicit by writing ˆ p ( λ )( y | x ) .
This makes the expressions for the expected loss of CCAT slightly more complicated.
We ﬁrst derive the Bayes optimal classiﬁer and its loss for CCAT.
We introduce
[Equation]
[Equation]
and express the logarithm of the predicted probabilities (conﬁdences) of class 1 and 2 in terms of these quantities.
[Equation]
For our conﬁdence-calibrated adversarial training one ﬁrst has to solve
[Equation]
We get with the expressions above (note that we have a binary classiﬁcation problem):
[Equation]
if
[Equation]
else .
[Equation]
Note that in CCAT for 50% of the batch we use the standard cross-entropy loss and for the other 50% we use
[Equation]
The corresponding expected loss is then given by
[Equation]
=
[Equation]
, where p ( x = 0) = p 0 and p ( x = (cid:15) )
= 1 − p ( x = 0) =
1 − p 0 .
With the true conditional probabilities p ( y = s | x = x ) we get
[Equation]
( s ) (cid:107) ))
( y = j | x = x ) log (˜ p ( y = j
| x = x + δ ∗ x ( s )))
For our problem it holds with p ( y = 2 | x = 0)
=
p ( y = 1 | x = (cid:15) )
=
1 (by assumption).
Thus,
E [ L (ˆ p y ( λ ( δ ))( x ) , ˜ p ( x ))
| x = 0] = − 2 (cid:88)
j = ˆ p 2 ( λ ( (cid:107) δ ∗ 0 (2) (cid:107) ))
( y = j | x = 0) log (˜ p ( y =
j | x = x + δ ∗ 0 (2)))
E [ L (ˆ p y ( λ ( δ ))( x ) , ˜ p ( x ))
| x = (cid:15) ]
= − 2 (cid:88)
j = ˆ p 1 ( λ ( (cid:107) δ ∗ (cid:15) (1) (cid:107) ))
( y = j | x = (cid:15) ) log (˜ p ( y =
j | x = x + δ ∗ (cid:15) (1)))
As (cid:107) δ ∗ x ( y ) (cid:107) is either 0 or (cid:107) (cid:15) (cid:107) and λ (0)
= 1 we use in the following for simplicity the notation λ
= λ ( (cid:107) (cid:15) (cid:107) ) .
Moreover, note that ˆ
[Equation]
if [Equation] [Equation] else , where K is the number of classes.
Thus, K = 2 in our example and we note that λ
+ (1 − λ )
= 1+ λ 2 .
With this we can write the total loss (remember that we have half normal cross-entropy loss and half the loss for the adversarial part with the modiﬁed “labels”) as
L ( a, b ) = p 0 (cid:104)
[Equation]
where we have omitted a global factor 1 2 for better readability (the last row is the cross-entropy loss) .
We distinguish two sets in the optimization.
First we consider the case a ≥ b .
Then it is easy to see that in order to minimize the loss we have a = b .
[Equation]
This yields e a = 1 − p
p 0 or a = log (cid:0) 1 − p
p 0 (cid:1) and the minimum for a ≥ b is attained on the boundary of the domain of a ≤ b .
The other case is a ≤ b .
We get ∂ a
[Equation]
=
[Equation]
This yields the solution
[Equation]
It is straightforward to check that a ∗ < b ∗ for all 0 < p 0 < 1 , indeed we have
[Equation]
if 0 < p 0 < 1 and note that λ < 1 by assumption.
We have a ∗ < 0 and thus g 2 (0) > g 1 (0) (Bayes optimal decision for x = 0 ) if
[Equation]
and b ∗ > 0 and thus g 1 ( (cid:15) ) > g 2 ( (cid:15) )
(Bayes optimal decision for x = (cid:15) )
if
[Equation]
Thus we recover the Bayes classiﬁer if
[Equation]
Now we consider the approach by Madry et al.
(2018) with 100% adversarial training.
The expected loss can be written as
[Equation]
[Equation]
This yields in terms of the parameters a, b the expected loss:
[Equation]
The expected loss is minimized if a = b as then both maxima are minimal.
This results in the expected loss
[Equation]
The critical point is attained at a ∗ =
b ∗ = log (cid:16) 1 − p
p 0 (cid:17) .
This is never Bayes optimal for all 0 < p 0 < 1 .
Note that b ∗ = a ∗ > 0 if p 0 < 1 − p 0 and thus the error is given by p 0 , similar a ∗ =
b ∗ <
if 1 − p 0 < p 0 and thus the error is given by 1 − p 0 .
We get an error of min { p 0 , 1 − p 0 } of the Bayes optimal solution of 100% adversarial training.
Next we consider 50% adversarial plus 50% clean training.
The expected loss
[Equation]
can be written as
[Equation]
We make a case distinction.
If a ≥ b , then the loss reduces to
[Equation]
MNIST:
[Table]
τ @
[Table]
% TPR ( L ∞
[Table]
with (cid:15) = 0 . 3 )
Optimization momentum+backtrack momentum –
Initialization zero rand zero zero
Iterations T
AT 0.
0.4 AT Conf 0.
1.0 CCAT 0.
SVHN:
[Table]
with
[Table] [Table]
[Table]
( L ∞ attack with (cid:15) = 0 . 03 )
Optimization momentum+backtrack momentum –
Initialization zero rand zero zero
Iterations T
AT 38.
30.8 AT Conf 27.
23.8 CCAT 4.
Cifar10:
[Table]
with
[Table]
for
[Table] [Table]
% TPR ( L ∞ attack with (cid:15) = 0 .
Optimization momentum+
[Table]
momentum –
Initialization zero rand zero zero
Iterations T
AT 60.
[Table]
Solving for the critical point yields a ∗ = log (cid:16)
1 − p 0 p 0 (cid:17)
=
b ∗ .
Next we consider the set a ≤ b .
This yields the loss
[Equation]
Solving for the critical point yields a ∗ = log (cid:16)
1 − p 0 p 0 (cid:17)
= b ∗ which fulﬁlls a ≤ b .
Actually, it coincides with the solution found already for 100% adversarial training.
One does not recover the
Bayes classiﬁer for any 0 < p 0 < 1 .
Moreover, we note that
[Equation]
Thus, we classify x = 0 correctly, if p 0 > 12 and x = (cid:15) correctly if p 0 < 12 .
Thus the error is given by min { p 0 , 1 − p 0 } .
B E XPERIMENTS
We give additional details on our experimental setup, speciﬁcally regarding attacks, training and the used evaluation metrics.
Afterwards, we include additional experimental results, including ablation studies, results for 98% true positive rate (TPR), results per attack, results per corruption on MNISTC (Mu & Gilmer, 2019) and Cifar10-C (Hendrycks & Dietterich, 2019) and a comparison with (Pang et al., 2018) .
( t − 1)
+ (1 − β ) g
( t ) 22: { “try” the update step and see if objective increases: } 23:
ˆ δ ( t ) :
= δ ( t ) + γg ( t ) 24: clip ˆ δ ( t ) i to [ − (cid:15), (cid:15) ] 25: clip x i + ˆ δ ( t ) i to [0 , 1] 26: ˆ v ( t ) :
= F ( x + ˆ δ ( t ) , y ) 27: { only keep the update if the objective increased; otherwise decrease learning rate: } 28: if ˆ v ( t ) ≥ v ( t ) then 29: δ ( t +1) : = ˆ δ ( t ) 30: else 31: γ := γ / α 32: end if 33: end for 34: return ˜ x , ˜ v B. A TTACKS
Complementary to the description of the projected gradient descent (PGD) attack by Madry et al. (2018) and our adapted attack, we provide a detailed algorithm in Alg. 2. We note that the objective maximized in (Madry et al., 2018) is [Equation]
[Equation]
where L denotes the cross-entropy loss, f ( · ; w ) denotes the model and ( x, y ) is an input-label pair from the test set.
Our adapted attack, in contrast, maximizes
[Equation]
[Equation]
Note that the maximum over labels, i.e., max k (cid:54) =
y is explicitly computed during optimization; this means that in contrast to (Goodfellow et al., 2019), we do not run K targeted attack and subse-
MNIST ( L ∞ attack with (cid:15) = 0 . 3 ) 0 .
Test Conﬁdence
AT 0 . 2 0 .
Adversarial Conﬁdence
AT 0 . 2 0 .
Test Conﬁdence
CCAT, ρ exp =  0 .
Adversarial Conﬁdence
CCAT, ρ
[Figure]
=
Cifar10 ( L ∞ attack with (cid:15) = 0 . 03 ) 0 .
Test Conﬁdence
AT 0 . 2 0 .
Adversarial Conﬁdence
AT 0 . 2 0 .
Test Conﬁdence
CCAT, ρ pow =  0 .
Adversarial Conﬁdence
CCAT, ρ pow =
-case adversarial examples across all tested L ∞ attacks.
quently take the maximum-conﬁdence one, where K is the number of classes. We denote these two variants as PGD-CE and PGD-Conf, respectively. Deviating from (Madry et al., 2018), we initialize δ uniformly over directions and norm (instead of uniform initialization over the volume of the (cid:15) -ball):
[Equation]
where δ (cid:48) is sampled from a standard Gaussian and u ∈ [0 , 1] from a uniform distribution.
We also consider zero initialization, i.e., δ = 0 .
For random initialization we always consider multiple attempts, 10 for PGD-Conf and 50 for PGD-CE, with zero initialization, we use only 1 attempt.
Both PGD-CE and PGD-Conf can also be applied using the L 2 , L 1 and L 0 norms following the description above.
Then, gradient normalization, i.e., the signum in Line 10 in Alg. 2 for the L ∞ norm, the projection, and the the initialization need to be adapted.
For the L 2 norm, the gradient is normalized by dividing by the L 2 norm; for the L 1 norm only the 1% largest values (in absolute terms) of the gradient are kept and normalized by their L 1 norm; and for the L 0 norm, the gradient is normalized by the L 1 norm.
Projection is easily implemented for the L 2 norm, while we follow the algorithm of Duchi et al.
(2008) for the L 1 project; for the L 0 projection, only the (cid:15)
-largest values are kept.
Similarly, initialization for L 2 and L 1 are simple by randomly choosing a direction (as in
Eq.
and then normalizing by their norm.
For L 0 , we randomly choose pixels with probability 2 / 3 (cid:15)/ ( HWD ) and set them to a uniformly random values u ∈ [0 , 1] , where H × W × D is the image size.
In experiments, we found that tuning the learning rate for PGD with L 1 and L 0 constraints (independent of the objective, i.e., Eq.
(9) or Eq.
(10) ) is much more difﬁcult.
Additionally, PGD using the L 0 norm seems to get easily stuck in sub-optimal local optima.
Alg. 2 also gives more details on the employed momentum and backtracking scheme.
These two “tricks” add two additional hyper-parameters to the number of iterations T and the learning rate γ , namely the momentum parameter β and the learning rate factor α .
After each iteration, the computed update, already including the momentum term, is only applied if this improves the objective.
This is checked through an additional forward pass.
If not, the learning rate is divided by α , and the update is rejected.
Alg. 2 includes this scheme as an algorithm for an individual test example x with label y for brevity; however, extending it to work on batches, which is used in our paper, is straight-forward.
In practice, for PGD-CE, with T = 200 iterations, we use β = 0 .
9 and α = 1 . 25 ; for PGD-Conf, with T = 2000 iterations, we use β = 0 .
9 and α = 1 . 1 .
We also give more details on the used black-box attacks.
For random sampling, we apply Eq.
T = 5000 times in order to maximize Eq.
We also implemented the black-box attack of Ilyas et al.
(2018a)
using a population of 50 and variance of 0 .
1 for estimating the gradient in Line 10 of
Alg. 2; a detailed algorithm is provided in (Ilyas et al., 2018a).
We use a learning rate of 0 .
001 (note that the gradient is signed, as in (Madry et al., 2018)) and also integrated a momentum with β = 0 .
MNIST ( L ∞ attack with (cid:15) = 0 . 3 )
[Figure]
s i t i v
[Figure]
t
[Figure] [Figure]
[Figure]
and backtracking as described in Alg. 2 with α = 1 . 1 and T = 2000 iterations. We use zero and random initialization; in the latter case we allow 10 random retries.
For the simple black-box attack we follow the algorithmic description in (Narodytska & Kasiviswanathan, 2017) considering only axis-aligned perturbations of size (cid:15) per pixel.
We run the attack for T = 2000 iterations and allow 10 random retries.
Finally, we use a variant of the cube attack proposed in (Andriushchenko, 2019).
We run the attack for T = 5000 iterations with a probability of change of 0 .
We emphasize that, except for (Ilyas et al., 2018a), these attacks are not gradient-based and do not approximate the gradient.
B. T RAINING
As described in Sec. 4, we follow the ResNet-20 architecture by He et al.
(2016) implemented in PyTorch (Paszke et al., 2017).
For training we use a batch size of 100 and train for 100 and 200 epochs on MNIST and SVHN/Cifar10, respectively: this holds for normal training, adversarial training (AT) and conﬁdence-calibrated adversarial training (CCAT).
For the latter two, we use
PGD-CE and PGD Conf, respectively, for T = 40 iterations, momentum and backtracking ( β = 0 .
9 , α = 1 . 5 ).
For PGD-CE we use a learning rate of 0 .
01 and 0 .
005 on MNIST, SVHN and
Cifar10.
For PGD-Conf we use a learning rate of 0 . 05 .
For training, we use standard stochastic gradient descent, starting with a learning rate of 0 . 1 on MNIST/SVHN and 0 . 075 on Cifar10.
The learning rate is multiplied by 0 .
95 after each epoch.
We do not use weight decay; but the network includes batch normalization (Ioffe & Szegedy, 2015).
On SVHN and Cifar10, we use random cropping, random ﬂipping (only Cifar10) and contrast augmentation during training.
We always train on 50% clean and 50% adversarial examples per batch, i.e., each batch contains both clean and adversarial examples which is important when using batch normalization.
B. E VALUATION M ETRICS
For reproducibility and complementing the discussion in the main paper, we describe the used evaluation metrics and evaluation procedure in more detail.
Adversarial examples are computed on the ﬁrst 1000 examples of the test set; the used conﬁdence threshold is computed on the last 1000 examples of the test set; test errors are computed on all test examples minus the last 1000.
As we consider multiple attacks, and some attacks allow multiple random attempts, we always consider the worst case adversarial example per test example and across all attacks/attempts; the worst-case is selected based on conﬁdence.
ROC AUC: To compute ROC curves, and the area under the curve, i.e., ROC AUC, we deﬁne negatives as successful adversarial examples (on correctly classiﬁed test examples) and positives as the corresponding correctly classiﬁed test examples.
The ROC AUC as well as the curve itself can easily be calculated using (Pedregosa et al., 2011).
Practically, the generated curve could be used to directly estimate a threshold corresponding to a pre-determined true positive rate (TPR).
However, this requires interpolation; after trying several constant interpolation schemes, we concluded that the [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
nﬁd
[Table] [Table]
l a
[Table]
nﬁd e n ce p e r C l a ss
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Table]
[Figure] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]
[Table]
[Figure]
nﬁd n
C l
[Figure]
[Table]
[Figure]
results are distorted signiﬁcantly, especially for TPRs close to 100% . Thus, we followed a simpler scheme as mentioned above: on a held out validation set of size 1000 (the last 1000 samples of the test set), we sorted the corresponding conﬁdences, and picked the conﬁdence threshold in order to obtain the desired TPR, e.g., 99% , on this set exactly.
MNIST (
[Figure]
with (cid:15)
[Figure]
[Figure]
SVHN (PGD-Conf, L ∞ with (cid:15) = 0 . 03 )
[Figure]
Cifar10 (PGD-Conf, L ∞ with (cid:15) = 0 . 03 )
[Figure]
Robust Test Error: For clarity, we repeat our conﬁdence-integrated deﬁnition of the robust test error:
[Equation]
As described in the main paper, Eq.
(12) quantiﬁes the performance of a classiﬁer with reject-option at a speciﬁc conﬁdence threshold τ on both clean and adversarial examples.
The regular robust test error, corresponding to Eq.
(12) at τ = 0 , can also be written as
[Equation]
[Table]
[Figure]
o nﬁ d e n ce p e r C l a ss y =2 f y = ˜
y
{+=
+}
f ˜ y
=0 .y
= 4 f y = Classes 1, 2, 3, 4, 5, 6, 7, 8, 9, The robust test error is easy to handle as it quantiﬁes overall performance, including generalization on clean examples (ﬁrst term in Eq.
(13)) and robustness on adversarial examples corresponding to correctly classiﬁed clean examples (second term in Eq.
Additionally, the robust test error lies in [0 , 1] .
Generalizing Eq.
(13) to τ > 0 is non-trivial due to the following considerations:
First, when integrating a conﬁdence threshold, the reference set (i.e., the denominator) needs to be adapted.
Otherwise, the metric will not reﬂect the actual performance after thresholding, i.e., rejecting examples – speciﬁcally, the values are not comparable across different conﬁdence thresholds τ .
Then, Eq.
(13) might be adapted as follows:
[Equation]
Note that in the nominator, we only consider clean and adversarial examples with conﬁdence above the threshold τ , i.e., c ( x n ) ≥ τ and c (˜ x n ) ≥ τ .
Similarly, the denominator has been adapted accordingly, as after rejection, the reference set changes, too.
However, this formulation has a problem when adversarial examples obtain higher conﬁdence than the original clean examples.
Thus, second, we need to account for the case
where c (˜ x n ) >
τ ≥ c ( x n ) , i.e., an adversarial example obtains a higher conﬁdence than the corresponding clean example.
Then, the nominator may exceed the denominator, resulting in a value larger than one.
To mitigate this problem, we need to include exactly this case in the denominator.
Formalizing this case, we see that it corresponds exactly to the second term in the denominator or Eq.
[Equation]
Overall, with Eq.
(12) we obtain a metric that lies within [0 , 1] , is comparable across thresholds τ and, for τ = 0 , reduces to the regular robust test error as used in related work (Madry et al., 2018). [Table] [Table]
B. A BLATION S TUDY
Complementing Sec. 4, we include ablation studies for MNIST, SVHN and Cifar10; for our attack in Tab. 6 and training in Tab. 9.
Regarding the proposed attack
[Table]
i.e., PGD-Conf, using momentum and backtracking, Tab. 6 shows that the main observations for SVHN can be transferred to MNIST and Cifar10.
Only the improvement of backtracking and momentum over just using momentum cannot be conﬁrmed on MNIST.
Note that for fair comparison, T iterations with backtracking are equivalent to 3 / 2 T iterations without backtracking; which is why we include results for T = 60 and T = 300 .
However, the importance of using enough iterations, i.e., T = 2000 , and zero initialization to attack CCAT is still clearly visible.
Interestingly, against AT on SVHN, more iterations are also beneﬁcial, while this is not required on MNIST and Cifar10.
Tab. 9 also reports results for CCAT with different transitions, cf.
Eq.
(6), and values for ρ .
As mentioned before, on SVHN and Cifar10, power transition with ρ pow = 10 works best; for larger ρ performance stagnates.
It is also important to note that the power transition does not preserve a bias towards to true label, i.e., for the maximum possible perturbation ( (cid:107) δ (cid:107) ∞
=
(cid:15) ), Eq.
(6) forces the network to predict a purely uniform distribution.
This is in contrast to the exponential transition, where the true one-hot distribution always receives a non-zero weight.
On MNIST, we found this to work considerably better.
B. A NALYSIS
For further analysis, Fig. 5 shows conﬁdence histograms for AT and CCAT on MNIST and Cifar10.
The conﬁdence histograms for CCAT reﬂect the expected behavior: adversarial examples are mostly successful in changing the label, which is supported by high RErr values for conﬁdence threshold τ = 0 , but their conﬁdence is pushed towards a uniform distributions.
For AT, in contrast, successful adversarial examples – fewer in total – generally obtain high conﬁdence; this results in conﬁdence [Table] [Table] [Table]
thresholding being not effective for AT.
This behavior, however, is less pronounced on MNIST.
Here, the exponential transition results in adversarial examples with conﬁdence slightly higher than uniform conﬁdence, i.e., 0 .
This might be the result of preserving a bias towards the true label through Eq.
In fact, for lower ρ , we found that this behavior is pronounced, until, for very small ρ , the behavior of AT is obtained.
In Fig.
7, we plot the probabilities for all ten classes along an adversarial direction.
We note that these directions do not necessarily correspond to successful adversarial examples.
Instead, we chose the ﬁrst 10 test examples on SVHN.
The adversarial examples were obtained using our L ∞ PGDConf attack with T = 2000 iterations and zero initialization for (cid:15) = 0 .
For AT, we usually observe a change in predictions along these directions; some occur within (cid:107) δ (cid:107) ∞ ≤ (cid:15) , corresponding to successful adversarial examples, some occur for (cid:107) δ (cid:107) ∞ > (cid:15) , corresponding to unsuccessful adversarial examples (within (cid:15) ).
However, AT always assigns high conﬁdence.
Thus, when allowing larger adversarial perturbations at test time, robustness of AT reduces signiﬁcantly.
For CCAT, in contrast, there are only few such cases; more often, the model achieves a near uniform prediction for small (cid:107) δ (cid:107) ∞ and extrapolates this behavior beyond the (cid:15) -ball used for training.
On SVHN, this behavior successfully allows to generalize the robustness to larger adversarial perturbations.
Furthermore, these plots illustrate why using more iterations at test time, and using techniques such as momentum and backtracking, are necessary to ﬁnd adversarial examples as the objective becomes more complex compared to AT.
Fig.
8 shows concrete adversarial examples against AT and CCAT on MNIST and SVHN.
Among the ﬁrst 200 test examples, we picked the ﬁrst six examples where AT is mistaken and (non-overlapping) six examples where CCAT has difﬁculties, i.e., assigns high conﬁdence.
On SVHN, for example,
CCAT is able to detect adversarial examples with low conﬁdence where AT assigns incorrect classes.
On MNIST, in contrast, CCAT, cannot recover mistakes made by AT.
Similarly, there exist examples where AT assigns the correct class while CCAT assigns the wrong class with very high conﬁdence.
On MNIST, we additionally found adversarial examples against CCAT to represent more “localized” changes in the L ∞ norm compared to AT.
Overall, however, we did not ﬁnd any systematic difference between strong adversarial against AT and CCAT. In Fig.
9, on MNIST, we additionally illustrate the advantage of CCAT with respect to the toy example in Proposition 1.
Here, we consider the case where the (cid:15) -balls of two training or test examples (in different classes) overlap.
As we show in Proposition 1, adversarial training is not able to handle such cases, resulting in the trade-off between accuracy in robustness reported in the literature (Tsipras et al., 2018; Stutz et al., 2019;
Raghunathan et al., 2019; Zhang et al., 2019). This is because adversarial training enforces high-conﬁdence predictions on both (cid:15) -balls (corresponding to different classes), resulting in an obvious conﬂict. CCAT, in contrast, enforces uniform predictions throughout the largest parts of both (cid:15) -balls, resolving the conﬂict. Finally, Fig.
9 also shows that
CCAT is indeed able to extrapolate the uniform predictions beyond the (cid:15) -ball used during training.
B. R ESULTS
In the following, we present and discuss complementary results corresponding to the main results of our paper as presented in Tab. 3 and 4.
To this end, we consider requiring only 98% TPR instead of 99% TPR, and most importantly, break down our analysis by the different white-box and black-box attacks used.
Main results for 98% TPR:
Tab. 7 reports results (corresponding to Tab. 3) requiring only 98% TPR.
This implies, that compared to 99% TPR, up to 1% more correctly classiﬁed test examples can be rejected.
For relatively simple tasks such as MNIST and SVHN, where Err values are low, this is a signiﬁcant “sacriﬁce”.
However, as can be seen, robustness in terms of RErr only improves slightly.
We found that the same holds for 95% TPR, however, rejecting more than 2% correctly classiﬁed examples seems prohibitive large for the considered datasets.
Per-Attack Results: Finally Tab. 10, 11 and 12 we break down the results of Tab.
3 regarding the used attacks.
For simplicity we focus on PGD-CE and PGD-Conf while reporting the used blackbox attacks together, i.e., taking the worst-case adversarial examples across all black-box attacks.
For comparison, we also include non-thresholded Err and RErr for the results from the main paper.
On MNIST, where AT performs very well in practice, it is striking that for 4 / 3 (cid:15) = 0 .
4 even blackbox attacks are able to reduce robustness completely, resulting in high RErr.
This observation also transfers to SVHN and Cifar10.
For CCAT, black-box attacks are only effective on Cifar10, where they result in roughly 87% RErr with τ @ 99% TPR.
For the L 2 , L 1 and L 0 attacks we can make similar observations, except on MNIST, where AT seems similarly robust against L 1 and L 0 attacks as CCAT.
Across all L p norms, it can also be seen that PGD-CE performs signiﬁcantly worse against our CCAT compared to AT, which shows that it is essential to optimize the right objective to evaluate the robustness of defenses and adversarially trained models , i.e., maximize conﬁdence against CCAT
Results on Corrupted MNIST/Cifar10:
We also conducted experiments on MNIST-C (Mu &
Gilmer, 2019) and Cifar10-C (Hendrycks & Dietterich, 2019).
These datasets are variants of
MNIST and Cifar10 that contain common perturbations of the original images obtained from various types of noise, blur or transformations; examples include zoom or motion blue, Gaussian and shot noise, rotations, translations and shear.
Tab.
14 presents the per-corruption results on MNISTC and Cifar10-C. Here, all includes all corruptions and mean reports the average results across all corruptions; note that, due to the thresholding, leaving different numbers of corrupted examples after detection, the distinction between all and mean is meaningful.
Striking is the performance of CCAT on noise corruptions such as gaussian noise or shot noise .
Here, CCAT is able to detect 100% of the corrupted examples (cf.
the true negative rate (TNR) in Tab. 14), resulting in a thresholded Err of 0% .
This is in stark contrast to AT, exhibiting a Err of roughly 15% after detection on Cifar10-C.
On the remaining corruptions, CCAT is able to perform slightly better than
AT, which is often due to higher detection rate, i.e., higher ROC AUC.
On, Cifar10, the generally lower Err of CCAT also contributes to the results.
Overall, this illustrates that CCAT is able to preserve the inductive bias of predicting near-uniform distribution on noise similar to L ∞ adversarial examples as seen during training.
Comparison with (Pang et al., 2018):
We also compare AT and CCAT with the approach by Pang et al.
We implemented the reverse cross-entropy error in PyTorch following the ofﬁcial code 1 .
The reverse cross-entropy loss encourages the non-maximum predictions (i.e., in the case of correct classiﬁcation, the probabilities of the “other” classes) to be uniform.
Then, adversarial examples are detected based on statistics of the predicted distributions.
We used the non-maximum element entropy (nonME) detector instead of the K-density detector (Feinman et al., 2017) which has been shown to be ineffective against adaptive attacks in (Carlini & Wagner, 2017a).
We note that (Pang et al., 2018) do not train on adversarial examples as AT or CCAT.
Our evaluation metrics need not be changed; however, thresholding is done on the nonME detector instead of on the conﬁdence, as for CCAT.
In Tab. 8, it can be seen, that the approach works considerably well on
MNIST, regarding both L ∞ and L 2 attacks.
Again, we report the worst-case results across PGDCE, PGD-Conf and the considered black-box attacks.
However, on SVHN and Cifar10, we are able to break to robustness of the approach, reaching 92% (thresholded)
RErr on SVHN and 99% RTE on Cifar10.
On SVHN, the models performs slightly better against L 2 attacks, with 81% , but can not be considered robust anymore.
[Table] [Table]
[Table] [Table]
we use
[Table] [Table]
[Table] [Table]
[Table]
PGD Conf (
[Table]
2 , (cid:15) = 3 )
AT 0.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
PGD
[Table] [Table]
= 3 )
AT
[Table] [Table]
[Table] [Table]
[Table]
Black-Box
[Table] [Table]
AT
[Table] [Table]
[Table] [Table]
[Table]
AT
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
PGD
[Table]
AT
[Table] [Table]
[Table] [Table]
[Table]
35.7 CCAT, ρ pow = 1.
[Table]
[Table] [Table]
PGD CE ( L 1 , (cid:15) = 10 . 00 )
AT 0.
15.5 CCAT, ρ
[Table]
= 0.
[Table] [Table]
[Table]
12.1 CCAT, ρ pow = 1.
[Table]
[Table] [Table]
Worst-Case
[Table] [Table]
AT
[Table] [Table]
PGD Conf ( L 0 , (cid:15) = 15 )
AT 0.
[Table] [Table]
[Table] [Table]
[Table] [Table]
PGD CE ( L 0 , (cid:15) = 15 )
AT 0.
[Table] [Table]
[Table] [Table]
17.7 CCAT, ρ
[Table]
= 0.
[Table] [Table]
[Table] [Table]
18.6 CCAT,
[Table]
= 0.
[Table] [Table]
[Table] [Table]
[Table]
Black-Box ( L 0 , (cid:15) = 15 )
AT 1.
93.9 CCAT
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]