R ETHINKING U NCERTAINTY IN D EEP L EARNING : W HETHER AND H OW IT I MPROVES R OBUSTNESS
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep neural networks (DNNs) are known to be prone to adversarial attacks, for which many remedies are proposed.
While adversarial training (AT) is regarded as the most robust defense, it suffers from poor performance both on clean examples and under other types of attacks, e.g. attacks with larger perturbations.
Meanwhile, regularizers that encourage uncertain outputs, such as entropy maximization (EntM) and label smoothing (LS) can maintain accuracy on clean examples and improve performance under weak attacks, yet their ability to defend against strong attacks is still in doubt.
In this paper, we revisit uncertainty promotion regularizers, including EntM and LS, in the ﬁeld of adversarial learning.
We show that EntM and LS alone provide robustness only under small perturbations.
Contrarily, we show that uncertainty promotion regularizers complement AT in a principled manner, consistently improving performance on both clean examples and under various attacks, especially attacks with large perturbations.
We further analyze how uncertainty promotion regularizers enhance the performance of AT from the perspective of Jacobian matrices ∇ X f ( X ; θ ) , and ﬁnd out that EntM effectively shrinks the norm of Jacobian matrices and hence promotes robustness.
I NTRODUCTION
Deep neural networks (DNNs) have achieved tremendous success in numerous ﬁelds such as image recognition (Russakovsky et al., 2015) and audio recognition (Graves & Jaitly, 2014).
However, as pointed out by (Szegedy et al., 2013), DNNs are vulnerable to adversarial attacks , where slightly perturbed adversarial examples can easily fool DNN classiﬁers.
The ubiquitous existence of adversarial examples (Kurakin et al., 2016) casts doubts on real-world DNN applications, especially those at high stake, such as access control.
Therefore, many techniques are proposed aiming to enhance the robustness of DNNs to adversarial attacks (Papernot et al., 2016; Kannan et al., 2018).
Regarding defenses to adversarial attacks, adversarial training (AT) (Goodfellow et al., 2014; Madry et al., 2017) is commonly recognized as the most effective defense.
However, it illustrates poor performance on clean examples and under attacks stronger than what they are trained on (Song et al., 2018).
Recent works (Rice et al., 2020) attribute such shortcomings to overﬁtting, but solutions, or even mitigation to them remain open problems.
Meanwhile, many regularization techniques are proposed to improve robustness, among which two closely related regularizers, entropy maximization (EntM) and label smoothing (LS) show performance improvements under weak attacks (Pang et al., 2019; Shafahi et al., 2019) without compromising accuracy on clean examples.
However, as stated by (Uesato et al., 2018; Athalye et al., 2018), strong and various kinds of attacks should be used to better approximate the adversarial risk of a defense.
Therefore, it remains doubtful how they perform under stronger attacks by themselves and how their adversarial risk is.
One appealing property of both EntM and LS is that they both penalize over-conﬁdent predictions and promote prediction uncertainty (Pereyra et al., 2017; Szegedy et al., 2016).
Therefore, both of them are used as regularizers that mitigate overﬁtting in multiple applications (M¨uller et al., 2019).
However, it remains unclear whether EntM and LS can effectively regularize AT, where many other regularizers are ineffective (Rice et al., 2020).
Therefore, in this paper, we perform an extensive empirical study on uncertainty promotion regularizers, i.e. entropy maximization and label smoothing, in the domain of adversarial machine learning.
We carry out experiments on both regularizers, with and without AT, on multiple datasets and under various adversarial settings.
We found out that, although neither EntM and LS are able to provide consistent robustness by themselves , both regularizers complement AT and improve its performances consistently.
The improvement is most evident under attacks with larger perturbations (e.g. 9% improvement under perturbation ε = 16 / 255 for models trained with 8 / 255 AT.
See Table 1 and 3).
In addition, we investigate the underlying mechanisms about how EntM and LS complement AT, and attribute the improvements to the shrunken norm of Jacobian matrix (cid:107)∇ X f ( X ; θ ) (cid:107) ( ∼ 10 times.
See Sect. 6), which indicates better numerical stability and hence better adversarial robustness than AT.
To summarize, we make the following contributions:
We carry out an extensive empirical study about whether uncertainty promotion regularizers, i.e. entropy maximization (EntM) and label smoothing (LS), provide better robustness, and ﬁnd out that while neither of them provides consistent robustness under strong attacks alone , both techniques serve as effective regularizers that improve the performance of AT consistently on multiple datasets, especially under large perturbations.
We provide further analysis on uncertainty promotion regularizers from the perspective of Jacobian matrices, and ﬁnd out that by promoting prediction uncertainty, the norm of Jacobian matrix (cid:107)∇ X f ( X ; θ ) (cid:107) is signiﬁcantly shrunken, leading to better numerical stability and adversarial robustness.
R ELATED W ORK AND D ISCUSSIONS
Adversarial Attacks and Defenses.
(Szegedy et al., 2013; Goodfellow et al., 2014) pointed our the vulnerability of DNNs to adversarial examples, and proposed an efﬁcient attack, the FGSM, to generate such examples.
Since then, as increasingly strong adversaries Carlini & Wagner (2017) are proposed, AT (Madry et al., 2017) is considered as the most effective defense remaining.
More recently, (Zhang et al., 2019) propose TRADES, extending AT and showing better robustness than Madry et al.
We refer readers to (Yuan et al., 2019; Biggio & Roli, 2018) for more comprehensive surveys on adversarial attacks and defenses.
However, all AT methods illustrate common drawbacks.
Speciﬁcally, Rice et al.
(2020) shows that
AT suffers from severe overﬁtting that cannot be easily mitigated, which leads to multiple undesirable consequences.
For example, AT performs poorly on clean examples, and Song et al.
(2018) shows that AT overﬁts on the adversary it is trained on and performs poorly under other attacks.
Uncertainty Promotion Techniques in Machine Learning.
The principle of maximum entropy is widely accepted in not only physics, but also machine learning, such as reinforcement learning (Ziebart et al., 2008; Haarnoja et al., 2018).
In supervised learning, uncertainty promotion techniques, including entropy maximization (Pereyra et al., 2017) and label smoothing (Szegedy et al., 2016) act on model outputs and can improve generalization in many applications, such as image classiﬁcation and language modeling (Pereyra et al., 2017; M ¨ uller et al., 2019; Szegedy et al., 2016).
There are only several works that study uncertainty promotion techniques in the ﬁeld of adversarial machine learning.
ADP (Pang et al., 2019) claims that diversity promotion in ensembles can promote robustness, for which entropy maximization is used.
Essentially, ADP consists of multiple mutually independent models that in total mimic the behavior of entropy maximization.
However, ADP only shows improvements under weak attacks, e.g. PGD with 10 iterations.
Shafahi et al.
(2019) claims that label smoothing can improve robustness beyond AT by itself , yet the claim remains to be further investigated under a wider range of adversaries, e.g. adaptive attacks and black-box attacks.
As stated by (Uesato et al., 2018; Athalye et al., 2018)
that, one should use strong and various attackers for better approximation of the adversarial risk (Uesato et al., 2018), it remains unknown how much robustness EntM or LS alone can provide under strong and diverse attacks.
Moreover, neither of these works study the relationship between AT and uncertainty promotion, which is an important open problem since many regularizers fail to mitigate the overﬁtting of AT (Rice et al., 2020).
We also discuss knowledge distillation (KD)
(Hinton et al., 2015) as another line of work that introduces uncertainty. KD is a procedure that trains a student model to ﬁt the outputs of a teacher network, which are also ’soft’ labels and introduce uncertainty. However, KD alone has been shown not to improve robustness (Carlini & Wagner, 2016). Also, while (Goldblum et al., 2020) proposed adversarially robust distillation (ARD), combining AT and KD, ARD primarily aims to build ro- bust models upon robust teacher models, while we primarily study uncertainty and do not rely on pre-trained teacher models.
Therefore we consider ARD to be orthogonal to our work.
P RELIMINARIES
Notations.
We denote a dataset D =
{ X ( i ) , y ( i ) } ni =1 , where X ( i ) ∈
R d are input data, y ( i )
∈ R C are one-hot vectors for labels.
We denote a DNN parameterized by θ as f ( X ; θ ) :
R d → R C , which outputs logits on classes [ C ] .
We denote f σ ( X ; θ ) =
σ ◦ f
( X ; θ ) , the network followed by softmax activation.
We denote the cross entropy loss as CE(ˆ y, y )
= − (cid:80) Ci =
y i log ˆ y i , the Shannon entropy as H ( p ) =
− (cid:80) Ci =1 p i log p i and the Kullback-Leibler divergence as D KL ( p (cid:107) q )
= (cid:80) Ci =1 p
i log p
i q i .
Adversarial Attacks and Adversarial Training.
Given a data-label pair ( X, y ) and a neural network f
( X ; θ ) , adversarial attacks aim to craft an adversarial example X ( adv ) , (cid:107) X
( adv ) − X (cid:107) ≤ ε , so as to fool f
( X ; θ ) .
Such goal can be formulated as the problem:
[Equation]
[Equation]
) , subject to (cid:107) X (cid:48) − X (cid:107) ≤ ε.
[Equation]
where L atk is some loss function, generally taken as CE .
Eqn.
1 can generally be solved via iterative optimization, such as Projected Gradient Descent (PGD) (Madry et al., 2017).
Adversarial Training (AT) is generally considered the most effective defense against adversarial attacks, where the training set consists of both clean examples and adversarial examples, and the following objective is minimized (Goodfellow et al., 2014),
[Equation]
where the inner maximization is commonly solved via PGD.
We speciﬁcally denote AT which uses
PGD to solve the inner problem as PAT (Madry et al., 2017).
TRADES (Zhang et al., 2019) extends
PAT by optimizing Eqn. 3 in similar min-max styles, leading to better robustness than PAT.
[Equation]
M ETHODOLOGY 4. E NTROPY M AXIMIZATION
Given an example ( X, y ) and a neural network f ( X ; θ ) , the proposed entropy maximization (EntM) minimizes the cross entropy loss, while maximizing the Shannon entropy of the output probability.
[Equation]
where λ is a hyperparameter controlling the strength of entropy maximization.
For normal training, we minimize over E ( X,y )
∼D [ L entm ( X, y )] ; for AT, such as PAT and TRADES, we replace CE with
L entm in Eqn. 2, which for PAT yields Eqn. 5, and similarly for TRADES:
[Equation]
The entropy maximization term penalizes for over-conﬁdent outputs and encourages uncertain predictions, which is different from CE encouraging one-hot outputs.
Such a formulation is also seen in (Pereyra et al., 2017; Dubey et al., 2018) in general deep learning and ﬁne-grained classiﬁcation.
4. C ONNECTION WITH L ABEL S MOOTHING
Label smoothing is also a widely used technique that prevents assigning over-conﬁdent predictions.
Given a data-label pair ( X, y ) , and denote the uniform distribution over C classes as u C , the objective of label smoothing can be formulated as
[Equation]
Label smoothing and entropy maximization are intrinsically similar.
Speciﬁcally, both label smoothing and entropy maximization penalize over-conﬁdent outputs by:
[Equation]
Therefore EntM and LS are connected, with differences in the asymmetric D KL ( f σ ( X ; θ ) (cid:107) u C ) and
D KL ( u C (cid:107) f
σ ( X ; θ )) .
In our paper, we carry out experiments on LS and EntM for comparison.
Q UANTITATIVE E XPERIMENTS 5. E XPERIMENTAL S ETTINGS AND R ESULTS
Datasets and Models We utilize four commonly used datasets for evaluation, CIFAR-10, CIFAR100, SVHN and MNIST.
We train ResNet18, a common selection in adversarial training (Madry et al., 2017) for CIFAR-10, SVHN and CIFAR-100, and a four-layer CNN for MNIST.
Details about training settings and architectures can be found in the Appendix.
Comparison Models We study the robustness of the following models without adversarial training.
• Normal models trained using standard cross entropy.
We denote them as Normal .
• ADP
(Pang et al., 2019), where they built ensembles of models, regularized by maximizing their ensemble diversity.
Entropy maximization is involved in the ensemble diversity term.
We use 3 models as the ensemble, and take the recommended parameters (2 , 0 . 5) .
• Models trained with EntM in Eqn.
We set λ = 2 for all datasets following ADP who also set the entropy term to 2.
• Models trained with label smoothing ( LS ).
We choose γ such that the non-maximal predictions are the same as EntM on each dataset, i
e. γ = 0 .
74 for CIFAR-10 and SVHN, and γ = 0 .
84 for CIFAR-100.
We also study the robustness of the following models with adversarial training.
All adversaries are taken as L ∞ adversary with ε = 8 / 255 , step size 2 / 255 with 7 steps for CIFAR-10, CIFARand SVHN, and ε = 48 / 255 , step size 6 / 255 with 10 steps for MNIST.
• AT (Eqn. 2) with PGD adversary, ( PAT ).
We take α = 0 .
5 as in (Goodfellow et al., 2014).
• PAT with EntM objective (Eqn. 5), PAT-EntM .
We set identical λ as in EntM..
• PAT with label smoothing (Eqn. 6), PAT-LS .
We set identical γ values as in LS.
• TRADES (Zhang et al., 2019), which shows better robustness than PAT.
We take β = 3 .
Threat models.
Without explicit mentioning, we focus on untargeted L ∞ adversaries.
We consider
PGD adversaries with CE objective (referred to PGD and FGSM), and also CW adversaries (Carlini & Wagner, 2017), i.e. adversaries using CW objective and optimized by PGD 2 .
We consider both white and black-box settings.
In black-box settings, adversarial examples are crafted on a proxy model trained on identical data and then transferred to the target model (Papernot et al., 2017).
We show test accuracy under various attacks on CIFAR-10, CIFAR-100, SVHN and MNIST in
Table 1, 2, 3 and 9 3 .
We use FGSM k to denote FGSM with ε = k/ 255 , PGD p − k as p -step PGD attacks with ε = k/ 255 , step size k/ 2550 , and CW p − k as similar to PGD p − k except for the attack objective.
We adopt random initialization with Gaussian X (cid:48)
=
X + N (0 , 0 . 005) with 5 restarts for all attacks.
For black-box attacks, adversarial examples are crafted on a Normal model for defenses without AT, and on a PAT model for those with AT.
Best accuracies higher than random are bolded.
[Table] [Table]
5. H OW MUCH DO U NCERTAINTY P ROMOTION REGULARIZERS IMPROVE ROBUSTNESS ? 5.2. W ITHOUT A DVERSARIAL T RAINING
We ﬁrst analyze the case where no adversarial training is used, namely, EntM, ADP and LS, whose robustness under diverse and strong attacks is under question.
We make the following ﬁndings.
• Under attacks, EntM and LS perform better than ADP in CIFAR-10 and CIFAR-100, and in SVHN, LS performs better than ADP, while EntM performs similarly as ADP.
Therefore we focus our analysis on EntM and LS instead of ADP.
• EntM and LS show gradient obfuscation under weak attacks.
On CIFAR-10 and SVHN, white-box PGD10 attacks succeed less often than black-box ones (See Table 1 and 3, White-Box and Black-Box PGD10), which is a clear indication of gradient obfuscation (Athalye et al., 2018), i.e. gradient-based attacks with few steps perform even worse than gradient-free attacks because gradients around a data point are made useless by the model.
Gradient
obfuscation
on
EntM
and
LS
leads us to use adversaries with more steps to better approximate their true robustness, as multi-step adversaries are more likely to escape from the area where gradient obfuscation happens and ﬁnd adversarial examples.
[Table]
• EntM and LS improve robustness only under small perturbations.
We carry out evaluations on PGD40.
It can be shown that under PGD40-8 and 16, EntM, LS and ADP fail to perform better than
random guessing on all datasets, and improvements over random can only be seen under PGD40-4 on CIFAR-10 and SVHN (Table
and 3).
Therefore, the claims by (Pang et al., 2019; Shafahi et al., 2019) that ADP and LS improve robustness are conditional, and that EntM, LS and ADP are only robust under small perturbations.
5.2. W ITH A DVERSARIAL T RAINING
We then study uncertainty promotion regularizers in conjunction with AT and make the following ﬁndings.
• PAT-EntM and PAT-LS outperform PAT on clean examples by 1% on CIFAR-10, 3% on CIFAR-100, and under strong attacks like PGD40, outperform PAT by 2-9% on CIFAR-10, and 3-6% on CIFAR-100.
See Clean and PGD40 in Table 1, 3, 2.
• PAT-EntM achieves generally similar or better performance than its counterpart PAT-LS, e.g. on CIFAR-10 and CIFAR-100, PAT-EntM achieves 1% better adversarial accuracy, while PAT-LS achieves < 1% better accuracy on clean examples, see Table 1 and 2.
Therefore for the rest of the paper we primarily focus on PAT-EntM. • Robustness improvements are most evident under large perturbations.
For example, accuracies under PGD40-16 improved by 9% on CIFAR-10 and SVHN from PAT to PAT-EntM, see Table 1 and 3, which is exactly the opposite from where AT is not used.
• In terms of accuracy under attacks, PAT-EntM outperforms stronger baseline TRADES by over 3% on CIFAR-10 and CIFAR-100, and achieve similar performances as TRADES on SVHN, further showing the effectiveness of uncertainty promotion regularizers.
• We also tested all AT defenses under CW on CIFAR-10 and CIFAR-100, see Table 1 and 2.
Results show that PAT-EntM shows better robustness under CW attacks than PAT (as much as 7% on CIFAR-10 and 3% on CIFAR-100).
Also, PAT-EntM
shows better robustness than
TRADES (except for CIFAR-10, CW40-8).
The results under a different kind of adversary further demonstrates the robustness improvement provided by EntM.
Following the suggestions of (Athalye et al., 2018), we perform the following sanity checks to make sure PAT-EntM is not only disabling weak attacks like EntM and LS, and better approximate and evaluate its adversarial risk.
Perturbation-Accuracy Curve.
We expand perturbations to test whether the robust accuracy monotonically decreases, and whether sufﬁciently large perturbations reduce accuracy to 0.
We utilize L ∞ PGD with ε = k/ 255 , step size 2 / 255 and k iterations for k ∈ [10 , 150] , on PAT, PATEntM and TRADES models, on CIFAR-10 and CIFAR-100.
We show the results in Fig.
It can be shown that in both datasets and under all ε , PAT-EntM outperforms PAT and TRADES.
Moreover,
[Figure] [Figure]
# step curve on CIFAR-10 and CIFAR-100.
ε = 8 / 255 , 16 /under
sufﬁciently large ε , all accuracies drop to 0, which is another evidence that PAT-EntM is improving robustness rather than obfuscating gradients.
Attacks with more iterations.
We perform attacks with ﬁxed ε and more iterations to make sure attacks converge, and to reliably evaluate the robustness of PAT-EntM. We take ε = 8 / 255 , 16 / 255 , with # steps k ∈ [8 , 256] , step size max(1 / 510 , 2 ε/k ) on CIFAR-10 and CIFAR-100.
We show the results in Fig.
It can be shown that more iterations lead to stronger attacks, and when the attacks converge at 200 steps, PAT-EntM still achieves consistent performance increase (over 4% on both CIFAR-10 and CIFAR-100).
Moreover, PAT-EntM performs better than TRADES in all cases, which further endorses the robustness improvement provided by EntM.
Adaptive Attacks.
We perform adaptive attacks on PAT-EntM. Due to its connection with label smoothing, we leverage CE with smoothed labels in Eqn.
6 as the loss function L atk in Eqn.
We search for the best parameter γ for each attack from [0 , 1] at an interval of 0.1.
We consider ε = 8 / 255 , 16 / 255 on CIFAR-10 and CIFAR-100, and plot corresponding curves also in Fig.
It can be shown in Fig.
2 that with properly selected smooth parameter γ , adaptive attacks are able to reduce robust accuracies compared to non-adaptive ones.
However, even under strong adaptive attacks, we can still see performance gain of PAT-EntM over PAT (e.g. 7% on CIFAR-10 and 4% on CIFAR-100, ε = 16 / 255 ) and in most cases over TRADES, except for CIFAR-10, ε = 8 / 255 , where PAT-EntM is < 0.5% less robust than TRADES under strong adaptive attacks (45.05% and 45.21%).
However, PAT-EntM is 3.8% more accurate on clean examples than TRADES, and therefore we still consider PAT-EntM more competitive than TRADES.
5. H YPERPARAMETER A NALYSIS
We vary the hyperparameter λ in Eqn.
4 and 5 to see how the uncertainty level λ inﬂuences accuraries on both clean and adversarial examples.
We vary λ ∈ [0 , 10] on CIFAR-10 and CIFAR-100.
We plot the accuracies on clean examples and under PGD40-8 in Fig.
The results show that on both datasets, accuracies on both clean and adversarial examples simultaneously increase with λ from 0.1 to 5.
This is an appealing property and contrary to the belief that accuracy and robustness are at odds with each other (Zhang et al., 2019; Tsipras et al., 2018).
We also observe in Fig.
3 that on both datasets, λ = 2 and λ = 5 achieve similar performances (within 0.5%) in terms of both accuracy and robustness, which shows that EntM is not highly sensitive to hyperparameters. [Figure]
H OW U NCERTAINTY P ROMOTION W ORKS - F URTHER A NALYSIS
In this section we dig deeper into how uncertainty promotion works, which may bring deeper insights towards more robust algorithms.
For brevity, we use f ( X ) as a shorthand for f ( X ; θ ) and omit θ .
We focus on the ﬁrst-order approximation of f ( X ) ,
[Equation]
If the approximation holds well locally, we have the following inequality:
[Equation]
where (cid:107)∇ X f ( X ) (cid:107) 2 is the spectral norm (also the largest singular value) of the Jacobian matrix.
It can be shown by Eqn.
11 that (cid:107)∇ X f ( X ) (cid:107) 2 locally upper bounds the local Lipschitz constant of f at point X (denoted as l f,X ).
Denoting M f,X
= f ( X ) y − max i (cid:54) =
y f ( X ) i as the decision margin, Tsuzuku et al.
(2018) shows the relationship between M f,X , Lipschitzness, and robustness.
Theorem 1.
Denote the global Lipschitz constant of f ( X ; θ ) as l f .
The following holds for any data point X , and any perturbation δ .
[Equation]
Theorem 1 shows that, for an arbitrary noise δ , as long as (cid:107) δ (cid:107) 2 ≤ M f,X √ 2 l f , the model f will output correct predictions on X + δ .
Therefore, we focus on the metric normalized margin , deﬁned as M f,X l f for analysis.
In practice, we replace the global Lipschitz constant l f with l f,X at each point X , which is upper bounded by (cid:107)∇ X f ( X ) (cid:107) 2 . We then compute M
[Equation]
X f
[Equation]
X ) (cid:107) 2 for analysis.
However, considering the fact that f ( X ) is non-linear, we deﬁne another term Q f ( X (cid:48) , X ) describing how much f ( X ; θ ) deviates from the linear approximation, to account for cases where Eqn. 9 approximates poorly.
[Equation]
We sample 2,000 test samples ( X, y ) from CIFAR-10, and study the relationship among normalized margin
[Equation]
2 , non-linearity Q f ( X
[Equation]
, X ) , and robustness .
We use L 2 adversaries in correspondence with Theorem 1.
We list related results in Table 4.
With EntM, (cid:107)∇ X f ( X ) (cid:107) 2 is shrunken by over 10 times regardless whether AT is used, leading to a 4-time increase of normalized margin in both EntM and PAT-EntM.
However, while EntM alone achieves high regularized margin, it also massively compromises local linearity compared to Normal, as shown by the high Q f ( X ( adv ) , X ) .
Therefore, the area where the linear approximation in Eqn.
9-11 holds is limited, thereby unable to guarantee robustness.
By contrast, PAT-EntM only worsens local linearity slightly compared to PAT, and therefore the enlarged normalized margin contributes to better robustness.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
We further show that normalized margin, along with local linearity explains the improvement.
We leverage an inﬁnite L 2 adversary to attack each example X until X ( adv ) causes misclassiﬁcation.
We then compute the distortions (cid:107) X ( adv ) − X (cid:107) 2 , and study the relationship between distortions and normalized margins.
A more robust model should require a larger distortion to successfully attack.
First, we plot the normalized margins and distortions in Fig 4(a), and show their Pearson correlations in Table 4.
It can be shown that on Normal, PAT, PAT-EntM, the correlation is much higher than that on EntM. It is also shown in Fig.
4(a)
that although EntM show similar normalized margins as
PAT-EntM, the distortions required are still small.
On the contrary, the larger normalized margins of PAT-EntM lead to larger distortions and hence better robustness than PAT.
We further separate points with Q f ( X ( adv ) , X ) > 65 on EntM, which is the top 40% in terms of Q f ( X ( adv ) , X ) , and plot them separately in Fig. 4(b).
It can be shown that samples with higher local non-linearity show higher normalized margins as PAT-EntM, but no higher distortion and hence no better robustness.
All results show that normalized margins, combined with local linearity, explains robustness.
C ONCLUSION
In this paper, we revisit uncertainty promotion regularizers in the ﬁeld of adversarial learning.
We ﬁnd out that uncertainty promotion regularizers alone are causing gradient obfuscation that may improve performance under weak attacks, and that they alone only provide inconsistent robustness against small perturbations.
Contrarily, our extensive experiments demonstrate that uncertainty promotion regularizers augment AT, improving accuracy on clean examples and enhancing robustness, especially under large perturbations.
We further demonstrate that both good local linearity and shrunken norm of Jacobian matrices contribute to better robustness shown by PAT-EntM than PAT.
We hope that this paper would again underscore the necessity to evaluate under strong attacks, and raise attention to further insights about why uncertainty promotion regularizers work.
We consider theoretical investigations of why they work well alongside AT as an important future work.
A D ETAILED E XPERIMENTAL S ETTINGS A. A RCHITECTURES
We train ResNet18 for CIFAR-10, CIFAR-100 and SVHN.
We use the same architecture of
ResNet18 as in TRADES 4 .
The input size for ResNet18 is 32 × 32 × 3 .
We train a four-layer CNN for MNIST.
The detailed architecture is:
• Input: 28 × 28 × 1 grayscale images.
• 5x5 convolution with stride 5 and 10 ﬁlters.
ReLU activation.
• Max pooling with stride 2.
• 5x5 convolution with stride 5 and 20 ﬁlters.
ReLU activation.
• Max pooling with stride 2.
• Flatten.
• FC layer with input shape 320 and output 10.
ReLU activation.
• FC layer with input shape 50 and output shape 10.
A. T RAINING P ARAMETER S ETTINGS Settings/Models
[Table]
We list all training settings in Table 5.
For each quantitative result, we train 3 models
[Table]
and each PGD is repeated 3 times for average.
For datasets
[Table]
we perform standard data augmentation techniques, including random cropping and random horizontal ﬂipping (on CIFAR-10 and CIFAR-100)
[Table]
B F ULL E XPERIMENTAL R ESULTS
[Table]
We show full experimental results in 6, 7, 8 and 9.
Although in MNIST, the improvement is less signiﬁcant (1% or below)
[Table]
and TRADES signiﬁcantly outperforms both PAT and PAT-EntM, PATEntM still achieves improvement over AT in terms of robustness under various attacks.
C G RADIENT O BFUSCATION OF LS AND E NT M WITHOUT AT
In this section we present more evidence regarding gradient obfuscation.
[Table] [Table]
[Table] [Table]
[Table]
C. R ANDOM S EARCHING
We present other evidence that EntM, LS and ADP alone may sometimes lead to gradient obfuscation.
We sample 1000 images from the test set of CIFAR-10.
We carry out random search attacks for 10,000 times on each image where EntM, LS and ADP succeeded in defending
[Table]
and list the success rate of random attacks in Table 10.
In many seemingly successful defenses (15% on LS, 6% on ADP and EntM)
[Table]
gradient-based attackers are obfuscated that they cannot ﬁnd adversarial examples while such examples do exist.
For reference, a PAT model shows no sign of gradient obfuscation.
We also perform exactly the same experiments for PAT and PAT-EntM. For samples where PAT and
PAT-EntM succeeded in defending, none of them can be successfully attacked via a 10,000-time random search attack.
It endorses that PAT-EntM is not suffering from gradient obfuscation.
[Table] [Table]
C. L OSS S URFACE V ISUALIZATION
We choose data-label pairs ( X, y ) from the CIFAR-10 test set, and plot its local loss surface at X (cid:48)
=
X + (cid:15) 1 d 1 + (cid:15)
d 2 to visually show the properties of EntM and PAT-EntM. d 1 = sign ( ∇ X CE( f σ ( X ; θ ) , y ) is the gradient sign direction, and d 2 is a random direction with (cid:107) d 2 (cid:107) ∞
= 1 .
We choose (cid:15) 1 , (cid:15) 2 ∈ [ − 0 .
Among the ﬁrst 8 samples in the CIFAR-10 test set, we select 2 of them and plot them in Fig.
It can be shown that EntM alone is likely (25% would be a rough estimation. )
to create a ’hole’ in a small neighborhood of X , thereby causing gradient-based iterative attackers to stuck.
This result corresponds with the fact that weak gradient-based attackers cannot achieve a high success rate due to gradient obfuscation.
It also corresponds to the results in Table 4 that EntM alone may compromise local linearity by creating a highly curved surface.
We also plot the decision margin M f,X (cid:48) at X (cid:48)
=
X + (cid:15) 1 d 1 + (cid:15)
d 2 in Fig.
It can be shown that although EntM alone creates a small neighborhood where the margin is large, the area of the neighborhood is small, leaving large areas where M f,X = 0 .
It shows that EntM alone cannot ensure robustness, especially when perturbations are large, and corresponds with results in Table 1, 2, 3.
D TRADES WITH E NT M
We also carry out experiments on CIFAR-10 and CIFAR-100 to see how EntM works with TRADES.
We take λ = 2 and list the results in Table 11 and 12.
It can be shown that TRADES-EntM improves accuracy on clean examples by 6% on CIFAR-100 and 1.7% on CIFAR-10 compared to TRADES, but the robustness improvements are less evident than PAT-EntM, e.g. less than 1% improvement compared to TRADES under attacks on CIFAR-10, and 2-4% improvements on CIFAR-100.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]
CIFARPAT
TRADES
PAT-EntM
TRADES-EntM
White-Box
Clean 86.
FGSM 70.
PGD1068.
PGD1048.
PGD1019.
PGD2067.
PGD2044.
PGD2012.
PGD4067.
PGD4043.
PGD4010.
Black-Box
PGD1076.
PGD1063.
PGD2076.
PGD2062.
[Table]
PAT
TRADES
PAT
[Table]
TRADES
[Table]
White-Box
Clean 59.
FGSM 38.
PGD1037.
PGD1022.
PGD107.
PGD2036.
PGD2020.
PGD205.
PGD4036.
PGD4019.
PGD404.
Black-Box
PGD1049.
PGD1038.
PGD2049.
PGD2038.