P ROPER N ETWORK I NTERPRETABILITY H ELPS A DVERSARIAL R OBUSTNESS IN C LASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent works have empirically shown that there exist adversarial examples that can be hidden from neural network interpretability ( namely, making network interpretation maps visually similar) , and interpretability is itself susceptible to adversarial attacks.
In this paper, we theoretically show that with the correct measurement of interpretation, it is actually difﬁcult to hide adversarial examples, as conﬁrmed by experiments on MNIST, CIFAR-10 and Restricted ImageNet.
Spurred by that, we develop a novel defensive scheme built only on robust interpretation (without resorting to adversarial loss minimization).
We show that our defense achieves high classiﬁcation robustness, outperforming state-of-the-art adversarial training methods against attacks of large perturbation while attaining high interpretation robustness under various settings of adversarial attacks
I NTRODUCTION
It has become widely known that convolutional neural networks (CNNs) are vulnerable to adversarial examples , namely, perturbed inputs with intention to mislead networks’ prediction (Szegedy et al., 2014; Goodfellow et al., 2015; Papernot et al., 2016a; Carlini & Wagner, 2017; Chen et al., 2018;
Su et al., 2018).
The vulnerability of CNNs has spurred extensive research on adversarial attack and defense.
To design adversarial attacks, most work has focused on creating either imperceptible input perturbations (Goodfellow et al., 2015; Papernot et al., 2016a; Carlini & Wagner, 2017;
Chen et al., 2018) or adversarial patches robust to the physical environment (Eykholt et al., 2018; Brown et al., 2017; Athalye et al., 2017).
Many defense methods have also been developed to prevent CNNs from misclassiﬁcation when facing adversarial attacks.
Examples include defensive distillation (Papernot et al., 2016b), training with adversarial examples (Goodfellow et al., 2015), input gradient or curvature regularization (Ross & Doshi-Velez, 2018; Moosavi-Dezfooli et al., 2019), adversarial training via robust optimization (Madry et al., 2018), and TRADES to trade adversarial robustness off against accuracy (Zhang et al., 2019).
Besides studying adversarial effects on network prediction decisions, this work explores the connection between adversarial robustness and network interpretability, and provides novel insights on when and how interpretability helps the robustness.
Having a prediction might not be enough for many real-world machine learning applications.
It is also crucial to demystify why CNNs make certain decisions.
Thus, the problem of network interpretation arises.
Various methods have been proposed to understand the mechanism of decision making by CNNs.
One category of methods justify a prediction decision by assigning importance values to reﬂect the inﬂuence of individual pixels or image sub-regions on the ﬁnal classiﬁcation.
Examples include pixel-space sensitivity map methods (Simonyan et al., 2013; Zeiler & Fergus, 2014;
Springenberg et al., 2014; Smilkov et al., 2017; Sundararajan et al., 2017) and class-discriminative localization methods (Zhou et al., 2016; Selvaraju et al., 2017; Chattopadhay et al., 2018; Petsiuk et al., 2018), where the former evaluates the sensitivity of a network classiﬁcation decision to pixel variations at the input, and the latter localizes which parts of an input image were looked at by the network for making a classiﬁcation decision.
Sensitivity map methods include vanilla gradient (Simonyan et al., 2013), deconvolution (Zeiler & Fergus, 2014), guided backpropagation (Springenberg et al., 2014), SmoothGrad (Smilkov et al., 2017), integrated gradient (IG) (Sundararajan et al., 2017) to name a few.
They highlight ﬁne-grained details in the image but are not class-discriminative for visual explanation.
By contrast, localization approaches like class activation map (CAM)
(Zhou et al., 2016), GradCAM (Selvaraju et al., 2017), GradCAM++ (Chattopadhay et al., 2018) and RISE (Petsiuk et al., 2018) are highly class-discriminative, namely, localizing image sub-regions reasoned for a prediction class. We refer readers to Sec. 2 for some representative interpretation methods.
Besides interpreting CNNs via feature importance maps, some methods zoom into the internal response of neural networks.
Examples include network dissection (Bau et al., 2017), which evaluates the alignment between individual hidden units and semantic concepts, and learning perceptuallyaligned representations from robust training (Engstrom et al., 2019).
Very recently, some works (Xu et al., 2019b;a; Zhang et al., 2018; Subramanya et al., 2018; Ghorbani et al., 2019; Dombrowski et al., 2019; Chen et al., 2019) are beginning to study adversarial robustness by exploring the spectrum between classiﬁcation accuracy and network interpretability.
It was shown in (Xu et al., 2019b;a) that an imperceptible adversarial perturbation to fool classiﬁers can lead to a signiﬁcant change in a class-speciﬁc network interpretability map, e.g., CAM.
Thus, it was argued that such an interpretability discrepancy can be used as a helpful metric to differentiate adversarial examples from benign inputs.
However, the work (Zhang et al., 2018; Subramanya et al., 2018) showed that under certain conditions, generating an attack (which we call interpretability sneaking attack, ISA ) that fools the classiﬁer as well as its coupled interpreter (in terms of keeping interpretability map highly similar to that of benign input) is not signiﬁcantly more difﬁcult than generating adversarial inputs deceiving the classiﬁer only.
Besides investigating robustness in classiﬁcation through the lens of interpretability, the work (Ghorbani et al., 2019;
Dombrowski et al., 2019) studied the robustness in network interpretation maps, showing that which can signiﬁcantly be manipulated via imperceptible input perturbations but keeping the classiﬁer’s decision intact .
We call this type of threat model attack against interpretability (AAI) .
The existing work had no agreement on the relationship between robustness in interpretation and robustness in classiﬁcation.
Spurred by that, we attempt to explore this relationship from both attack and defense perspectives.
The most relevant work to ours is (Chen et al., 2019), which robustiﬁed network interpretation with the aid of integrated gradient (IG), an axiomatic attribution map.
It proposed a robust attribution training, which was shown as a principled generalization of previous formulations of robust classiﬁcation and an effective defense against AAI.
In this paper, we ﬁrst investigate when ISA is possible, and then relate our insights on ISA to robust classiﬁcation and robust interpretability.
Different from the previous work, our paper contains the following contributions.
We provide an answer to the question of when adversarial examples can bypass interpretability discrepancy.
We show that enforcing stealthiness of adversarial examples from network interpretation could be challenging.
Its difﬁculty relies on how one measures the interpretability discrepancy caused by input perturbations.
We propose an (cid:96) 1 norm based 2-class interpretability discrepancy measure and theoretically show that constraining it helps adversarial robustness.
We develop an interpretability-aware robust training method and empirically show that interpretability alone can be used to defend adversarial attacks for both misclassifcation and misinterpretation.
Compared to the IG-based robust attribution training (Chen et al., 2019), our approach is simpler in implementation, and provides better robustness even as facing a strong adversary.
P RELIMINARIES AND M OTIVATION : I NTERPRETABILITY OF CNN S FOR J USTIFYING A C LASSIFICATION D ECISION
To explain what and why CNNs predict, we consider two types of network interpretation methods: a) class activation map (CAM)
(Zhou et al., 2016; Selvaraju et al., 2017; Chattopadhay et al., 2018) and b) pixel sensitivity map (PSM) (Simonyan et al., 2013; Springenberg et al., 2014;
Smilkov et al., 2017; Sundararajan et al., 2017; Yeh et al., 2019).
Let f ( x ) ∈ R C denote a CNN-based predictor that maps an input x ∈ R d to a probability vector of C classes.
Here f c ( x ) , the c th element of f ( x ) , denotes the classiﬁcation score (given by logit before the softmax) for class c .
Let L ( x , c ) denote an interpreter (CAM or PSM) that reﬂects where in x contributes to the classiﬁer’s decision on c .
CAM-type methods.
CAM (Zhou et al., 2016) produces a class-discriminative localization map for CNNs, which performs global averaging pooling over convolutional feature maps prior to the softmax.
Let the penultimate layer output K feature maps, each of which is denoted by a vector representation A k ∈ R u for channel k ∈
[ K ] .
Here [ K ] represents the integer set { 1 , 2 , . .
. , K } .
The i th entry of CAM L CAM ( x , c ) is given by
[Equation]
where w ck is the linear classiﬁcation weight that associates the channel k with the class c , and A k,i denotes the i th element of A k .
The rationale behind (1) is that the classiﬁcation score f c ( x ) can be written as the average of CAM values (Zhou et al., 2016), f c ( x )
=
(cid:80) ui
=1 [ L CAM ( x , c )] i .
For visual explanation, L CAM ( x , c ) is often up-sampled to the input dimension d using bi-linear interpolation.
GradCAM (Selvaraju et al., 2017) generalizes CAM for CNNs without the architecture ‘global average pooling → softmax layer’ over the ﬁnal convolutional maps.
Speciﬁcally, the weight w ck in (1) is given by the gradient of the classiﬁcation score f c ( x ) with respect to (w.r.t.)
the feature map
A k , w ck = 1 u (cid:80) ui
=1 ∂f c ( x ) ∂A
k,i .
GradCAM++ (Chattopadhay et al., 2018), a generalized formulation of GradCAM, utilizes a more involved weighted average of the (positive) pixel-wise gradients but provides a better localization map if an image contains multiple occurrences of the same class.
In this work, we focus on CAM since it is computationally light and our models used in experiments follow the architecture ‘global average pooling → softmax layer’.
PSM-type methods.
PSM uses calculations with gradients to assign importance scores to individ- ual pixels toward explaining the classiﬁcation decision about an input.
Examples of commonly-used approaches include vanilla gradient (Simonyan et al., 2013), guided backpropogation (Springenberg et al., 2014), SmoothGrad (Smilkov et al., 2017), and integrated gradient (IG) (Sundararajan et al., 2017).
In particular, IG satisﬁes the completeness attribution axiom that PSM ought to obey.
Specif- ically, it averages gradient saliency maps for interpolations between an input x and a baseline image a :
[Equation]
where m is the number of steps in the Riemman approximation of the integral.
The completeness axiom (Sundararajan et al., 2017, Proposition 1) states that (cid:80) di
{+=
+}
[ L IG ( x , c )] i = f c ( x ) − f c ( a ) , where the baseline image a is often chosen such that f c ( a ) ≈ 0 , e.g., the black image.
Note that CAM also satisﬁes the completeness axiom.
PSM is able to highlight ﬁne-grained details in the image, but is computationally intensive and not very class-discriminative compared to CAM (Selvaraju et al., 2017).
Interpretability discrepancy caused by adversarial perturbation.
Let x (cid:48)
=
x + δ represent an adversarial example w.r.t.
x , where δ denotes an adversarial perturbation .
By replacing the input image x with x (cid:48) , the CNN will be fooled from the true label t to the target (incorrect) label t (cid:48) .
It was recently shown in (Xu et al., 2019b;a) that the adversary could introduce an evident interpretability discrepancy w.r.t.
both the true and the target label in terms of L ( x , t ) vs. L ( x (cid:48) , t ) , and L ( x , t (cid:48) )
vs. L ( x (cid:48) , t (cid:48) ) .
An illustrative example is provided in Figure 1.
We see that an adversary suppresses the network interpretation w.r.t.
the true label but promotes the interpretation w.r.t.
the target label.
We also observe that compared to IG, CAM and GradCAM++ better localize class-speciﬁc discriminative regions.
These results reveal two observations on how measuring interpretation discrepancy affects classiﬁcation robustness: a) interpretability discrepancy may be used to detect adversarial examples, b) interpretability discrepancy itself may be vulnerable to adversarial perturbations.
In what follows, we explore the spectrum between adversarial robustness and interpretability from a uniﬁed perspective considering both the adversarial vulnerability of interpretability discrepancy and the value of interpretability discrepancy in a defense .
R OBUSTNESS IN C LASSIFICATION VS . I NTERPREATION M AP : A N A TTACK P ERSPECTIVE
In this section, we examine two types of threat models, interpretability sneaking attack (ISA) and attack against interpretability (AAI) .
Since an adversarial example designed for misclassiﬁcation
[Figure]
CAM
GradCAM++
IG
O r i g i n a l e xa m p l e x
A d v er s a r i a l e xa m p l e x (cid:48)
L ( · ,t )
L ( · ,t (cid:48) )
[Figure]
L ( · ,t )
L ( · ,t (cid:48) )
[Figure]
L ( · ,t )
L ( · ,t (cid:48) )
[Figure]
gives rise to interpretability discrepancy (which could then be used as a detector for the adversarial input), the problem of ISA arises: One may wonder whether or not it is easy to generate adversarial examples that mistake classiﬁcation but keep interpretation intact .
Such adversarial vulnerability could have serious consequences when classiﬁcation and interpretation are jointly used in tasks like medical diagnosis (Subramanya et al., 2018), and call into question the faithfulness of interpretation to network classiﬁcation.
One the other hand, it is suggested from interpretability discrepancy that an interpreter could be quite sensitive to input perturbations.
Thus, the problem of AAI arises: One may wonder if perturbed inputs could induce differing explanations but without altering predictions .
3. R ETHINKING ISA FROM PERSPECTIVE OF INTERPRETABILITY DISCREPANCY
Previous work (Zhang et al., 2018; Subramanya et al., 2018) showed that it is not difﬁcult to hide adversarial examples from network interpretation when the interpretability discrepancy is measured w.r.t.
a single class label (either the true label t or the target label t (cid:48) ).
However, we see from
Figure 1 that the adversary (against classiﬁcation) alters interpretability maps w.r.t.
both t and t (cid:48) .
This motivates us to rethink whether the single-class interpretability discrepancy measure is proper, and whether ISA is truly easy to bypass an interpretability discrepancy check.
We consider the following generic form of interpretability discrepancy
[Equation]
) where recall that x and x (cid:48) are natural and adversarial examples respectively, L represents an interpreter, e.g., CAM or IG, C denotes the set of class labels used in L , |C| is the cardinality of C , and we consider p ∈ { 1 , 2 } in this paper.
Clearly, a speciﬁcation of (3) relies on the choice of C and p .
We specify (3) with C = { t, t (cid:48) } and p = 1 , leading to (cid:96) 1 2-class interpretability discrepancy measure,
[Equation]
Rationale behind (cid:96) 1 2-class discrepancy measure.
Compared to the previous work (Zhang et al., 2018; Subramanya et al., 2018) using a single class label, we choose C = { t, t (cid:48) } 1 , motivated by the fact that an interpretability discrepancy occurs w.r.t.
both t and t (cid:48)
(Figure 1).
Moreover, although
Euclidean distance (namely, (cid:96) 2 norm or its square) is arguably one of the most commonly-used discrepancy metrics (Zhang et al., 2018), we show in Proposition 1 that the proposed interpretability discrepancy measure D 2 ,(cid:96) 1 ( x , x (cid:48) ) has a non-trivial lower bound for any successful adversarial attack.
This provides an explanation on why it could be difﬁcult to hide adversarial examples from network interpretability methods.
Moreover, (cid:96) 1 is an upper bound of the (cid:96) 2 norm and promotes the sparsity of interpretability discrepancy, which enforces pixels of L to stay intact when facing input perturbations.
Proposition 1.
Given a classiﬁer f ( x ) ∈ R C and its interpreter L ( x , c ) for c ∈ [ C ] , suppose that the interpreter satisﬁes the completeness axiom, namely, (cid:80) i
[ L ( x , c )]
i =
f c ( x ) .
For a natural example x and an adversarial example x (cid:48) with prediction t and t (cid:48) respectively, D 2 ,(cid:96) 1 ( x , x (cid:48) ) in (4) has the lower bound
[Equation]
Proof : See proof in Appendix A.
[Equation]
Proposition 1 connects D 2 ,(cid:96) 1 ( x , x (cid:48) ) with the classiﬁcation margin f t ( x ) − f t (cid:48)
( x ) .
Thus, if a classiﬁer has a large classiﬁcation margin on the natural example x , it will be difﬁcult to ﬁnd a successful adversarial attack with small interpretability discrepancy.
In other words, constraining the interpretability discrepancy prevents misclassiﬁcation since generating adversarial examples becomes infeasible under D 2 ,(cid:96) 1 ( x , x (cid:48) ) < 12 ( f t ( x ) − f t (cid:48)
( x )) .
Also, the completeness condition of L suggests specifying (4) with CAM (1) or IG (2).
Here we focus on CAM due to its light computation.
Design of ISA.
We pose the following optimization problem for design of ISA, which not only fools a classiﬁer’s decision but also minimizes the resulting interpretability discrepancy, minimize δ
[Equation]
subject to (cid:107) δ (cid:107) [Equation] ≤ (cid:15), (6) where λ > 0 is a regularization parameter that strikes a balance between the success of an attack and its resulting interpretability discrepancy, τ ≥
(e.g., 0 . 1 used in the paper) is a tolerance on the classiﬁcation margin of a successful attack between the target label t (cid:48) and the non-target top-1 prediction label arg max j (cid:54) =
t (cid:48)
f j ( x + δ ) , D 2 ,(cid:96) 1 was deﬁned by (4), and (cid:15) > 0 is a (pixellevel) perturbation size.
In (6), the ﬁrst term of the objective corresponds to a C&W-type attack loss (Carlini & Wagner, 2017), which reaches − τ if the attack succeeds in misclassiﬁcation.
To ﬁnd ISA of minimum interpretability discrepancy, we perform a bisection on λ until there exists no successful attack that can be found when λ further decreases.
Although we focus on (cid:96) ∞ attack in (6), but a similar formulation applies to (cid:96) 1 and (cid:96) 2 attacks.
Attacks are found using PGD, with subgradients taken at non-differentiable points.
We consider only targeted attacks to better evaluate the effect on interpretability of target classes, although this approach can be extended to an untargeted setting (e.g., by using the target label-free interpretability discrepancy measure introduced in the next section).
Difﬁculty of generating ISA versus interpretability discrepancy measure.
Through an illustrative example in Figure 2, we elaborate on how the choice of interpretability discrepancy measure plays a crucial role on the difﬁculty of hiding adversarial examples from network interpretation.
We refer readers to Sec.
5 for more experimental results.
We generate ISA under different speciﬁcations of (3) for different values of perturbation size (cid:15) .
Compared to (cid:96) 1 / (cid:96) 2 1-class (true class t ), (cid:96) 2 2-class, and (cid:96) 1 / (cid:96) 2 all-class
we see that the (cid:96) 1 2-class interpretability discrepancy (4) cannot be easily mitigated even as the attack power (in terms of (cid:15) ) increases.
This is veriﬁed by a) its high interpretability discrepancy score and b) its ﬂat slope of discrepancy score against (cid:15) in Figure 2-(a)&(b).
Figure 2-(c) further shows CAMs of adversarial examples w.r.t.
the true label t and the target label t (cid:48) generated by (cid:96) 1 1/2/all-class ISAs.
We note that both 1-class measure and all-class measure could give a false sense of ease of hiding adversarial examples.
For (cid:96) 1 1-class ISA, although the interpretability discrepancy w.r.t.
t is minimized, the discrepancy w.r.t.
t (cid:48) remains large, supported by the observation that (cid:96) 1 1-class ISA even yields a smaller correlation between L ( x (cid:48) , t (cid:48) ) and L ( x , t (cid:48) ) than PGD attack.
Similarly, although the averaged discrepancy over all classes is minimized for (cid:96) 1 all-class ISA, discrepancies w.r.t.
speciﬁc labels such as t and t (cid:48) do not necessarily reduce.
This illustrates that although ISA may be found, with the proper choice of discrepancy measure, ISA with a low discrepancy becomes quite difﬁcult.
3. A TTACK AGAINST
I NTERPRETABILITY (
Different from ISA, AAI produces input perturbations to maximize the interpretability discrepancy while keeping the classiﬁcation decision intact.
Thus, AAI provides a means to evaluate the ad- [Figure] versarial robustness in interpretability.
Since t = arg max i f [Equation] x ) = [Equation]
f
[Equation]
x
[Equation]
) =
[Equation]
in AAI, the 2-class interpretability discrepancy measure (4) reduces to its 1-class version. The problem of generating AAI is then cast as
[Equation]
=
t f j ( x + δ ) − f t ( x + δ ) , 0 } − D 1 ( x , x + δ ) subject to (cid:107) δ (cid:107) ∞ ≤ (cid:15), (7) where the ﬁrst term is a hinge loss to enforce f t ( x + δ ) ≥ max j (cid:54) =
t f j ( x + δ ) , namely, arg max i f i ( x (cid:48) )
=
t (unchanged prediction under input perturbations), and D 1 denotes a 1-class interpretability discrepancy measure, e.g., D 1 ,(cid:96) 1 from (4), or the top- k pixel difference between interpretability maps (Ghorbani et al., 2019).
Similar to (6), the regularization parameter λ in (7) strikes a balance between stealthiness in classiﬁcation and variation in interpretability.
Experiments in Sec.
5 will show that the state-of-the-art defense methods against adversarial examples do not necessarily preserve robustness in interpretability as (cid:15) increases, athough the prediction is not altered.
I NTERPRETABILITY -A WARE R OBUST T RAINING : D EFENSE P ERSPECTIVE
We recall from Sec.
3.1 that adversarial examples that intend to fool a classiﬁer could be difﬁcult to evade the (cid:96) 1 2-class interpretability discrepancy.
Thus, constraining the interpretability discrepancy helps to prevent misclassiﬁcation.
Motivated by this observation about ISA, we introduce a interpretability based defense method that penalizes interpretability discrepancy to achieve high classiﬁcation robustness.
As an additional beneﬁt, it also robustiﬁes the classiﬁer against
AAI, where the adversary maximizes the interpretability discrepancy.
Target label-free interpretability discrepancy.
Different from attack generation, the (cid:96) 1 2-class discrepancy measure (4) cannot directly be used by a defender since the target label t (cid:48)
speciﬁed by the adversary is not known a priori .
To circumvent this issue, we propose to approximate the in- terpretability discrepancy w.r.t.
the target label by weighting discrepancies from all non-true classes according to their importance in prediction.
This modiﬁes (4) to
[Equation]
(8) where the softmax function
[Equation]
adjusts the importance of non-true labels according to their classiﬁcation conﬁdence.
Clearly, when x (cid:48) succeeds in misclassiﬁcation, the top-1 predicted class of x (cid:48) becomes the target label and the resulting interpretability discrepancy is most penalized.
Interpretability-aware robust training.
We propose to train a classiﬁer against the worst-case interpretability discrepancy (8), yielding the min-max optimization problem minimize θ
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] (9) where θ denotes the model parameters, D train denotes the training dataset, f train is the training loss (e.g., cross-entropy loss), γ > 0 is a regularization parameter, and for ease of notation we omit the parameters θ and t in ˜ D
( x , x (cid:48) ) .
Here γ controls the tradeoff between clean accuracy and robustness; see Appendix C for experiments analyzing this tradeoff.
In problem (9), the inner maximization is only used to evaluate the worst-case interpretability discrepancy.
Thus, it is different from adversarial training (Madry et al., 2018), where the training loss is replaced with the adversarial loss maximize x (cid:48) :
(cid:107) x (cid:48) − x (cid:107) ∞ ≤ (cid:15) f train ( θ , x (cid:48) ; x , t ) .
The formulation (9) allows us to examine whether or not robust interpretation is directly beneﬁcial to robust classiﬁcation.
For completeness, we will also provide experiment results on a modiﬁed formulation of (9) with the use of the adversarial loss.
Difference to (Chen et al., 2019).
The recent work (Chen et al., 2019) proposed improving adversarial robustness by leveraging robust IG attributions.
However, different from (Chen et al., 2019), our approach is motivated by the importance of (cid:96) 1 2-class interpretability discrepancy measure.
We will show in Sec. 5 that the incorporation of interpretability discrepancy w.r.t.
target class labels, namely, the second term in (8), plays a key role in boosting classiﬁcation and interpretation robustness.
This is because robust interpretability implies robust classiﬁcation only when interpretation maps are measured with a proper metric.
We will also show that our proposed method is sufﬁcient to improve adversarial robustness even in the absence of adversarial loss, while the robust attribution regularization method (Chen et al., 2019) becomes ineffective when the attack becomes stronger.
Last but not the least, beyond IG, our proposed theory and method apply to any network interpretation method with completeness axiom.
The use of CAM avoids Riemman approximation used in IG and thus simpliﬁes the implementation during robust training.
E XPERIMENTS
In this section, we empirically show the effectiveness of our proposed methods in various attack and defense settings.
For ISA, we examine how the interpretability discrepancy measure plays a role in hiding adversarial examples from network interpretation.
For AAI, we evaluate its attack success rate under the natural and various robust models.
For interpretability-aware robust training, we demonstrate its advantages in a) defending against PGD attacks with different steps and perturbation sizes (Madry et al., 2018; Athalye et al., 2018), b) defending against unforeseen adversarial attacks (Kang et al., 2019), c) rendering robustness in interpretability, and d) computation efﬁciency compared to the IG-based robust attribution regularization method (Chen et al., 2019).
Datasets, CNN models, and experiment setting.
We evaluate networks trained on the MNIST,
CIFAR-10 and a restricted ImageNet (R-ImageNet) dataset used in (Engstrom et al., 2019).
We consider three models, Small (for MNIST and CIFAR), Pool (for MNIST) and WResnet (for CIFAR and R-ImageNet): 1) a small CNN architecture consisting of three convolutional layers of 16, and 100 ﬁlters (Small), 2) a CNN architecture with two convolutional layers of 32 and 64 ﬁlters each followed by max-pooling which is adapted from (Madry et al., 2018)
(Pool), and 3) a Wide Resnet from (Madry et al., 2018)
(WResnet).
We refer readers to Appendix B for more details.
In the following experiments, we consider 5 baselines from the literature: i) standard training ( Normal ), ii) adversarial training ( Adv ) (Madry et al., 2018), iii) TRADES (Zhang et al., 2019), iv)
IG-Norm that uses IG-based robust attribution regularization (Chen et al., 2019), v) IG-Sum-Norm (namely, IG-Norm with adversarial loss).
Additionally, we consider 3 variants of our method: vi) the proposed interpretability-aware robust training method (9) (we call Int ), vii)
Int using (cid:96) 1 1 -class discrepancy ( Int-1-class ), and viii)
Int with adversarial loss ( Int-Adv ).
For interpretability-based methods, the regularization parameter γ is set to 0.01. In Appendix C, we explore different settings of the parameter γ for our method to demonstrate that we can achieve a range of different points on the robustness-accuracy tradeoff.
[Table]
Evaluating ISA.
We evaluate the effect of interpretability discrepancy measure on ease of ﬁnding
ISAs.
Spurred by Figure 2, such an effect is quantiﬁed by calculating minimum discrepancy required in generating ISAs against different values of perturbation size (cid:15) in (6).
We conduct experiments over 4 network interpretation methods: i) CAM, ii) GradCAM++, iii) IG, and iv) internal representation at the penultimate (pre-softmax) layer (denoted by Repr ).
In order to fairly compare among different interpretation methods, we compute a normalized discrepancy score (NDS) extended from (3) and a normalized slope (NSL) that measures the relative change of NDS for (cid:15) ∈
[ˇ (cid:15), ˆ (cid:15) ] .
We refer readers to Appendix D for more details.
In Table 1, we present NDS and NSL of ISAs generated under different realizations of interpretability discrepancy measure (3), each of which is given by a combination of interpretation method (CAM, GradCAM++, IG or Repr), (cid:96) p norm ( p ∈ { 1 , 2 } ) and number of interpreted classes.
Note that Repr is independent of the number of classes, and thus we report NDS and NSL corresponding to Repr in the 2-class column of Table 1.
Given an (cid:96) p norm and an interpretation method, we consistently ﬁnd that the use of 2-class measure achieves the largest NDS and smallest NSL at the same time.
This implies that the 2-class discrepancy measure increases the difﬁculty of ISA to evade network interpretability check.
Moreover, given a class number and an interpretation method, we see that NDS under (cid:96) 1 norm is greater than that under (cid:96) 2 norm, since the former is naturally an upper bound of the latter.
Also, the use of (cid:96) 1 norm often yields a smaller value of NSL, implying that the (cid:96) 1 -norm based discrepancy measure is more resistant to ISA.
Furthermore, by ﬁxing the combination of (cid:96) 1 norm and 2 classes, we observe that IG is the most resistant to ISA due to their relatively high NDS and low ISA, and Repr yields the worst performance.
However, compared to
CAM, the computation cost of IG increases dramatically as the input dimension, the model size, and the number of steps in Riemman approximation increase.
We ﬁnd that it becomes infeasible to generate ISA using IG for WResnet under R-ImageNet within 200 hours.
Evaluating AAI.
We specify problem (7) under the top-k attack setting (Ghorbani et al., 2017), where the top-k intersection between the original and adversarial interpretability maps is minimized.
The strength of AAI is then measured by the Kendall’s Tau order rank correlation between the aforementioned two interpretability maps (Chen et al., 2019).
The higher the correlation is, the more robust the model is against AAI. Reported rank correlations are averaged over a test set.
[Figure]
In Table 2, we present the performance of AAI under multiple perturbation sizes to attack models trained using different training methods.
For the model Small under MNIST, we evaluate AAI over 5 baselines (Normal, Adv, TRADES, IG-Norm, IG-Sum-Norm) and 3 variants of our method (Int, Int-1-class and Int-Adv).
For a larger model WResnet under CIFAR-10, the training methods using IG (IG-Norm and IG-Norm-Sum) are excluded due to their prohibitive computation cost.
The insights learnt from Table 2 are summarized as below.
First, the normally trained model (Normal) does not automatically give robustness guarantees in interpretability, particularly for AAI with (cid:15) ≥ 0 .
Second, the methods Adv, TRADES, IG-Sum-Norm and IntAdv that uses adversarial loss offer certain robustness against AAI but the performance gets worse as the perturbation size (cid:15) increases.
Third, in the absence of adversarial loss, the baseline IG-Norm becomes less robust as (cid:15) increases.
By contrast, our proposed method Int is consistently more robust and its advantage becomes more evident as (cid:15) increases, including when (cid:15) is increased beyond the value used for training.
Evaluating interpretability-aware robust training.
We previously showed in Table 2 that interpretability-aware robust training (Int, Int-Adv) often achieves more robust interpretability than state-of-the-art adversarial training methods particularly for large perturbation size.
We next provide a thorough evaluation on how interpretability helps adversarial robustness against misclassiﬁcation.
Robustness versus efﬁciency.
In Figure 3, we present the training time (left y-axis) and the adversarial test accuracy (right y-axis) for different training methods (x-axis), which are ranked in a decreasing order of computation complexity.
Here the adversarial test accuracy (ATA) is measured using 200 -step ( (cid:96) ∞ -norm) perturbation sizes (cid:15) = 0 .
3 and 0 .
4 on the Small MNIST model (Madry et al., 2018).
Note that all AT-type methods (IG-Norm-Sum, Int-Adv, TRADES and Adv) offer robust classiﬁcation at (cid:15) = 0 . 3 with ATA around 80% .
Among non-AT but interpretability promoted defensive schemes (IG-Norm, Int, Int-1class), we ﬁnd that only the proposed Int yields competitive ATA, and outperforms all AT-type methods except Int-Adv at (cid:15) = 0 .
Particularly, IG-Norm degrades signiﬁcantly in robust classiﬁcation when PGD attack becomes stronger (also see results in Table 4).
Moreover, the non-robustness of Int-1-class veriﬁes the importance of 2-class interpretability discrepancy measure on rendering robust classiﬁcation.
Last but not the least, Adv, TRADES and Int-based methods have similar computational complexity, but IG-based methods make training time (per epoch) signiﬁcantly higher, 3 times more than Int-Adv even under the Small MNIST model.
Int does not cause obfuscated gradients and importance of 2-class measure.
It was shown in (Athalye et al., 2018;
Carlini, 2019) that some defense methods cause obfuscated gradients , which give a false sense of security.
There exist two characteristic behaviors of obfuscated gradients: a) One-step attacks perform better than iterative attacks; b) Increasing distortion bound does not increase success.
Motivated by that, we evaluate our interpretability-aware robust training methods under PGD attacks with a) different steps and b) different perturbation sizes.
Tablereports ATA of interpretability-aware robust training relative to various baselines, where 200 -step PGD attacks are conducted for (cid:15) ∈ { 0 , 0 . 05 , 0 . 1 , and 8 / 255 on CIFAR for robust training.
As we can see, ATA decreases as (cid:15) increases.
Thus, our methods do not exhibit the behavior a) of obfuscated gradients.
We also observe that compared to Adv and TRADES, Int achieves reasonable but worse ATA as (cid:15) < 0 . 3 on MNIST and (cid:15) ≤ 8 / 255 on CIFAR. As (cid:15) used in PGD attack achieves the value used for robust training, Int achieves ATA 0 .
790 against ATA 0 .
890 from Adv on MNIST, but outperforms Adv on CIFAR ( 0 .
270 vs 0 .
Interestingly, the advantage of Int becomes more evident as the adversary becomes stronger, i.e., (cid:15) > 0 .
3 on MNIST and (cid:15) > 8 / 255 on CIFAR.
We highlight that such a robust classiﬁcation is achieved by promoting robustness of interpretability alone (without using adversarial loss).
It is worth mentioning that IG-Norm fails to defend PGD attack with (cid:15) = 0 .
3 for the Small MNIST model.
We further note that Int-1-class performs much worse than Int, supporting the importance of using 2-class discrepancy measure (see Prop. 1).
Besides robust classiﬁcation, IG-Norm and Int-1-class are also not sufﬁcient to render robustness in interpretation (Table 2).
Table 4 shows ATA of interpretability-aware robust training against k -step PGD attacks (examining behavior b) of obfuscated gradients), where k ∈ { 1 , 10 , 100 , 200 } .
As we can see, ATA decreases as k increases.
This also suggests that the high robust accuracy from our methods is not a result of obfuscated gradients.
Similar to Table 3, we see that compared to Int, IG-Norm and Int-1-class are insufﬁcient to defend PGD attacks with k ≥ 100 .
Beyond (cid:96) ∞ -norm PGD attacks.
In Table 5, we present ATA of interpretability-aware robust training and various baselines for defending attacks (Gabor, Snow, JPEG (cid:96) ∞ , JPEG (cid:96) 2 , and JPEG (cid:96) 1 ) recently proposed in (Kang et al., 2019).
These attacks are called ‘unforseen attacks’ since they are not met by PGD-based robust training and often induce larger perturbations than conventional
PGD attacks.
For robust training methods without resorting to adversarial loss, we ﬁnd that Int signiﬁcantly outperforms IG-Norm especially under Snow and JPEG (cid:96) p attacks.
Int also yields quite competitive robustness compared to AT-type methods on most attacks. [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
C ONCLUSION
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
In this paper, we investigate the connection between network interpretability and adversarial robustness.
We show theoretically and empirically that with the correct choice of discrepancy measure, it is difﬁcult to hide adversarial examples from interpretation.
We leverage this discrepancy measure to develop a interpretabilityaware robust training method that displays 1) high classiﬁcation robustness in a variety of settings and 2) high robustness of interpretation.
A PPENDIX A P ROOF OF P ROPOSITION
For ∀ c ∈ [ C ] , by the completeness axiom we have f c ( x ) =
(cid:80) i
[ L ( x , c )] i .
Using this fact, we obtain that
[Equation]
Similarly, we have
[Equation] [Equation]
Adding (10) and (11) rearranging yields
[Equation]
Since f t (cid:48) ( x (cid:48) ) − f t ( x (cid:48) ) ≥
[Equation]
we then have (cid:107) L ( x (cid:48)
[Equation]
, t ) − L ( x (cid:48) , t ) (cid:107) 1 ≥ f t ( x ) − f t (cid:48)
( x ) , which is equivalent to (5).
The considered network model has a global average pooling layer followed by a fully connected layer at the end of the network. For R-ImageNet, we use only a normally trained network, used for evaluating ISA.
During adversarial training of all baselines, 40 adversarial steps are used for MNIST and 10 steps for CIFAR.
To ensure stability of all training methods, the size of perturbation is increased during training from 0 to a ﬁnal value of 0 . 3 on MNIST and 8 /on CIFAR. MNIST and R-ImageNet networks are trained for 100 epochs and CIFAR networks are trained for 200 epochs. A batch size of 50 is used for MNIST, 128 for CIFAR and 64 for RImagenet.
For all methods, training is performed using Adam with an initial learning rate of 0.0001 for MNIST and 0.001 for CIFAR and R-Imagenet, with the learning decayed by × 1 /at training steps 40000 and 60000 for CIFAR and 8000 and 16000 for R-Imagenet.
C E XPERIMENTS ON R EGULARIZATION P ARAMETER γ
We conduct experiments for evaluating the sensitivity of the regularization parameter γ in our proposed apporoach (namely, Int) under a Small MNIST model.
Adversarial test accuracy (ATA) and clean test accuracy results are plotted in Figure A1.
As illustrated, using different values of the hyperparameter γ controls the tradeoff between clean accuracy and ATA, with smaller γ yielding higher clean accuracy, but lower ATA (a value of γ = 0 corresponds to normal training).
We note that with the model tested, ATA stops increasing at a value of γ = 0 .
Beyond this value, clean accuracy continues to decrease while ATA slightly decreases.
These results indicate that by choosing an appropriate γ , it is possible to smoothly interpolate between normal training and maximally robust Int training.
D D ETAILS ON NDS AND NSL
In order to fairly compare among different interpretation methods, we compute a normalized discrepancy score (NDS) extended from (3) :
D norm = 1 |C| (cid:80) i ∈C
[Equation]
A smaller value of NDS implies the less difﬁculty for ISA to hide adversarial examples.
A strong ISA is also expected to resist interpretability discrepancy even as the perturbation size (cid:15) increases; see Figure 2-(a)&(b).
To quantify this, we compute an additional quantity called normalized slope (NSL) that measures the relative change of NDS for (cid:15) ∈
[ˇ (cid:15), ˆ (cid:15) ] :
S
[Equation]
=
[Equation]
The smaller NDS is, the less strong ISA is to evade network interpretation change. In our experiment, we choose ˇ (cid:15) = (cid:15) ∗ and ˆ (cid:15) = 1 .
6 (cid:15) ∗ , where (cid:15) ∗ is the minimum perturbation size required for a successful PGD attack for each image x .
Here we perform binary search over (cid:15) to ﬁnd its smallest value for misclassiﬁcation.
Reported NDS and
NSL statistics are averaged over a test set.
E A DDITIONAL T ABLES
[Table]
TRADES
[Table]
Small
[Table]
TRADES
[Table]
[Table]
[Table]
MNIST, Pool
Normal 0.
Adv 0.
TRADES 0.
IG-Norm 0. IG-Norm-Sum 0.
Int 0.
Int-Adv 0.
Int-1-class 0.
(cid:15) =
CIFAR-10, Small
Normal 0.
Adv 0.
TRADES 0.
IG-Norm 0. IG-Norm-Sum 0.
Int 0.
Int-Adv 0.
Int-1-class 0.
Method
[Table]
MNIST, Pool
[Table]
TRADES
[Table]
Sum
[Table]
, Small
[Table]
TRADES
[Table]
Int
[Table]
[Table] [Table]