C ERTIFIABLY
R OBUST
I NTERPRETATION
IN
D EEP
L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep learning interpretation is essential to explain the reasoning behind model predictions.
Understanding the robustness of interpretation methods is important especially in sensitive domains such as medical applications since interpretation results are often used in downstream tasks.
Although gradient-based saliency maps are popular methods for deep learning interpretation, recent works show that they can be vulnerable to adversarial attacks.
In this paper, we address this problem and provide a certiﬁable defense method for deep learning interpretation.
We show that a sparsiﬁed version of the popular SmoothGrad method, which computes the average saliency maps over random perturbations of the input, is certiﬁably robust against adversarial perturbations.
We obtain this result by extending recent bounds for certiﬁably robust smooth classiﬁers to the interpretation setting.
Experiments on ImageNet samples validate our theory.
I NTRODUCTION 1. M OTIVATION
The growing use of deep learning in a wide range of highly-sensitive applications, such as autonomous driving, medicine, ﬁnance, and even the legal system (Teichmann et al., 2018; Quellec et al., 2017; Fischer & Krauss, 2018; Nissan, 2017), raises concerns about human trust in machine learning systems.
Interpretations, which explain why certain predictions are made, are critical for establishing this trust between users and the machine learning system.
Moreover, the interpretation results themselves are often used for downstream tasks.
For example, gradient-based saliency maps have been used in medical applications: Quellec et al.
(2017) uses gradient-based interpretation techniques to highlight signs of retinal disease in retinal fundus photographs (see Figure 1-b.)
Similarly, BenTaieb et al.
(2016) uses a gradient-based saliency map as part of a pipeline to automatically analyse histopathology slides of colon tumours.
Techniques like these are also used in medical research to annotate datasets (Lahiani et al., 2019).
Outside of the medical ﬁeld, gradientbased interpretations can be used in general-purpose image segmentation tasks (Hong et al., 2015).
However, Ghorbani et al.
(2019) has shown that several gradient-based interpretation methods are sensitive to adversarial examples, obtained by adding a small perturbation to the input image.
These adversarial examples maintain the original class label while greatly distorting the saliency map (Figure 1-a: see ‘Gradient’ interpretation method).
Although adversarial attacks and defenses on image classiﬁcation have been studied extensively in recent years (Szegedy et al., 2014; Uesato et al., 2018;
Goodfellow et al., 2015; Athalye et al., 2018a;b; Buckman et al., 2018; Kurakin et al., 2017; Papernot & McDaniel, 2016; Carlini & Wagner, 2017; Madry et al., 2018), less attention has been paid to effectively defending deep learning interpretation against adversarial examples (Lee et al., 2019;
Etmann et al., 2019).
This is partially due to the difﬁculty of protecting high-dimensional saliency maps compared to defending a class label, as well as to the lack of a ground truth for interpretation.
Interestingly, we have observed that the standard adversarial training used for the classiﬁcation robustness does not lead to robust interpretations (see Section 4).
This calls for better understanding the robustness of interpretation methods and developing defenses with performance guarantees.
[Figure]
Interpretation of
[Figure]
1. P ROPOSED A PPROACH AND M AIN C ONTRIBUTIONS
In the last couple of years, several approaches have been proposed for interpreting neural network outputs (Simonyan et al., 2014;
Sundararajan et al., 2017; Alvarez-Melis & Jaakkola, 2018; Adebayo et al., 2018; Huang et al., 2017).
Speciﬁcally, Simonyan et al.
(2014) computes the elementwise absolute value of the gradient of the largest class score with respect to the input.
To deﬁne some notation, let g ( x ) be this most basic form of the gradient-based saliency map, for an input image x ∈ R n .
For simplicity, we also assume that elements of g ( x ) have been linearly normalized to be between 0 and 1 .
g ( x ) represents, to a ﬁrst order linear approximation, the importance of each pixel in determining the class label.
Numerous variations of this method have been introduced, which we review in the appendix.
A popular saliency map method which extends the basic gradient method is SmoothGrad (Smilkov et al., 2017), which takes the average gradient over random perturbations of the input.
Formally, we deﬁne the smoothing function as:
[Equation]
where (cid:15) has a normal distribution (i.e. (cid:15) ∼ N( 0 ,σ 2 I ) ).
We will discuss other smoothing functions in
Section 2.3 while the empirical smoothing function which computes the average over ﬁnitely many perturbations of the input will be discussed in Section 2.2.
We refer to the basic method described in the above equation as the scaled SmoothGrad 1 .
Since a ground truth label for interpretation is not available, we use a similarity metric between the original and perturbed saliency maps as an estimate of the interpretation robustness.
Ghorbani et al.
(2019) introduced a top-K overlap metric for this purpose: deﬁne R ( x , ˜ x ,K ) as the number of overlapping elements between top K largest elements of saliency maps of x and the perturbed image ˜ x .
Ghorbani et al.
(2019) developed an L ∞ attack on R ( x , ˜ x ,K ) , which we adapt to the Lcase to test empirical robustness: see the appendix for details.
This top-K overlap metric is naturally motivated: absolute magnitude has little meaning in saliency maps, which are nearly always presented as normalized.
Moreover, it is common (Sundararajan et al., 2017) to clip the highest values when displaying saliency maps, so that more than a few pixels are clearly visible.
This suggests that relative ranks of salience values (and not their absolute values) are important for interpretation.
Note that for an input x , R ( x , ˜ x ,K ) depends on the speciﬁc perturbation ˜ x .
We deﬁne R ∗ ( x ,K ) as the robustness measure with respect to the worst perturbation of x .
That is,
[Equation]
For deep learning models, this optimization is non-convex in general.
Thus, characterizing the true robustness of interpretation methods is likely intractable for most modern classiﬁer models.
In this paper, we show that a lower bound on the true robustness value of an interpretation method (i.e. a robustness certiﬁcate) can be computed efﬁciently.
In other words, for a given input x , we compute a robustness certiﬁcate R cert such that R cert ( x ,K ) ≤ R ∗ ( x ,K ) .
To establish the robustness certiﬁcate for saliency map methods, we prove the following result for a general function h ( . )
whose range is between 0 and 1:
Theorem 2.
Let h ( x ) be the output of an interpretation method whose range is between 0 and 1 and let ¯ h be its smoothed version deﬁned as in equation 1.1.
Let ¯ h
i ( x ) and ¯ h
[ i ] ( x ) be the i -th element and the i -th largest elements of ¯ h ( x ) , respectively.
Let Φ be the cdf of the normal distribution.
If
[Equation]
then for the smoothed interpretation method, we have R cert ( x ,K ) ≥ i .
Intuitively, this means that, if there is a sufﬁciently large gap between the i -th largest element of the smoothed saliency map and its ( 2 K − i ) -th
largest element, then we can certify that at least i elements in the top K largest elements of the original smoothed saliency map will also be in the top K elements of adversarially perturbed saliency map.
We present a more general version of this result with empirical expectations for smoothing in Section 2, as well as another rank-based robustness certiﬁcate in Section 3.
The proof of this bound relies on an extension of Cohen et al.
(2019) which addresses certiﬁed robustness in the classiﬁcation case.
Proofs for all theorems are given in the appendix.
Evaluating the robustness certiﬁcate for the scaled SmoothGrad method on ImageNet samples produces vacuous bounds (Figure 1-c).
This motivated us to develop variations of SmoothGrad with larger robustness certiﬁcates.
One such variation is Sparsiﬁed SmoothGrad which is deﬁned by smoothing a sparsiﬁcation function that maps the largest elements of g ( x ) to one and the rest to zero.
Sparsiﬁed SmoothGrad obtains a considerably larger value of the robustness certiﬁcate (Figure 1-c) while producing high-quality saliency maps (Figure 3).
We study other variations of Sparsiﬁed
SmoothGrad in Section 2.
A considerably different certiﬁcation result for robust interpretation is provided by Lee et al.
That work provides a certiﬁed L 2 radius in which a ﬁrst-order gradient saliency map is exactly identical to the saliency map for the unperturbed image.
The method depends on the network structure being locally linear (ReLU-based): the inner radius of the locally linear region is directly calculated.
By contrast, our method makes no demands on the structure of the classiﬁer, and could be applied to saliency maps generated by methods other than computing simple gradients: any bounded saliency function can be used as h ( x ) in our Theorem 1.
Additionally, our certiﬁcation method is computationally inexpensive enough to allow certiﬁcates to be computed on ImageNet-scale samples.
S MOOTHING FOR C ERTIFIABLY R OBUST I NTERPRETATION 2. N OTATION
We introduce the following notations to indicate Gaussian smoothing: for a function h , we deﬁne population and empirical smoothed functions, respectively, as:
[Equation]
In other words, ¯ h ( x ) represents the expected value of h ( x ) when smoothed under normal perturbations of (cid:15) with some standard deviation σ while ˜
h ( x ) represents an empirical estimate of ¯ h ( x ) using q samples.
We call σ 2 the smoothing variance and q the number of smoothing perturbations.
We use v i to denote the i th element of the vector v .
Similarly h i ( x ) denotes the i th element of the output
h ( x ) .
We also deﬁne, for any h ( x ) , rank ( h ( x ) ,i ) as the ordinal rank of h i ( x ) in h ( x ) (in the descending order): rank ( h ( x ) ,i ) =
j denotes that h i ( x ) is the j th largest element in h ( x ) .
We use x [ i ] to denote the i th largest element in x .
If i is not an integer, the ceiling of i is used.
2. R OBUSTNESS C ERTIFICATE
In order to derive a robustness certiﬁcate for saliency maps, we present an extension of the classiﬁcation robustness result of Cohen et al.
(2019) to real-valued functions, rather than discrete classiﬁcation functions.
In our case, we will apply this to the saliency map vector g .
First, we deﬁne a ﬂoor function to simplify notation.
Deﬁnition 2.1.
The Floor function is a function L ∶ [ 0 , 1 ] → [ 0 , 1 ] , such that
[Equation]
where ρ denotes the L 2 norm of the adversarial distortion and σ 2 denotes the smoothing variance.
Φ is the cdf function for the standard normal distribution and Φ − 1 is its inverse.
Below (Theorem 1) is a general result which can be used to derive robustness certiﬁcates for interpretation methods.
In particular, it is used to derive our robustness certiﬁcate, Theorem 2, later in the section:
Theorem 1.
Let h ∶ R n → [ 0 , 1 ] n be a real-valued function.
Let L ( . ) be the ﬂoor function deﬁned as in equation 2.1 with parameters σ 2 and ρ .
Using σ 2 ∈ R as the smoothing variance for h , ∀ i,j ∈ [ n ] , x , ˜ x ∈ R n where ∥ x − ˜ x
∥ ≤ ρ :
[Equation]
Note that this theorem is valid for any general function.
However, we will use it for our case where ¯ h ( x ) is a smoothed saliency map.
Theorem 1 states that, for a given saliency map vector ¯ h ( x ) , if
L ( ¯ h i ( x )) ≥ ¯ h j ( x ) , then if x is perturbed inside an L 2 norm ball of radius at most ρ , ¯ h i ( ˜ x ) ≥ ¯ h j ( ˜ x ) .
This result extends Theorem 1 in Cohen et al.
(2019) in two ways: ﬁrst, it provides a guarantee about the difference in the values of two quantities, which in general might not be related, while the original result compared probabilities of two mutually exclusive events.
Second, we are considering a real-valued function h , rather than a classiﬁcation output which can only take discrete values.
This bound can be compared directly to Lecuyer et al.
(2019)’s result which similarly concerns unrelated elements in a vector.
Just as in the classiﬁcation case (as noted by Cohen et al.
(2019)), Theoremgives a signiﬁcantly tighter bound than that of Lecuyer et al.
(see details in the appendix).
[Figure]
We can extend Theorem 1 to use empirical estimates of smoothed functions.
Following Lecuyer et al.
(2019), we derive upper and lower bounds of the expected value function ¯ h ( x ) in terms of ˜ h ( x ) , by applying Hoeffding’s Lemma.
To present our result for the empirical case, we ﬁrst deﬁne an empirical ﬂoor function to derive a similar lower bound when the population mean is estimated using a ﬁnite number of samples:
Deﬁnition 2.2.
The Empirical Floor function is a function ˆ L ∶ [ 0 , 1 ] → [ 0 , 1 ] , such that for given values of ρ, σ, p, q, n , where ρ denotes the maximum L 2 distortion, σ 2 denotes the smoothing variance, p denotes the probability bound, q denotes the number of perturbations, and n is the size of input of the function:
[Equation]
where c ∶=
[Equation]
Corollary 1.
Let h ∶ R n → [ 0 , 1 ] n be a function such that for given values of q , σ , ∀ i,j ∈ [ n ] , x , ˜ x ∈
R n , ∥ x − ˜ x
∥ 2 ≤ ρ , with probability at least p ,
[Equation]
Note that unlike the population case, this certiﬁcate bound is probabilistic.
Also note that this bound compares the empirical expectation of the saliency map at an observed point to the true population expectation at all nearby points.
This convention is in line with recent works providing probabilistic certiﬁcates for smoothed classiﬁers (Cohen et al., 2019; Salman et al., 2019).
Theorem 1 allows us to derive certiﬁcates for the top- K overlap (denoted by R ).
In particular:
Theorem 2.
∀ x , ˜ x ∈ R n , ∥ x − ˜ x
∥ 2 ≤ ρ, σ ∈ R , q ∈ N , deﬁne R cert ( x ,K ) as the largest i ≤ K such that ˆ L ( ˜ h
[ i ] ( x )) ≥ ˜
h
[ 2 K − i ] ( x ) .
Then, with probability at least p ,
[Equation]
where R ( x , ˜ x , K ) denotes the top- K overlap between ˜ h
( x ) and ¯ h ( ˜ x ) .
Intuitively, if there is a sufﬁciently large gap between the i th and ( 2 K − i ) th largest elements of an empirical smoothed saliency map, then we can certify that the overlap between top K elements of the observed saliency map and any possible perturbed population smoothed saliency map is at least i with probability at least p .
If the observed saliency map is possibly adversarially corrupted up to a radius of ρ , it is thus still guaranteed to be similar to the “ground truth” population saliency map.
Note that it requires some modiﬁcations of SmoothGrad (Smilkov et al., 2017) for our bounds to be directly applicable.
Smilkov et al.
(2017) in particular deﬁnes two methods, which we will call
SmoothGrad and Quadratic SmoothGrad.
SmoothGrad takes the mean over samples of the signed gradient values, with absolute value typically taken after smoothing for visualization.
Quadratic
SmoothGrad takes the mean of the elementwise squares of gradient values.
Both methods, therefore, require modiﬁcation for our bounds to be applied: we deﬁne scaled SmoothGrad ˜ g
( x ) , such that g ( x ) is the elementwise absolute value of the gradient, linearly scaled so that the largest element is one.
We can similarly deﬁne a scaled Quadratic SmoothGrad.
We can then apply Theorem 2 directly to scaled SmoothGrad (or scaled Quadratic SmoothGrad), simply by scaling the components of g ( x ) (or g ( x )⊙ g ( x ) ) to lie in the interval [ 0 , 1 ] .
However, we observe that this gives vacuous bounds for both of them when using the suggested hyperparameters from Smilkov et al.
One issue is that the suggested value for q (number of perturbations) is 50 which is too small to give useful bounds in
Corollary 1.
For a standard size image from the ImageNet dataset ( n = 224 × 224 × 3 = 150 , 528 ) , with p = 0 . 95 , this gives c = 0 .
395 (using Deﬁnition 2.2).
Note that even for a small ρ :
[Equation]
Thus the gap between z and ˆ L ( z ) is at least 0 .
We can see from Corollary 1 that a gap of 0 .
79 (on a scale of 1) is far too large to be of any practical use.
We instead take q = 2 13 , which gives a more manageable estimation error of c = 0 .
However, we found that even with this adjustment, the bounds computed using Theorem 2 are not satisfactory for either scaled SmoothGrad and or scaled
Quadratic SmoothGrad (see details in the appendix).
This prompted the development of Sparsiﬁed
SmoothGrad described in Section 2.3.
2. S PARSIFIED S MOOTH G RAD AND ITS R ELAXATION
Scaled SmoothGrad and Quadratic SmoothGrad give vacuous robustness certiﬁcates using our proposed method, as demonstrated in Figure 1.
We therefore develop a new method, Sparsiﬁed SmoothGrad , which has (1) non-vacuous robustness certiﬁcates at ImageNet scale (Figure 2a), (2) similar high-quality visual output to SmoothGrad (Figure 3), and (3) theoretical guarantees that aid in setting its hyper-parameters (Section 3).
The Sparsiﬁed SmoothGrad is deﬁned as ˜ g
[ τ ] , where g [ τ ] is deﬁned as follows:
[Equation]
In other words, τ controls the degree of sparsiﬁcation : a fraction τ of elements (the largest τn elements of g ( x ) ) are assigned to 1 , and the rest are set to 0 .
For some applications, it may be desirable to have at least some differentiable elements in the computed saliency map.
For this purpose, we also propose Relaxed Sparsiﬁed SmoothGrad:
[Equation]
Here, τ controls the degree of sparsiﬁcation and γ controls the degree of clipping : a fraction γ of elements are clipped to 1.
Elements neither clipped nor sparsiﬁed are linearly scaled between 0 and 1 .
Note that Relaxed Sparsiﬁed SmoothGrad is a generalization of Sparsiﬁed SmoothGrad.
With no clipping ( γ = 0 ), we again achieve nearly-vacuous results.
However, with only a small degree of clipping ( γ = 0 . 01 ), we achieve results very similar (although slightly worse) than sparsifed SmoothGrad; see Figure 2b.
We use Relaxed Sparsiﬁed SmoothGrad in this paper to test the performance of ﬁrst-order adversarial attacks against Sparsiﬁed SmoothGrad-like techniques.
R ANK C ERTIFICATE FOR THE P ROPOSED S PARSIFIED S MOOTH G RAD
In this section, we show that if the median rank of a saliency map element over smoothing perturbations is sufﬁciently small (i.e. near the top rank), then for an adversarially perturbed input, that [Figure] element will certiﬁably remain near the top rank of the proposed Sparsiﬁed SmoothGrad method with high probability.
This provides a theoretical justiﬁcation for the certiﬁable robustness of the proposed Sparsiﬁed SmoothGrad method.
To present this result, we ﬁrst deﬁne the certiﬁed rank of an element in the saliency map as follows:
Deﬁnition 3.1 (Certiﬁed Rank) .
For a given input x and a given saliency map method (denoted by h ∶ R n → R n ), let the maximum adversarial distortion be ρ , i.e. ∥ ˜ x − x ∥ 2 ≤ ρ .
Then, for a probability p , the certiﬁed rank for an element at index i (denoted by rank cert ( x ,i ) ) is deﬁned as the minimum k such that the following condition holds:
[Equation]
If the i -th element of the saliency map has a certiﬁed rank of k , using Corollary 1, we will have:
[Equation]
with probability at least p.
That is, the i th element of the population smoothed saliency map is guaranteed to be as large as the smallest n − k + 1 elements of the smoothed saliency map of any adversarially perturbed input.
Note that certiﬁed rank depends on the particular perturbations used to generate the smoothed saliency map ˜ h ( x ) .
In the following result, we show that if the median rank of a gradient element at index i , over a set of randomly generated perturbations, is less than a speciﬁed threshold value, then the certiﬁed rank of that element in the Sparsiﬁed SmoothGrad saliency map generated using those perturbations can be upper bounded.
Theorem 3.
Let U be the set of q random perturbations for a given input x using the smoothing variance σ 2 .
Using the Sparsiﬁed SmoothGrad method, with probability p , we have
[Equation]
where τ is the sparsiﬁcation parameter of the Sparsiﬁed SmoothGrad method.
For instance, if ρ ≪ σ and for sufﬁciently large number of smoothing perturbations (i.e. q → ∞ ), we have ˆ L ( 1 / 2 ) → 1 / 2 .
If we set τ = K /( 2 n ) , then for indices whose median ranks are less than or equal to K / 2 , their certiﬁed ranks will be less than or equal to K .
That is, even after adversarially perturbing the input, they will certiﬁably remain among the top K elements of the
Sparsiﬁed SmoothGrad saliency map.
We present a more general form of this result in the appendix
[Figure]
E MPIRICAL R ESULTS
To test the empirical robustness of Sparsiﬁed SmoothGrad, we used an L 2 attack on R ( x , K ) adapted from the L ∞ attack deﬁned by Ghorbani et al.
; see the appendix for details of our proposed attack.
We test Relaxed Sparsiﬁed SmoothGrad ( γ = 0 .
01 ,τ = 0 . 1 ) , rather than Sparsiﬁed
SmoothGrad because our attack is gradient-based and Sparsiﬁed SmoothGrad has no deﬁned gradients.
We tested on ResNet-18 with CIFAR-10 with the attacker using a separately-trained, fully differentiable version of ResNet-18, with SoftPlus activations in place of ReLU.
We present our results in Figure 4.
We observe that our method is signiﬁcantly more robust than the SmoothGrad method while its robustness is on par with the Quadratic SmoothGrad method with the same number of smoothing perturbations.
We note that our robustness certiﬁcate appears to be loose for the large perturbation magnitudes used in these experiments; see the appendix for a direct comparison.
The attack introduced by Ghorbani et al.
(and its L 2 extension that we introduced) does not change the the original classiﬁcation of the image, so protecting against adversarial attacks on classiﬁcation is an insufﬁcient defence.
To demonstrate this, we compare a model trained adversarially for classiﬁcation with a baseline model.
Speciﬁcally, we adversarially train a simple
CNN classiﬁcation model on MNIST (see appendix for model architecture) using a state-of-the-art
L 2 adversarial training procedure provided by Rony et al.
We used the code available for this adversarial training procedure directly.
While, as predicted by Tsipras et al.
(2018), the resulting model produced high-quality saliency maps, the model was not signiﬁcantly more robust to adversarial perturbation tailored for the interpretation than a naively trained classiﬁer (Figure 5, top- K overlap of 58.5 % with adversarial training for classiﬁcation vs. 57 .
3% on the baseline).
This highlights that adversarial training for classiﬁcation does not sufﬁce to provide adversarial robustness for interpretation.
In appendix D, we propose an adversarial training approach which successfully defends interpretations against adversarial attack (top- K overlap greater than 80 % under the same attack) while also producing high-quality saliency maps.
However, unlike the proposed Sparsiﬁed SmoothGrad, this approach is not certiﬁably robust.
C ONCLUSION
In this work, we studied the robustness of deep learning interpretation against adversarial attacks.
We introduced a sparsiﬁed variant of the popular SmoothGrad method which computes the average saliency maps over random perturbations of the input.
By establishing an easy-to-compute robustness certiﬁcate for the interpretation problem, we showed that the proposed Sparsiﬁed SmoothGrad is certiﬁably robust to adversarial attacks while producing high-quality saliency maps.
We provided experiments on ImageNet samples validating our theory
A. G ENERAL F ORM AND P ROOF OF T HEOREM
We note that Theorem 3 can be used to derive a more general bound for any saliency map method that for an input x , ﬁrst maps g ( x ) to an elementwise function that only depends on the rank of the current element in g ( x ) and not on the individual value of the element.
We denote the composition of the gradient function and this elementwise function as g [ rank ] .
The only properties that the function must satisfy is that it must be monotonically decreasing and non-negative.
Thus, we have the following statement:
Theorem 3.
Let T be the threshold value and let U be the set of q random perturbations for a given input x using the smoothing variance σ 2 and let p be the probability bound.
If i is an element index such that:
[Equation]
Then:
[Equation]
Furthermore:
[Equation]
are both independent of x .
Thus RHS is a constant
[Equation]
Proof.
Let the elementwise function be f ∶ N → R + , i.e f takes the rank of the element as the input and outputs a real number.
Furthermore, we assume that f is a non-negative monotonically decreasing function.
Thus g [ rank ]
i
[Equation]
i
[Equation]
We use f ( i ) to denote the constant value that f maps elements of rank i to.
Note that g [ rank ] [ i ] ( x ) is the i th largest element of g [ rank ] ( x ) .
Since f is a monotonically decreasing function:
[Equation] [Equation]
] [ i
[Equation]
Thus g [ rank ] [ i ] ( x ) is independent of x , we simply use g [ rank ] [ i ] (⋅) to denote f ( i ) , i.e: g [ rank ] [ i ] (⋅) =
f ( i ) ∀ i ∈ [ n ]
Because Median (cid:15) ∈ U [ rank ( g ( x + (cid:15) ) , i )] ≤ T , for at least half of sampling instances (cid:15) in U , rank ( g ( x + (cid:15) ) ,i ) ≤ T .
So in these instances g [ rank ] i
[Equation]
f
[Equation]
The remaining half or fewer elements are mapped to other nonnegative values.
Thus the sample mean:
[Equation]
Using Corollary 1, ¯ g [ rank ] i ( x ) is certiﬁably as large as all elements with indices j such that:
[Equation]
Now we will ﬁnd an upper bound on the number of elements with indices j such that:
[Equation]
Because all the ranks from 1 to n will occur in every sample in U, we have:
[Equation]
Thus strictly fewer than ∑ nk = 1 g [ rank ] [ k ] (
⋅)/ ˆ L ( g [ rank ] [ T ] (⋅)/ 2 ) elements will have mean greater than ˆ L ( g [ rank ] [ T ] (⋅)/ 2 ) .
Hence, ¯ g i ( x ) is certiﬁably at least as large as n − (∑ nk = 1 g [ rank ] [ k ] (
⋅)/ ˆ L ( g [ rank ] [ T ] (⋅)/ 2 ))
+ 1 elements, which by the deﬁnition of rank cert ( x ,i ) yields the result.
Theorem 3 in the main text follows trivially, because in the Sparsiﬁed SmoothGrad case, ∑ nk = 1 g [ τ ] [ k ] (⋅)
=
T , and g [ τ ] [ T ] (⋅) = 1 .
Note that this represents the tightest possible realization of this general theorem.
B R ELATED W ORKS
Sundararajan et al.
(2017) deﬁnes a baseline, which represents an input absent of information and determines feature importance by accumulating gradient information along the path from the baseline to the original input.
Alvarez-Melis & Jaakkola (2018) builds interpretable neural networks by learning basis concepts that satisfy an interpretability criteria.
Adebayo et al.
(2018) proposes methods to assess the quality of saliency maps.
Although these methods can produce visually pleasing results, they can be sensitive to noise and adversarial perturbations.
Szegedy et al.
(2014) introduced adversarial attacks for classiﬁcation in deep learning.
That work dealt with L 2 attacks, and uses L-BFGS optimization to minimize the norm of the perturbation.
Carlini & Wagner (2017) provide an L 2 attack for classiﬁcation which is often considered state of the art.
One strategy to make classiﬁers more robust to adversarial attacks is randomized smoothing.
Lecuyer et al.
(2019) use randomized smoothing to develop certiﬁably robust classiﬁers in both the L 1 and L 2 norms.
They show that if Gaussian smoothing is applied to class scores, a gap between the highest smoothed class score and the next highest smoothed score implies that the highest smoothed class score will still be highest under all perturbations of some magnitude.
This guarantees that the smoothed classiﬁer will be robust under adversarial perturbation.
Li et al.
and Cohen et al.
(2019) consider a related formulation.
Cohen gives a bound that is tight in the case of linear classiﬁers and gives signiﬁcantly larger certiﬁed radii.
In their formulation, the unsmoothed classiﬁer c is treated as a black box outputting just a discrete class label.
The smoothed classiﬁer outputs the class observed with greatest frequency over noisy samples.
Salman et al.
(2019) considers adversarial attacks against smoothed classiﬁers.
In the last couple of years, several approaches have been proposed to for interpreting neural network outputs.
Simonyan et al.
(2014) computes the gradient of the class score with respect to the input.
Smilkov et al.
(2017) computes the average gradient-based importance values generated from several noisy versions of the input.
Sundararajan et al.
(2017) deﬁnes a baseline, which represents an input absent of information and determines feature importance by accumulating gradient information along the path from the baseline to the original input.
Alvarez-Melis & Jaakkola (2018) builds interpretable neural networks by learning basis concepts that satisfy an interpretability criteria.
Adebayo et al.
(2018) proposes methods to assess the quality of saliency maps.
Although these methods can produce visually pleasing results, they can be sensitive to noise and
adversarial perturbations
(Ghorbani et al., 2019; Kindermans et al., 2019).
As mentioned in Section 1, several approaches have been introduced for interpreting image classiﬁcation by neural networks (Simonyan et al., 2014; Smilkov et al., 2017; Sundararajan et al., 2017;
Shrikumar et al., 2017).
C L 2 A TTACK ON S ALIENCY M APS
We developed an L 2 norm attack on R cert , based on Ghorbani et al.
(2019)’s
L ∞ attack.
Our algorithm is presented as Algorithm 1.
We deviate from Ghorbani et al.
(2019)’s attack in the following ways.
: • We use gradient descent, rather than gradient sign descent: this is a direct adaptation to the L 2 norm.
• We initialize learning rate as ρ ∥∇ D ( x 0 )∥ 2 , and then decrease learning rate with increasing iteration count, proportionately (for the most part) to the reciprocal of the iteration count.
These are both standard practices for gradient descent.
• We use random initialization and random restarts, also standard optimization practices.
• If a gradient descent step would cross a decision boundary, we use backtracking line search to reduce the learning rate until the step stays on the correct-class side.
This allows the optimization to get arbitrarily close to decision boundaries without crossing them.
We measured the effectiveness of our attack ( Q = 100 ,P = 20 ,T = 5 ) against a slight modiﬁcation of Ghorbani et al.
(2019)’s attack, in which the image was projected (if necessary) onto the L 2 ball at every iteration, and also clipped to ﬁt within image box constraints (this was not mentioned in
Ghorbani et al.
(2019)’s original algorithm).
For this attack, we set the ( L ∞ ) learning rate parameter at ρ / 500 , and ran for up to 100 iterations.
We also tested against random perturbations.
For random perturbations, up to 100 points were tested until a point in the correct class was identiﬁed.
We tested these attacks on both “vanilla gradient” and SmoothGrad saliency maps.
See Figure 6.
Experimental conditions are as described in Section F for experiments on CIFAR-10.
In this ﬁgure, for each attack magnitude, we discard any image on which any optimization method failed.
[Figure]
Deﬁne D z
−∑ i
2: for t
1 ,...,T do 3: loop 4: δ ← Uniformly random vector on L 2 = ρ sphere.
5: x 0 ← x + δ 6:
Clip x 0 such that it falls within image box constraints.
7: if c ( x 0 ) =
c ( x ) then break inner loop 8: if Q iterations have passed over all t cycles sampling then 9: break outer loop 10: end if 11: end loop 12: α ← ρ ∥∇ D ( x 0 )∥13: for p = 1 ,...,P do 14: loop 15: x p ← x p − 1 + α ∇ D ( x p − 1 ) 16:
If necessary, project x p such that ∥ x p − x ∥ 2 ≤ ρ 17:
Clip x p such that it falls within image box constraints.
18: if c ( x p ) =
c ( x ) then 19: break inner loop 20: else 21: α ← α22: end if 23: end loop 24: α ← pαp +25: end for 26: ˜ x t = arg max z ∈{ x 1 ,..., x P } D ( z ) 27: end for 28: if random sampling failed at every iteration then fail 29: ˜ x = arg max z ∈{ ˜ x 1 ,..., ˜ x T } D ( z )
D A DVERSARIAL T RAINING FOR R OBUST S ALIENCY M APS
Adversarial training has been used extensively for making neural networks robust against adversarial attacks on classiﬁcation (Madry et al., 2018).
The key idea is to generate adversarial examples for a classiﬁcation model, and then re-train the model on these adversarial examples.
In this section, we present an adversarial training approach for fortifying deep learning interpretations so that the saliency maps generated by the model (during test time) are robust against adversarial examples.
We focus on “vanilla gradient” saliency maps, although the technique presented here can potentially be applied to any saliency map method which is differentiable w.r.t.
the input.
We solve the following optimization problem for the network weights (denoted by θ ):
[Equation]
[Table]
[Equation]
where ˜ x is an adversarial perturbation for the saliency map generated from x .
To generate ˜ x , we developed an L 2 attack on saliency maps by extending the L ∞ attack of Ghorbani et al.
(see the details in Section C).
(cid:96) cls ( x ,y ) is the standard cross entropy loss, and λ is the regularization parameter to encourage consistency between saliency maps of the original and adversarially perturbed images.
We observe that the proposed adversarial training signiﬁcantly improves the robustness of saliency maps.
Aggregate empirical results are presented in Figure 7, and examples of saliency maps are presented in Figure 8.
It is notable that the quality of the saliency maps is greatly improved for unperturbed inputs, by adversarial training.
We note that this observation is related to the observation made in Tsipras et al.
(2018) showing that adversarial training for classiﬁcation improves the quality of the gradient-based saliency maps.
We observe that even for very large value of λ , only a slight reduction in classiﬁcation accuracy occurs due to the added regularization term.
E A DVERSARIAL T RAINING A RCHITECTURE D ETAILS
We use the Adam optimizer and generate new adversarial examples after each batch of training according to the updated model.
We use a simple convolutional neural network, with SoftPlus activations to ensure differentiability of the saliency map, on the MNIST data set (Figure 9).
Adversarial perturbations of norm up to ρ = 10 standard deviations of pixel intensity were used.
The adversarial attack used was the L 2 attack described in Algorithm 1, with P = 15 ,Q = 100 ,T = 3 .
Training was performed for 30 epochs using 48,000 images from the MNIST training set, testing was on the entire MNIST test set.
Instances where Algorithm 1 failed were not counted in the averages of saliency map robustness, and were rare.
(Highest frequency was for λ = 0 , at 0.11%).
We used the implementation of Adam Optimizer provided with PyTorch (Paszke et al., 2017), with default training parameters.
These are (Table 2):
[Table]
F D ESCRIPTION OF E XPERIMENTS IN M AIN T EXT
For ImageNet experiments (Figures 1-b and 2-a,b), we use ResNet-50, using the model pre-trained on ImageNet that is provided by torchvision.models , and images were pre-processed according to the recommended procedure for that model.
In all of these ﬁgures, data are from the ILSV RCvalidation set, sample size is 64, and the main data lines represent the 60 th percentile in the sample of the calculated robustness certiﬁcate.
Error bars represent the 48 th and 72 th percentile values, corresponding to a 95% conﬁdence interval for the population quantile.
For CIFAR-10 experiments, we train a ResNet-18 model on the CIFAR-10 training set (with pixel intensities normalized to σ = 1 ,µ = 0 in each channel) using Stochastic Gradient Descent with
Momentum as implemented by PyTorch (Paszke et al., 2017).
The following training parameters were used:
[Table]
Twenty percent of the training data was used for validation, and early stopping was used to maximize accuracy relative to this validation set.
For adversarial attacks on CIFAR samples, we train a version of ResNet-18 with SoftMax activations instead of ReLU.
The adversarial attack used was the Lattack described in Algorithm 1, with P = 20 ,Q = 100 ,T = 5 .
When adversarially attacking images smoothed with q = 8192 perturbations with this model, fewer perturbations are are used (512).
In these experiments, the sample size is 144.
Error bars represent the 95% conﬁdence interval of the population mean.
In Figure 4, instances where the adversarial attack failed were not counted at each point.
For MNIST experiments in Section 4, the model architecture, attack method, and baseline training method are the same as used for the adversarial training experiments, detailed in appendix
E above.
The adversarially training for classiﬁcation uses the implementation provided in Rony et al.
(2019), with default arguments.
G A DDITIONAL E XAMPLE I MAGES
See Figures 10 and 11.
H B OUNDS FOR S CALED S MOOTH G RAD , Q UADRATIC S MOOTH G RAD , AND R ELAXED S PARSIFIED S MOOTH G RAD WITH γ =
In the text, we mention that we achieve vacuous bounds for Scaled SmoothGrad, Quadratic SmoothGrad, and Relaxed Sparsiﬁed SmoothGrad with γ = 0 .
Here are these bounds (Figure 12):
Note that equivalent negligibly
I A DDITIONAL I MAGES FROM A DVERSARIAL T RAINING E XPERIMENT (A PPENDIX D)
See Figure 13.
J C OMPARISON OF B OUNDS TO E MPIRICAL P ERFORMANCE FOR R ELAXED S PARSIFIED S MOOTH G RAD .
We present a detailed view of Figure 4, for small magnitude perturbations, with the robustness certiﬁcate shown.
(Figure 14)
K C OMPARISON TO B OUNDS IN L ECUYER ET AL . (2019)
Lecuyer et al.
(2019) approaches the classiﬁcation case for certiﬁed robustness by smoothing, by using bounds directly comparably to Theorem 1, but applying them to the class score elements, rather than the saliency map elements: bounds are certiﬁed by demonstrating that the top class score is certiﬁably larger than all other class scores.
However, as noted by Cohen et al.
, these bounds are rather loose, and Cohen et al.
(2019) gives signiﬁcantly tighter bounds speciﬁcally for classiﬁcation case, which we extend to apply to interpretation.
In Figure 15, we compare our bounds for interpretation to a straightforward application of Lecuyer et al.
(2019)’s results for class scores to saliency scores.
Note that Lecuyer et al.
(2019)’s results have a free parameter, for which we numerically maximize the bound.