Original: A N EVALUATION OF QUALITY AND ROBUSTNESS OF SMOOTHED EXPLANATIONS
Revised: A N EVALUATION OF QUALITY AND ROBUSTNESS OF SMOOTHED EXPLANATIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Explanation methods play a crucial role in helping to understand the decisions of deep neural networks (DNNs) to develop trust that is critical for the adoption of predictive models.
Revised: Explanation methods play a crucial role in helping to understand the decisions of deep neural networks (DNNs) to develop trust that is critical for the adoption of predictive models.

Original: However, explanation methods are easily manipulated through visually imperceptible perturbations that generate misleading explanations.
Revised: However, explanation methods are easily manipulated through visually imperceptible perturbations that generate misleading explanations.

Original: The geometry of the decision surface of the DNNs has been identiﬁed as the main cause of this phenomenon and several smoothing approaches have been proposed to build more robust explanations.
Revised: The geometry of the decision surface of the DNNs has been identiﬁed as the main cause of this phenomenon and several smoothing approaches have been proposed to build more robust explanations.

Original: In this work we provide a thorough evaluation of the quality and robustness of the explanations derived by smoothing approaches.
Revised: In this work we provide a thorough evaluation of the quality and robustness of the explanations derived by smoothing approaches.

Original: Their different properties are evaluated with extensive experiments, which reveal the settings where the smoothed explanations are better and also worse than the explanations derived by the common Gradient method.
Revised: Their different properties are evaluated with extensive experiments, which reveal the settings where the smoothed explanations are better and also worse than the explanations derived by the common Gradient method.

Original: By making the connection with the literature on adversarial attacks, we further show that such smoothed explanations are robust primarily against additive (cid:96) p -norm attacks.
Revised: By making the connection with the literature on adversarial attacks, we further show that such smoothed explanations are robust primarily against additive (cid:96) p -norm attacks.

Original: However, a combination of additive and non-additive attacks can still manipulate these explanations, which reveals shortcomings in their robustness properties
Revised: However, a combination of additive and non-additive attacks can still manipulate these explanations, which reveals shortcomings in their robustness properties

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Explanation methods attribute a numerical value to each data feature in order to quantify its relative importance towards the model’s prediction.
Revised: Explanation methods attribute a numerical value to each data feature in order to quantify its relative importance towards the model’s prediction.

Original: Such attributions help to better understand and trust complex models like deep neural networks (DNNs).
Revised: Such attributions help to better understand and trust complex models like deep neural networks (DNNs).

Original: In safety-critical tasks, such an understanding is a prerequisite to the deployment of DNNs, because a domain expert will never make important decisions based on a model’s prediction unless that model is trustworthy.
Revised: In safety-critical tasks, such an understanding is a prerequisite to the deployment of DNNs, because a domain expert will never make important decisions based on a model’s prediction unless that model is trustworthy.

Original: Moreover, explanations can help to understand the reasons behind the decision of a model, and when it comes to model debugging, they can reveal the presence of any spurious data correlations that may lead to faulty predictions during inference (Ribeiro et al., 2016).
Revised: Moreover, explanations can help to understand the reasons behind the decision of a model, and when it comes to model debugging, they can reveal the presence of any spurious data correlations that may lead to faulty predictions during inference (Ribeiro et al., 2016).

Original: In the context of image classiﬁcation with deep neural networks, several explanation methods have been proposed based on the gradient with respect to input, also called gradient-based explanations (Baehrens et al., 2010; Bach et al., 2015; Selvaraju et al., 2017; Sundararajan et al., 2017;
Revised: In the context of image classiﬁcation with deep neural networks, several explanation methods have been proposed based on the gradient with respect to input, also called gradient-based explanations (Baehrens et al., 2010; Bach et al., 2015; Selvaraju et al., 2017; Sundararajan et al., 2017;

Original: Springenberg et al., 2015).
Revised: Springenberg et al., 2015).

Original: The explanation generated by these methods, a saliency map , highlights the parts of the image that contributed to the prediction.
Revised: The explanation generated by these methods, a saliency map , highlights the parts of the image that contributed to the prediction.

Original: Recent work has shown that gradient-based explanations of neural networks can be fragile and can be easily manipulated via adversarially perturbed inputs (Ghorbani et al., 2019;
Revised: Recent work has shown that gradient-based explanations of neural networks can be fragile and can be easily manipulated via adversarially perturbed inputs (Ghorbani et al., 2019;

Original: Dombrowski et al., 2019; Heo et al., 2019; Viering et al., 2019;
Revised: Dombrowski et al., 2019; Heo et al., 2019; Viering et al., 2019;

Original: Kindermans et al., 2019).
Revised: Kindermans et al., 2019).

Original: That is, one can ﬁnd a small-norm perturbation to be added to an input ( often imperceptible), such that the focus of the explanation changes towards irrelevant features while the model’s output remains unchanged.
Revised: That is, one can ﬁnd a small-norm perturbation to be added to an input ( often imperceptible), such that the focus of the explanation changes towards irrelevant features while the model’s output remains unchanged.

Original: This, in turn, can make explanations inappropriate to help end-users gain trust in a model’s prediction.
Revised: This, in turn, can make explanations inappropriate to help end-users gain trust in a model’s prediction.

Original: The large curvature of the decision surface of neural networks has been identiﬁed as one of the causes of fragility for gradient-based explanations (Ghorbani et al., 2019; Dombrowski et al., 2019;
Revised: The large curvature of the decision surface of neural networks has been identiﬁed as one of the causes of fragility for gradient-based explanations (Ghorbani et al., 2019; Dombrowski et al., 2019;

Original: Wang et al., 2020).
Revised: Wang et al., 2020).

Original: To make explanations more robust, a class of approaches proposed smoothing the explanation or making the decision surface of neural networks more smooth (Wang et al., 2020;
Revised: To make explanations more robust, a class of approaches proposed smoothing the explanation or making the decision surface of neural networks more smooth (Wang et al., 2020;

Original: Dombrowski et al., 2019; Ivankay et al., 2020).
Revised: Dombrowski et al., 2019; Ivankay et al., 2020).

Original: We refer to these approaches as smoothing approaches .
Revised: We refer to these approaches as smoothing approaches .

Original: It is worth mentioning that similar methods have been proposed in the context of adversarial robustness, with the aim of ﬂattening the decision surface of neural networks in order to reach more robust predictions (Moosavi-Dezfooli et al., 2019; Qin et al., 2019).
Revised: It is worth mentioning that similar methods have been proposed in the context of adversarial robustness, with the aim of ﬂattening the decision surface of neural networks in order to reach more robust predictions (Moosavi-Dezfooli et al., 2019; Qin et al., 2019).

Original: Here, we provide a thorough investigation of the explanations derived by smoothing approaches in terms of explanation quality and robustness .
Revised: Here, we provide a thorough investigation of the explanations derived by smoothing approaches in terms of explanation quality and robustness .

Original: We employ various tests to assess the quality of these explanations.
Revised: We employ various tests to assess the quality of these explanations.

Original: Each test evaluates a desirable property for explanations, such as: sensitivity to changes in the model, ﬁdelity to the predictor function, etc.
Revised: Each test evaluates a desirable property for explanations, such as: sensitivity to changes in the model, ﬁdelity to the predictor function, etc.

Original: In terms of robustness, we show that explanations derived by smoothing approaches only provide robustness against additive (cid:96) p norm attacks.
Revised: In terms of robustness, we show that explanations derived by smoothing approaches only provide robustness against additive (cid:96) p norm attacks.

Original: Speciﬁcally, in this work, we show that compared to additive attacks, attacks based on the combination of spatial transformation (Xiao et al., 2018) and/or color transformation (Laidlaw &
Revised: Speciﬁcally, in this work, we show that compared to additive attacks, attacks based on the combination of spatial transformation (Xiao et al., 2018) and/or color transformation (Laidlaw &

Original: Feizi, 2019) together with additive perturbations are more effective in manipulating these explanations.
Revised: Feizi, 2019) together with additive perturbations are more effective in manipulating these explanations.

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: • We study the effectiveness of smoothing approaches to achieve robust explanations.
Revised: • We study the effectiveness of smoothing approaches to achieve robust explanations.

Original: We present results on evaluating both the quality and robustness properties of smoothed explanations.
Revised: We present results on evaluating both the quality and robustness properties of smoothed explanations.

Original: • We assess the quality of smoothed explanations via presenting the results of various quality tests.
Revised: • We assess the quality of smoothed explanations via presenting the results of various quality tests.

Original: Our results demonstrate the pros and cons of smoothed explanations with respect to the following quality aspects: sensitivity to model parameters, class discriminativeness, Inﬁdelity, and sparseness.
Revised: Our results demonstrate the pros and cons of smoothed explanations with respect to the following quality aspects: sensitivity to model parameters, class discriminativeness, Inﬁdelity, and sparseness.

Original: • We present results for different combination of additive and non-additive attacks, and show that they are able to manipulate explanations derived by smoothing approaches more successfully.
Revised: • We present results for different combination of additive and non-additive attacks, and show that they are able to manipulate explanations derived by smoothing approaches more successfully.

Original: Combining different types of perturbations to achieve stronger attacks has been a topic of investigation in the context of adversarial examples (Jordan et al., 2019).
Revised: Combining different types of perturbations to achieve stronger attacks has been a topic of investigation in the context of adversarial examples (Jordan et al., 2019).

Original: To the best of our knowledge, this is the ﬁrst time such attacks have been used in the context of explanations.
Revised: To the best of our knowledge, this is the ﬁrst time such attacks have been used in the context of explanations.

Original: Related works.
Revised: Related works.

Original: There have been several works aiming to make explanations more robust.
Revised: There have been several works aiming to make explanations more robust.

Original: These works mostly focused on either modifying the explanation method itself or modifying the predictor model to achieve robust explanations.
Revised: These works mostly focused on either modifying the explanation method itself or modifying the predictor model to achieve robust explanations.

Original: Wang et al.
Revised: Wang et al.

Original: (2020) introduced Uniform Gradient, which is similar to Smooth Gradient unless it uses Uniform noise, and showed that it can hardly be manipulated by additive attacks.
Revised: (2020) introduced Uniform Gradient, which is similar to Smooth Gradient unless it uses Uniform noise, and showed that it can hardly be manipulated by additive attacks.

Original: Dombrowski et al.
Revised: Dombrowski et al.

Original: (2019) proved that a network with soft-plus activations has a more robust Gradient explanation compared to a ReLU network, given that the parameter β of the soft-plus function is chosen to be sufﬁciently small.
Revised: (2019) proved that a network with soft-plus activations has a more robust Gradient explanation compared to a ReLU network, given that the parameter β of the soft-plus function is chosen to be sufﬁciently small.

Original: Consequently, they proposed the β -smoothing approach in which they substitute the ReLU activations of a trained network by softplus functions with a small β parameter.
Revised: Consequently, they proposed the β -smoothing approach in which they substitute the ReLU activations of a trained network by softplus functions with a small β parameter.

Original: Wang et al.
Revised: Wang et al.

Original: (2020) introduced a regularization term called
Revised: (2020) introduced a regularization term called

Original: Smooth Surface Regularization (SSR) to the training objective of a DNN.
Revised: Smooth Surface Regularization (SSR) to the training objective of a DNN.

Original: This training objective penalizes the large curvature of a DNN by regularizing the eigenvalue of the input hessian with the maximum absolute value.
Revised: This training objective penalizes the large curvature of a DNN by regularizing the eigenvalue of the input hessian with the maximum absolute value.

Original: Moreover, they showed that adversarial training (Madry et al., 2018) also leads to more robust explanations.
Revised: Moreover, they showed that adversarial training (Madry et al., 2018) also leads to more robust explanations.

Original: This fact can also be deduced from the results of (MoosaviDezfooli et al., 2019) as they showed that adversarial training leads to a signiﬁcant decrease in the curvature of the loss surface with respect to inputs.
Revised: This fact can also be deduced from the results of (MoosaviDezfooli et al., 2019) as they showed that adversarial training leads to a signiﬁcant decrease in the curvature of the loss surface with respect to inputs.

Original: Anders et al.
Revised: Anders et al.

Original: (2020) proposed an attack in which they adversarially manipulate the model instead of the input in order to manipulate the explanation.
Revised: (2020) proposed an attack in which they adversarially manipulate the model instead of the input in order to manipulate the explanation.

Original: Then they propose a modiﬁcation to the existing explanation methods to make them more robust against such manipulated models.
Revised: Then they propose a modiﬁcation to the existing explanation methods to make them more robust against such manipulated models.

Original: Lakkaraju et al.
Revised: Lakkaraju et al.

Original: (2020) proposed a framework for generating robust and stable black box explanations based on adversarial training.
Revised: (2020) proposed a framework for generating robust and stable black box explanations based on adversarial training.

Original: Chen et al.
Revised: Chen et al.

Original: (2019) introduced a regularization term to the training objective of neural networks to achieve robust Integrated Gradient explanations.
Revised: (2019) introduced a regularization term to the training objective of neural networks to achieve robust Integrated Gradient explanations.

Original: Finally, Dombrowski et al.
Revised: Finally, Dombrowski et al.

Original: (2020) developed a theoretical framework to derive bounds on the maximum manipubality of explanations and proposed three different techniques to boost the robustness of explanations.
Revised: (2020) developed a theoretical framework to derive bounds on the maximum manipubality of explanations and proposed three different techniques to boost the robustness of explanations.

Original: In this work, we show that the robustness of smoothed explanations can be affected by employing a combination of additive and non-additive attacks.
Revised: In this work, we show that the robustness of smoothed explanations can be affected by employing a combination of additive and non-additive attacks.

Original: Furthermore, we present a through evaluation of the different quality aspects of smoothed explanations.
Revised: Furthermore, we present a through evaluation of the different quality aspects of smoothed explanations.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: First, we provide the deﬁnition of an explanation map and then brieﬂy describe the explanation methods we used in this paper.
Revised: First, we provide the deﬁnition of an explanation map and then brieﬂy describe the explanation methods we used in this paper.

Original: Then we continue with introducing the attacks to explanations and the smoothing approaches we are going to study in this paper.
Revised: Then we continue with introducing the attacks to explanations and the smoothing approaches we are going to study in this paper.

Original: Consider a model f : R d → R K which classiﬁes an input x ∈ R d into one of the K classes.
Revised: Consider a model f : R d → R K which classiﬁes an input x ∈ R d into one of the K classes.

Original: An explanation map , denoted by h f ( x ) : R d → R d , associates a score to each feature of the input indicating the relevance of that feature towards the model’s prediction.
Revised: An explanation map , denoted by h f ( x ) : R d → R d , associates a score to each feature of the input indicating the relevance of that feature towards the model’s prediction.

Original: For instance, in the context of image classiﬁcation, saliency maps associate a score to each pixel of the input image resulting in a heatmap that highlights important regions of the image leading to the model prediction.
Revised: For instance, in the context of image classiﬁcation, saliency maps associate a score to each pixel of the input image resulting in a heatmap that highlights important regions of the image leading to the model prediction.

Original: In this work, we focus on the gradient-based explanations and mainly on the Gradient method.
Revised: In this work, we focus on the gradient-based explanations and mainly on the Gradient method.

Original: Given a model f and an input x , the Gradient explanation is deﬁned as ∇ x f ( x ) .
Revised: Given a model f and an input x , the Gradient explanation is deﬁned as ∇ x f ( x ) .

Original: Since other gradient-based explanation methods make use of the gradients with respect to input, we argue that our results could be extended to those explanation methods as well.
Revised: Since other gradient-based explanation methods make use of the gradients with respect to input, we argue that our results could be extended to those explanation methods as well.

Original: We will also consider two smoothed variants, namely Smooth (Smilkov et al., 2017) and Uniform Gradient (Wang et al., 2020) methods.
Revised: We will also consider two smoothed variants, namely Smooth (Smilkov et al., 2017) and Uniform Gradient (Wang et al., 2020) methods.

Original: 2. A TTACKS TO MANIPULATE EXPLANATIONS
Revised: 2. A TTACKS TO MANIPULATE EXPLANATIONS

Original: Similarly to common adversarial attacks (Goodfellow et al., 2015; Moosavi-Dezfooli et al., 2016;
Revised: Similarly to common adversarial attacks (Goodfellow et al., 2015; Moosavi-Dezfooli et al., 2016;

Original: Szegedy et al., 2014), recent work has shown that explanations can also be manipulated by adding a small and almost imperceptible perturbation to the input (Ghorbani et al., 2019; Dombrowski et al., 2019).
Revised: Szegedy et al., 2014), recent work has shown that explanations can also be manipulated by adding a small and almost imperceptible perturbation to the input (Ghorbani et al., 2019; Dombrowski et al., 2019).

Original: We refer to this class of attacks as explanation attacks .
Revised: We refer to this class of attacks as explanation attacks .

Original: There have been various formulations for explanation attacks (Ghorbani et al., 2019; Dombrowski et al., 2019).
Revised: There have been various formulations for explanation attacks (Ghorbani et al., 2019; Dombrowski et al., 2019).

Original: In this work, we will use the formulation introduced by Dombrowski et al.
Revised: In this work, we will use the formulation introduced by Dombrowski et al.

Original: In this attack, the attacker tries to ﬁnd a perturbed input for which the explanation is manipulated to be very similar to a given target explanation map while the output of the model remains approximately unchanged.
Revised: In this attack, the attacker tries to ﬁnd a perturbed input for which the explanation is manipulated to be very similar to a given target explanation map while the output of the model remains approximately unchanged.

Original: Note that the target map could be any heatmap in general; however, we used the explanation of a target image as a target map in this work.
Revised: Note that the target map could be any heatmap in general; however, we used the explanation of a target image as a target map in this work.

Original: Below, we will give a formal deﬁnition of this attack.
Revised: Below, we will give a formal deﬁnition of this attack.

Original: Deﬁnition 1 (Targeted manipulation attack) .
Revised: Deﬁnition 1 (Targeted manipulation attack) .

Original: An explanation h f ( x ) for model f ( x ) is vulnerable to attack at input x if there exist a perturbed input x adv , such that h f ( x adv ) is similar to a given target map h t but the model’s output remains unchanged.
Revised: An explanation h f ( x ) for model f ( x ) is vulnerable to attack at input x if there exist a perturbed input x adv , such that h f ( x adv ) is similar to a given target map h t but the model’s output remains unchanged.

Original: An attacker ﬁnds x adv by minimizing the following objective function:
Revised: An attacker ﬁnds x adv by minimizing the following objective function:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term in (1) ensures the similarity of the manipulated explanation to the target map, the second term ensures the similarity between the model output for the original and perturbed inputs, and the third term regularizes the perturbation to ensure perceptual similarity between the original and perturbed images.
Revised: The ﬁrst term in (1) ensures the similarity of the manipulated explanation to the target map, the second term ensures the similarity between the model output for the original and perturbed inputs, and the third term regularizes the perturbation to ensure perceptual similarity between the original and perturbed images.

Original: Note that L reg is deﬁned by the attacker according to the type of the perturbation.
Revised: Note that L reg is deﬁned by the attacker according to the type of the perturbation.

Original: The relative weighting of the terms in (1) is controlled by the hyper-parameters γ 1 and γ 2 .
Revised: The relative weighting of the terms in (1) is controlled by the hyper-parameters γ 1 and γ 2 .

Original: 2. T OWARDS ROBUST EXPLANATIONS
Revised: 2. T OWARDS ROBUST EXPLANATIONS

Original: Recent works have tried to deﬁne the robustness of explanations in terms of the sensitivity of input gradients to changes in the input data (Wang et al., 2020; Dombrowski et al., 2019).
Revised: Recent works have tried to deﬁne the robustness of explanations in terms of the sensitivity of input gradients to changes in the input data (Wang et al., 2020; Dombrowski et al., 2019).

Original: Wang et al.
Revised: Wang et al.

Original: (2020) deﬁne the robustness of explanations by the Lipschitz continuity coefﬁcient of the input gradients; a smaller coefﬁcient means that the explanation is less sensitive to the changes in the input and hence more robust.
Revised: (2020) deﬁne the robustness of explanations by the Lipschitz continuity coefﬁcient of the input gradients; a smaller coefﬁcient means that the explanation is less sensitive to the changes in the input and hence more robust.

Original: In this regard, a class of approaches to generate robust explanations have been proposed in the recent works, which are either based on smoothing out the explanation maps or ﬂattening the decision boundary of the model itself.
Revised: In this regard, a class of approaches to generate robust explanations have been proposed in the recent works, which are either based on smoothing out the explanation maps or ﬂattening the decision boundary of the model itself.

Original: Broadly, these approaches can be classiﬁed into two categories: (1) Post-hoc approaches do not require retraining of the network and can be applied as a post-processing step.
Revised: Broadly, these approaches can be classiﬁed into two categories: (1) Post-hoc approaches do not require retraining of the network and can be applied as a post-processing step.

Original: (2) Ad-hoc approaches to robust explanations require retraining of the network and hence are more costly.
Revised: (2) Ad-hoc approaches to robust explanations require retraining of the network and hence are more costly.

Original: In this work, we consider Smooth Gradient (Smilkov et al., 2017), Uniform Gradient (Wang et al., 2020), and β -smoothing (Dombrowski et al., 2019) as post-hoc approaches.
Revised: In this work, we consider Smooth Gradient (Smilkov et al., 2017), Uniform Gradient (Wang et al., 2020), and β -smoothing (Dombrowski et al., 2019) as post-hoc approaches.

Original: The ﬁrst two methods involve smoothing the explanation map, while the third one smooths the decision surface of the model.
Revised: The ﬁrst two methods involve smoothing the explanation map, while the third one smooths the decision surface of the model.

Original: All three approaches act on pre-trained models, and hence are characterized as post-hoc.
Revised: All three approaches act on pre-trained models, and hence are characterized as post-hoc.

Original: Among the ad-hoc methods, we study the explanations generated by adversarially trained networks, and networks trained with curvature regularization (CURE) (Moosavi-Dezfooli et al., 2019), which is a similar approach to SSR (Wang et al., 2020) 1 .
Revised: Among the ad-hoc methods, we study the explanations generated by adversarially trained networks, and networks trained with curvature regularization (CURE) (Moosavi-Dezfooli et al., 2019), which is a similar approach to SSR (Wang et al., 2020) 1 .

Original:  E VALUATING POST - HOC APPROACHES
Revised:  E VALUATING POST - HOC APPROACHES

Original: Here, we begin by evaluating the quality of explanations derived by post-hoc approaches that do not require retraining of the network.
Revised: Here, we begin by evaluating the quality of explanations derived by post-hoc approaches that do not require retraining of the network.

Original: Then, we evaluate the robustness of these explanations by presenting results on effective non-additive attacks to manipulate them.
Revised: Then, we evaluate the robustness of these explanations by presenting results on effective non-additive attacks to manipulate them.

Original: For all of the experiments in this section, we used a VGG-16 network trained on ImageNet (Russakovsky et al., 2015), and for generating the explanation maps we used the Captum (Kokhlikyan et al., 2020) package.
Revised: For all of the experiments in this section, we used a VGG-16 network trained on ImageNet (Russakovsky et al., 2015), and for generating the explanation maps we used the Captum (Kokhlikyan et al., 2020) package.

Original: Moreover, for the β -smoothing approach we always set β = 0 . 8 as suggested in (Dombrowski et al., 2019).
Revised: Moreover, for the β -smoothing approach we always set β = 0 . 8 as suggested in (Dombrowski et al., 2019).

Original: 3. Q UALITY OF EXPLANATIONS OF POST - HOC APPROACHES
Revised: 3. Q UALITY OF EXPLANATIONS OF POST - HOC APPROACHES

Original: To evaluate and compare the quality of the explanations, we use various quality tests presented in the literature.
Revised: To evaluate and compare the quality of the explanations, we use various quality tests presented in the literature.

Original: In general, assessing the quality of an explanation is a challenging task and each quality test only evaluates a speciﬁc quality aspect of an explanation.
Revised: In general, assessing the quality of an explanation is a challenging task and each quality test only evaluates a speciﬁc quality aspect of an explanation.

Original: Therefore the assemblage of quality tests helps to understand which quality aspects of the explanations are improved and which are deteriorated by the smoothing approaches.
Revised: Therefore the assemblage of quality tests helps to understand which quality aspects of the explanations are improved and which are deteriorated by the smoothing approaches.

Original: Adebayo et al.
Revised: Adebayo et al.

Original: (2018) argued that it is desired for an explanation to be sensitive to the changes in the model parameters.
Revised: (2018) argued that it is desired for an explanation to be sensitive to the changes in the model parameters.

Original: They proposed a model parameter randomization test to assess this sensitivity.
Revised: They proposed a model parameter randomization test to assess this sensitivity.

Original: In this test, the parameters of a model are progressively randomized from the top layer (logits) to the bottom layers.
Revised: In this test, the parameters of a model are progressively randomized from the top layer (logits) to the bottom layers.

Original: In each step of randomization, the explanation from the resulting model is compared against the explanation from the original model.
Revised: In each step of randomization, the explanation from the resulting model is compared against the explanation from the original model.

Original: Randomizing the model parameters means losing what the model has learned from the data during training.
Revised: Randomizing the model parameters means losing what the model has learned from the data during training.

Original: Therefore, we expect a ”good” explanation to be destroyed in this process.
Revised: Therefore, we expect a ”good” explanation to be destroyed in this process.

Original: However, if an explanation is insensitive to the randomization of the model parameters, then it is not deemed appropriate for debugging the model under erroneous predictions.
Revised: However, if an explanation is insensitive to the randomization of the model parameters, then it is not deemed appropriate for debugging the model under erroneous predictions.

Original: The visual results of this test for Gradient explanation and post-hoc approaches are shown in Figure 1.
Revised: The visual results of this test for Gradient explanation and post-hoc approaches are shown in Figure 1.

Original: More examples of this test can be found in the Appendix.
Revised: More examples of this test can be found in the Appendix.

Original: One can observe that the explanations derived from post-hoc approaches show less sensitivity to the randomization of model parameters than compared to the Gradient method.
Revised: One can observe that the explanations derived from post-hoc approaches show less sensitivity to the randomization of model parameters than compared to the Gradient method.

Original: This can also be veriﬁed by the Spearman rank correlation between the original and randomized explanations shown in Figure 2.
Revised: This can also be veriﬁed by the Spearman rank correlation between the original and randomized explanations shown in Figure 2.

Original: We observe that for the smoothed explanation methods, the original and randomized explanations have a high rank correlation after the randomization of the top layers of the network.
Revised: We observe that for the smoothed explanation methods, the original and randomized explanations have a high rank correlation after the randomization of the top layers of the network.

Original: These results highlight that using
Revised: These results highlight that using

Original: Smooth Gradient, Uniform Gradient, and β -smoothing to achieve a more robust explanation can come at the expense of having explanations that are less sensitive to model parameters.
Revised: Smooth Gradient, Uniform Gradient, and β -smoothing to achieve a more robust explanation can come at the expense of having explanations that are less sensitive to model parameters.

Original: Class sensitivity of explanations.
Revised: Class sensitivity of explanations.

Original: A good visual explanation should be able to localize the image regions relevant to the target category, i.e., it should be class discriminative (Selvaraju et al., 2017).
Revised: A good visual explanation should be able to localize the image regions relevant to the target category, i.e., it should be class discriminative (Selvaraju et al., 2017).

Original: This is particularly signiﬁcant when dealing with images containing more than one object.
Revised: This is particularly signiﬁcant when dealing with images containing more than one object.

Original: To assess the class discriminativeness of an explanation we used a quality test equivalent to the pointing game (Zhang et al., 2016).
Revised: To assess the class discriminativeness of an explanation we used a quality test equivalent to the pointing game (Zhang et al., 2016).

Original: We sampled images from the MS COCO dataset (Lin et al., 2014), containing two objects that are also present among the ImageNet class labels.
Revised: We sampled images from the MS COCO dataset (Lin et al., 2014), containing two objects that are also present among the ImageNet class labels.

Original: For this test we only keep
Revised: For this test we only keep

Original: ”obj [Table] smoothing
Revised: Note

Original: [Table]
Revised: [Table]

Original: the samples for which one of the objects in the image is the top predicted class by the network and the other object is among the top 20 predicted classes by the network.
Revised: the samples for which one of the objects in the image is the top predicted class by the network and the other object is among the top 20 predicted classes by the network.

Original: We compute the explanation maps for each of the class labels corresponding to the objects.
Revised: We compute the explanation maps for each of the class labels corresponding to the objects.

Original: Using the segmentation mask of the objects provided in the dataset as ground truth, we compute what percentage of the top-20 values in the explanation maps generated for each target category are inside the corresponding segmentation masks.
Revised: Using the segmentation mask of the objects provided in the dataset as ground truth, we compute what percentage of the top-20 values in the explanation maps generated for each target category are inside the corresponding segmentation masks.

Original: The results of this test are shown in table 1 and a visual depiction of this test is given in Figure 3.
Revised: The results of this test are shown in table 1 and a visual depiction of this test is given in Figure 3.

Original: These results indicate that the smoothed explanation methods are less discriminatory when generated for the target class label that has a lower probability.
Revised: These results indicate that the smoothed explanation methods are less discriminatory when generated for the target class label that has a lower probability.

Original: This suggests that in terms of class discriminativeness of explanations, the post-hoc smoothing approaches investigated in this paper are inferior to the Gradient method.
Revised: This suggests that in terms of class discriminativeness of explanations, the post-hoc smoothing approaches investigated in this paper are inferior to the Gradient method.

Original: Sparseness of explanations.
Revised: Sparseness of explanations.

Original: To create explanations that are human-accessible, it is advantageous to have a sparse explanation map (Molnar, 2019), i.e, only the features that are truly predictive of the model output should have signiﬁcant contributions, and irrelevant features should have negligible contributions.
Revised: To create explanations that are human-accessible, it is advantageous to have a sparse explanation map (Molnar, 2019), i.e, only the features that are truly predictive of the model output should have signiﬁcant contributions, and irrelevant features should have negligible contributions.

Original: Sparse explanations are more concise because they only include features with signiﬁcant contribution making it simpler for end-users to understand the reasons for a speciﬁc prediction of the model (Chalasani et al., 2020).
Revised: Sparse explanations are more concise because they only include features with signiﬁcant contribution making it simpler for end-users to understand the reasons for a speciﬁc prediction of the model (Chalasani et al., 2020).

Original: To measure the sparseness of an explanation map, we applied the Gini Index on the absolute value of the ﬂattened explanation maps.
Revised: To measure the sparseness of an explanation map, we applied the Gini Index on the absolute value of the ﬂattened explanation maps.

Original: The Gini Index is a metric that measures the sparseness of a vector with non-negative values (Hurley & Rickard, 2009).
Revised: The Gini Index is a metric that measures the sparseness of a vector with non-negative values (Hurley & Rickard, 2009).

Original: By deﬁnition, the Gini Index take values in [0 , 1] with higher values indicating more sparseness.
Revised: By deﬁnition, the Gini Index take values in [0 , 1] with higher values indicating more sparseness.

Original: Table 2 shows the average Gini Index of the Gradient, Smooth Gradient, Uniform Gradient, andβ -smoothing computed for 100 randomly sampled images from ImageNet. The results show that
Revised: Table 2 shows the average Gini Index of the Gradient, Smooth Gradient, Uniform Gradient, and

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: compared to the Gradient method, Smooth Gradient and Uniform Gradient provide less concise explanations, whereas β -smoothing actually improves the sparseness of the explanations as compared the method.

Original: Explanation Inﬁdelity.
Revised: Explanation Inﬁdelity.

Original: Introduced in Yeh et al.
Revised: Introduced in Yeh et al.

Original: (2019), this metric captures how the predictor function changes in response to signiﬁcant perturbations to the input and is deﬁned as the expected difference between the two terms: 1) the dot product of the input perturbation and the explanation and 2) the difference between function values after signiﬁcant perturbations to the input.
Revised: (2019) this metric captures how the predictor function changes in response to signiﬁcant perturbations to the input and is deﬁned as the expected difference between the two terms: 1) the dot product of the input perturbation and the explanation and 2) the difference between function values after signiﬁcant perturbations to the input.

Original: The metric generalizes the completeness
Revised: The metric generalizes the completeness

Original: axiom (Shrikumar et al., 2017; Sundararajan et al., 2017) because it allows for different types of perturbations which could be of interest depending on the problem and the dataset.
Revised: axiom (Shrikumar et al., 2017; Sundararajan et al., 2017) because it allows for different types of perturbations which could be of interest depending on the problem and the dataset.

Original: We use the inﬁdelity metric to compare the effect of post-hoc smoothing approaches on the ﬁdelity of explanations to the predictor function.
Revised: We use the inﬁdelity metric to compare the effect of post-hoc smoothing approaches on the ﬁdelity of explanations to the predictor function.

Original: As suggested in (Yeh et al., 2019), we used the square removal perturbation to compute the inﬁdelity of explanations for randomly selected images from ImageNet.
Revised: As suggested in (Yeh et al., 2019), we used the square removal perturbation to compute the inﬁdelity of explanations for randomly selected images from ImageNet.

Original: Table 3 shows the results for the post-hoc approaches.
Revised: Table 3 shows the results for the post-hoc approaches.

Original: A lower inﬁdelity value indicates better ﬁdelity of the explanation to the predictor function.
Revised: A lower inﬁdelity value indicates better ﬁdelity of the explanation to the predictor function.

Original: The results suggest that the degree of smoothing used to robustify explanations, also improves their inﬁdelity.
Revised: The results suggest that the degree of smoothing used to robustify explanations, also improves their inﬁdelity.

Original: Therefore with respect to the Inﬁdelity metric, all of the smoothed explanations investigated in this section are superior to the Gradient method.
Revised: Therefore with respect to the Inﬁdelity metric, all of the smoothed explanations investigated in this section are superior to the Gradient method.

Original: This ﬁnding is also in line with the results of Yeh et al.
Revised: This ﬁnding is also in line with the results of Yeh et al.

Original: (2019), i.e., that modest smoothing improves the inﬁdelity of explanations.
Revised: (2019), i.e., that modest smoothing improves the inﬁdelity of explanations.

Original: 3. R OBUSTNESS OF EXPLANATIONS OF POST - HOC APPROACHES
Revised: 3. R OBUSTNESS OF EXPLANATIONS OF POST - HOC APPROACHES

Original: Now, we will evaluate the robustness of Smooth Gradient, Uniform Gradient, and β -smoothing explanations.
Revised: Now, we will evaluate the robustness of Smooth Gradient, Uniform Gradient, and β -smoothing explanations.

Original: We present attacks composed of additive and non-additive perturbations, and show that they are more effective than additive attacks to manipulate explanations.
Revised: We present attacks composed of additive and non-additive perturbations, and show that they are more effective than additive attacks to manipulate explanations.

Original: The non-additive attacks we employed are spatial transformation attacks (Xiao et al., 2018), and recoloring attacks (Laidlaw & Feizi, 2019).
Revised: The non-additive attacks we employed are spatial transformation attacks (Xiao et al., 2018), and recoloring attacks (Laidlaw & Feizi, 2019).

Original: See the Appendix B for a brief description of each of these attacks.
Revised: See the Appendix B for a brief description of each of these attacks.

Original: In the rest of this paper, we refer to the additive attack as Delta , spatial transformation attack as StAdv , and recoloring attack as Recolor .
Revised: In the rest of this paper, we refer to the additive attack as Delta , spatial transformation attack as StAdv , and recoloring attack as Recolor .

Original: We used the projected gradient descent (PGD) algorithm to optimize the objective function (1) 2 .
Revised: We used the projected gradient descent (PGD) algorithm to optimize the objective function (1) 2 .

Original: In our experiments, we evaluate three combinations of attacks, namely Delta, Delta+StAdv, and
Revised: In our experiments, we evaluate three combinations of attacks, namely Delta, Delta+StAdv, and

Original: Delta+StAdv+Recolor, against the explanation of a VGG-16 network trained on ImageNet (Russakovsky et al., 2015).
Revised: Delta+StAdv+Recolor, against the explanation of a VGG-16 network trained on ImageNet (Russakovsky et al., 2015).

Original: See Appendix C.1 for the details about the (cid:96) ∞ norm for each type of the perturbations and the hyper-parameters used in each attack setting.
Revised: See Appendix C.1 for the details about the (cid:96) ∞ norm for each type of the perturbations and the hyper-parameters used in each attack setting.

Original: We use two metrics to evaluate the attacks: (1) The Cosine Distance metric (cosd) to evaluate the similarity between the target and manipulated explanations (Wang et al., 2020).
Revised: We use two metrics to evaluate the attacks: (1) The Cosine Distance metric (cosd) to evaluate the similarity between the target and manipulated explanations (Wang et al., 2020).

Original: A lower cosine distance corresponds to a lower (cid:96) 2 distance between the target and manipulated explanations indicating a higher similarity.
Revised: A lower cosine distance corresponds to a lower (cid:96) 2 distance between the target and manipulated explanations indicating a higher similarity.

Original: The range of the values for cosd is between 0 and 1 .
Revised: The range of the values for cosd is between 0 and 1 .

Original: The LPIPS metric for quantifying the perceptual similarity between images (Zhang et al., 2018).
Revised: The LPIPS metric for quantifying the perceptual similarity between images (Zhang et al., 2018).

Original: A lower LPIPS value indicates higher similarity.
Revised: A lower LPIPS value indicates higher similarity.

Original: Figure 4 shows the cosine distance between the target and manipulated explanations, and the perceptual similarity (LPIPS) between the perturbed and original images for each attack setting.
Revised: Figure 4 shows the cosine distance between the target and manipulated explanations, and the perceptual similarity (LPIPS) between the perturbed and original images for each attack setting.

Original: We can observe that Delta+StAdv, and Delta+StAdv+Recolor attacks are more effective than Delta attacks to manipulate β -smoothing explanations, i.e, with a less perceptible perturbation (lower LPIPS value), we can reach a cosd value between manipulated and target explanations very close to the cosd value when attacking the Gradient method.
Revised: We can observe that Delta+StAdv, and Delta+StAdv+Recolor attacks are more effective than Delta attacks to manipulate β -smoothing explanations, i.e, with a less perceptible perturbation (lower LPIPS value), we can reach a cosd value between manipulated and target explanations very close to the cosd value when attacking the Gradient method.

Original: The effect of the non-additive attacks is less signiﬁcant on the Smooth and Uniform Gradient methods, however we can still observe improvements in the cosd values under these attacks.
Revised: The effect of the non-additive attacks is less signiﬁcant on the Smooth and Uniform Gradient methods, however we can still observe improvements in the cosd values under these attacks.

Original: Taken together, these results show that Smooth Gradient, Uniform
Revised: Taken together, these results show that Smooth Gradient, Uniform

Original: Gradient, and β -smoothing explanations are more vulnurale to non-additive attacks and hence such attacks should be considered as a threat to the robustness of these methods .
Revised: Gradient, and β -smoothing explanations are more vulnurale to non-additive attacks and hence such attacks should be considered as a threat to the robustness of these methods .

Original: As an example, we can visually see the effectiveness of Delta+StAdv+Recolor attack against differnt explanation methods in Figure 5.
Revised: As an example, we can visually see the effectiveness of Delta+StAdv+Recolor attack against differnt explanation methods in Figure 5.

Original:  E VALUATING AD - HOC APPROACHES
Revised:  E VALUATING AD - HOC APPROACHES

Original: Here, we recreate the experiments of Section 3 for the ad-hoc approaches.
Revised: Here, we recreate the experiments of Section 3 for the ad-hoc approaches.

Original: We study the explanations of networks trained with curvature regularization (CURE) (Moosavi-Dezfooli et al., 2019), and adversarial training (Madry et al., 2018).
Revised: We study the explanations of networks trained with curvature regularization (CURE) (Moosavi-Dezfooli et al., 2019), and adversarial training (Madry et al., 2018).

Original: Training with CURE, regularizes the eigenvalue of the input hessian with maximum absolute value and is similar to SSR, which was shown to improve the robustness of explanations against additive attacks (Wang et al., 2020).
Revised: Training with CURE, regularizes the eigenvalue of the input hessian with maximum absolute value and is similar to SSR, which was shown to improve the robustness of explanations against additive attacks (Wang et al., 2020).

Original: Adversarial training also smooths the decision surface and can provide more robust explanations.
Revised: Adversarial training also smooths the decision surface and can provide more robust explanations.

Original: For the experiments in this section, we used a ResNet-18 network trained with CURE and an adversarially trained ResNet-18 network trained on adversarial examples with (cid:96) ∞ norm of the perturbations upper bounded by 8 / 255 (Engstrom et al., 2019).
Revised: For the experiments in this section, we used a ResNet-18 network trained with CURE and an adversarially trained ResNet-18 network trained on adversarial examples with (cid:96) ∞ norm of the perturbations upper bounded by 8 / 255 (Engstrom et al., 2019).

Original: Both networks are trained on CIFARdataset (Krizhevsky, 2012).
Revised: Both networks are trained on CIFARdataset (Krizhevsky, 2012).

Original: 4. Q UALITY OF EXPLANATIONS OF AD - HOC APPROACHES
Revised: 4. Q UALITY OF EXPLANATIONS OF AD - HOC APPROACHES

Original: Cascade randomization of model parameters.
Revised: Cascade randomization of model parameters.

Original: We evaluate the sensitivity of explanations of the networks trained with CURE and adversarial training using the cascade randomization of model parameters test.
Revised: We evaluate the sensitivity of explanations of the networks trained with CURE and adversarial training using the cascade randomization of model parameters test.

Original: The Spearman rank correlation between the original and randomized explanations is shown in Figure 6.
Revised: The Spearman rank correlation between the original and randomized explanations is shown in Figure 6.

Original: These Results show the explanation of an adversarially trained network is less sensitive to model parameters.
Revised: These Results show the explanation of an adversarially trained network is less sensitive to model parameters.

Original: This suggests that the explanation of an adversarially trained network cannot be helpful to debug a model when it is making a wrong prediction.
Revised: This suggests that the explanation of an adversarially trained network cannot be helpful to debug a model when it is making a wrong prediction.

Original: Sparseness of explanations.
Revised: Sparseness of explanations.

Original: We compare the sparseness of the explanations derived by ad-hoc approaches, using the Gini Index metric.
Revised: We compare the sparseness of the explanations derived by ad-hoc approaches, using the Gini Index metric.

Original: Table 4 compares the Gini Index for the explanations of networks trained with different training objectives.
Revised: Table 4 compares the Gini Index for the explanations of networks trained with different training objectives.

Original: These results show that adversarial training helps to improve the sparseness of explanations as compared to standard training.
Revised: These results show that adversarial training helps to improve the sparseness of explanations as compared to standard training.

Original: Hence the explanations of an adversarially trained network are more concise .
Revised: Hence the explanations of an adversarially trained network are more concise .

Original: This is in line with the results of Chalasani et al.
Revised: This is in line with the results of Chalasani et al.

Original: (2020) as well.
Revised: (2020) as well.

Original: However, the rsults of Table 4 indicates that training a network with CURE does not help to improve the sparseness of explanations as compared to standard training.
Revised: However, the rsults of Table 4 indicates that training a network with CURE does not help to improve the sparseness of explanations as compared to standard training.

Original: [Table]
Revised: [Table]

Original: Explanation Inﬁdelity.
Revised: Explanation Inﬁdelity.

Original: To compare the ﬁdelity of explanations derived by ad-hoc approaches to the predictor function, we used the Inﬁdelity metric with square perturbation (Yeh et al., 2019).
Revised: To compare the ﬁdelity of explanations derived by ad-hoc approaches to the predictor function, we used the Inﬁdelity metric with square perturbation (Yeh et al., 2019).

Original: Table 5 shows the results for randomly selected images from CIFAR-10.
Revised: Table 5 shows the results for randomly selected images from CIFAR-10.

Original: A lower inﬁdelity value indicates better ﬁdelity of the explanation to the predictor function.
Revised: A lower inﬁdelity value indicates better ﬁdelity of the explanation to the predictor function.

Original: From these results, we can observe that training a network with CURE and adversarial training helps to improve the explanation Inﬁdelity.
Revised: From these results, we can observe that training a network with CURE and adversarial training helps to improve the explanation Inﬁdelity.

Original: Therefore with respect to the Inﬁdelity metric, the ad-hoc smoothing approaches investigated in this section improve the explanation Inﬁdelity as compared to standard training.
Revised: Therefore with respect to the Inﬁdelity metric, the ad-hoc smoothing approaches investigated in this section improve the explanation Inﬁdelity as compared to standard training.

Original: [Table]
Revised: [Table]

Original: 4. R OBUSTNESS OF EXPLANATIONS OF AD - HOC APPROACHES
Revised: 4. R OBUSTNESS OF EXPLANATIONS OF AD - HOC APPROACHES

Original: Now, we evaluate the improvement of robustness via ad-hoc approaches.
Revised: Now, we evaluate the improvement of robustness via ad-hoc approaches.

Original: We present results for
Revised: We present results for

Original: Delta, Delta+StAdv, and Delta+StAdv+Recolor attacks against explanations of the networks trained with CURE and adversarial training.
Revised: Delta, Delta+StAdv, and Delta+StAdv+Recolor attacks against explanations of the networks trained with CURE and adversarial training.

Original: Figure 7 shows the results of these attacks.
Revised: Figure 7 shows the results of these attacks.

Original: For the adversarially trained network, we can observe that non-additive attacks can more effectively manipulate explanations compared to the additive attacks.
Revised: For the adversarially trained network, we can observe that non-additive attacks can more effectively manipulate explanations compared to the additive attacks.

Original: However, even with the strongest attack setting we still cannot get close to the cosd value reached by attacking the explanation of the network trained in standard way.
Revised: However, even with the strongest attack setting we still cannot get close to the cosd value reached by attacking the explanation of the network trained in standard way.

Original: For the attacks against the explanation of the network trained with CURE, the effect of non-additive attacks are less signiﬁcant in terms of the cosd value, however we can still observe that such attacks can reach similar cosd values with perceptually less visible perturbations.
Revised: For the attacks against the explanation of the network trained with CURE, the effect of non-additive attacks are less signiﬁcant in terms of the cosd value, however we can still observe that such attacks can reach similar cosd values with perceptually less visible perturbations.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have evaluated two aspects of smoothed explanations: a) explanation quality, and b) robustness of explanation.
Revised: We have evaluated two aspects of smoothed explanations: a) explanation quality, and b) robustness of explanation.

Original: In terms of explanation quality, we performed a thorough evaluation of four quality aspects: sensitivity to model parameters, class discriminativeness, sparseness, and inﬁdelity.
Revised: In terms of explanation quality, we performed a thorough evaluation of four quality aspects: sensitivity to model parameters, class discriminativeness, sparseness, and inﬁdelity.

Original: Our results show that the smoothed explanations investigated in this paper perform worse than those of the Gradient method in terms of sensitivity to model parameters and class discriminativeness.
Revised: Our results show that the smoothed explanations investigated in this paper perform worse than those of the Gradient method in terms of sensitivity to model parameters and class discriminativeness.

Original: On the other hand, we show that using such smoothing methods helps to improve explanation Inﬁdelity and sparseness.
Revised: On the other hand, we show that using such smoothing methods helps to improve explanation Inﬁdelity and sparseness.

Original: We further looked at the robustness of explanations, when inputs are perturbed by a combination of additive and non-additive attacks.
Revised: We further looked at the robustness of explanations, when inputs are perturbed by a combination of additive and non-additive attacks.

Original: To the best of our knowledge, this is the ﬁrst time such attacks are used to manipulate explanations.
Revised: To the best of our knowledge, this is the ﬁrst time such attacks are used to manipulate explanations.

Original: Our experimental results highlighted the fact that non-additive attacks are still a threat for explanation methods, including the smoothed ones.
Revised: Our experimental results highlighted the fact that non-additive attacks are still a threat for explanation methods, including the smoothed ones.

Original: These results also point us to the fact that many problems in explanation robustness can be addressed by making analogies with the area of prediction robustness.
Revised: These results also point us to the fact that many problems in explanation robustness can be addressed by making analogies with the area of prediction robustness.

Original: As these two areas are closely related, the solutions already explored in prediction robustness can be potentially helpful to study explanation robustness.
Revised: As these two areas are closely related, the solutions already explored in prediction robustness can be potentially helpful to study explanation robustness.

Original: This will be the focus of our future work.
Revised: This will be the focus of our future work.