Original: D ECOY - ENHANCED S ALIENCY M APS
Revised: D ECOY - ENHANCED S ALIENCY M APS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Saliency methods can make deep neural network predictions more interpretable by identifying a set of critical features in an input sample, such as pixels that contribute most strongly to a prediction made by an image classiﬁer.
Revised: Saliency methods can make deep neural network predictions more interpretable by identifying a set of critical features in an input sample, such as pixels that contribute most strongly to a prediction made by an image classiﬁer.

Original: Unfortunately, recent evidence suggests that many saliency methods poorly perform, especially in situations where gradients are saturated, inputs contain adversarial perturbations, or predictions rely upon inter-feature dependence.
Revised: Unfortunately, recent evidence suggests that many saliency methods poorly perform, especially in situations where gradients are saturated, inputs contain adversarial perturbations, or predictions rely upon inter-feature dependence.

Original: To address these issues, we propose a framework that improves the robustness of saliency methods by following a two-step procedure.
Revised: To address these issues, we propose a framework that improves the robustness of saliency methods by following a two-step procedure.

Original: First, we introduce a perturbation mechanism that subtly varies the input sample without changing its intermediate representations.
Revised: First, we introduce a perturbation mechanism that subtly varies the input sample without changing its intermediate representations.

Original: Using this approach, we can gather a corpus of perturbed data samples while ensuring that the perturbed and original input samples follow the same distribution.
Revised: Using this approach, we can gather a corpus of perturbed data samples while ensuring that the perturbed and original input samples follow the same distribution.

Original: Second, we compute saliency maps for the perturbed samples and propose a new method to aggregate saliency maps.
Revised: Second, we compute saliency maps for the perturbed samples and propose a new method to aggregate saliency maps.

Original: With this design, we offset the gradient saturation inﬂuence upon interpretation.
Revised: With this design, we offset the gradient saturation inﬂuence upon interpretation.

Original: From a theoretical perspective, we show that the aggregated saliency map not only captures inter-feature dependence but, more importantly, is robust against previously described adversarial perturbation methods.
Revised: From a theoretical perspective, we show that the aggregated saliency map not only captures inter-feature dependence but, more importantly, is robust against previously described adversarial perturbation methods.

Original: Following our theoretical analysis, we present experimental results suggesting that, both qualitatively and quantitatively, our saliency method outperforms existing methods, in a variety of applications.
Revised: Following our theoretical analysis, we present experimental results suggesting that, both qualitatively and quantitatively, our saliency method outperforms existing methods, in a variety of applications.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks (DNNs) deliver remarkable performance in an increasingly wide range of application domains, but they often do so in an inscrutable fashion, delivering predictions without accompanying explanations.
Revised: Deep neural networks (DNNs) deliver remarkable performance in an increasingly wide range of application domains, but they often do so in an inscrutable fashion, delivering predictions without accompanying explanations.

Original: In a practical setting such as automated analysis of pathology images, if a patient sample is classiﬁed as malignant, then the physician will want to know which parts of the image contribute to this diagnosis.
Revised: In a practical setting such as automated analysis of pathology images, if a patient sample is classiﬁed as malignant, then the physician will want to know which parts of the image contribute to this diagnosis.

Original: Thus, in general, a DNN that delivers interpretations alongside its predictions will enhance the credibility and utility of its predictions for end users (Lipton, 2016).
Revised: Thus, in general, a DNN that delivers interpretations alongside its predictions will enhance the credibility and utility of its predictions for end users (Lipton, 2016).

Original: In this paper, we focus on a popular branch of explanation methods, often referred to as saliency methods , which aim to ﬁnd input features ( e.g., image pixels or words) that strongly inﬂuence the network predictions (Simonyan et al., 2013; Selvaraju et al., 2016; Binder et al., 2016; Shrikumar et al., 2017; Smilkov et al., 2017; Sundararajan et al., 2017; Ancona et al., 2018).
Revised: In this paper, we focus on a popular branch of explanation methods, often referred to as saliency methods , which aim to ﬁnd input features ( e.g., image pixels or words) that strongly inﬂuence the network predictions (Simonyan et al., 2013; Selvaraju et al., 2016; Binder et al., 2016; Shrikumar et al., 2017; Smilkov et al., 2017; Sundararajan et al., 2017; Ancona et al., 2018).

Original: Saliency methods typically rely on back-propagation from the network’s output back to its input to assign a saliency score to individual features so that higher scores indicate higher importance to the output prediction.
Revised: Saliency methods typically rely on back-propagation from the network’s output back to its input to assign a saliency score to individual features so that higher scores indicate higher importance to the output prediction.

Original: Despite attracting increasing attention, saliency methods suffer from several fundamental limitations:
Revised: Despite attracting increasing attention, saliency methods suffer from several fundamental limitations:

Original: • Gradient saturation (Sundararajan et al., 2017; Shrikumar et al., 2017; Smilkov et al., 2017) may lead to the problem that the gradients of important features have small magnitudes, breaking down the implicit assumption that important features, in general, correspond to large gradients.
Revised: • Gradient saturation (Sundararajan et al., 2017; Shrikumar et al., 2017; Smilkov et al., 2017) may lead to the problem that the gradients of important features have small magnitudes, breaking down the implicit assumption that important features, in general, correspond to large gradients.

Original: This issue can be triggered when the DNN outputs are ﬂattened in the vicinity of important features.
Revised: This issue can be triggered when the DNN outputs are ﬂattened in the vicinity of important features.

Original: • Importance isolation (Singla et al., 2019) refers to the problem that gradient-based saliency methods evaluate the feature importance in an isolated fashion, implicitly assuming that the other features are ﬁxed.
Revised: • Importance isolation (Singla et al., 2019) refers to the problem that gradient-based saliency methods evaluate the feature importance in an isolated fashion, implicitly assuming that the other features are ﬁxed.

Original: • Perturbation sensitivity (Ghorbani et al., 2017; Kindermans et al., 2017; Levine et al., 2019) refers to the observation that even imperceivable, random perturbations or a simple shift transformation of the input data may lead to a large change in the resulting saliency scores.
Revised: • Perturbation sensitivity (Ghorbani et al., 2017; Kindermans et al., 2017; Levine et al., 2019) refers to the observation that even imperceivable, random perturbations or a simple shift transformation of the input data may lead to a large change in the resulting saliency scores.

Original: In this paper, we tackle these limitations by proposing a decoy-enhanced saliency score.
Revised: In this paper, we tackle these limitations by proposing a decoy-enhanced saliency score.

Original: At a high level, our method generates the saliency score of an input by aggregating the saliency scores of multiple perturbed copies of this input.
Revised: At a high level, our method generates the saliency score of an input by aggregating the saliency scores of multiple perturbed copies of this input.

Original: Speciﬁcally, given an input sample of interest, our method ﬁrst generates a population of perturbed samples, referred to as decoys , that perfectly mimic the neural network’s intermediate representation of the original input.
Revised: Speciﬁcally, given an input sample of interest, our method ﬁrst generates a population of perturbed samples, referred to as decoys , that perfectly mimic the neural network’s intermediate representation of the original input.

Original: These decoys are used to model the variation of an input sample originating from either sensor noise or adversarial attacks.
Revised: These decoys are used to model the variation of an input sample originating from either sensor noise or adversarial attacks.

Original: The decoy construction procedure draws inspiration from the knockoffs , proposed recently by Barber & Candès (2015) in the setting of error-controlled feature selection, where the core idea is to generate knockoff features that perfectly mimic the empirical dependence structure among the original features.
Revised: The decoy construction procedure draws inspiration from the knockoffs , proposed recently by Barber & Candès (2015) in the setting of error-controlled feature selection, where the core idea is to generate knockoff features that perfectly mimic the empirical dependence structure among the original features.

Original: In brief, the current paper makes three primary contributions.
Revised: In brief, the current paper makes three primary contributions.

Original: First, we propose a framework to perturb input samples to produce corresponding decoys that preserve the input distribution, in the sense that the intermediate representations of the original input data and the decoys are indistinguishable.
Revised: First, we propose a framework to perturb input samples to produce corresponding decoys that preserve the input distribution, in the sense that the intermediate representations of the original input data and the decoys are indistinguishable.

Original: We formulate decoy generation as an optimization problem, applicable to diverse deep neural network architectures.
Revised: We formulate decoy generation as an optimization problem, applicable to diverse deep neural network architectures.

Original: Second, we develop a decoy-enhanced saliency score by aggregating the saliency maps of generated decoys.
Revised: Second, we develop a decoy-enhanced saliency score by aggregating the saliency maps of generated decoys.

Original: By design, this score naturally offsets the impact of gradient saturation.
Revised: By design, this score naturally offsets the impact of gradient saturation.

Original: From a theoretical perspective, we show how the proposed score can simultaneously reﬂect the joint effects of other dependent features and achieve robustness to adversarial perturbations.
Revised: From a theoretical perspective, we show how the proposed score can simultaneously reﬂect the joint effects of other dependent features and achieve robustness to adversarial perturbations.

Original: Third, we demonstrate empirically that the decoy-enhanced saliency score outperforms existing saliency methods, both qualitatively and quantitatively, on three real-world applications.
Revised: Third, we demonstrate empirically that the decoy-enhanced saliency score outperforms existing saliency methods, both qualitatively and quantitatively, on three real-world applications.

Original: We also quantify our method’s advantage over existing saliency methods in terms of robustness against various adversarial attacks.
Revised: We also quantify our method’s advantage over existing saliency methods in terms of robustness against various adversarial attacks.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: A variety of saliency methods have been proposed in the literature.
Revised: A variety of saliency methods have been proposed in the literature.

Original: Some, such as edge detectors and
Revised: Some, such as edge detectors and

Original: Guided Backpropagation (Springenberg et al., 2014) are independent from the predictive model (Nie et al., 2018; Adebayo et al., 2018).
Revised: Guided Backpropagation (Springenberg et al., 2014) are independent from the predictive model (Nie et al., 2018; Adebayo et al., 2018).

Original: 1 Others are designed only for speciﬁc architectures ( i.e., GradCAM (Selvaraju et al., 2016) for CNNs, DeConvNet for CNNs with ReLU activations (Zeiler &
Revised: 1 Others are designed only for speciﬁc architectures ( i.e., GradCAM (Selvaraju et al., 2016) for CNNs, DeConvNet for CNNs with ReLU activations (Zeiler &

Original: Fergus, 2014)).
Revised: Fergus, 2014)).

Original: In this paper, instead of exhaustively evaluating all saliency methods, we apply our method to the three saliency methods that do depend on the predictor ( i.e., passing the sanity checks in Adebayo et al.
Revised: In this paper, instead of exhaustively evaluating all saliency methods, we apply our method to the three saliency methods that do depend on the predictor ( i.e., passing the sanity checks in Adebayo et al.

Original: and Sixt et al.
Revised: and Sixt et al.

Original: and are applicable to diverse DNN architectures:
Revised: and are applicable to diverse DNN architectures:

Original: • The vanilla gradient method (Simonyan et al., 2013) simply calculates the gradient of the class score with respect to the input x , which is deﬁned as E grad ( x ; F c ) =
Revised: • The vanilla gradient method (Simonyan et al., 2013) simply calculates the gradient of the class score with respect to the input x , which is deﬁned as E grad ( x ; F c ) =

Original: (cid:79) x F c ( x ) .
Revised: (cid:79) x F c ( x ) .

Original: • The SmoothGrad method (Smilkov et al., 2017) seeks to reduce noise in the saliency map by averaging over explanations of the noisy copies of an input, deﬁned as E sg ( x ; F c )
Revised: • The SmoothGrad method (Smilkov et al., 2017) seeks to reduce noise in the saliency map by averaging over explanations of the noisy copies of an input, deﬁned as E sg ( x ; F c )

Original: = 1 N (cid:80) Ni =
Revised: = 1 N (cid:80) Ni =

Original: 1 E grad ( x + g i ; F c ) with noise vectors g i ∼ N (0 , σ 2 ) .
Revised: 1 E grad ( x + g i ; F c ) with noise vectors g i ∼ N (0 , σ 2 ) .

Original: • The integrated gradient method 2 (Sundararajan et al., 2017) starts from a baseline input x 0 and sums over the gradient with respect to scaled versions of the input ranging from the baseline to the observed input, deﬁned as E ig ( x ; F c ) =
Revised: • The integrated gradient method 2 (Sundararajan et al., 2017) starts from a baseline input x 0 and sums over the gradient with respect to scaled versions of the input ranging from the baseline to the observed input, deﬁned as E ig ( x ; F c ) =

Original: ( x − x 0 ) × (cid:82) 1 0 (cid:79) x F c ( x 0 + α ( x − x 0 )) dα .
Revised: ( x − x 0 ) × (cid:82) 1 0 (cid:79) x F c ( x 0 + α ( x − x 0 )) dα .

Original: We do not empirically compare to several other categories of methods.
Revised: We do not empirically compare to several other categories of methods.

Original: Counterfactual-based methods work under the same setup as saliency methods, providing explanations for the predictions of a pretrained DNN model (Sturmfels et al., 2020).
Revised: Counterfactual-based methods work under the same setup as saliency methods, providing explanations for the predictions of a pretrained DNN model (Sturmfels et al., 2020).

Original: These methods identify the important subregions within an input image by perturbing the subregions (by adding noise, rescaling (Sundararajan et al., 2017), blurring (Fong & Vedaldi, 2017), or inpainting (Chang et al., 2019)) and measuring the resulting changes in the predictions (Ribeiro et al., 2016; Lundberg & Lee, 2017;
Revised: These methods identify the important subregions within an input image by perturbing the subregions (by adding noise, rescaling (Sundararajan et al., 2017), blurring (Fong & Vedaldi, 2017), or inpainting (Chang et al., 2019)) and measuring the resulting changes in the predictions (Ribeiro et al., 2016; Lundberg & Lee, 2017;

Original: Chen et al., 2018; Fong &
Revised: Chen et al., 2018; Fong &

Original: Vedaldi, 2017; Dabkowski & Gal, 2017; Chang et al., 2019; Yousefzadeh & O’Leary, 2019; Goyal et al., 2019).
Revised: Vedaldi, 2017; Dabkowski & Gal, 2017; Chang et al., 2019; Yousefzadeh & O’Leary, 2019; Goyal et al., 2019).

Original: Although these methods do identify meaningful subregions in practice, they exhibit several limitations.
Revised: Although these methods do identify meaningful subregions in practice, they exhibit several limitations.

Original: First, counterfactual-based methods implicitly assume that regions containing the object most contribute to the prediction (Fan et al., 2017).
Revised: First, counterfactual-based methods implicitly assume that regions containing the object most contribute to the prediction (Fan et al., 2017).

Original: However, Moosavi-Dezfooli et al.
Revised: However, Moosavi-Dezfooli et al.

Original: (2017) showed that counterfactual-based methods are also vulnerable to adversarial attacks, which force these methods to output unrelated background rather than the meaningful objects as important subregions. 
Revised: (2017) showed that counterfactual-based methods are also vulnerable to adversarial attacks, which force these methods to output unrelated background rather than the meaningful objects as important subregions. DecoyGenerator ... SaliencyMethods

Original: Second, the counterfactual images may be potentially far away from the training distribution, causing ill-deﬁned classiﬁer behavior (Burns et al., 2019;
Revised: Original image Pretrained network Decoy images Label-dependent gradients

Revised: Saliency maps of decoys

Revised: Decoy-enhanced saliency maps ...

Revised: Patch masks (A)

Revised: Original image

Revised: Decoy image (B)

Revised: Identical intermediaterepresentation

Revised: Network

Revised: Second, the counterfactual images may be potentially far away from the training distribution, causing ill-deﬁned classiﬁer behavior (Burns et al., 2019;

Original: Hendrycks & Dietterich, 2019).
Revised: Hendrycks & Dietterich, 2019).

Original: In addition to these limitations, counterfactual-based methods and our decoy-based method are fundamentally different in three ways.
Revised: In addition to these limitations, counterfactual-based methods and our decoy-based method are fundamentally different in three ways.

Original: First, the former seeks the minimum set of features to exclude in order to minimize the prediction score or to include in order to maximize the prediction score (Fong & Vedaldi, 2017), whereas our approach aims to characterize the inﬂuence of each feature on the prediction score.
Revised: First, the former seeks the minimum set of features to exclude in order to minimize the prediction score or to include in order to maximize the prediction score (Fong & Vedaldi, 2017), whereas our approach aims to characterize the inﬂuence of each feature on the prediction score.

Original: Second, counterfactual-based methods explicitly consider the decision boundary by comparing each image to the closest image on the other side of the boundary.
Revised: Second, counterfactual-based methods explicitly consider the decision boundary by comparing each image to the closest image on the other side of the boundary.

Original: In contrast, the proposed method only considers the decision boundary implicitly by calculating the gradient’s variants.
Revised: In contrast, the proposed method only considers the decision boundary implicitly by calculating the gradient’s variants.

Original: Third, unlike counterfactual images, which could potentially be out-of-distribution, decoys are plausibly constructed in the sense that their intermediate representations are indistinguishable from the original input data by design.
Revised: Third, unlike counterfactual images, which could potentially be out-of-distribution, decoys are plausibly constructed in the sense that their intermediate representations are indistinguishable from the original input data by design.

Original: Because of these limitations and differences, we do not compare our method with counterfactual-based methods.
Revised: Because of these limitations and differences, we do not compare our method with counterfactual-based methods.

Original: In addition to saliency methods and counterfactual-based methods, several other types of interpretation methods have been proposed that either aim for a different goal or have a different setup.
Revised: In addition to saliency methods and counterfactual-based methods, several other types of interpretation methods have been proposed that either aim for a different goal or have a different setup.

Original: For example, recent research ( e.g., Ribeiro et al.
Revised: For example, recent research ( e.g., Ribeiro et al.

Original: ; Lundberg & Lee (2017);
Revised: ; Lundberg & Lee (2017);

Original: Chen et al.
Revised: Chen et al.

Original: (2018; 2019b)) designed techniques to explain a black-box model, where the model’s internal weights are inaccessible.
Revised: (2018; 2019b)) designed techniques to explain a black-box model, where the model’s internal weights are inaccessible.

Original: Koh & Liang (2017) and some follow-up work (Yeh et al., 2018; Koh et al., 2019) tried to ﬁnd the training points that are most inﬂuential for a given test sample.
Revised: Koh & Liang (2017) and some follow-up work (Yeh et al., 2018; Koh et al., 2019) tried to ﬁnd the training points that are most inﬂuential for a given test sample.

Original: Some other efforts have been made to train a more interpretable DNN classiﬁer (Fan et al., 2017; Zołna et al., 2019;
Revised: Some other efforts have been made to train a more interpretable DNN classiﬁer (Fan et al., 2017; Zołna et al., 2019;

Original: Alvarez-Melis & Jaakkola, 2018; Toneva & Wehbe, 2019), synthesize samples that represent the model predictions (Ghorbani et al., 2019; Chen et al., 2019a)), or identifying noise-tolerant features (Ikeno & Hara, 2018; Schulz et al., 2020).
Revised: Alvarez-Melis & Jaakkola, 2018; Toneva & Wehbe, 2019), synthesize samples that represent the model predictions (Ghorbani et al., 2019; Chen et al., 2019a)), or identifying noise-tolerant features (Ikeno & Hara, 2018; Schulz et al., 2020).

Original: However, due to the task and setup differences, we do not consider these methods in this paper.
Revised: However, due to the task and setup differences, we do not consider these methods in this paper.

Original:  M ETHODS 3. P ROBLEM SETUP
Revised:  M ETHODS 3. P ROBLEM SETUP

Original: Consider a multi-label classiﬁcation task in which a pre-trained neural network model implements a function F : R d (cid:55)→ R C that maps from the given input x ∈ R d to C predicted classes.
Revised: Consider a multi-label classiﬁcation task in which a pre-trained neural network model implements a function F : R d (cid:55)→ R C that maps from the given input x ∈ R d to C predicted classes.

Original: The score for each class c ∈ { 1 , · · · , C } is F c ( x ) , and the predicted class is the one with maximum score, i.e., arg max c ∈{ 1 , ··· ,C } F c ( x ) .
Revised: The score for each class c ∈ { 1 , · · · , C } is F c ( x ) , and the predicted class is the one with maximum score, i.e., arg max c ∈{ 1 , ··· ,C } F c ( x ) .

Original: A saliency method aims to assign to each feature a saliency score , encoded in a saliency map E ( x ; F c ) :
Revised: A saliency method aims to assign to each feature a saliency score , encoded in a saliency map E ( x ; F c ) :

Original: R d (cid:55)→ R d , in which the features with higher scores represent higher “importance” relative to the ﬁnal prediction.
Revised: R d (cid:55)→ R d , in which the features with higher scores represent higher “importance” relative to the ﬁnal prediction.

Original: Given a pre-trained neural network model F with L layers, an input x , and a saliency method E such that E ( x ; F ) is a saliency map of the same dimensions as x , the proposed scores can be obtained in two steps: generating decoys and aggregating the saliency maps of the decoys.
Revised: Given a pre-trained neural network model F with L layers, an input x , and a saliency method E such that E ( x ; F ) is a saliency map of the same dimensions as x , the proposed scores can be obtained in two steps: generating decoys and aggregating the saliency maps of the decoys (See Fig. 1(A) that illustrates the workﬂow of creating decoy-enhanced saliency maps).

Original: 3. D ECOY DEFINITION
Revised: 3. D ECOY DEFINITION

Original: Say that F (cid:96) :
Revised: Say that F (cid:96) :

Original: R d (cid:55)→ R d (cid:96) is the function instantiated by the given network, which maps from an input x ∈ R d to its intermediate representation F (cid:96) ( x ) ∈ R d (cid:96) at layer (cid:96) ∈ { 1 , 2 , · · · , L } .
Revised: R d (cid:55)→ R d (cid:96) is the function instantiated by the given network, which maps from an input x ∈ R d to its intermediate representation F (cid:96) ( x ) ∈ R d (cid:96) at layer (cid:96) ∈ { 1 , 2 , · · · , L } .

Original: A vector ˜ x ∈ R d is said to be a decoy of x ∈ R d at a speciﬁed layer (cid:96) if the following swappable condition is satisﬁed:F
Revised: A vector ˜ x ∈ R d is said to be a decoy of x ∈ R d at a speciﬁed layer (cid:96) if the following swappable condition is satisﬁed:

Original: [Equation]
Revised: [Equation]

Original: [Equation] , K ) ) , for swappablefeatures K [Equation] { 1 , · · · , d } .
Revised: , for swappable

Original: [Equation]
Revised: [Equation]

Original: Here, the swap (˜ x , K ) operation swaps features between x and ˜ x based on the elements in K .
Revised: Here, the swap (˜ x , K ) operation swaps features between x and ˜ x based on the elements in K .

Original: In this work, K represents a small meaningful feature set, which represents a small region/segment in an image or a group of words (embeddings) in a sentence.
Revised: In this work, K represents a small meaningful feature set, which represents a small region/segment in an image or a group of words (embeddings) in a sentence.

Original: Take an image recognition task for example.
Revised: Take an image recognition task for example.

Original: Assume K = { 10 } and ˜ x is a zero matrix,
Revised: Assume K = { 10 } and ˜ x is a zero matrix,

Original: then x swap (˜ x , K ) indicates a new image that is identical to x except that the tenth pixel is set to zero.
Revised: then x swap (˜ x , K ) indicates a new image that is identical to x except that the tenth pixel is set to zero.

Revised: An illustrative explanation of swap operator is shown in

Original:  Using the swappable condition, we aim to ensure that the original image x and its decoy ˜ x are indistinguishable in terms of the intermediate representation at layer (cid:96) .
Revised: Fig. 1(B). Using the swappable condition, we aim to ensure that the original image x and its decoy ˜ x are indistinguishable in terms of the intermediate representation at layer (cid:96) .

Original: Note in particular that the construction of decoys relies solely on the ﬁrst (cid:96) layers of the neural network F 1 , F 2 , · · · , F (cid:96) and is independent of the succeeding layers F (cid:96) +1 , · · · , F L .
Revised: Note in particular that the construction of decoys relies solely on the ﬁrst (cid:96) layers of the neural network F 1 , F 2 , · · · , F (cid:96) and is independent of the succeeding layers F (cid:96) +1 , · · · , F L .

Original: As such, ˜ x is conditionally independent of the classiﬁcation task F ( x ) given the input x ; i.e., ˜ x | = F ( x ) | x .
Revised: As such, ˜ x is conditionally independent of the classiﬁcation task F ( x ) given the input x ; i.e., ˜ x | = F ( x ) | x .

Original: 3. D ECOY GENERATION
Revised: 3. D ECOY GENERATION

Original: To identify decoys satisfying the swappable condition, we solve the following optimization problem:
Revised: To identify decoys satisfying the swappable condition, we solve the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: Here, ( · ) + = max( · , 0) , and the operators (cid:107)·(cid:107) 1 and (cid:107)·(cid:107) ∞ correspond to the L 1 and L ∞ norms, respectively.
Revised: Here, ( · ) + = max( · , 0) , and the operators (cid:107)·(cid:107) 1 and (cid:107)·(cid:107) ∞ correspond to the L 1 and L ∞ norms, respectively.

Original: M ∈ { 0 , 1 } d is a speciﬁed binary mask.
Revised: M ∈ { 0 , 1 } d is a speciﬁed binary mask.

Original: And the value of each feature in the decoy ˜ x is restricted to lie in a legitimate value range i.e., [ x min , x max ] ( e.g., the pixel value should lie in [0, 255]).
Revised: And the value of each feature in the decoy ˜ x is restricted to lie in a legitimate value range i.e., [ x min , x max ] ( e.g., the pixel value should lie in [0, 255]).

Original: We impose the constraint (cid:107) F (cid:96) (˜ x ) − F (cid:96) ( x ) (cid:107) ∞ ≤ (cid:15) , which ensures that the generated decoy satisﬁes the swappable condition described in Eqn. 1.
Revised: We impose the constraint (cid:107) F (cid:96) (˜ x ) − F (cid:96) ( x ) (cid:107) ∞ ≤ (cid:15) , which ensures that the generated decoy satisﬁes the swappable condition described in Eqn. 1.

Original: It should be noted that we take ˜ x and x to be indistinguishable except for the swappable features indicated by the mask ( i.e., x swap (˜ x , K ) =
Revised: It should be noted that we take ˜ x and x to be indistinguishable except for the swappable features indicated by the mask ( i.e., x swap (˜ x , K ) =

Original: ˜ x ).
Revised: ˜ x ).

Original: As is shown later in Section 3.4, our decoy-enhanced saliency score is deﬁned to capture the empirical range of the decoy saliencies.
Revised: As is shown later in Section 3.4, our decoy-enhanced saliency score is deﬁned to capture the empirical range of the decoy saliencies.

Original: Here, we ﬁrst need to estimate the upper/lower ends of the legitimate decoys.
Revised: Here, we ﬁrst need to estimate the upper/lower ends of the legitimate decoys.

Original: To achieve this, in Eqn. 2, we maximize the deviation between ˜ x and x from both the positive and negative directions, i.e., s = +

Revised: To achieve this, in Eqn. 2, we maximize the deviation between ˜ x and x from both the positive and negative directions, i.e., s = +

Original: and s = − 1 .
Revised: and s = − 1 .

Original: By using this objective function, for each mask
Revised: By using this objective function, for each mask

Original: M , we can compute two decoys—one for the positive deviation ( i.e., s = +1 ) and the other for the negative one ( i.e., s = − 1 ).
Revised: M , we can compute two decoys—one for the positive deviation ( i.e., s = +1 ) and the other for the negative one ( i.e., s = − 1 ).

Original: For details of the optimization procedure, see Section A6.
Revised: To solve the optimization function in Eqn. 2, we employ three commonly adopted methods – lagrange multiplier, projected gradient, and change-of-variable (Carlini & Wagner, 2017) – to transform the original objective function into the following form: [Equation] [Table] [Equation]

Revised: where λ > 0 is the lagrange multiplier.

Revised: ˆ x i = arctanh (2˜ x i − 1) , for all i ∈ { 1 , 2 , · · · , d } .

Revised: τ > 0 is introdued to approximate the L ∞ norm in Eqn. 2.

Revised: After obtaining ˆ x by solving Eqn. 3, we compute ˜ x and map it back to the original feature value range [ x min , x max ] .

Revised: More details about how to transform

Revised: Eqn. 2 into Eqn.

Revised: 3 can be found in Section A6.

Original:  D ECOY - ENHANCED SALIENCY SCORES
Revised: 3. D ECOY - ENHANCED SALIENCY SCORES

Original: Given an input sample x and a swappable patch with size K , we  obtain n unique masks by sliding the swappable patch across the input with a certain stride.
Revised: Given an input sample x and a swappable patch with size P , we can obtain ( √ d − P + stride )unique masks by sliding the swappable patch across the input with a certain stride.

Revised: For computational efﬁciency, we aggregate m masks into one decoy sample and optimize these masks jointly by solving one decoy sample from Eqn. 3.

Original: Then, we can generate 2 n decoys for that sample, denoted (cid:8) ˜ x 1 , ˜ x 2 , · · · , ˜ x 2 n (cid:9) .
Revised: Then, we can generate 2 n decoys for that sample. We denote these decoys as (cid:8) ˜ x 1 , ˜ x 2 , · · · , ˜ x 2 n (cid:9) .

Revised: Here, n = (cid:106) ( √ d − P + stride ) 2 /m (cid:107) .

Original: For these decoys, we can then apply a given saliency method E to yield the corresponding decoy saliency maps (cid:8) E (˜ x 1 ; F ) , E (˜ x 2 ; F ) , · · · , E (˜ x 2 n ; F ) (cid:9) .
Revised: For these decoys, we can then apply a given saliency method E to yield the corresponding decoy saliency maps (cid:8) E (˜ x 1 ; F ) , E (˜ x 2 ; F ) , · · · , E (˜ x 2 n ; F ) (cid:9) .

Original: With these decoy saliency maps in hand, for each feature x i in x , we can characterize its saliency score variation by using a population of saliency scores 
Revised: With these decoy saliency maps in hand, for each feature x i in x , we can characterize its saliency score variation by using a population of saliency scores ˜

Original: ˜ E i = (cid:8) E (˜ x 1 ; F c ) i , E (˜ x 2 ; F c ) i , · · · , E (˜ x 2 n ; F c ) i (cid:9) .
Revised: E i = (cid:8) E (˜ x 1 ; F c ) i , E (˜ x 2 ; F c ) i , · · · , E (˜ x 2 n ; F c ) i (cid:9) .

Original: In this work, we deﬁne the decoy-enhanced saliency score Z i for each feature x i as
Revised: In this work, we deﬁne the decoy-enhanced saliency score Z i for each feature x i as

Original: [Equation]
Revised: [Equation]

Original: Here, Z i is determined by the empirical range of the decoy saliency scores.
Revised: Here, Z i is determined by the empirical range of the decoy saliency scores.

Original: Ideally, important features will have large values and unimportant ones will have small values.
Revised: Ideally, important features will have large values and unimportant ones will have small values.

Original: Note that the proposed method is designed speciﬁcally for nonlinear models in need of interpretation.
Revised: Note that the proposed method is designed speciﬁcally for nonlinear models in need of interpretation.

Original: As is discussed in Section A7, it exhibits ill-deﬁned on linear models.
Revised: As is discussed in Section A7, it cannot output meaningful saliency maps on linear models.

Original: It should also be noted that by sliding the swappable patch across the input and ensembling the obtained decoy-enhanced saliency maps, we could capture the saliency of each feature.
Revised: It should also be noted that by sliding the swappable patch across the input and ensembling the obtained decoy-enhanced saliency maps, we could capture the saliency of each feature.

Original: The motivations of manipulating at a patch level rather than the entire input are capturing the local dependency structure and enabling batch operations for better efﬁciency.
Revised: The motivations of manipulating at a patch level rather than the entire input are capturing the local dependency structure and enabling batch operations for better efﬁciency.

Original: 3. T HEORETICAL INSIGHTS
Revised: 3. T HEORETICAL INSIGHTS

Original: In this section, we analyze the saliency score method in a theoretical fashion.
Revised: In this section, we analyze the saliency score method in a theoretical fashion.

Original: In particular, we take a convolutional neural network with the ReLU activation function as an example to discuss why the proposed interpretation method can account for inter-feature dependence while also improving explanatory robustness.
Revised: In particular, we take a convolutional neural network with the ReLU activation function as an example to discuss why the proposed interpretation method can account for inter-feature dependence while also improving explanatory robustness.

Original: It should be noted that, while we conduct our theoretical analysis in the setting of CNNs with a speciﬁc activation function, the conclusions drawn from the theoretical analysis can easily be extended to other feed-forward neural architectures and other activation functions ( e.g., sigmoid and tanh).
Revised: It should be noted that, while we conduct our theoretical analysis in the setting of CNNs with a speciﬁc activation function, the conclusions drawn from the theoretical analysis can easily be extended to other feed-forward neural architectures and other activation functions ( e.g., sigmoid and tanh).

Original: For analysis of other neural architectures, see Section A9.
Revised: For analysis of other neural architectures, see Section A9.

Original: Consider a CNN with L hidden blocks, with each layer (cid:96) containing a convolutional layer with a ﬁlter of size √ s (cid:96) × √ s (cid:96) and a max pooling layer with pooling size √ s (cid:96) × √ s (cid:96) .
Revised: Consider a CNN with L hidden blocks, with each layer (cid:96) containing a convolutional layer with a ﬁlter of size √ s (cid:96) × √ s (cid:96) and a max pooling layer with pooling size √ s (cid:96) × √ s (cid:96) .

Original: (We set the pooling size the same as the kernel size in each block for simplicity.)
Revised: (We set the pooling size the same as the kernel size in each block for simplicity.)

Original: The input to this CNN is x ∈ R d , unrolled from a √ d × √ d matrix.
Revised: The input to this CNN is x ∈ R d , unrolled from a √ d × √ d matrix.

Original: Similarly, we also unroll each convolutional ﬁlter into g (cid:96) ∈
Revised: Similarly, we also unroll each convolutional ﬁlter into g (cid:96) ∈

Original: R s (cid:96) , where g (cid:96) is indexed as ( g (cid:96) )
Revised: R s (cid:96) , where g (cid:96) is indexed as ( g (cid:96) )

Original: j for j ∈ J (cid:96) .
Revised: j for j ∈ J (cid:96) .

Original: Here, J (cid:96) corresponds to the index shift in matrix form from the top-left to bottom-right element.
Revised: Here, J (cid:96) corresponds to the index shift in matrix form from the top-left to bottom-right element.

Original: For example, a 3 × 3 convolutional ﬁlter ( i.e., s (cid:96) = 9 ) is indexed by J (cid:96) =
Revised: For example, a 3 × 3 convolutional ﬁlter ( i.e., s (cid:96) = 9 ) is indexed by J (cid:96) =

Original: (cid:110) −√
Revised: (cid:110) −√

Original: d − 1 , −√ d, −√ d + 1 , − 1 , 0 , 1 , √ d − 1 , √ d, √ d + 1 (cid:111) .
Revised: d − 1 , −√ d, −√ d + 1 , − 1 , 0 , 1 , √ d − 1 , √ d, √ d + 1 (cid:111) .

Original: The output of the network is the probability vector p ∈ R C generated by the softmax function, where C is the total number of classes.
Revised: The output of the network is the probability vector p ∈ R C generated by the softmax function, where C is the total number of classes.

Original: Such a network can be represented as
Revised: Such a network can be represented as

Original: [Equation]
Revised: [Equation]

Original: where relu ( · ) and pool ( · ) indicate the ReLU and pooling operators, m (cid:96) ∈ R d (cid:96) is the output of the block (cid:96) ( m 0 = x ), and ( g (cid:96) ∗ m (cid:96) − 1 ) ∈ R d
Revised: where relu ( · ) and pool ( · ) indicate the ReLU and pooling operators, m (cid:96) ∈ R d (cid:96) is the output of the block (cid:96) ( m 0 = x ), and ( g (cid:96) ∗ m (cid:96) − 1 ) ∈ R d

Original: (cid:96) − 1 represents a convolutional operation on that block.
Revised: (cid:96) − 1 represents a convolutional operation on that block.

Original: We assume for simplicity that the convolution retains the input shape.
Revised: We assume for simplicity that the convolution retains the input shape.

Original: Consider an input x and its decoy ˜ x , generated by swapping features in K .
Revised: Consider an input x and its decoy ˜ x , generated by swapping features in K .

Original: For each feature i ∈ K , we have the following theorem for the decoy-enhanced saliency score Z i :
Revised: For each feature i ∈ K , we have the following theorem for the decoy-enhanced saliency score Z i :

Original: Theorem 1.
Revised: Theorem 1.

Original: In the aforementioned setting, Z i is bounded by
Revised: In the aforementioned setting, Z i is bounded by

Original: [Equation]
Revised: [Equation]

Original: Here, C 1 > 0 is a bounded constant, and H x is the Hessian of F c ( x ) on x where ( H x ) i,k = ∂
Revised: Here, C 1 > 0 is a bounded constant and H x is the Hessian of F c ( x ) on x where ( H x ) i,k = ∂

Original: 2 F c ∂ x i ∂ x k .
Revised: 2 F c ∂ x i ∂ x k .

Original: ˜ x + and ˜ x − refer to the decoy that maximizes and minimizes E (˜ x ; F c ) , respectively.
Revised: ˜ x + and ˜ x − refer to the decoy that maximizes and minimizes E (˜ x ; F c ) , respectively.

Original: See Section A for the proof.
Revised: See Section A for the proof.

Original: Theorem 1 implies that the proposed saliency score is determined by the secondorder Hessian ( ( H x ) i,k ) in the same swappable feature set.
Revised: Theorem 1 implies that the proposed saliency score is determined by the secondorder Hessian ( ( H x ) i,k ) in the same swappable feature set.

Original: The score explicitly models the feature dependencies in the swappable feature set via this second-order Hessian, potentially capturing meaningful patterns such as edges, texture, etc.
Revised: The score explicitly models the feature dependencies in the swappable feature set via this second-order Hessian, potentially capturing meaningful patterns such as edges, texture, etc.

Original: In addition to enabling representation of inter-feature dependence, Theorem 1 sheds light on the robustness of the proposed saliency score against adversarial attack.
Revised: In addition to enabling representation of inter-feature dependence, Theorem 1 sheds light on the robustness of the proposed saliency score against adversarial attack.

Original: To illustrate the robustness improvement of our method, we introduce the following proposition.
Revised: To illustrate the robustness improvement of our method, we introduce the following proposition.

Original: The proof of this proposition as well as in-depth analysis can be found in Section A8.
Revised: The proof of this proposition as well as in-depth analysis can be found in Section A8.

Original: Proposition 1.
Revised: Proposition 1.

Original: Given an input x and the corresponding adversarial sample ˆ x , if both | x i − ˜ x i | ≤
Revised: Given an input x and the corresponding adversarial sample ˆ x , if both | x i − ˜ x i | ≤

Original: C 

Revised: C

Original: δ i and (cid:12)(cid:12)(cid:12) ˆ x
Revised: δ i and (cid:12)(cid:12)(cid:12) ˆ x

Original: i −
Revised: i −

Original: ˜ˆ x i (cid:12)(cid:12)(cid:12) ≤ C 2 δ i can  obtain where C 2 > 0 is a bounded constant, then the following relation can be guaranteed.
Revised: ˜ˆ x i (cid:12)(cid:12)(cid:12) ≤ C 2 δ i can be obtain where C 2 > 0 is a bounded constant and δ i = | E (ˆ x , F ) i − E ( x , F ) i | , then the following relation can be guaranteed.

Original: [Equation]
Revised: [Equation]

Original: Given an adversarial sample ˆ x ( i.e., the perturbed x ), we say a saliency method is not robust against ˆ x if the deviation of the corresponding explanation δ i = | E (ˆ x , F ) i − E ( x , F ) i | (for all i ∈ { 1 , 2 , · · · , d } ) is large.
Revised: Given an adversarial sample ˆ x ( i.e., the perturbed x ), we say a saliency method is not robust against ˆ x if the deviation of the corresponding explanation δ i = | E (ˆ x , F ) i − E ( x , F ) i | (for all i ∈ { 1 , 2 , · · · , d } ) is large.

Original: According to the proposition above, we can easily discover that the deviation of our decoy-enhanced saliency score is always no larger than that of other saliency methods when a certain condition is satisﬁed.
Revised: According to the proposition above, we can easily discover that the deviation of our decoy-enhanced saliency score is always no larger than that of other saliency methods when a certain condition is satisﬁed.

Original: This indicates that, when the condition holds, our saliency method can guarantee a stronger resistance to the adversarial perturbation.
Revised: This indicates that, when the condition holds, our saliency method can guarantee a stronger resistance to the adversarial perturbation.

Original: To ensure the  conditions | x i − ˜ x i | ≤ C 2 δ i and (cid:12)(cid:12)(cid:12) ˆ x i − ˜ˆ
Revised: To ensure the satisﬁcation of conditions | x i − ˜ x i | ≤ C 2 δ i and (cid:12)(cid:12)(cid:12) ˆ x i − ˜ˆ

Original: x i (cid:12)(cid:12)(cid:12) ≤ C 2 δ i obtain, we can further introduce the corresponding condition as a constraint to Eqn. 2.
Revised: x i (cid:12)(cid:12)(cid:12) ≤ C 2 δ i , we can further introduce the corresponding condition as a constraint to Eqn. 2.

Original: In the following section, without further clariﬁcation, the saliency scores used in our evaluation are all derived with this constraint imposed.
Revised: In the following section, without further clariﬁcation, the saliency scores used in our evaluation are all derived with this constraint imposed.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: To evaluate the effectiveness of our proposed method, we perform extensive experiments on deep learning models that target three tasks: image classiﬁcation, sentiment analysis, and network intrusion detection.
Revised: To evaluate the effectiveness of our proposed method, we perform extensive experiments on deep learning models that target three tasks: image classiﬁcation, sentiment analysis, and network intrusion detection.

Original: The performance of our approach is assessed both qualitatively and quantitatively.
Revised: The performance of our approach is assessed both qualitatively and quantitatively.

Original: The results show that our proposed method identiﬁes intuitively more coherent saliency maps than the state-of-the-art saliency methods alone.
Revised: The results show that our proposed method identiﬁes intuitively more coherent saliency maps than the state-of-the-art saliency methods alone.

Original: The method also achieves quantitatively better alignment to truly important features and demonstrates stronger robustness to adversarial manipulation.
Revised: The method also achieves quantitatively better alignment to truly important features and demonstrates stronger robustness to adversarial manipulation.

Original: The description of the datasets and experimental setup can be found in Section A10.[Figure] [Figure] [Figure]
Revised: The description of the datasets and experimental setup can be found in Section A10.

Original: [Figure]

Original: [Figure]

Original: 4. S ALIENCY BENCHMARK
Revised: 4. S ALIENCY BENCHMARK

Original: As mentioned in Section [Figure] we apply our decoy enhancement method to three saliency methods: vanilla gradient, SmoothGrad, and integrated gradient.
Revised: As mentioned in Section 2, we apply our decoy enhancement method to three saliency methods: vanilla gradient, SmoothGrad, and integrated gradient.

Revised: Here, we applied the default setup for the integrated gradient (a zero baseline) and SmoothGrad.

Revised: Section A14 shows that our method can also improve the performance of the variants of the integrated gradient/SmoothGrad and GradCAM.

Original: In each case, the decoy-enhanced saliency scores are post-processed in the following way before qualitative and quantitative evaluations.
Revised: In each case, the decoy-enhanced saliency scores are post-processed in the following way before qualitative and quantitative evaluations.

Original: First, we follow the existing methods (Simonyan et al., 2013) and compute the absolute saliency scores.

Original: For images, to obtain a single importance score for each pixel, we use the maximum absolute saliency score across all color channels. To avoid outlier features with extremely high saliency values leading to almost zero saliency scores for the other features, we then winsorized outlier saliencyvalues to a relatively high value (the 95 th percentile), as suggested by Smilkov et al.
Revised: To rule out the bias introduced by the saliency values and ensure a fair comparison, we constructed a binary saliency map by retaining only the top- K features ranked by each method. Speciﬁcally, we set the saliency value of the selected features as 1 and the rest features as 0 . In this section, we choose the K as the top 20 percent of all features.

Original: (2017) before linearly scaling to the range [0 , 1] .
Revised: Note that Section A16 shows that subtly varying K does not inﬂuence the experiment conclusions.

Original: To demonstrate that all three methods, when enhanced with decoys, still depend on the predictor, we carry out a sanity check on the ImageNet dataset.
Revised: To demonstrate that all three methods, when enhanced with decoys, still depend on the predictor, we carry out a sanity check on the ImageNet dataset.

Original: The results show that our decoy enhanced-saliency methods pass the sanity check (see Section A11 for details).
Revised: The results show that our decoy enhanced-saliency methods pass the sanity check (see Section A11 for details).

Original: 4. P ERFORMANCE IN VARIOUS APPLICATIONS
Revised: 4. P ERFORMANCE IN VARIOUS APPLICATIONS

Original: To comprehensively evaluate our proposed approach against the baselines mentioned above, we focus on two criteria.
Revised: To comprehensively evaluate our proposed approach against the baselines mentioned above, we focus on two criteria.

Original: First, we aim to achieve qualitative coherence of the identiﬁed saliency map.
Revised: First, we aim to achieve qualitative coherence of the identiﬁed saliency map.

Original: Intuitively, we prefer a saliency method that highlights features that align closely with the predictions ( e.g., highlights the object of interest in an image or the words indicating the sentiment of the sentence).
Revised: Intuitively, we prefer a saliency method that highlights features that align closely with the predictions ( e.g., highlights the object of interest in an image or the words indicating the sentiment of the sentence).

Original: Second,  to quantify the correctness of the saliency maps produced by the corresponding saliency method, we use the ﬁdelity metric (Dabkowski & Gal, 2017), deﬁned as SF ( E ( · ; F c ) , x ) =
Revised: Second, we aim to quantify the correctness of the saliency maps produced by the corresponding method. To do it, we use the ﬁdelity metric (Dabkowski & Gal, 2017), deﬁned as: [Equation] [Figure]

Original: − log F c ( E ( x ; F c ) ◦ x ) F c ( E ( x ; F c ) ◦ x ) , where c indicates the predicted class of input x , and E ( x ; F c ) is the normalized saliency map described above.
Revised: ound [Figure] where c indicates the predicted class of input x , and E ( x ; F c ) is the top- K -retained binary saliency map described above.

Original: E ( x ; F c ) ◦ x performs entry-wise multiplication between E ( x ; F c ) and x , encoding the overlap between the object of interest and the concentration of the saliency map.
Revised: E ( x ; F c ) ◦ x performs entry-wise multiplication between E ( x ; F c ) and x , encoding the overlap between the object of interest and the concentration of the saliency map.

Original: The rationale behind this metric  is as follows.
Revised: The rationale behind this metric utilization is as follows.

Original: By viewing the saliency score of the feature as its contribution to the predicted class, a good saliency method will weight important features morehighly than less important ones and thus give rise to higher predicted class scores and lower metric values.
Revised: By viewing the saliency score of the feature as its contribution to the predicted class, a good saliency method will highlight more important features and thus give rise to higher predicted class scores and lower metric values.

Original: Note that we subtract the mean saliency E ( x ; F c ) to eliminate the inﬂuence of bias in E ( x ; F c ) and exclude trivial cases such as E ( x ; F c )

Original: = 1 .

Original: 4.2. P ERFORMANCE ON THE I MAGE N ET DATASET
Revised: 4.2. P ERFORMANCE ON THE I MAGE N ET DATASET

Original: We applied our decoy-enhanced saliency score to randomly sampled images from the ImageNet dataset (Russakovsky et al., 2015), with a pretrained VGG16 model (Simonyan & Zisserman, 2014).[Figure]
Revised: We applied our decoy-enhanced saliency score to randomly sampled images from the ImageNet dataset (Russakovsky et al., 2015), with a pretrained VGG16 model (Simonyan & Zisserman, 2014).

Original: See Section A12 for applicability of our method to diverse CNN architectures such as AlexNet (Krizhevsky et al., 2012) and ResNet (He et al., 2016).
Revised: See Section A12 for applicability of our method to diverse CNN architectures such as AlexNet (Krizhevsky et al., 2012) and ResNet (He et al., 2016).

Original: The 3 × 3 image patches are treated as swappable features in generating decoys.
Revised: The 3 × 3 image patches are treated as swappable features in generating decoys.

Original: A side-by-side comparison (Figure 1(A)) suggests that decoys consistently help to reduce noise and produce more visually coherent saliency maps.
Revised: A side-by-side comparison (Fig. 2(A)) suggests that decoys consistently help to reduce noise and produce more visually coherent saliency maps.

Original: For example, the original integrated gradient method highlights the region of dog head in a scattered format.
Revised: For example, the original integrated gradient method highlights the region of dog head in a scattered format, which is also revealed by the difference plot.

Original: In contrast, the decoy-enhanced integrated gradient method not only highlights the missing body but also identiﬁes the dog head with more details such as ears, cheek, and nose (See Section A18 for more visualization examples).
Revised: In contrast, the decoy-enhanced integrated gradient method not only highlights the missing body but also identiﬁes the dog head with more details such as ears, cheek, and nose (See Section A18 for more visualization examples).

Original: The visual coherence is also quantitatively supported by the saliency ﬁdelity (Figure 1(B) blue and orange bars.).
Revised: The visual coherence is also quantitatively supported by the saliency ﬁdelity score.

Original: To further evaluate the necessity of the two steps ( i.e., decoy generation and aggregation) in our method, we carried out a control experiment by replacing either step with alternatives.
Revised: To further evaluate the necessity of the two steps ( i.e., decoy generation and aggregation) in our method, we carried out a control experiment by replacing either step with alternatives.

Original: Speciﬁcally, as alternatives to the decoy generation, we used an image in which all pixel values are  replaced with a single mean pixel value.
Revised: Speciﬁcally, as alternatives to the decoy generation, we used an image in which all pixel values are either replaced with a single mean pixel value or contaminated with Gaussian white noise.

Original: Regarding the decoy aggregation, we calculated the mean saliency score as the alternative.
Revised: Regarding the decoy aggregation, we calculated the mean saliency score as the alternative.

Original: As shown in Figure 1(B) above, our method, which incorporate both steps, reports the best performance.
Revised: As shown in Fig. 2(B), our method, which incorporate both steps, reports the best performance.

Original: This validates the effectiveness of each of our designs.
Revised: This validates the effectiveness of each of our designs.

Revised: Recall that the number of decoys n is decided by the patch size ( P ), stride, and the number of multiple masks in one decoy ( m ).

Revised: Here, we keep stride as 1 and vary n by selecting different P and

Revised: M .

Revised: Fig.

Revised: 2(C) shows that our method achieves stable ﬁdelity scores across the substantial variations of decoy numbers.

Original:  4.2. P ERFORMANCE ON THE S TANFORD S ENTIMENT T REEBANK (SST) DATASET
Revised: The sensitivity test of other hyper-parameters can be found in Section A16. [Figure] 4.2. P ERFORMANCE ON THE S TANFORD S ENTIMENT T REEBANK [Table]

Original: We also applied our decoy-enhanced saliency score to randomly sampled sentences from the Stanford
Revised: We also applied our decoy-enhanced saliency score to randomly sampled sentences from the Stanford

Original: Sentiment Treebank (SST) (Russakovsky et al., 2015).
Revised: Sentiment Treebank (SST) (Russakovsky et al., 2015).

Original: We train a two-layer CNN (Kim, 2014) which takes the pretrained word embeddings as input (Pennington et al., 2014) (see A10 for experimental details).
Revised: We train a two-layer CNN (Kim, 2014) which takes the pretrained word embeddings as input (Pennington et al., 2014) (see A10 for experimental details).

Original: As suggested by Guan et al.
Revised: As suggested by Guan et al.

Original: (2019), the average saliency value of all dimensions of a word embedding is regarded as the word-level saliency value.
Revised: (2019), the average saliency value of all dimensions of a word embedding is regarded as the word-level saliency value.

Original: The embeddings of the words are treated as swappable features when generating decoys.
Revised: The embeddings of the words are treated as swappable features when generating decoys.

Original: As shown in Figure 2(A) and (B), a side-by-side comparison suggests that decoys consistently help to produce semantically more meaningful saliency maps.
Revised: As shown in Fig. 3(A), a side-by-side comparison suggests that decoys consistently help to produce semantically more meaningful saliency maps.

Original: For example, in a sentence with negative sentiment, keywords associated with negation, such as ’no’ and ’not’, are more highlighted by decoy-enhanced saliency methods.
Revised: For example, in a sentence with negative sentiment, keywords associated with negation, such as ’no’ and ’not’, are more highlighted by decoy-enhanced saliency methods.

Original: The semantic coherence is also quantitatively supported by the saliency ﬁdelity (Figure 2(C)).
Revised: The semantic coherence is also quantitatively supported by the saliency ﬁdelity (Fig. 3(B)).

Original: The constant-decoy ( i.e., the mean embedding ofall the sentences) baseline also achieves worse ﬁdelity than the decoys generated by Eqn. 2. Note that other decoy proxies ( e.g., random noise, blurring) are invalid for text data.
Revised: We also tested the alternatives mentioned above: constant (replacing the decoy generation with the mean embedding of the whole dictionary) and noise perturbation with range aggregation, and decoy with mean aggregation. Fig. 3(B) shows that our method outperforms these alternatives.

Original: 4. R OBUSTNESS TO ADVERSARIAL ATTACKS
Revised: 4. R OBUSTNESS TO ADVERSARIAL ATTACKS

Original: Next we investigate the robustness of our method to adversarial manipulations of images.
Revised: Next we investigate the robustness of our method to adversarial manipulations of images.

Original: In particular, we focus on three popular adversarial attacks (Ghorbani et al., 2017): (1) the top- k attack, which seeks to decrease the scores of the top k most important features, (2) the target attack, which aims to increase the importance of a pre-speciﬁed region in the input image, and (3) the mass-center attack, which aims to spatially change the center of mass of the original saliency map.
Revised: In particular, we focus on three popular adversarial attacks (Ghorbani et al., 2017): (1) the top- k attack, which seeks to decrease the scores of the top k most important features, (2) the target attack, which aims to increase the importance of a pre-speciﬁed region in the input image, and (3) the mass-center attack, which aims to spatially change the center of mass of the original saliency map.

Original: Here, we specify the bottom-right 4 × 4 region of the original image for the target attack and select k = 5000 in the top- k attack.
Revised: Here, we specify the bottom-right 4 × 4 region of the original image for the target attack and select k = 5000 in the top- k attack.

Original: We use the sensitivity metric (Alvarez-Melis & Jaakkola, 2018) to quantify the robustness of a saliency method E to adversarial attack, deﬁned as SS ( E ( · , F c ) , x , ˆ x ) = (cid:107) ( E ( x ,F c ) − E (ˆ x ,F c )) (cid:107) 2 (cid:107) x − ˆ x (cid:107),
Revised: We use the sensitivity metric (Alvarez-Melis & Jaakkola, 2018) to quantify the robustness of a saliency method E to adversarial attack, deﬁned as:

Original: [Figure]
Revised: [Equation]

Original: where ˆ x is the perturbed image of x .
Revised: where ˆ x is the perturbed image of x .

Original: A small sensitivity value means that similar inputs do not lead to substantially different saliency maps
Revised: A small sensitivity value means that similar inputs do not lead to substantially different saliency maps.

Original: [Figure] As shown in Figure 3(A) [Figure]
Revised: As shown in Fig.

Original:  a side-by-side comparison suggests that decoys consistently yield low sensitivity scores and help to produce more visually coherent saliency maps, mitigating the impact of various adversarial attacks.
Revised: 4(A), a side-by-side comparison suggests that decoys consistently yield low sensitivity scores and help to produce more visually coherent saliency maps, mitigating the impact of various adversarial attacks.

Original: More examples can be found in Section A18.
Revised: More examples can be found in Section A18.

Original: The visual coherence and robustness to adversarial attacks are also quantitatively supported by Figure 3(B) ∼ (D).
Revised: The visual coherence and robustness to adversarial attacks are also quantitatively supported by Fig. 4(B) ∼ (D).

Original: As is mentioned above, we also did experiments on a MLP trained with a network intrusion dataset and show the results in Section A13.
Revised: As is mentioned above, we also did experiments on a MLP trained with a network intrusion dataset and show the results in Section A13.

Original: The results are consistent with those on CNNs, which conﬁrm our method’s applicability to the widely-used feed-forward networks. 
Revised: The results are consistent with those on CNNs, which conﬁrm our method’s applicability to the widely-used feed-forward networks. [Figure]

Original:  D ISCUSSION AND CONCLUSION
Revised:  D ISCUSSION AND CONCLUSION

Original: In this work, we propose a method for computing, from a given saliency method, decoy-enhanced saliency scores that yield more accurate and robust saliency maps.
Revised: In this work [Figure] we propose a method for computing, from a given saliency method, decoy-enhanced saliency scores that yield more accurate and robust saliency maps.

Original: We formulate the decoy generation as an optimization problem, applicable to diverse deep neural network architecture.
Revised: We formulate the decoy generation as an optimization problem, applicable to diverse deep neural network architecture.

Original: We demonstrate the superior performance of our method relative to three standard saliency methods, both qualitatively and quantitatively, even in the presence of various adversarial perturbations to the image.
Revised: We demonstrate the superior performance of our method relative to three standard saliency methods, both qualitatively and quantitatively, even in the presence of various adversarial perturbations to the image.

Original: From a theoretical perspective, by deriving a closed-form solution, we show that the proposed score can provably compensate for the limitations of existing saliency methods by reﬂecting the joint effects from other dependent features and maintaining robustness to adversarial perturbations. 
Revised: From a theoretical perspective, by deriving a closed-form solution, we show that the proposed score can provably compensate for the limitations of existing saliency methods by reﬂecting the joint effects from other dependent features and maintaining robustness to adversarial perturbations. Fig.

Revised: 2(C) shows our method can achieve a decent performance with only a small number of decoys.

Original: Although decoy generation introduces extra computational overhead on top of existing saliency methods, the run time of generating one decoy is faster than existing saliency methods (See Section A15). Since decoy generation can run parallelly in a batch mode, theextra overhead does not jeopardize our method’s application to large datasets.
Revised: Section A15 further shows the runtime of generating one decoy is marginal compared to existing saliency methods. This indicates that our technique can improve the existing saliency methods without introducing too much computational overhead. With the parallel computing enabled by multiple GPUs, our method can be much faster, which further decreases the overhead and escalates our method’s practicability (See Section A15 for a detailed runtime analysis).

Original: Our method has three hyperparameters: swappable feature size K , network layer (cid:96) , and initial Lagrange multiplier λ .
Revised: Our method mainly introduce three hyperparameters: swappable feature size K , network layer (cid:96) , and initial Lagrange multiplier λ .

Original: In Section A16, we show that our method is insensitive to the substantial variation of hyperparameters.
Revised: In Section A16, we show that our method is insensitive to the substantial variation of hyperparameters.

Original: We generate decoys by using Eqn. 2.
Revised: We generate decoys by using Eqn. 2.

Original: While there are other widely used perturbation methods ( e.g., random noise, blurring, and inpainting), they are not suitable for generating decoys.
Revised: While there are other widely used perturbation methods ( e.g., random noise, blurring, and inpainting), they are not suitable for generating decoys.

Revised: First, Section 4 shows that some general pertrbations ( i.e., random noise and constant perturbation) obtain worse ﬁdelity than decoy.

Original: First, without ensuring the swappable condition in Eqn. 1, they cannot provide a theoretical guarantee for robustness improvement.
Revised: Second, without ensuring the swappable condition in Eqn. 1, they cannot provide a theoretical guarantee for robustness improvement.

Original: Second, methods like blurring and inpainting are not well-deﬁned for applications beyond computer vision.
Revised: Third, methods like blurring and inpainting are not well-deﬁned for applications beyond computer vision.

Original: This work points to several promising directions for future research.
Revised: This work points to several promising directions for future research.

Revised: First, ( E ( x ; F c ) ◦ x ) may be out-of-distribution and thus fails our ﬁdelity metric.

Revised: We will investigate more rigorous metrics and use other benchmark datasets ( e.g., BAM (Yang & Kim, 2019)) for evaluation.

Original: First, a possible extension is to customize our method to recurrent neural networks and to inputs with categorical/discrete features.
Revised: Second, a possible extension is to customize our method to recurrent neural networks and to inputs with categorical/discrete features.

Original: Second, recent work (Bansal et al., 2020; Chen et al., 2019c) shows that adversarial training can improve the interpretability of a DNN model.
Revised: Third, recent work (Bansal et al., 2020; Chen et al., 2019c) shows that adversarial training can improve the interpretability of a DNN model.

Original: It is worth exploring whether our method could further enhance the quality of saliency maps derived from these adversarially retrained classiﬁers.
Revised: It is worth exploring whether our method could further enhance the quality of saliency maps derived from these adversarially retrained classiﬁers.

Original: A third promising direction could be reframing interpretability as hypothesis testing and using decoys to deliver a set of salient features, subject to false discovery rate control at some pre-speciﬁed level (Burns et al., 2019;
Revised: A fourth promising direction could be reframing interpretability as hypothesis testing and using decoys to deliver a set of salient features, subject to false discovery rate control at some pre-speciﬁed level (Burns et al., 2019;

Original: Lu et al., 2018).
Revised: Lu et al., 2018).

Original: A I MPLEMENTATION DETAILS
Revised: A I MPLEMENTATION DETAILS

Original: The optimization function proposed to generate decoys is non-differentiable and very difﬁcult to solve; hence, we instead solve an alternate formulation with the help of the following tricks.
Revised: The optimization function proposed to generate decoys is non-differentiable and very difﬁcult to solve; hence, we instead solve an alternate formulation with the help of the following tricks.

Original: First, we introduce a Lagrange multiplier λ > 0 and augment the ﬁrst constraint in the optimization function as a penalty in the objective function.
Revised: First, we introduce a Lagrange multiplier λ > 0 and augment the ﬁrst constraint in the optimization function as a penalty in the objective function.

Original: This will rule out the hyper-parameter (cid:15) in Eqn. 2.
Revised: This will rule out the hyper-parameter (cid:15) in Eqn. 2.

Original: Second, we use projected gradient descent during the optimization to eliminate the mask constraint ( i e., (˜ x − x ) ◦ (1 − M ) = 0 ).
Revised: Second, we use projected gradient descent during the optimization to eliminate the mask constraint ( i e., (˜ x − x ) ◦ (1 − M ) = 0 ).

Original: Speciﬁcally, after each standard gradient descent step, we enforce ˜ x
Revised: Speciﬁcally, after each standard gradient descent step we enforce ˜ x

Original: = ˜ x ◦ M + x ◦ (1 − M ) .
Revised: = ˜ x ◦ M + x ◦ (1 − M ) .

Original: Third, we use the change-of-variable trick (Carlini & Wagner, 2017) to eliminate the feature value constraint ( i e., ˜ x ∈ [ x min , x max ] d ).
Revised: Third, we use the change-of-variable trick (Carlini & Wagner, 2017) to eliminate the feature value constraint ( i.e., ˜ x ∈ [ x min , x max ] d ).

Original: Instead of directly optimizing ˜ x, we ﬁrst normalize it to [0 , 1] and introduce ˆ x satisfying ˜ x i = 12 (tanh( ˆ x i ) + 1) , for all i ∈ { 1 , 2 , · · · , d } .
Revised: Instead of directly optimizing ˜ x we ﬁrst normalize it to [0 , 1] and introduce ˆ x satisfying ˜ x i = 1 2 (tanh( ˆ x i ) + 1) , for all i ∈ { 1 , 2 , · · · , d } .

Original: Because tanh(ˆ x i ) ∈ [ − 1 , 1] implies ˜ x i ∈ [0 , 1] , any solution to ˆ x is naturally valid.
Revised: Because tanh(ˆ x i ) ∈ [ − 1 , 1] implies ˜ x i ∈ [0 , 1] , any solution to ˆ x is naturally valid.

Original: It should be noted that other transformations for this third step are also possible but were not explored in this paper.
Revised: It should be noted that other transformations for this third step are also possible but were not explored in this paper.

Original: Putting these ideas together, we minimize the following objective function:
Revised: Putting these ideas together, we minimize the following objective function:

Original: [Equation]
Revised: [Equation]

Original: where λ > 0 is initialized small and repeatedly doubled until the optimization succeeds.
Revised: where λ > 0 is initialized small and repeatedly doubled until the optimization succeeds.

Original: Because the L ∞ norm is not fully differentiable, we adopt the approximation trick introduced by Carlini &
Revised: Because the L ∞ norm is not fully differentiable, we adopt the approximation trick introduced by Carlini &

Original: Wagner (2017) and solve the following formulation:
Revised: Wagner (2017) and solve the following formulation:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where τ > 0 .
Revised: where τ > 0 .

Original: In this paper, we follow the selection strategy proposed in Carlini & Wagner (2017) and initialize τ = 1 .
Revised: In this paper, we follow the selection strategy proposed in Carlini & Wagner (2017) and initialize τ = 1 .

Original: After each iteration, if the second term is zero, then we reduce τ by a factor of 0 .
Revised: After each iteration, if the second term is zero, then we reduce τ by a factor of 0 .

Original: 95 and repeat
Revised: 95 and repeat

Original: ; otherwise, we terminate the optimization.
Revised: ; otherwise, we terminate the optimization.

Original: After obtaining ˆ x , we compute ˜ x and map it back to the original feature value range [ x min , x max ] .
Revised: After obtaining ˆ x , we compute ˜ x and map it back to the original feature value range [ x min , x max ] .

Original: Note that Eqn. 8 can be efﬁciently solved by any ﬁrst-order optimization method without introducing too much computational overhead.
Revised: Note that Eqn. 3 can be efﬁciently solved by any ﬁrst-order optimization method without introducing too much computational overhead.

Original: In practice, the average run time of solving it is 62 .
Revised: In practice, the average run time of solving it is 62 .

Original: 3% shorter than the fastest, vanilla gradient method.
Revised: 3% shorter than the fastest, vanilla gradient method.

Original: A P ROOF OF T HEOREM
Revised: A P ROOF OF T HEOREM

Original: Before proving Theorem 1, we ﬁrst state and prove the following lemma.
Revised: Before proving Theorem 1, we ﬁrst state and prove the following lemma.

Original: Lemma 1.
Revised: Lemma 1.

Original: Consider an input x and its decoy ˜ x , generated by replacing the original features with swappable features in K , |K| = K .
Revised: Consider an input x and its decoy ˜ x , generated by replacing the original features with swappable features in K , |K| = K .

Original: The partial derivative of F c (˜ x ) w.r.t.
Revised: The partial derivative of F c (˜ x ) w.r.t.

Original: to ˜ x i for i ∈ K is
Revised: to ˜ x i for i ∈ K is

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The second-order Taylor expansion of the predicted F c ( x ) for target class c around x is as follows:
Revised: The second-order Taylor expansion of the predicted F c ( x ) for target class c around x is as follows:

Original: [Equation]
Revised: [Equation]

Original:  where ∆ = x − ˜ x .
Revised: (13) where ∆ = x − ˜ x .

Original: By deﬁnition of the decoys in Section 2.2 (
Revised: By deﬁnition of the decoys in Section 2.2 (

Original: i.e., F c ( x ) =
Revised: i.e., F c ( x ) =

Original: F c (˜ x ) ), the following equation holds:
Revised: F c (˜ x ) ), the following equation holds:

Original: [Equation]
Revised: [Equation]

Original: From the above equation, we can see that, for a linear model, the linearity zeroes out the gradient of the decoys, causingthe ill-deﬁned behavior of our method.
Revised: From the above equation, we can see that, for a linear model, the linearity zeroes out the gradient of the decoys, causing our method to output zero saliency scores for all input features.

Original: This does not dilute the signiﬁcance of our method because linear models are self-explainable and do not need extra post hoc explanation tools.
Revised: We clariﬁed in

Original: We clariﬁed this in Section 3.4 that our method is mainly deﬁned for non-linear models since non-linear models are those most in need of interpretation.
Revised: Section 3.4 that our method is mainly deﬁned for non-linear complicated models.

Original: Given a swappable patch of size K × 1 starting from position i 1 , then ∆ =
Revised: Given a swappable patch of size K × 1 starting from position i 1 , then ∆ =

Original: [0 , ..., x i 1 − ˜ x
Revised: [0 , ..., x i 1 − ˜ x

Original: i 1 , ..., x i K − ˜ x
Revised: i 1 , ..., x i K − ˜ x

Original: i K , 0 , ..., 0] .
Revised: i K , 0 , ..., 0] .

Original: As such, we have
Revised: As such, we have

Original: [Equation]
Revised: [Equation]

Original: Plugging Eqn.
Revised: Plugging Eqn.

Original: (12) into Eqn.
Revised: (15) into Eqn.

Original: (11), we have
Revised: (14), we have

Original: [Equation]
Revised: [Equation]

Original: Then we can derive
Revised: Then we can derive

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: First, we can derive | ˜ x
Revised: First, we can derive | ˜ x

Original: i − x i | is bounded by 2 max ( x max , | x min | ) .
Revised: i − x i | is bounded by 2 max ( x max , | x min | ) .

Original: We also have | ˜ x
Revised: We also have | ˜ x

Original: i + k − x i + k |in that we can always ﬁnd a small perturbation to each feature in x such that (cid:107) F (cid:96) (˜ x ) −
Revised: i + k − x i + k |in that we can always ﬁnd a small perturbation to each feature in x such that (cid:107) F (cid:96) (˜ x ) −

Original: F (cid:96) ( x ) (cid:107) ∞ ≤ (cid:15) .
Revised: F (cid:96) ( x ) (cid:107) ∞ ≤ (cid:15) .

Original: In addition, both gradient and Hessian are bounded by some Lipschitz constant (Szegedy et al., 2013).
Revised: In addition, both gradient and Hessian are bounded by some Lipschitz constant (Szegedy et al., 2013).

Original: As a result, we can always ﬁnd a constant C , such that C ≥
Revised: As a result, we can always ﬁnd a constant C , such that C ≥

Original: [Equation]
Revised: [Equation]

Original: For the case K = 1 , we have
Revised: For the case K = 1 , we have

Original: [Equation]
Revised: [Equation]

Original: Now we prove Theorem 1 from Section 3.5.
Revised: Now we prove Theorem 1 from Section 3.5.

Original: Consider a CNN with L hidden blocks, with each layer (cid:96) containing a convolutional layer with a ﬁlter of size √ s (cid:96) × √ s (cid:96) and a max pooling layer with pooling size √ s (cid:96) × √ s (cid:96) .
Revised: Consider a CNN with L hidden blocks, with each layer (cid:96) containing a convolutional layer with a ﬁlter of size √ s (cid:96) × √ s (cid:96) and a max pooling layer with pooling size √ s (cid:96) × √ s (cid:96) .

Original: The input to this CNN is x ∈ R d , unrolled from a √ d × √ d matrix.
Revised: The input to this CNN is x ∈ R d , unrolled from a √ d × √ d matrix.

Original: Similarly, we also unroll each convolutional ﬁlter into g (cid:96) ∈
Revised: Similarly, we also unroll each convolutional ﬁlter into g (cid:96) ∈

Original: R s (cid:96) , where g (cid:96) is indexed as ( g (cid:96) )
Revised: R s (cid:96) , where g (cid:96) is indexed as ( g (cid:96) )

Original: j for j ∈ J (cid:96) .
Revised: j for j ∈ J (cid:96) .

Original: Here, J (cid:96) corresponds to the index shift in matrix form from the top-left to bottom-right element.
Revised: Here, J (cid:96) corresponds to the index shift in matrix form from the top-left to bottom-right element.

Original: The output of the network is the probability vector p ∈ R C generated by the softmax function, where C is the total number of classes.
Revised: The output of the network is the probability vector p ∈ R C generated by the softmax function, where C is the total number of classes.

Original: Such a network can be represented as
Revised: Such a network can be represented as

Original: [Equation]
Revised: [Equation]

Original: where relu ( · ) and pool ( · ) indicate the ReLU and pooling operators, m (cid:96) ∈ R d (cid:96) is the output of the block (cid:96) ( m 0 = x ), and ( g (cid:96) ∗ m (cid:96) − 1 ) ∈ R d
Revised: where relu ( · ) and pool ( · ) indicate the ReLU and pooling operators, m (cid:96) ∈ R d (cid:96) is the output of the block (cid:96) ( m 0 = x ), and ( g (cid:96) ∗ m (cid:96) − 1 ) ∈ R d

Original: (cid:96) − 1 represents a convolutional operation on that block.
Revised: (cid:96) − 1 represents a convolutional operation on that block.

Original: Consider an input x and its decoy ˜ x , generated by swapping features in K .
Revised: Consider an input x and its decoy ˜ x , generated by swapping features in K .

Original: For each feature i ∈ K , we have the following theorem for the decoy-enhanced saliency score Z i :
Revised: For each feature i ∈ K , we have the following theorem for the decoy-enhanced saliency score Z i :

Original: Theorem 1.
Revised: Theorem 1.

Original: In the aforementioned setting, Z i is bounded by
Revised: In the aforementioned setting, Z i is bounded by

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The gradient of p c with respect to x can be written as follows, using the denominator layout notation of the derivative of a vector:
Revised: The gradient of p c with respect to x can be written as follows, using the denominator layout notation of the derivative of a vector:

Original: [Equation]
Revised: [Equation]

Original: 17) where
Revised: (20) where

Original: [Equation]
Revised: [Equation]

Original: (18) and (cid:40)
Revised: (21) and (cid:40)

Original: [Equation]
Revised: [Equation]

Original: if c
Revised: if c

Original: [Equation]
Revised: [Equation]

Original: otherwise .
Revised: otherwise .

Original: [Equation]
Revised: [Equation]

Original: Then we can write ∂ p c ∂ o as follows:
Revised: Then we can write ∂ p c ∂ o as follows:

Original: [Equation]
Revised: [Equation]

Original: 20) where ˆ P · c corresponds to the c -th column of ˆ P and ˆ P = diag ( p ) − pp T .
Revised: 23) where ˆ P · c corresponds to the c -th column of ˆ P and ˆ P = diag ( p ) − pp T .

Original: We then deﬁne B (cid:96) =
Revised: We then deﬁne B (cid:96) =

Original: ∂ m (cid:96)
Revised: ∂ m (cid:96)

Original: [Equation]
Revised: [Equation]

Original: m
Revised: m

Original: [Equation]
Revised: [Equation]

Original: −as B (cid:96) ∈ R d (cid:96) − 1 × d (cid:96) .
Revised: −as B (cid:96) ∈ R d (cid:96) − 1 × d (cid:96) .

Original: In the following, we compute B (cid:96) .
Revised: In the following, we compute B (cid:96) .

Original: First, we can have
Revised: First, we can have

Original: [Equation]
Revised: [Equation]

Original: if ˆ j − n ∈ J (cid:96) , and n = argmax n (cid:48) ∈ ˆ j + J (cid:96) ( g (cid:96) ∗ m (cid:96) − 1 ) n (cid:48) ,
Revised: if ˆ j − n ∈ J (cid:96) , and n = argmax n (cid:48) ∈ ˆ j + J (cid:96) ( g (cid:96) ∗ m (cid:96) − 1 ) n (cid:48) ,

Original: [Equation]
Revised: [Equation]

Original: otherwise , (21) where ˆ j represents the center of the pooling patch in relu ( g (cid:96) ∗ m (cid:96) − 1 ) , which results in ( m (cid:96) )
Revised: otherwise , (24) where ˆ j represents the center of the pooling patch in relu ( g (cid:96) ∗ m (cid:96) − 1 ) , which results in ( m (cid:96) )

Original: j .
Revised: j .

Original: Then we can compute
Revised: Then we can compute

Original: [Equation]
Revised: [Equation]

Original: otherwise , (22) where ( a (cid:96) )
Revised: otherwise , (25) where ( a (cid:96) )

Original: n = 1 { ( relu ( g (cid:96) ∗ m (cid:96) − 1 ) n ) ≥ 0 } .
Revised: n = 1 { ( relu ( g (cid:96) ∗ m (cid:96) − 1 ) n ) ≥ 0 } .

Original: If we change the activation function to either sigmoid or tanh, then ( a (cid:96) ) n in Eqn.
Revised: If we change the activation function to either sigmoid or tanh, then ( a (cid:96) ) n in Eqn.

Original: 22 will be replaced with the derivative of either function.
Revised: 25 will be replaced with the derivative of either function.

Original: For the sigmoid activation function σ ( x ) , the derivative is σ ( x )(1 − σ ( x )) , with a range of [0 , 14 ] .
Revised: For the sigmoid activation function σ ( x ) , the derivative is σ ( x )(1 − σ ( x )) , with a range of [0 , 14 ] .

Original: For the tanh activation function tanh ( x ) , the derivative is 1 − tanh ( x ) 2 , with a range of [0 , 1] .
Revised: For the tanh activation function tanh ( x ) , the derivative is 1 − tanh ( x ) 2 , with a range of [0 , 1] .

Original: We conclude that the derivative of both sigmoid and tanh are bounded by a value no larger than 1.
Revised: We conclude that the derivative of both sigmoid and tanh are bounded by a value no larger than 1.

Original: Combining Eqn. 21 with 22, we have
Revised: Combining Eqn. 24 with 25, we have

Original: [Equation]
Revised: [Equation]

Original: otherwise .
Revised: otherwise .

Original: [Equation]
Revised: [Equation]

Original: For simplicity, we rewrite the non-zero condition as n ∈ ˆ J (cid:96) .
Revised: For simplicity, we rewrite the non-zero condition as n ∈ ˆ J (cid:96) .

Original: Plugging B (cid:96) , (cid:96) = 1 , ..., L , into Eqn. 17, we can obtain the partial derivative (cid:79) x p c .
Revised: Plugging B (cid:96) , (cid:96) = 1 , ..., L , into Eqn. 20, we can obtain the partial derivative (cid:79) x p c .

Original: Further, we compute each element in the Hessian matrix H ij as follows:
Revised: Further, we compute each element in the Hessian matrix H ij as follows:

Original: [Equation]
Revised: [Equation]

Original: 24) and
Revised: 27) and

Original: [Equation]
Revised: [Equation]

Original: if c (cid:48)
Revised: if c (cid:48)

Original: =
Revised: =

Original: c ,
Revised: c ,

Original: [Equation]
Revised: [Equation]

Original: otherwise .
Revised: otherwise .

Original: Now we compute ((cid:81)
Revised: Now we compute (

Original: [Equation]
Revised: [Equation]

Original: B
Revised: B

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: ) jn L as
Revised: L as

Original: [Equation]
Revised: [Equation]

Original: (26) where
Revised: 29) where

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: [Equation] ] , (27) and where C n 2 = ( g 2 ) n 2 − 2 (cid:80) n 1 ∈ ˆ J 1 g n − 1 .
Revised: ] , (30) and where C n 2 = ( g 2 ) n 2 − 2 (cid:80) n 1 ∈ ˆ J 1 g n − 1 .

Original: Here, we redeﬁne ˆ J 1 as the set of indices such that ( B 1 ) jn 1 (cid:54) = 0 for n 1 ∈ ˆ J 1 .
Revised: Here, we redeﬁne ˆ J 1 as the set of indices such that ( B 1 ) jn 1 (cid:54) = 0 for n 1 ∈ ˆ J 1 .

Original: As such, we can compute ( B 1 ) j · (cid:81) L − 1 (cid:96) =
Revised: As such, we can compute ( B 1 ) j · (cid:81) L − 1 (cid:96) =

Original: 2 B (cid:96) as
Revised: 2 B (cid:96) as

Original: [Equation]
Revised: [Equation]

Original: B
Revised: B

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Plugging Eqn.
Revised: Plugging Eqn.

Original: 28 into Eqn.
Revised: 31 into Eqn.

Original: 26, we have
Revised: 29, we have

Original: [Equation]
Revised: [Equation]

Original: Plugging Eqn. 29 into Eqn.
Revised: Plugging Eqn. 32 into Eqn.

Original: 24, we have
Revised: 27, we have

Original: [Equation]
Revised: [Equation]

Original: where C j is a linear combination of g 1 , ..., g L , W L +1 , which is bounded.
Revised: where C j is a linear combination of g 1 , ..., g L , W L +1 , which is bounded.

Original: H ij equals the multiplication of two components—the summation of neurons activated by x and a gradient
Revised: H ij equals the multiplication of two components—the summation of neurons activated by x and a gradient

Original: ∂ ˆ P · c ∂ x i .
Revised: ∂ ˆ P · c ∂ x i .

Original: Given the total number of neurons in a CNN is a constant (denoted by C T ), we have 0 ≤ (cid:16)(cid:80) L(cid:96)
Revised: Given the total number of neurons in a CNN is a constant (denoted by C T ), we have 0 ≤ (cid:16)(cid:80) L(cid:96)

Original: =1 (cid:80) n (cid:96) ∈
Revised: =1 (cid:80) n (cid:96) ∈

Original: ˆ J (cid:96) ( a (cid:96) ) n (cid:96) (cid:17) ≤ C T .
Revised: ˆ J (cid:96) ( a (cid:96) ) n (cid:96) (cid:17) ≤ C T .

Original: Then, we have | ( H x ) ij | ≤ C T | C j ∂ ˆ P · c ∂ x i | .
Revised: Then, we have | ( H x ) ij | ≤ C T | C j ∂ ˆ P · c ∂ x i | .

Original: Since the derivatives of both sigmoid and tanh are no larger than 1 , this inequality also applies to the network with these two functions as the activation function.
Revised: Since the derivatives of both sigmoid and tanh are no larger than 1 , this inequality also applies to the network with these two functions as the activation function.

Original: Similarly, for the Hessian ( H ˜ x ) ij of a decoy ˜ x , we also have | ( H ˜ x ) ij ≤ C T | C j ∂ ˆ P · c ∂ ˜ x
Revised: Similarly, for the Hessian ( H ˜ x ) ij of a decoy ˜ x , we also have | ( H ˜ x ) ij ≤ C T | C j ∂ ˆ P · c ∂ ˜ x

Original: i | .
Revised: i | .

Original: Given the inequality of ( H ˜ x ) ij and ( H ˜ x ) ij , we can obtain that | ( H ˜ x ) ij − ( H x ) ij | ≤ 2 C T max ( | ˜ C j ∂ ˆ P · c ∂ ˜ x i | , | C j ∂ ˆ P · c ∂ x i ) | , where ∂ ˆ P · c ∂ x i is given by Eqn. 25.
Revised: Given the inequality of ( H ˜ x ) ij and ( H ˜ x ) ij , we can obtain that | ( H ˜ x ) ij − ( H x ) ij | ≤ 2 C T max ( | ˜ C j ∂ ˆ P · c ∂ ˜ x i | , | C j ∂ ˆ P · c ∂ x i ) | , where ∂ ˆ P · c ∂ x i is given by Eqn. 28.

Original: Recalling that P c is within [0 , 1] , the gradient ∂ P c ∂ x i is bounded by some Lipschitz constant (Szegedy et al., 2013), we can obtain that ∂ ˆ P · c ∂ x i is bounded by some constant.
Revised: Recalling that P c is within [0 , 1] , the gradient ∂ P c ∂ x i is bounded by some Lipschitz constant (Szegedy et al., 2013), we can obtain that ∂ ˆ P · c ∂ x i is bounded by some constant.

Original: Finally, we can derive that | ( H ˜ x ) ij − ( H x ) ij | ≤ C C , where C C represents the upper bound.
Revised: Finally, we can derive that | ( H ˜ x ) ij − ( H x ) ij | ≤ C C , where C C represents the upper bound.

Original: Now, we derive the decoy-enhanced saliency score Z i for x i , given a population of saliency scores ˜
Revised: Now, we derive the decoy-enhanced saliency score Z i for x i , given a population of saliency scores ˜

Original: E i = (cid:8) E (˜ x 1 ; F ) i , E (˜ x 2 ; F ) i , · · · , E (˜ x 2 n ; F ) i (cid:9) .
Revised: E i = (cid:8) E (˜ x 1 ; F ) i , E (˜ x 2 ; F ) i , · · · , E (˜ x 2 n ; F ) i (cid:9) .

Original: Let ˜ x
Revised: Let ˜ x

Original: + , ˜ x − ∈ (cid:8) ˜ x 1 , ˜ x 2 , · · · , ˜ x 2 n (cid:9) denotes the decoy which maximizes and minimize E (˜ x ; F ) i , respectively.
Revised: + , ˜ x − ∈ (cid:8) ˜ x 1 , ˜ x 2 , · · · , ˜ x 2 n (cid:9) denotes the decoy which maximizes and minimize E (˜ x ; F ) i , respectively.

Original: According to Lemma 1, the partial derivative (cid:79) ˜
Revised: According to Lemma 1, the partial derivative (cid:79) ˜

Original: x
Revised: x

Original: i p c has the following relationship
Revised: i p c has the following relationship

Original: [Equation]
Revised: [Equation]

Original: Then, we can derive
Revised: Then [Equation] we can derive

Original: [Equation]
Revised: [Equation]

Revised: −

Revised: [Equation]

Revised: [Equation]

Revised: k

Original: [Equation]
Revised: [Equation]

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: And
Revised: And

Original: Z i 
Revised: Z i =

Revised: [Equation]

Revised: x

Original: [Equation]
Revised: [Equation]

Original:  Combining Eqn. 34 with Eqn. 35, we have (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
Revised: i [Equation] Combining Eqn. 37 with Eqn. 38, we have (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original:  k 
Revised: [Equation] (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:88) k [Equation] (˜

Original: [Equation]
Revised: [Equation]

Original:  − ˜
Revised: k − ˜

Original: [Equation]
Revised: [Equation]

Original:  (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤C
Revised: − [Equation] [Equation] i (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤

Original: [Equation]
Revised: [Equation]

Original: Recall that (˜ x + k − ˜ x − k ) is bounded by a upper-bound, we can obtain that there exist a constant C 1 , such that C 1 ≥ 
Revised: Recall that (˜ x + k − ˜ x − k ) is bounded by a upper-bound, we can obtain that there exist a constant C 1 , such that C 1 ≥ 12 C C (cid:80) k ∈K (˜ x + k − ˜ x − k ) + 2 C .

Revised: Note that this upper bound is data speciﬁc, and we leave the exploration on its tightness as a part of future works.

Original: [Equation]
Revised: [Equation]

Original: A P ROOF OF P ROPOSITION
Revised: A P ROOF OF P ROPOSITION

Original: Proposition 1.
Revised: Proposition 1.

Original: Given an input x and its corresponding adversarial sample ˆ x , if both | x i − ˜ x i | ≤ C [-
-] δ i and (cid:12)(cid:12)(cid:12) ˆ x
Revised: Given an input x and its corresponding adversarial sample ˆ x , if both | x i − ˜ x i | ≤ C [-
-]2 δ i and (cid:12)(cid:12)(cid:12) ˆ x

Original: i −
Revised: i −

Original: ˜ˆ x i (cid:12)(cid:12)(cid:12) ≤ C 2 δ i can obtain (where C 2 > 0 is a bounded constant), then the following relation can be guaranteed.
Revised: ˜ˆ x i (cid:12)(cid:12)(cid:12) ≤ C 2 δ i can obtain where C 2 > 0 is a bounded constant and δ i = | E [Equation] x , F

Original: [Equation]
Revised: [Equation]

Revised: i − E ( x , F ) i | , then the following relation can be guaranteed.

Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Recall the goal of the attack against saliency maps is to subtly perturb an input sample such that the added perturbation does not change the output of the classiﬁer (Ghorbani et al., 2017) but force a saliency method to output a less meaningful saliency map ( i.e., highlighting features that are irrelevant to the classiﬁer prediction).
Revised: Recall the goal of the attack against saliency maps is to subtly perturb an input sample such that the added perturbation does not change the output of the classiﬁer (Ghorbani et al., 2017) but force a saliency method to output a less meaningful saliency map ( i.e., highlighting features that are irrelevant to the classiﬁer prediction).

Original: To achieve this goal, when generating an adversarial sample ˆ x from the given input x , an attacker needs to impose the following constraint (cid:107) ˆ x − x (cid:107) ∞ ≤ (cid:15) .
Revised: To achieve this goal, when generating an adversarial sample ˆ x from the given input x , an attacker needs to impose the following constraint (cid:107) ˆ x − x (cid:107) ∞ ≤ (cid:15) .

Original: Suppose we have an adversarial sample ˆ x satisﬁes this constraint.
Revised: Suppose we have an adversarial sample ˆ x satisﬁes this constraint.

Original: Then, we can assume (ˆ x − x ) i
Revised: Then, we can assume (ˆ x − x ) i

Original: = ˆ (cid:15) i , where | ˆ (cid:15) i | ≤ (cid:15) , for i
Revised: = ˆ (cid:15) i , where | ˆ (cid:15) i | ≤ (cid:15) , for i

Original: = 1 , 2 , ..., d .
Revised: = 1 , 2 , ..., d .

Original: In addition, we can compute saliency maps E (ˆ x , F ) and E ( x , F ) for ˆ x and x by using an existing saliency method.
Revised: In addition, we can compute saliency maps E (ˆ x , F ) and E ( x , F ) for ˆ x and x by using an existing saliency method.

Original: Given both saliency maps, we can further compute the difference between E ( ˆ x , F ) and E ( x , F ) as
Revised: Given both saliency maps, we can further compute the difference between E (ˆ x , F ) and E ( x , F ) as

Original: [Equation]
Revised: [Equation]

Original: Based on the Eqn. 2 in Section 3.3, when generating the decoys ˜ x , we ensure the classiﬁer’s predictions for those decoys are as same as that of the x .
Revised: Based on the Eqn. 2 in Section 3.3, when generating the decoys ˜ x , we ensure the classiﬁer’s predictions for those decoys are as same as that of the x .

Original: In this work, we achieve this by bounding the difference between the hidden representations of ˜ x and x .
Revised: In this work, we achieve this by bounding the difference between the hidden representations of ˜ x and x .

Original: As is discussed in Section A7, to preserve the same prediction c for ˜ x and x , one has to ensure | F c (˜ x ) − F c ( x ) | is bounded.
Revised: As is discussed in Section A7, to preserve the same prediction c for ˜ x and x , one has to ensure | F c (˜ x ) − F c ( x ) | is bounded.

Original: This implies the difference between ˜ x and x is bounded within (cid:15) .
Revised: This implies the difference between ˜ x and x is bounded within (cid:15) .

Original: Here, (cid:15) i represents the maximum difference between ˜ x
Revised: Here, (cid:15) i represents the maximum difference between ˜ x

Original: i and x i at the i th dimension.
Revised: i and x i at the i th dimension.

Original: As is mentioned above, the adversarial sample ˆ x does not change the classiﬁer’s prediction.
Revised: As is mentioned above, the adversarial sample ˆ x does not change the classiﬁer’s prediction.

Original: Therefore, we could imply ˆ (cid:15) i ≤ (cid:15) i , for i = 1 , 2 , ..., d .
Revised: Therefore, we could imply ˆ (cid:15) i ≤ (cid:15) i , for i = 1 , 2 , ..., d .

Original: Now, suppose we obtain a set of decoys for x and have their corresponding saliency maps, i.e., (cid:8)
Revised: Now, suppose we obtain a set of decoys for x and have their corresponding saliency maps, i.e., (cid:8)

Original: E (˜ x 1 ; F ) i , E (˜ x 2 ; F ) i , · · · , E (˜ x 2 n ; F ) i ) (cid:9) .
Revised: E ( ˜ x 1 ; F ) i , E ( ˜ x 2 ; F ) i , · · · , E ( ˜ x 2 n ; F ) i ) (cid:9) .

Original: Let ˜ x + ∈ (cid:8) ˜ x 1 , ˜ x 2 , · · · , ˜ x n (cid:9) denote the decoys which maximize E (˜ x ; F ) i and let ˜ x − denote the decoys which minimize E (˜ x ; F ) i .
Revised: Let ˜ x + ∈ (cid:8) ˜ x 1 , ˜ x 2 , · · · , ˜ x n (cid:9) denote the decoys which maximize E (˜ x ; F ) i and let ˜ x − denote the decoys which minimize E (˜ x ; F ) i .

Original: Similarly, we can also have the corresponding decoys
Revised: Similarly, we can also have the corresponding decoys

Original: ˜ˆ x − and ˜ˆ x − for the adversarial sample
Revised: ˜ˆ x − and ˜ˆ x − for the adversarial sample

Original: ˆ x as well as their corresponding saliency maps.
Revised: ˆ x as well as their corresponding saliency maps.

Original: With both the decoys and saliency maps for the input sample x and its adversarial sample ˆ x , we can compute the difference between ( Z ˆ x ) i and ( Z x ) i as 
Revised: With both the decoys and saliency maps for the input sample x and its adversarial sample ˆ x , we can compute the difference between ( Z ˆ x ) i and ( Z x ) i as [Equation]

Original: [Equation]
Revised: [Equation]

Original: To guarantee an improvement in robustness against the adversarial perturbation, we have to ensure that | ( Z ˆ x ) i − ( Z x ) i | − | ( E (ˆ x , F ) − E ( x , F ))
Revised: To guarantee an improvement in robustness against the adversarial perturbation, we have to ensure that | ( Z ˆ x ) i − ( Z x ) i | − | ( E (ˆ x , F ) − E ( x , F ))

Original: i | ≤ 0 , for i = 1 , 2 ..., d .
Revised: i | ≤ 0 , for i = 1 , 2 ..., d .

Original: That is,
Revised: That is,

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) [Equation] j = ( H 
Revised: As is discussed in Section A7, | ( H x ) ij | ≤

Original: [Equation]
Revised: [Equation]

Original: ij (cid:16) ( ˜ˆ x [Equation] j − ˜ˆ x
Revised: With this, we can have

Original: [Equation]
Revised: [Equation]

Original: j ) − (˜ x + j − ˜

Original: [Equation]

Original: [Equation]

Original: As is discussed in Section A7, | ( H x ) ij | ≤ C C .

Original: With this, we can have

Original: [Equation]
Revised: [Equation]

Original: By plugging Eqn. 41 into Eqn. 40, we conclude that as long as (cid:12)(cid:12)(cid:12) (˜ˆ x + j − ˜ˆ x 
Revised: By plugging Eqn. 44 into Eqn. 43, we conclude that as long as (cid:12)(cid:12)(cid:12) (˜ˆ x + j − ˜ˆ x − j ) − (˜ x + j − ˜ x − j )

Original: [Equation]
Revised: [Equation]

Original: ) −

Original: [Equation]

Original: − ˜ x −

Original: [Equation]

Original: [Equation]

Original: ≤ 1 C c d
Revised: ≤ 1 C c d

Original: [Equation]
Revised: [Equation]

Original: dj ={+=
+}
Revised: dj {+=
+}

Original: [Equation] H x ) ij ˆ (cid:15) j (cid:12)(cid:12)(cid:12) , our method could guarantee to improve the robustness against the adversarial perturbations.
Revised: ( H x ) ij ˆ (cid:15) j (cid:12)(cid:12)(cid:12) , our method could guarantee to improve the robustness against the adversarial perturbations.

Original: Let δ i =
Revised: Let δ i =

Original: | E (ˆ x , F ) i − E ( x , F ) i | .
Revised: | E (ˆ x , F ) i − E ( x , F ) i | .

Original: If we can ensure that | x i − ˜ x i | ≤ 1 4 C c d δ i and (cid:12) (cid:12) (cid:12) ˆ x i − ˜ˆ 
Revised: If we can ensure that | x i − ˜ x i | ≤ 1 4 C c d δ i and (cid:12)(cid:12)(cid:12) ˆ x i − ˜ˆ x i (cid:12)(cid:12)(cid:12) ≤ [Equation]

Original: x i (cid:12) (cid:12) (cid:12) ≤ 1 4 C c d δ i , we can have (cid:12) (cid:12) ˜ x
Revised: [Table] [Equation] c d δ i , we can have (cid:12)(cid:12) ˜ x

Original: + j − ˜ x − j (cid:12) (cid:12) ≤ 1 2 C c d δ i and (cid:12) (cid:12) (cid:12)
Revised: + j − ˜ x − j (cid:12)(cid:12) ≤ 1 2 C c d δ i and (cid:12)(cid:12)(cid:12)

Original: ˜ˆ x + j −
Revised: ˜ˆ x + j −

Original: ˜ˆ x − j (cid:12) (cid:12) (cid:12) ≤ 1 2 C c d δ i .
Revised: ˜ˆ x − j (cid:12)(cid:12)(cid:12) ≤ 1 2 C c d δ i .

Original: Thus, the aforementioned condition can be satisﬁed, i.e., (cid:12)(cid:12)(cid:12) (˜ˆ x + j −
Revised: Thus, the aforementioned condition can be satisﬁed, i.e., (cid:12)(cid:12)(cid:12) (˜ˆ x + j −

Original: ˜ˆ x − j ) − (˜ x + j − ˜ x − j ) (cid:12)(cid:12)(cid:12) ≤ 1 C c d δ i .
Revised: ˜ˆ x − j ) − (˜ x + j − ˜ x − j ) (cid:12)(cid:12)(cid:12) ≤ 1 C c d δ i .

Original: By setting
Revised: By setting

Original: C 2 = 1 4 C c d , we could obtain the robustness conditions in Proposition 1.
Revised: C 2 = 1 4 C c d , we could obtain the robustness conditions in Proposition 1.

Original: [Equation]
Revised: [Equation]

Original: A C OROLLARY
Revised: A C OROLLARY

Original: Consider a multilayer perceptron with L fully-connected hidden layers and a decoy swappable size
Revised: Consider a multilayer perceptron with L fully-connected hidden layers and a decoy swappable size

Original: K × 1 .
Revised: K × 1 .

Original: The input of this MLP is x ∈ R d .
Revised: The input of this MLP is x ∈ R d .

Original: For each hidden layer, we use the ReLU activation function.
Revised: For each hidden layer, we use the ReLU activation function.

Original: Similar to the CNN mentioned above, the output of this CNN is p ∈ R C .
Revised: Similar to the CNN mentioned above, the output of this CNN is p ∈ R C .

Original: The network can be represented as:
Revised: The network can be represented as:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where W (cid:96) ∈
Revised: where W (cid:96) ∈

Original: R
Revised: R

Original: d (cid:96) − 1 × d (cid:96) , for (cid:96) ∈ { 1 , · · · , L + 1 } represents the weights of the neural network, and b (cid:96) ∈ R d (cid:96) represents the biases, where d 0 = d and d L +1 = C .
Revised: d (cid:96) − 1 × d (cid:96) , for (cid:96) ∈ { 1 , · · · , L + 1 } represents the weights of the neural network, and b (cid:96) ∈ R d (cid:96) represents the biases, where d 0 = d and d L +1 = C .

Original: m (cid:96) ∈ R d (cid:96) is the output of each hidden layer, with m 0 = x and o ∈ R C is the logits.
Revised: m (cid:96) ∈ R d (cid:96) is the output of each hidden layer, with m 0 = x and o ∈ R C is the logits.

Original: The entry-wise softmax operator for target class c is deﬁned as p c =
Revised: The entry-wise softmax operator for target class c is deﬁned as p c =

Original: [Equation]
Revised: [Equation]

Original: Corollary 1.
Revised: Corollary 1.

Original: For the above MLP, Z i is also bounded by:
Revised: For the above MLP, Z i is also bounded by:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Proof.
Revised: Proof.

Original: Based on the proof of Theorem 1, the gradient of p c with respect to x can be written as follows
Revised: Based on the proof of Theorem 1, the gradient of p c with respect to x can be written as follows

Original: [Equation]
Revised: [Equation]

Original: where B (cid:96) =
Revised: where B (cid:96) =

Original: ∂ m (cid:96) ∂ m (cid:96) − 1 , B (cid:96) ∈ R d (cid:96) − 1 × d (cid:96) .
Revised: ∂ m (cid:96) ∂ m (cid:96) − 1 , B (cid:96) ∈ R d (cid:96) − 1 × d (cid:96) .

Original: ˆ P · c is also deﬁned as ˆ P = diag ( p ) − pp T .
Revised: ˆ P · c is also deﬁned as ˆ P = diag ( p ) − pp T .

Original: In the following, we compute B l .
Revised: In the following, we compute B l .

Original: First, we can compute ( B 1 ) ij , in which
Revised: First, we can compute ( B 1 ) ij , in which

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where ( a 1 ) j = 1 { ( W T 1 x + b 1 ) j ≥ 0 } .
Revised: where ( a 1 ) j = 1 { ( W T 1 x + b 1 ) j ≥ 0 } .

Original: Similar, we can also compute ( B (cid:96) ) ij , for (cid:96) = 2 , 3 , ..., L
Revised: Similar, we can also compute ( B (cid:96) ) ij , for (cid:96) = 2 , 3 , ..., L

Original: [Equation]
Revised: [Equation]

Original: (cid:96) ) j , (46) where ( a (cid:96) )
Revised: [Equation] where ( a (cid:96) )

Original: j = 1 { ( W T(cid:96)
Revised: j = 1 { ( W T(cid:96)

Original: x
Revised: x

Original: + b (cid:96) )
Revised: + b (cid:96) )

Original: j ≥ 0 } .
Revised: j ≥ 0 } .

Original: Then, we compute the each element in the Hessian matrix H ij .
Revised: Then, we compute the each element in the Hessian matrix H ij .

Original: Speciﬁcally, based on Eqn. 24, we have
Revised: Speciﬁcally, based on Eqn. 27, we have

Original: [Equation]
Revised: [Equation]

Original: where ∂ ˆ P · c ∂ x i is the same with Eqn.25.
Revised: where ∂ ˆ P · c ∂ x i is the same with Eqn.

Original: Now, we compute ( (cid:81) L(cid:96)
Revised: Now, we compute ( (cid:81) L(cid:96)

Original: =1 B l )
Revised: =1 B l )

Original: jn L as
Revised: jn L as

Original: [Equation]
Revised: [Equation]

Original: 48) where ( B 1 ) j · = [( W 1 ) j 1 ( a 1 ) 1 , ( W 1 ) j 2 ( a 1 ) 2 , ..., ( W 1 ) jd 1 ( a 1 ) d 1 ] and ( B1 ) j · B 2 = [( a 2 )
Revised: (51) where ( B 1 ) j · = [( W 1 ) j 1 ( a 1 ) 1 , ( W 1 ) j 2 ( a 1 ) 2 , ..., ( W 1 ) jd 1 ( a 1 ) d 1 ] and ( B

Original: [Equation]
Revised: [Equation]

Original:  where ( C 2 ) n 2 ,n 1 = ( W 2 ) n 1 ,n 2 ( W 1 ) j,n 1 .
Revised: 52) where ( C 2 ) n 2 ,n 1 = ( W 2 ) n 1 ,n 2 ( W 1 ) j,n 1 .

Original: For simplicity, we can rewrite (cid:80) d 1 n 1 =1 ( C 2 ) n 2 ,n 1 ( a 1 ) n 

Revised: For simplicity, we can rewrite (cid:80) d 1 n 1 =1 ( C 2 ) n 2 ,n 1 ( a 1 ) n

Original: =
Revised: =

Original: ( C 2 ) n 2 (cid:80) d 1 n 1 =1 ( a 1 ) n 1 .
Revised: ( C 2 ) n 2 (cid:80) d 1 n 1 =1 ( a 1 ) n 1 .

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: As such, we can compute ( B 1 )
Revised: As such, we can compute ( B 1 )

Original: [Equation] [Equation] B
Revised: j

Original: [Equation]
Revised: [Equation]

Original:  as 
Revised: L − 1 (cid:96) = 2 B (cid:96) as ( B

Original: [Equation]
Revised: [Equation]

Original: B [Equation]
Revised: j ·

Original: [Equation]
Revised: [Equation]

Original: Plugging Eqn.
Revised: Plugging Eqn.

Original: 51 into Eqn.
Revised: 54 into Eqn.

Original: 20, we have
Revised: 23, we have

Original: [Equation]
Revised: [Equation]

Original: [Equation] Finally, we can obtain that [Equation] (cid:96)
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: d (cid:96) (cid:88) n (cid:96) [Equation] [Equation] n (cid:96)
Revised: we can obtain that

Original: [Equation]
Revised: [Equation]

Original: (cid:33) [Equation] (53) where C j is a linear combination of the elements in ( W 1 ) j · , W 2 , ..., W L +1 .
Revised: where C j is a linear combination of the elements in ( W 1 ) j · , W 2 , ..., W L +1 .

Original: Note that the Hessian derived from the MLP has a similar form with the Hessian derived from the
Revised: Note that the Hessian derived from the MLP has a similar form with the Hessian derived from the

Original: CNN in Eqn. 30, i.e., the summation of neurons activated by x multiplying the gradient.
Revised: CNN in Eqn. 33, i.e., the summation of neurons activated by x multiplying the gradient.

Original: Here, the summation of neurons activated by x is again bounded by the total number of neurons in the network.
Revised: Here, the summation of neurons activated by x is again bounded by the total number of neurons in the network.

Original: The gradient ∂ ˆ P · c ∂ x i is bounded by a Lipschitz constant.
Revised: The gradient ∂ ˆ P · c ∂ x i is bounded by a Lipschitz constant.

Original: Similarly, we also have the following inequality for ( H ˜ x ) ij and ( H x ) ij , i.e., | ( H ˜ x ) ij − ( H x ) ij | ≤ C M .
Revised: Similarly, we also have the following inequality for ( H ˜ x ) ij and ( H x ) ij , i.e., | ( H ˜ x ) ij − ( H x ) ij | ≤ C M .

Original: Similar to Theorem 1,let ˜ x + , ˜ x − ∈ (cid:8) ˜ x 1 , ˜ x 2 , · · · , ˜ x 2 n (cid:9) denotes the decoy which maximizes and minimize E (˜ x ; F ) i , respectively.
Revised: Similar to Theorem 1,let ˜ x + , ˜ x − ∈ (cid:8) ˜ x 1 , ˜ x 2 , · · · , ˜ x 2 n (cid:9) denotes the decoy which maximizes and minimize E (˜ x ; F ) i , respectively.

Original: Based on Eqn. 31 to Eqn. 36, we have
Revised: Based on Eqn. 34 to Eqn. 39, we have

Original: [Equation]
Revised: [Equation]

Original: C 2 ≥ 12 C M (cid:80) k ∈K (˜ x + k − ˜ x − k )
Revised: C 2 ≥ 12 C M (cid:80) k ∈K (˜ x + k − ˜ x − k )

Original: + 2 C .
Revised: + 2 C .

Original: Slightly different for CNN, MLP sometimes is used to process the input that does not have a strong local dependency.
Revised: Slightly different for CNN, MLP sometimes is used to process the input that does not have a strong local dependency.

Original: In this case, we can set the swappable path size K = 1 .
Revised: In this case, we can set the swappable path size K = 1 .

Original: Then, Eqn. 54 can reformulated as (cid:12)(cid:12)
Revised: Then, Eqn. 57 can reformulated as (cid:12)(cid:12)

Original: Z i − 12 (cid:12)(cid:12) (
Revised: Z i − 12 (cid:12)(cid:12) (

Original: ˜ x + i − ˜ x − i )
Revised: ˜ x + i − ˜ x − i )

Original: ( H x ) i,i (cid:12)(cid:12)(cid:12)(cid:12)
Revised: ( H x ) i,i (cid:12)(cid:12)(cid:12)(cid:12)

Original: ≤ C 2 .
Revised: ≤ C 2 .

Original: As we can observe from this equation, our proposed saliency score is still able to compensate for the gradient saturation problem. 
Revised: As we can observe from this equation, our proposed saliency score is still able to compensate for the gradient saturation problem. [Equation]

Original: (cid:3)
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: A D ATASETS AND EXPERIMENT SETUP
Revised: A D ATASETS AND EXPERIMENT SETUP

Original: In this section, we introduce the datasets used in our experiments and the neural network trained on each dataset, followed by our choices of hyper-parameters when explaining each model.
Revised: In this section, we introduce the datasets used in our experiments and the neural network trained on each dataset, followed by our choices of hyper-parameters when explaining each model.

Original: ImageNet.
Revised: ImageNet.

Original: We randomly select a subset of samples from the ImageNet validation set, which can be downloaded from the following link: http://www.image-net.org/ .
Revised: We randomly select a subset of samples from the ImageNet validation set, which can be downloaded from the following link: http://www.image-net.org/ .

Original: We adopt the most widely used preprocessing method for the selected images.
Revised: We adopt the most widely used preprocessing method for the selected images.

Original: Speciﬁcally, for each image, we resized it to 227 × 227 , converted it to BGR format, and subtract the mean value of each channel [103.939, 116.779, 123.68] from the image.
Revised: Speciﬁcally, for each image, we resized it to 227 × 227 , converted it to BGR format, and subtract the mean value of each channel [103.939, 116.779, 123.68] from the image.

Original: Rather than training our own networks, we downloaded a pretrained VGG16 model, AlexNet model, and ResNet_v1_50 model from the following link: https://github.com/tensorflow/models/tree/master/research/ slim and http://www.cs.toronto.edu/~guerzhoy/tf_alexnet/ .
Revised: Rather than training our own networks, we downloaded a pretrained VGG16 model, AlexNet model, and ResNet_v1_50 model from the following link: https://github.com/tensorflow/models/tree/master/research/ slim and http://www.cs.toronto.edu/~guerzhoy/tf_alexnet/ .

Original: We applied our proposed method to explain the predictions of these networks on the selected samples.
Revised: We applied our proposed method to explain the predictions of these networks on the selected samples.

Original: SST.
Revised: SST.

Original: We downloaded the Stanford Sentiment Treebank (SST1) from the following link: https: //github.com/harvardnlp/sent-conv-torch/tree/master/data .
Revised: We downloaded the Stanford Sentiment Treebank (SST1) from the following link: https: //github.com/harvardnlp/sent-conv-torch/tree/master/data .

Original: The data is spited into a training set of 76 , 961 samples and a testing set of 1 , 821 samples.
Revised: The data is spited into a training set of 76 , 961 samples and a testing set of 1 , 821 samples.

Original: We used a pretrained glove embedding to represent each word in the sentences (sample).
Revised: We used a pretrained glove embedding to represent each word in the sentences (sample).

Original: The embedding of each word is a vector of 100 dimensions.
Revised: The embedding of each word is a vector of 100 dimensions.

Original: The pretrained embedding matrix can be downloaded from the following link: http://nlp.stanford.edu/data/wordvecs/glove.6B.zip .
Revised: The pretrained embedding matrix can be downloaded from the following link: http://nlp.stanford.edu/data/wordvecs/glove.6B.zip .

Original: We trained a two-layer CNN with the embeddings as inputs.
Revised: We trained a two-layer CNN with the embeddings as inputs.

Original: The model achieves about 80% accuracy on the testing set.
Revised: The model achieves about 80% accuracy on the testing set.

Original: The preprocessed testing data and the pretrained model can be downloaded from the following link: https://tinyurl.com/y9noqj6l .
Revised: The preprocessed testing data and the pretrained model can be downloaded from the following link: https://tinyurl.com/y9noqj6l .

Original: We run our explanation method on the pretrained model with the testing samples.
Revised: We run our explanation method on the pretrained model with the testing samples.

Original: Network intrusion detection (IDS).
Revised: Network intrusion detection (IDS).

Original: We use a subset of CSE-CIC-IDS2018 dataset (Sharafaldin et al., 2018; for Cybersecurity, 2018), a network intrusion dataset contains the benign network trafﬁc traces and malicious traces generated by three types of attacks: Denial of Service (DoS)-Hulk, SSHBruteForce, and Inﬁltration.
Revised: We use a subset of CSE-CIC-IDS2018 dataset (Sharafaldin et al., 2018; for Cybersecurity, 2018), a network intrusion dataset contains the benign network trafﬁc traces and malicious traces generated by three types of attacks: Denial of Service (DoS)-Hulk, SSHBruteForce, and Inﬁltration.

Original: The training set contains 88 , 661 samples and the testing set has 22 ,samples.
Revised: The training set contains 88 , 661 samples and the testing set has 22 ,samples.

Original: Each sample is represented as a vector of 83 dimensions, where each feature represents the statistics of network trafﬁc ﬂows ( e.g., Number of packets, Number of bytes, Length of packets, etc).
Revised: Each sample is represented as a vector of 83 dimensions, where each feature represents the statistics of network trafﬁc ﬂows ( e.g., Number of packets, Number of bytes, Length of packets, etc).

Original:  The features are normalized within [0, 1] by using the scikit-learn MinMaxScaler function.
Revised: [Figure] The features are normalized within [0, 1] by using the scikit-learn MinMaxScaler function.

Original: We trained a two-layer MLP to classify whether an input is a benign trafﬁc or an attack (intrusion).
Revised: We trained a two-layer MLP to classify whether an input is a benign trafﬁc or an attack (intrusion).

Original: The model reaches 99% accuracy on the testing set.
Revised: The model reaches 99% accuracy on the testing set.

Original: After training the model, we randomly sampled a subset of 2 , 000 testing samples and used our method to derive explanations from the model predictions of samples in this subset.
Revised: After training the model, we randomly sampled a subset of 2 , 000 testing samples and used our method to derive explanations from the model predictions of samples in this subset.

Original: The dataset, model, and the descriptions of each feature can be found in https://tinyurl.com/y9noqj6l .
Revised: The dataset, model, and the descriptions of each feature can be found in https://tinyurl.com/y9noqj6l .

Original: Hyper-parameter choices.
Revised: Hyper-parameter choices.

Original: The hyper-parameter choices of the proposed method on three datasets are shown in Table A1.
Revised: The hyper-parameter choices of the proposed method on three datasets are shown in Table A1.

Original: In the table, (cid:96) is the index of the layer within the target model that is selected 
Revised: In the table, (cid:96) is the index of the layer within the target model that is selected to generate the decoy images.

Revised: The Lagrange multiplier λ controls the weight of (cid:107) F (cid:96) (˜

Original: [Figure]
Revised: [Table]

Original: to generate the decoy images. The hyper-parameter λ controls the weight of (cid:107) F (cid:96) (˜ x ) − F (cid:96) ( x ) (cid:107) ∞ .
Revised: ) − F (cid:96) [Table] ) (cid:107) ∞ .

Original: The patch_size and stride control the size and the stride step of each decoy patch.
Revised: The patch_size and stride control the size and the stride step of each decoy patch.

Original: τ is introduced by
Revised: τ is introduced by

Original: Eqn. 8 in Section A6.
Revised: Eqn. 3 in Section A6.

Original: Note that we set the swappable patch size of SST and IDS data as 1 , because their features may not have a strong local correlation.
Revised: Note that we set the swappable patch size of SST and IDS data as 1 , because their features may not have a strong local correlation.

Original: It should also be noted that we selected the swappable patch size of ImageNet data as the widely used convolutional kernel size 3 and stride size 1 .
Revised: It should also be noted that we selected the swappable patch size of ImageNet data as the widely used convolutional kernel size 3 and stride size 1 .

Revised: We set the number of patches (masks) in each decoy m as 100 for ImageNet, 1 for SST and

Revised: IDS.

Revised: When generating adversarial attack images, we applied the code released by the corresponding work (Ghorbani et al., 2017) and followed their default setup in our implementation.

Original: A S ANITY CHECK FOR DECOY - ENHANCED SALIENCY MAPS
Revised: A S ANITY CHECK FOR DECOY - ENHANCED SALIENCY MAPS

Original: As suggested by Adebayo et al.
Revised: As suggested by Adebayo et al.

Original: (2018), any valid saliency methods should pass the sanity check in the sense that the saliency method should be dependent on the learned parameters of the predictive model, instead of edge or other generic feature detectors.
Revised: (2018), any valid saliency methods should pass the sanity check in the sense that the saliency method should be dependent on the learned parameters of the predictive model, instead of edge or other generic feature detectors.

Original: We performed the model parameter randomization test (Adebayo et al., 2018) on the ImageNet dataset by comparing the output of the proposed saliency method on a pretrained VGG16 network with the output of the proposed saliency method on a weight-randomized VGG16 network.
Revised: We performed the model parameter randomization test (Adebayo et al., 2018) on the ImageNet dataset by comparing the output of the proposed saliency method on a pretrained VGG16 network with the output of the proposed saliency method on a weight-randomized VGG16 network.

Original: If the proposed saliency method indeed depends on the learned parameters of the model, it is expected that the outputs between the two cases differ substantially.
Revised: If the proposed saliency method indeed depends on the learned parameters of the model, it is expected that the outputs between the two cases differ substantially.

Original: Following the cascading randomization strategy (Adebayo et al., 2018), the weights of pretrained
Revised: Following the cascading randomization strategy (Adebayo et al., 2018), the weights of pretrained

Original: VGG16 network are randomized from the top to bottom layers in a cascading fashion.
Revised: VGG16 network are randomized from the top to bottom layers in a cascading fashion.

Original: This cascading randomization procedure is designed to destroy the learned weights successively.
Revised: This cascading randomization procedure is designed to destroy the learned weights successively.

Original: As illustrated in Figure A4, the cascading randomization destroys the decoy-enhanced saliency maps combined with three existing saliency methods, qualitatively.
Revised: As illustrated in Fig. A5, the cascading randomization destroys the decoy-enhanced saliency maps combined with three existing saliency methods, qualitatively.

Original: The conclusion is also supported by quantitative comparison measured by the structural similarity index (SSIM), shown in Figure A5.
Revised: The conclusion is also supported by quantitative comparison measured by the structural similarity index (SSIM), shown in Fig. A6.

Original: A A PPLICABILITY TO OTHER CNN ARCHITECTURES
Revised: A A PPLICABILITY TO OTHER CNN ARCHITECTURES

Original: In addition to the VGG16 model, we generated saliency maps for AlexNet (Krizhevsky et al., 2012) and ResNet (He et al., 2016) trained from the ImageNet dataset.
Revised: In addition to the VGG16 model, we generated saliency maps for AlexNet (Krizhevsky et al., 2012) and ResNet (He et al., 2016) trained from the ImageNet dataset.

Original: We visualize their saliency maps in Figure A6.
Revised: We visualize their saliency maps in Fig. A7.

Original: We observe that our method consistently outperforms the baseline methods, both 
Revised: We observe that our method consistently outperforms the baseline methods, both quantitatively and qualitatively.

Revised: Together with the results in Section 4, these results suggest that we

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Table]
Revised: [Figure] [Figure]

Original: quantitatively and qualitatively.

Original: Together with the results in Section 4.2.1, these results suggest that we can apply our decoy-enhanced saliency methods to various feed-forward network architectures and expect consistent performance.

Original: A P ERFORMANCES ON THE NETWORK INTRUSION DATASET .
Revised: A P ERFORMANCES ON THE NETWORK INTRUSION DATASET .

Original: Rather than visualizing the saliency scores through heatmaps, we apply the following to compare the saliency scores obtained by different methods qualitatively.
Revised: Rather than visualizing the saliency scores through heatmaps, we apply the following to compare the saliency scores obtained by different methods qualitatively.

Original: We ranked the features based on their saliency scores and compared the ranking obtained by the existing methods with that obtained by our decoy-enhanced method.
Revised: We ranked the features based on their saliency scores and compared the ranking obtained by the existing methods with that obtained by our decoy-enhanced method.

Original: “Minimum size of packet in forward direction”, “Minimum length of a packet”, “Minimum time between two packets sent in the forward direction” are ranked higher by our methods than the baselines.
Revised: “Minimum size of packet in forward direction”, “Minimum length of a packet”, “Minimum time between two packets sent in the forward direction” are ranked higher by our methods than the baselines.

Original: These features could capture the differences between benign and malicious trafﬁcs.
Revised: These features could capture the differences between benign and malicious trafﬁcs.

Original: This is because attackers usually tend to rapidly send small packages to discover the backdoors in the victim network system, while the benign users may send much larger packages with a longer interval between two packages.
Revised: This is because attackers usually tend to rapidly send small packages to discover the backdoors in the victim network system, while the benign users may send much larger packages with a longer interval between two packages.

Original: On the contrary, features that are not that useful for intrusion detection ( e.g., timestamp, Download and upload ratio) are wrongly pinpointed by the existing method.
Revised: On the contrary, features that are not that useful for intrusion detection ( e.g., timestamp, Download and upload ratio) are wrongly pinpointed by the existing method.

Original: However, our methods correctly assign lower importance to these features.
Revised: However, our methods correctly assign lower importance to these features.

Original: Table A2 shows the ﬁdelity comparisons of different saliency methods.
Revised: Table A2 shows the ﬁdelity comparisons of different saliency methods.

Original: We can observe that our decoys-enhanced methods outperform the original saliency methods.
Revised: We can observe that our decoys-enhanced methods outperform the original saliency methods.

Original: These results show that our method could pinpoint more accurate features and achieve a higher ﬁdelity than baselines.
Revised: These results show that our method could pinpoint more accurate features and achieve a higher ﬁdelity than baselines.

Original: We also evaluated the constant-decoy baseline, i.e., generating decoys by replacing each feature withits mean value.
Revised: We also evaluated three alternatives used in Section 4: constant perturbation with range aggregation, noise perturbation with range aggregation, decoys generation with mean aggregation.

Original: The results in Table A2 shows that constant-decoy cannot achieve comparable performance with our method, and even perform worse than the existing methods in some cases.
Revised: The results in Table A2 are consistant with those in Fig. 2 and Fig. 3, i.e., our method outperforms these baselines.

Original: In summary, the results on this dataset are aligned with those on the other datasets, conﬁrming our method’s applicability to multilayer perceptrons.
Revised: In summary, the results on this dataset are aligned with those on the other datasets, conﬁrming our method’s applicability to multilayer perceptrons.

Original: A D ECOYS ON THE V ARIANTS OF I NTEGRATED G RADIENT .
Revised: A D ECOYS ON O THER B ASELINES .

Original: In Section 4, we applied the original integrated gradient.
Revised: In Section 4, we evaluated our methods on three state-of-the-art saliency methods.

Original: Recent research (Sturmfels et al., 2020) suggests some variants that improve the performance of the integrated gradient.
Revised: Recent research (Sturmfels et al., 2020; Hooker et al., 2019) suggests some variants that improve the perfor- [Table] [Figure]

Original: Here, by using
Revised: mance of these baseline methods.

Original:  ImageNet data, we evaluate whether our decoy method could further improve the variants of the [Table]
Revised: Here, by using ImageNet data, we evaluate whether our decoy method could further improve these variants and another widely used saliency method.

Original: integrated gradient. Speciﬁcally, we consider two variants mentioned in Sturmfels et al.
Revised: Speciﬁcally, we consider two variants of the integrated gradient: integrated gradient with uniform baseline (Sturmfels et al., 2020) and Expected Gradient (Sturmfels et al., 2020); two variants of the SmoothGrad:

Original: (2020): integrated gradientwith uniform baseline and Expected Gradient.
Revised: VarGrad (Hooker et al., 2019) and Smoothgrad with range aggregation; and one existing saliency method: Grad-CAM (Selvaraju et al., 2016). For the variants of the integrated gradient and SmoothGrad, we kept the number of samples the same as the original version and used the default number suggested by existing works - 25 (See https://github.com/PAIR-code/saliency ). We will investigate whether increasing the sample numbers improve the existing saliency methods’ ﬁdelity and robustness in future work. Fig.

Original: Table A3 shows the  comparison ofexplanation ﬁdelity of each method with/without decoys.
Revised: A8 and Fig. A9 shows the qualitatively and quantitatively comparison of each method with/without decoys.

Original: As is depicted in the table, ourdecoy method consistently improves the ﬁdelity of the different variants of the integrated gradient (Column 2 vs. Column 3).
Revised: As is depicted in Fig. A8, our method helps knock off the noises and improve the visual quality of the saliency maps. Fig.

Original:  Together with the results on the vanilla gradient and SmoothGrad, they demonstrate the generalizability of our technique to different saliency methods. 
Revised: A9 further demonstrates the advantage of our method in explanation ﬁdelity. Together with the results in Section 4, they demonstrate the generalizability of our technique to different saliency methods. Note that our method only imposes a minor improvement on Grad-CAM both qualitatively and quantitatively.

Original: As part of future work we will  investigate the effectiveness of adding our technique to morevariants of our selected base saliency methods mentioned in Sturmfels et al.
Revised: As part of future work, we will explore how to customize our method for Grad-CAM and investigate the effectiveness of applying our technique to more saliency methods.

Original: A R UNTIME OF D ECOY G ENERATIONS
Revised: A R UNTIME OF D ECOY G ENERATIONS

Original: To evaluate the computational cost of our decoy generations, we carried out the run time comparison between optimizing one decoy and calculating three types of saliency methods, repeated 500 times with respect to different patch masks.
Revised: To evaluate the computational cost of our decoy generations, we carried out the run time comparison between optimizing one decoy and calculating three types of saliency methods, repeated 500 times with respect to different patch masks.

Original: As illustrated in Figure A7a, on average, optimizing one decoy is 62 .
Revised: As illustrated in Fig. A10a, on average, optimizing one decoy is 62 .

Original: 3% faster than the fastest vanilla gradient-based saliency method.
Revised: 3% faster than the fastest vanilla gradient-based saliency method.

Original: For other methods, the optimization is even less expensive, in a relative sense.
Revised: For other methods, the optimization is even less expensive, in a relative sense.

Original: As is mentioned in Section 3.4, our decoy generation, and the saliency map computation can be run parallelly in a batch mode.
Revised: As is mentioned in Section 3.4, our decoy generation, and the saliency map computation can be run parallelly in a batch mode.

Original: In the optimal case, where we have enough resources to compute each decoy saliency map ( i.e., E (˜ x i ; F ) ) parallelly, the overall runtime of generating one decoy-enhanced saliency map is the total time of generating one decoy and computing one saliency map using the existing methods.
Revised: In the optimal case, where we have enough resources to compute each decoy saliency map ( i.e., E (˜ x i ; F ) ) parallelly, the overall runtime of generating one decoy-enhanced saliency map is the total time of generating one decoy and computing one saliency map using the existing methods.

Original: Figure A7b shows the comparisons between the optimal decoy-enhanced saliency map generation time and the original saliency map generation time across three saliency methods. 
Revised: Fig. A10b shows the comparisons between the optimal decoy-enhanced saliency map generation time and the original saliency map generation time across three saliency methods. [Figure] [Figure]

Original: As we can observe from the ﬁgure, our method introduces negligible computational overhead over the existing methods.
Revised: As we can observe from the ﬁgure, our method introduces negligible computational overhead over the existing methods.

Original: Taking a step back, when the users have limited resources for running the decoy-enhanced saliencies in a fully parallel fashion.
Revised: Taking a step back, when the users have limited resources for running the decoy-enhanced saliencies in a fully parallel fashion.

Original: Since the decoy generation is faster than the existing methods, in most cases, partial parallel computing still does not signiﬁcantly increase the computational overhead.
Revised: As is shown in Fig. 2, our method is not sensitive to the variations in the number of generated decoys. More speciﬁcally, Fig.

Original: In our experiment, we used a server with 32 CPU cores and 8 Nvidia
Revised: 2 shows that we can obtain a decent performance by only solving 16 decoys on the ImageNet dataset.

Original: Ti GPUs.
Revised: In the worst-case scenario where a user cannot run decoy generation in parallel, our method’s computational overhead over the baselines is 24X for the Gradient approach, 16 X for the Integrated gradient method, and X for the SmoothGrad method. In most cases, where users could afford partial parallel computing, this overhead will be decreased linearly with the available computational resources.

Original: Our method introduces about 2X ∼ 5X overheard to the existing saliency methods.
Revised: For example, if a user has 4 GPUs, the overhead will drop to 6X for Gradient, 4X for Integrated gradient, and 4X for SmoothGrad.

Original: We argue that for an ensemble method, this overhead is acceptable.
Revised: We argue that for an ensemble method, this overhead is acceptable.

Revised: Besides, saliency generation is much lighter weight than training deep neural networks.

Revised: Even with 4X ∼ 6X overheads, the time of computing saliency maps is still much less than network training.

Original: In addition, our method can be exponentially faster on more powerful machines, which escalates the practicality of our method.[Figure] [Figure]
Revised: In addition, our method can be even faster on more powerful machines, which escalates the practicality of our method.

Original: A H YPER - PARAMETER SENSITIVITY
Revised: A H YPER - PARAMETER SENSITIVITY

Original: We also conduct experiments on the VGG16 to understand the impact of hyper-parameter choices on the performance of our optimization-based decoy generation method.
Revised: We also conduct experiments on the VGG16 to understand the impact of hyper-parameter choices on the performance of our optimization-based decoy generation method.

Original: Speciﬁcally, we focus on the choice of three hyper-parameters: network layer (cid:96) , initial Lagrange multiplier λ [Figure] and patch size.
Revised: Speciﬁcally, we focus on the choice of three hyper-parameters: network layer (cid:96) , initial Lagrange multiplier λ , and patch size.

Original: Accordingly [Figure] we ﬁrst varied the value of (cid:96) for VGG16 and compared the differences of the generated decoy saliencies from the three aforementioned saliency methods.
Revised: Accordingly, we ﬁrst varied the value of (cid:96) for VGG16 and compared the differences of the generated decoy saliencies from the three aforementioned saliency methods.

Original: In particular, we set it to range from the ﬁrst convolutional layer to the last pooling layer and demonstrate the generated decoy saliencies in Figure A13.
Revised: In particular, we set it to range from the ﬁrst convolutional layer to the last pooling layer and demonstrate the generated decoy saliencies in Fig. A19.

Original: Note that according to our design, only the convolutional layers and the pooling layers can be used to generate decoy images.
Revised: Note that according to our design, only the convolutional layers and the pooling layers can be used to generate decoy images.

Original: For each saliency method, Figure A13 demonstrates that the decoy saliencies generated from different layers for the same image are of similar qualities.
Revised: For each saliency method, Fig. A19 demonstrates that the decoy saliencies generated from different layers for the same image are of similar qualities.

Original: Figure A13 also shows the mean and standard derivation of the SF scores for each saliency method.
Revised: Fig. A also shows the mean and standard derivation of the SF scores for each saliency method.

Original: These quantitative results also support the conclusion that our approach is not sensitive to the layer. 
Revised: These quantitative results also support the conclusion that our approach is not sensitive to the layer. This

Original: This is likely because, as previous research has shown (Chan et al., 2015; Saxe et al., 2011), the ﬁnal classiﬁcation results of a DNN are not highly related to the hidden representations.
Revised: [Figure] [Figure] is likely because, as previous research has shown (Chan et al., 2015; Saxe et al., 2011), the ﬁnal classiﬁcation results of a DNN are not highly related to the hidden representations.

Original: As a result, generating decoy saliencies for the same sample with the same label from different layers should yield similar results.
Revised: As a result, generating decoy saliencies for the same sample with the same label from different layers should yield similar results

Original:  We also varied the initial Lagrange multiplier λ to be (cid:8) 10 1 , 10 2 , 10 3 , 10 4 , 10 5 (cid:9) and compared the differences of the generated decoy saliencies.
Revised: [Figure] We also varied the initial Lagrange multiplier λ to be (cid:8) 10 1 , 10 2 , 10 3 , 10 4 , 10 5 (cid:9) and compared the differences of the generated decoy saliencies.

Original: Figure A8 depicts the quantitative and qualitative comparison results.
Revised: Fig. A11 depicts the quantitative and qualitative comparison results.

Original: As shown in the ﬁgure, the different choices of initial λ all produce similar saliency maps, indicating a negligible inﬂuence upon our method.
Revised: As shown in the ﬁgure, the different choices of initial λ all produce similar saliency maps, indicating a negligible inﬂuence upon our method.

Original: Finally, we  increased the patch size to be { 3 , 5 , 7 , 9 , 11 } and showed the generated decoy saliencies in Figure A9.
Revised: Then, we ﬁxed m and increased the patch size to be { 3 , 5 , 7 , 9 , 11 } and showed the generated decoy saliencies in Fig. A12.

Original: The results show that varying the patch size within a certain range only imposes a negligible inﬂuence upon our method.
Revised: The results show that varying the patch size within a certain range only imposes a negligible inﬂuence upon our method.

Original: Note that, as is mentioned in Section 3.4, we generate decoy masks  by sliding the swappable patch across a given input.
Revised: Recall that in Section 3.4, we mention that decoy masks are generated by sliding the swappable patch across a given input.

Original: This means that given a  constant stride 1 , the number of decoys n is unique to the patch size K .
Revised: With a given constant stride 1 , the number of sliding windows is equal to ( √ d − P + 1) 2 .

Original: Speciﬁcally, n is inversely proportional to K .
Revised: In our implementation, to enable batch computing, we introduce m , which controls the number of sliding windows in each decoy. Then, the number of decoys is 2 (cid:106) ( √ d − P + 1) 2 /m (cid:107) . Fig. A12 shows the results of ﬁxing m as 100 and varying P .

Original: As such, we do not explicitly set n as our hyper-parameter and control it by adjusting K .
Revised: In Fig. 2(C), we substantially varied both P and m and showed that our method is insensitive to the variations in the number of decoys n . Note that the box bars with the same color in Fig. 2(C) are drawn by ﬁxing P and varying m .

Original: Given that our method is not sensitive to the choice of K , it also not sensitive to the variation of n .
Revised: Their slight difference indicates the robustness of our method in the variations of m .

Original: The results in Figure A13, A8, and A9 indicate we can expect to obtain stable decoy saliencies when the hyper-parameters are subtly varied.
Revised: The results in Fig. 2(C), A19, A11, and A12 indicate we can expect to obtain stable decoy saliencies when the hyper-parameters are subtly varied.

Original: This is a critical characteristic because users do not need to overly worry about setting very precise hyper-parameters to obtain a desired saliency map.
Revised: This is a critical characteristic because users do not need to overly worry about setting very precise hyper-parameters to obtain a desired saliency map.

Revised: In addition to the hyper-parameters introduced by our methods, we also test the sensitivity of ﬁdelity evaluation results to the choice of K in the topK normalization.

Revised: Speciﬁcally, we varied K to select top 10% and 40% important features and redrawn the ﬁdelity/sensitivity comparison ﬁgures in Fig.

Revised: 2(B)/

Revised: Fig.

Original:  A O BJECT LOCALIZATION
Revised: 4 (B) ∼ (D). The results in Fig. A13, A14, and A15 are aligned with those in Fig 2 and 4. A O BJECT LOCALIZATION

Revised: accuracy

Revised: VGG16 network thresholding strategies

Original: [Table]
Revised: [Table]

Original: We compare our method and the vanilla gradient method on the object localization task (Dabkowski & Gal, 2017; Fong & Vedaldi, 2017), where the model was trained with the class label only without access to any localization data.
Revised: We compare our method and the vanilla gradient method on the object localization task (Dabkowski & Gal, 2017; Fong & Vedaldi, 2017), where the model was trained with the class label only without access to any localization data.

Original: We carried out Imagenet ILSVRC’14 localization task (Russakovsky et al., 2015) which contains 50K ImageNet validation images with annotated bounding boxes as ground truth.
Revised: We carried out Imagenet ILSVRC’14 localization task (Russakovsky et al., 2015) which contains 50K ImageNet validation images with annotated bounding boxes as ground truth.

Revised: For each of the image in the dataset, we ﬁrst calculated the gradient-based saliency

Original: For each of the image in the dataset, we ﬁrst calculated the gradient-based saliency maps with and without using blurry decoys, based on pretrained VGG16 network.
Revised: [Figure] [Figure] [Figure] maps with and without using blurry decoys, based on pretrained VGG16 network.

Original: Following the preprocessing steps suggested by Dabkowski & Gal (2017); Fong & Vedaldi (2017), for each calculated saliency maps, a bounding box for the most dominant object is predicted by thresholding the corresponding saliency map and extracting the tightest bounding box that contains the whole remaining saliency map.
Revised: Following the preprocessing steps suggested by Dabkowski & Gal (2017); Fong & Vedaldi (2017), for each calculated saliency maps, a bounding box for the most dominant object is predicted by thresholding the corresponding saliency map and extracting the tightest bounding box that contains the whole remaining saliency map.

Original: We investigated three different thresholding strategies suggested by Fong & Vedaldi (2017), including value thresholding, energy thresholding, and mean thresholding.
Revised: We investigated three different thresholding strategies suggested by Fong & Vedaldi (2017), including value thresholding, energy thresholding, and mean thresholding.

Original: Following the evaluation protocol suggested by Dabkowski & Gal (2017); Fong & Vedaldi (2017), the extracted localization box has to have Intersect over Union (IoU) greater than 0 .
Revised: Following the evaluation protocol suggested by Dabkowski & Gal (2017); Fong & Vedaldi (2017), the extracted localization box has to have Intersect over Union (IoU) greater than 0 .

Original: 5 with any of the ground truth bounding boxes in order to consider the localization successful, otherwise it is regarded as failure.
Revised: 5 with any of the ground truth bounding boxes in order to consider the localization successful, otherwise it is regarded as failure.

Original: As shown in Table A4, in terms of accuracy, decoy-enhanced saliency maps perform better than vanilla saliency maps without decoys.
Revised: As shown in Table A3, in terms of accuracy, decoy-enhanced saliency maps perform better than vanilla saliency maps without decoys.

Original: A A DDITIONAL EXPERIMENTAL RESULTS
Revised: A A DDITIONAL EXPERIMENTAL RESULTS

Original: Figure A11, Figure A10, and Figure A12 provide more results of the ﬁdelity and robustness evaluation.
Revised: Fig. A17, Fig. A16, and Fig. A18 provide more results of the ﬁdelity and robustness evaluation.

Original: These results are consistent with those shown in the Section 4. 
Revised: These results are consistent with those shown in the Section 4. [Table] anything [Table]

Revised: anything

Original: [Table]
Revised: [Table]

Revised: anything

Revised: [Table]

Revised: No

Original: SGrad+Decoys
Revised: [Table] anything [Table] Gradient w/o decoy

Original: SF: 0.
Revised: Gradient w/ decoys IntGrad w/o decoy IntGrad w/ decoys SGrad w/o decoy

Original: SF: 0.
Revised: SGrad w/ decoys SF [Table] SF

Original: SF: 0.
Revised: [Table] SF [Table] SF

Revised: [Table]

Original: SF: 0.
Revised: SF [Table] most [Table]

Original: SF: 0.
Revised: movies have a bright sheen most new movies have a bright sheen most [Table] movies [Table]

Revised: a bright sheen most

Original: SF: 0.
Revised: [Table] movies [Table] a

Original: a well made and often lovely depiction of the mysteries of friendship a well made and often lovely depiction of the mysteries of friendship a well made and often lovely depiction of the mysteries of friendship a well made and often lovely depiction of the mysteries of friendship a well made and often lovely depiction of the mysteries of friendship a well made and often lovely depiction of the mysteries of friendship a gentle compassionate drama about grief and healing a gentle compassionate drama about grief and healing a gentle compassionate drama about grief and healing a gentle compassionate drama about grief and healing a gentle compassionate drama about grief and healing a gentle compassionate drama about grief and healing
Revised: [Table] sheen most [Table] movies

Revised: [Table]

Original: Grad
Revised: a [Table] sheen [Table]

Revised: movies

Original: Grad+Decoys
Revised: [Table] sheen Gradient w/o decoy Gradient w/ decoys

Original: IntGrad 
Revised: IntGrad w/o decoy IntGrad w/ decoys SGrad w/o decoy SGrad w/ decoys

Revised: SF

Revised: [Table]

Original: IntGrad+Decoys
Revised: SF [Table] SF [Table]

Revised: SF

Revised: [Table]

Revised: SF

Revised: [Table]

Revised: SF

Revised: [Table]

Revised: 055 as a singular character study it ‘s perfect as a singular character study it ‘s perfect as a singular character study it ‘s perfect as a singular character study it ‘s perfect as a singular character study it ‘s perfect as a singular character study it ‘s perfect

Revised: Gradient w/o decoy

Revised: Gradient w/ decoys

Revised: IntGrad w/o decoy

Revised: IntGrad w/ decoys

Revised: SGrad w/o decoy

Revised: SGrad w/ decoys an intelligent fiction about learning through cultural clash an intelligent fiction about learning through cultural clash an intelligent fiction about learning through cultural clash an intelligent fiction about learning through cultural clash an intelligent fiction about learning through cultural clash an intelligent fiction about learning through cultural clash

Revised: SF:

Revised: [Table]

Revised: SF

Revised: [Table]

Original: SGrad
Revised: SF: 0. SF: 0. SF: 0. SF: 0.

Original: SGrad+Decoys
Revised: Gradient w/o decoy

Original: SF: 1.
Revised: Gradient w/ decoys IntGrad w/o decoy IntGrad w/ decoys SGrad w/o decoy

Revised: SGrad w/ decoys

Original: SF: -2. SF: 1.
Revised: SF: 0. SF: 0.

Original: SF: -1. SF: 1.
Revised: SF: 0. SF: 0. SF: 0.

Original: SF: 0.
Revised: SF: 0.

Original: [Figure]
Revised: a well made and often lovely depiction of the mysteries of friendship a well made and often lovely depiction of the mysteries of friendship a well made and often lovely depiction of the mysteries of friendship a well made and often lovely depiction of the mysteries of friendship a well made and often lovely depiction of the mysteries of friendship a well made and often lovely depiction of the mysteries of friendship Gradient w/o decoy

Original: [Figure]
Revised: Gradient w/ decoys

Original: [Table]
Revised: IntGrad w/o decoy IntGrad w/ decoys SGrad w/o decoy SGrad w/ decoys

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: M
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: c on

Original: [Figure]
Revised: [Figure]

Original: on
Revised: [Figure] [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: M
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: poo

Original: [Figure]
Revised: [Figure]

Original: G r ad + D e c o ys I n t G r ad + D e c o ys
Revised: [Figure]

Original: S G r a d + D e c o ys
Revised: [Figure] [Figure]