Original: I NTERPRETATION OF N EURAL N ETWORK IS F RAGILE
Revised: I NTERPRETATION OF N EURAL N ETWORK IS F RAGILE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In order for machine learning to be deployed and trusted in many applications, it is crucial to be able to reliably explain why the machine learning algorithm makes certain predictions.
Revised: In order for machine learning to be deployed and trusted in many applications, it is crucial to be able to reliably explain why the machine learning algorithm makes certain predictions.

Original: For example, if an algorithm classiﬁes a given pathology image to be a malignant tumor, then the doctor may need to know which parts of the image led the algorithm to this classiﬁcation.
Revised: For example, if an algorithm classiﬁes a given pathology image to be a malignant tumor, then the doctor may need to know which parts of the image led the algorithm to this classiﬁcation.

Original: How to interpret black-box predictors is thus an important and active area of research.
Revised: How to interpret blackbox predictors is thus an important and active area of research.

Original: A fundamental question is: how much can we trust the interpretation itself?
Revised: A fundamental question is: how much can we trust the interpretation itself?

Original: In this paper, we show that interpretation of deep learning predictions is extremely fragile in the following sense: two perceptively indistinguishable inputs with the same predicted label can be assigned very different interpretations.
Revised: In this paper, we show that interpretation of deep learning predictions is extremely fragile in the following sense: two perceptively indistinguishable inputs with the same predicted label can be assigned very different interpretations.

Original: We systematically characterize the fragility of several widely-used feature-importance interpretation methods (saliency maps, relevance propagation, and DeepLIFT) on ImageNet and CIFAR-10.
Revised: We systematically characterize the fragility of several widely-used feature-importance interpretation methods (saliency maps, integrated gradient, and DeepLIFT) on ImageNet and CIFAR-10.

Original: Our experiments show that even small random perturbation can change the feature importance and new systematic perturbations can lead to dramatically different interpretations without changing the label.
Revised: Our experiments show that even small random perturbation can change the feature importance and new systematic perturbations can lead to dramatically different interpretations without changing the label.

Original: We extend these results to show that interpretations based on exemplars (e.g. inﬂuence functions) are similarly fragile.
Revised: We extend these results to show that interpretations based on exemplars (e.g. inﬂuence functions) are similarly fragile.

Original: Our analysis of the geometry of the Hessian matrix gives insight on why fragility could be a fundamental challenge to the current interpretation approaches.
Revised: Our analysis of the geometry of the Hessian matrix gives insight on why fragility could be a fundamental challenge to the current interpretation approaches.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Predictions made by machine learning algorithms play an important role in our everyday lives and can affect decisions in technology, medicine, and even the legal system (Rich, 2015; Obermeyer &
Revised: Predictions made by machine learning algorithms play an important role in our everyday lives and can affect decisions in technology, medicine, and even the legal system (Rich, 2015; Obermeyer &

Original: Emanuel, 2016).
Revised: Emanuel, 2016).

Original: As the algorithms become increasingly complex, explanations for why an algorithm makes certain decisions are ever more crucial.
Revised: As the algorithms become increasingly complex, explanations for why an algorithm makes certain decisions are ever more crucial.

Original: For example, if an AI system predicts a given pathology image to be malignant, then the doctor would want to know what features in the image led the algorithm to this classiﬁcation.
Revised: For example, if an AI system predicts a given pathology image to be malignant, then the doctor would want to know what features in the image led the algorithm to this classiﬁcation.

Original: Similarly, if an algorithm predicts an individual to be a credit risk, then the lender (and the borrower) might want to know why.
Revised: Similarly, if an algorithm predicts an individual to be a credit risk, then the lender (and the borrower) might want to know why.

Original: Therefore having interpretations for why certain predictions are made is critical for establishing trust and transparency between the users and the algorithm (Lipton, 2016).
Revised: Therefore having interpretations for why certain predictions are made is critical for establishing trust and transparency between the users and the algorithm (Lipton, 2016).

Original: Having an interpretation is not enough, however.
Revised: Having an interpretation is not enough, however.

Original: The explanation itself must be robust in order to establish human trust.
Revised: The explanation itself must be robust in order to establish human trust.

Original: Take the pathology predictor; an interpretation method might suggest that a particular section in an image is important for the malignant classiﬁcation (e.g. that section could have high scores in saliency map).
Revised: Take the pathology predictor; an interpretation method might suggest that a particular section in an image is important for the malignant classiﬁcation (e.g. that section could have high scores in saliency map).

Original: The clinician might then focus on that section for investigation, treatment or even look for similar features in other patients.
Revised: The clinician might then focus on that section for investigation, treatment or even look for similar features in other patients.

Original: It would be highly disconcerting if in an extremely similar image, visually indistinguishable from the original and also classiﬁed as malignant, a very different section is interpreted as being salient for the prediction.
Revised: It would be highly disconcerting if in an extremely similar image, visually indistinguishable from the original and also classiﬁed as malignant, a very different section is interpreted as being salient for the prediction.

Original: Thus, even if the predictor is robust (both images are correctly labeled as malignant), that the interpretation is fragile would still be highly problematic in deployment.
Revised: Thus, even if the predictor is robust (both images are correctly labeled as malignant), that the interpretation is fragile would still be highly problematic in deployment.

Original: Our contributions.
Revised: Our contributions.

Original: In this paper, we show that widely-used neural network interpretation methods are fragile in the following sense: perceptively indistinguishable images that have the same prediction label by the neural network can often be given substantially different interpretations.
Revised: In this paper, we show that widely-used neural network interpretation methods are fragile in the following sense: perceptively indistinguishable images that have the same prediction label by the neural network can often be given substantially different interpretations.

Original: We systematically investigate two classes of interpretation methods: methods that assign importance scores to each feature (this includes simple gradient (Simonyan et al., 2013), DeepLift (Shrikumar et al., 2017), and integrated gradient (Sundararajan et al., 2017)), as well as a method that assigns importance scores to each training example: inﬂuence functions (Koh & Liang, 2017).
Revised: We systematically investigate two classes of interpretation methods: methods that assign importance scores to each feature (this includes simple gradient (Simonyan et al., 2013), DeepLift (Shrikumar et al., 2017), and integrated gradient (Sundararajan et al., 2017)), as well as a method that assigns importance scores to each training example: inﬂuence functions (Koh & Liang, 2017).

Original: For both classes of interpretations, we show that the importance of individual features or training examples
Revised: For both classes of interpretations, we show that the importance of individual features or training examples

Original: O r i g i na l
Revised: O r i g i na l

Original: P e r t u r bed % (a) (
Revised: P e r t u r bed % (a) (

Original: [Figure]
Revised: [Figure]

Original: ) "hay":66.2% "hay":99.6%
Revised: ) "hay":66.2% "hay":99.6%

Original: SaliencyMap
Revised: SaliencyMap

Original: SaliencyMap (c)  is highly fragile to even small random perturbations to the input image.
Revised: SaliencyMap (c) increased is highly fragile to even small random perturbations to the input image.

Original: Moreover we show how targeted perturbations can lead to dramatically different global interpretations (Fig. 1).
Revised: Moreover we show how targeted perturbations can lead to dramatically different global interpretations (Fig. 1).

Original: Our ﬁndings highlight the fragility of popular interpretation methods, which has not been carefully considered in literature.
Revised: Our ﬁndings highlight the fragility of popular interpretation methods, which has not been carefully considered in literature.

Original: Fragility directly limits how much we can trust and learn from the interpretations.
Revised: Fragility directly limits how much we can trust and learn from the interpretations.

Original: It also raises a signiﬁcant new security concern.
Revised: It also raises a signiﬁcant new security concern.

Original: Especially in medical or economic applications, users often take the interpretation of a prediction as containing causal insight ( “this image is a malignant tumor likely because of the section with a high saliency score” ).
Revised: Especially in medical or economic applications, users often take the interpretation of a prediction as containing causal insight ( “this image is a malignant tumor likely because of the section with a high saliency score” ).

Original: An adversary could minutely manipute the input to draw attention away from relevant features or onto his/her desired features.
Revised: An adversary could minutely manipute the input to draw attention away from relevant features or onto his/her desired features.

Original: Such attacks might be especially hard to detect as the actual labels have not changed.
Revised: Such attacks might be especially hard to detect as the actual labels have not changed.

Original: While we focus on image data here because most of the interpretation methods have been motivated by images, the fragility of interpretation could be a much broader problem.
Revised: While we focus on image data here because most of the interpretation methods have been motivated by images, the fragility of interpretation could be a much broader problem.

Original: Fig.
Revised: Fig.

Original: 2 illustrates the intuition that when the decision boundary in the input feature space is complex, as is the case with deep nets, a small perturbation in the input can push the example into a region with very different loss contours.
Revised: 2 illustrates the intuition that when the decision boundary in the input feature space is complex, as is the case with deep nets, a small perturbation in the input can push the example into a region with very different loss contours.

Original: Because the feature importance is closely related to the gradient which is perpendicular to the loss contours, the importance scores can also be dramatically different.
Revised: Because the feature importance is closely related to the gradient which is perpendicular to the loss contours, the importance scores can also be dramatically different.

Original: We provide additional analysis of this in Section 5.
Revised: We provide additional analysis of this in Section 5.

Original:  I NTERPRETATION M ETHODS FOR N EURAL N ETWORK P REDICTIONS 2. F EATURE -I MPORTANCE I NTERPRETATION
Revised:  I NTERPRETATION M ETHODS FOR N EURAL N ETWORK P REDICTIONS 2. F EATURE -I MPORTANCE I NTERPRETATION

Original: This ﬁrst class of methods explains predictions in terms of the relative importance of features in a test input sample.
Revised: This ﬁrst class of methods explains predictions in terms of the relative importance of features in a test input sample.

Original: Given the sample x t ∈ R d and the network’s prediction l , we deﬁne the score of the predicted class S l ( x t ) to be the value of the l -th output neuron right before the softmax operation.
Revised: Given the sample x t ∈ R d and the network’s prediction l , we deﬁne the score of the predicted class S l ( x t ) to be the value of the l -th output neuron right before the softmax operation.

Original: We take l to be the class with the max score; i.e. the predicted class.
Revised: We take l to be the class with the max score; i.e. the predicted class.

Original: Feature-importance methods seek to ﬁnd the dimensions of input data point that most strongly affect the score, and in doing so, these methods assign an absolute saliency score to each input feature.
Revised: Feature-importance methods seek to ﬁnd the dimensions of input data point that most strongly affect the score, and in doing so, these methods assign an absolute saliency score to each input feature.

Original: Here we normalize the scores for each image by the sum of the saliency scores across the features.
Revised: Here we normalize the scores for each image by the sum of the saliency scores across the features.

Original: This ensures that any perturbations that we design change not the absolute feature saliencies (which may still preserve the ranking of different features), but their relative values.
Revised: This ensures that any perturbations that we design change not the absolute feature saliencies (which may still preserve the ranking of different features), but their relative values.

Original: We summarize three different methods to calculate the normalized saliency score, denoted by R ( x t ) .
Revised: We summarize three different methods to calculate the normalized saliency score, denoted by R ( x t ) .

Original: Simple gradient method Introduced in Baehrens et al.
Revised: Simple gradient method Introduced in Baehrens et al.

Original: and applied to deep neural networks in Simonyan et al.
Revised: and applied to deep neural networks in Simonyan et al.

Original: (2013), the simple gradient method applies a local linear approximation of the model to detect the sensitivity of the score to perturbing each of the input dimensions.
Revised: (2013), the simple gradient method applies a local linear approximation of the model to detect the sensitivity of the score to perturbing each of the input dimensions.

Original: Given
Revised: Given

Original: [Figure]
Revised: [Figure]

Original: input x t ∈ R d , the score is deﬁned as:
Revised: input x t ∈ R d , the score is deﬁned as:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Integrated gradients A signiﬁcant drawback of the simple gradient method is the saturation problem discussed by Shrikumar et al.
Revised: Integrated gradients A signiﬁcant drawback of the simple gradient method is the saturation problem discussed by Shrikumar et al.

Original: ; Sundararajan et al.
Revised: ; Sundararajan et al.

Original: Consequently, Sundararajan et al.
Revised: Consequently, Sundararajan et al.

Original: (2017) introduced the integrated gradients method where the gradients of the score with respect to  scaled fractions of the input are summed and then multiplied by the input.
Revised: (2017) introduced the integrated gradients method where the gradients of the score with respect to M scaled versions of the input are summed and then multiplied by the input.

Original: Letting x 0 be the reference point and ∆ x t = x t − x 0 , the feature importance vector is calculated by:
Revised: Letting x 0 be the reference point and ∆ x t = x t − x 0 , the feature importance vector is calculated by:

Original: [Equation]
Revised: [Equation]

Original: which is then normalized for our analysis.
Revised: which is then normalized for our analysis.

Original: Here the absolute value is taken for each dimension.
Revised: Here the absolute value is taken for each dimension.

Original: DeepLIFT DeepLIFT is an improved version of layer-wise relevance propagation (LRP) method (Bach et al., 2015).
Revised: DeepLIFT DeepLIFT is an improved version of layer-wise relevance propagation (LRP) method (Bach et al., 2015).

Original: LRP methods decompose the score S l ( x t ) backwards through the neural network.
Revised: LRP methods decompose the score S l ( x t ) backwards through the neural network.

Original: In each step, the score from the last layer is propagated to the previous layer, with the score being divided proportionally to magnitude of the activations of the neurons in the previous layer.
Revised: In each step, the score from the last layer is propagated to the previous layer, with the score being divided proportionally to magnitude of the activations of the neurons in the previous layer.

Original: The scores are propagated to the input layer, and the result is a relevance score assigned to each of the input dimensions.
Revised: The scores are propagated to the input layer, and the result is a relevance score assigned to each of the input dimensions.

Original: DeepLIFT (Shrikumar et al., 2017) deﬁnes a reference point in the input space and propagates relevance scores proportionally to the changes in the neuronal activations from the reference.
Revised: DeepLIFT (Shrikumar et al., 2017) deﬁnes a reference point in the input space and propagates relevance scores proportionally to the changes in the neuronal activations from the reference.

Original: We use DeepLIFT with the Rescale rule; see Shrikumar et al.
Revised: We use DeepLIFT with the Rescale rule; see Shrikumar et al.

Original: (2017) for details.
Revised: (2017) for details.

Original: 2. E XEMPLAR -B ASED M ETHODS : I NFLUENCE F UNCTIONS
Revised: 2. E XEMPLAR -B ASED M ETHODS : I NFLUENCE F UNCTIONS

Original: A complementary approach to interpreting the results of a neural network is to explain the prediction of the network in terms of its training examples , { ( x i , y i ) } .
Revised: A complementary approach to interpreting the results of a neural network is to explain the prediction of the network in terms of its training examples , { ( x i , y i ) } .

Original: Speciﬁcally, we can ask: which training examples, if up-weighted or down-weighted during training time, would have the biggest effect on the loss of the test example ( x t , y t ) ?
Revised: Speciﬁcally, we can ask: which training examples, if up-weighted or down-weighted during training time, would have the biggest effect on the loss of the test example ( x t , y t ) ?

Original: Koh & Liang (2017) proposed a method to calculate this value, called the inﬂuence, deﬁned by the following equation:
Revised: Koh & Liang (2017) proposed a method to calculate this value, called the inﬂuence, deﬁned by the following equation:

Original: [Equation]
Revised: [Equation]

Original: where z i def = ( x i , y i ) and z t is deﬁned analogously.
Revised: where z i def = ( x i , y i ) and z t is deﬁned analogously.

Original: L ( z, ˆ θ ) is the loss of the network with parameters set to ˆ θ for the (training or test) data point z .
Revised: L ( z, ˆ θ ) is the loss of the network with parameters set to ˆ θ for the (training or test) data point z .

Original: H ˆ θ def = 1 n (cid:80) ni
Revised: H ˆ θ def = 1 n (cid:80) ni

Original: =1 ∇ 2 θ L ( z i , ˆ θ ) is the empirical Hessian of the network calculated over the training examples.
Revised: =1 ∇ 2 θ L ( z i , ˆ θ ) is the empirical Hessian of the network calculated over the training examples.

Original: The training examples with the highest inﬂuence are understood as explaining why a network made a particular prediction for a test example.
Revised: The training examples with the highest inﬂuence are understood as explaining why a network made a particular prediction for a test example.

Original: 2. M ETRICS FOR I NTERPRETATION S IMILARITY
Revised: 2. M ETRICS FOR I NTERPRETATION S IMILARITY

Original: We consider two natural metrics for quantifying the similarity between interpretations for two different images.
Revised: We consider two natural metrics for quantifying the similarity between interpretations for two different images.

Original: • Spearman’s rank order correlation : Because interpretation methods rank all of the features or training examples in order of importance, it natural to use the rank correlation (Spearman, 1904) to compare the similarity between interpretations.
Revised: • Spearman’s rank order correlation : Because interpretation methods rank all of the features or training examples in order of importance, it natural to use the rank correlation (Spearman, 1904) to compare the similarity between interpretations.

Original: • Top-k intersection :
Revised: • Top-k intersection :

Original: In many settings, only the most important features or interpretations are of interest.
Revised: In many settings, only the most important features or interpretations are of interest.

Original: In these settings, we can compute the size of the intersection of the k most important features before and after perturbation.
Revised: In these settings, we can compute the size of the intersection of the k most important features before and after perturbation.

Original:  R ANDOM AND SYSTEMATIC PERTURBATIONS
Revised:  R ANDOM AND SYSTEMATIC PERTURBATIONS

Original: Problem statement For a given ﬁxed neural network N and input data point x t , the feature importance and inﬂuence function methods that we have described produce an interpretation
Revised: Problem statement For a given ﬁxed neural network N and input data point x t , the feature importance and inﬂuence function methods that we have described produce an interpretation

Original: I ( x t ; N ) .
Revised: I ( x t ; N ) .

Original: For feature importance, I ( x t ; N ) is a vector of feature scores; for inﬂuence function
Revised: For feature importance, I ( x t ; N ) is a vector of feature scores; for inﬂuence function

Original: I ( x t ; N ) is a vector of scores for training examples.
Revised: I ( x t ; N ) is a vector of scores for training examples.

Original: We would like to devise efﬁcient perturbations to change the interpretability of a test image.
Revised: We would like to devise efﬁcient perturbations to change the interpretability of a test image.

Original: Yet, the perturbations should be visually imperceptible and should not change the label of the prediction.
Revised: Yet, the perturbations should be visually imperceptible and should not change the label of the prediction.

Original: Formally, we deﬁne the problem as:
Revised: Formally, we deﬁne the problem as:

Original: [Equation]
Revised: [Equation]

Original: subject to: || δ || ∞ ≤
Revised: subject to: || δ || ∞ ≤

Original: [Equation]
Revised: [Equation]

Original: , Prediction ( x t + δ ; N ) =
Revised: , Prediction ( x t + δ ; N ) =

Original: Prediction ( x
Revised: Prediction ( x

Original: [Equation]
Revised: [Equation]

Original: N ) where D ( · ) measures the change in interpretation (e.g. how many of the top- k pixels are no longer the top- k pixels of the saliency map after the perturbation) and (cid:15) > 0 constrains the norm of the perturbation.
Revised: N ) where D ( · ) measures the change in interpretation (e.g. how many of the top- k pixels are no longer the top- k pixels of the saliency map after the perturbation) and (cid:15) > 0 constrains the norm of the perturbation.

Original: In this paper, we carry out three kinds of input perturbations.
Revised: In this paper, we carry out three kinds of input perturbations.

Original: Random sign perturbation As a baseline, we generate random perturbations in which each pixel is randomly perturbed by ± (cid:15) .
Revised: Random sign perturbation As a baseline, we generate random perturbations in which each pixel is randomly perturbed by ± (cid:15) .

Original: This is used to measure robustness against untargeted perturbations.
Revised: This is used to measure robustness against untargeted perturbations.

Original: Iterative attacks against feature-importance methods In Algorithm 1 we deﬁne two adversarial attacks against feature-importance methods, each of which consists of taking a series of steps in the direction that maximizes a differentiable dissimilarity function between the original and perturbed interpretation.
Revised: Iterative attacks against feature-importance methods In Algorithm 1 we deﬁne two adversarial attacks against feature-importance methods, each of which consists of taking a series of steps in the direction that maximizes a differentiable dissimilarity function between the original and perturbed interpretation.

Original: The top- k attack seeks to perturb the saliency map by decreasing the relative importance of the k most important features of the original image.
Revised: The top- k attack seeks to perturb the saliency map by decreasing the relative importance of the k most important features of the original image.

Original: (2) When the input data are images, the center of mass of the saliency map often captures the user’s attention.
Revised: (2) When the input data are images, the center of mass of the saliency map often captures the user’s attention.

Original: The mass-center attack is designed to result in the maximum spatial displacement of the center of mass of the saliency scores.
Revised: The mass-center attack is designed to result in the maximum spatial displacement of the center of mass of the saliency scores.

Original: Both of these attacks can be applied to any of the three feature-importance methods.
Revised: Both of these attacks can be applied to any of the three feature-importance methods.

Original: Gradient sign attack against inﬂuence functions We can obtain effective adversarial images for inﬂuence functions without resorting to interative procedures.
Revised: Gradient sign attack against inﬂuence functions We can obtain effective adversarial images for inﬂuence functions without resorting to interative procedures.

Original: We linearize (3) around the values of the current inputs and parameters.
Revised: We linearize (3) around the values of the current inputs and parameters.

Original: If we further constrain the L ∞ norm of the perturbation to (cid:15) , we obtain an optimal single-step perturbation:
Revised: If we further constrain the L ∞ norm of the perturbation to (cid:15) , we obtain an optimal single-step perturbation:

Original: [Equation]
Revised: [Equation]

Original: This perturbation can be applied to the pixels of a test image x t → x t + δ to increase or decrease the inﬂuence of a particular training example, x i .
Revised: This perturbation can be applied to the pixels of a test image x t → x t + δ to increase or decrease the inﬂuence of a particular training example, x i .

Original: The attack we use consists of applying the negative of the perturbation in (4) to decrease the inﬂuence of the 3 most inﬂuential training images of the original test image 1 .
Revised: The attack we use consists of applying the negative of the perturbation in (4) to decrease the inﬂuence of the 3 most inﬂuential training images of the original test image 1 .

Original: Of course, this affects the inﬂuence of all of the other training images as well.
Revised: Of course, this affects the inﬂuence of all of the other training images as well.

Original: Algorithm
Revised: Algorithm

Original: Input: test image x t , maximum norm of perturbation (cid:15) , normalized feature importance function R ( · ) , number of iterations M , step size α
Revised: Input: test image x t , maximum norm of perturbation (cid:15) , normalized feature importance function R ( · ) , number of iterations P , step size α

Original: Deﬁne a dissimilarity function D to measure the change between interpretations of two images: 
Revised: Deﬁne a dissimilarity function D to measure the change between interpretations of two images: D (

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  
Revised:  

Original: [Equation]
Revised: [Equation]

Original: (cid:80)
Revised: (cid:80)

Original: [Equation]
Revised: [Equation]

Original: for top-k attack |
Revised: for top-k attack |

Original: [Equation]
Revised: [Equation]

Original: for mass-center attack, where B is the set of the k largest dimensions a of R ( x t ) , and C ( · ) is the center of saliency mass b .
Revised: for mass-center attack, where B is the set of the k largest dimensions a of R ( x t ) , and C ( · ) is the center of saliency mass b .

Original: Initialize x 0 = x t for p ∈ { 1 , . . .
Revised: Initialize x 0 = x t for p ∈ { 1 , . . .

Original: , P } do
Revised: , P } do

Original: Perturb the test image in the direction of signed gradient c of the dissimilarity function:
Revised: Perturb the test image in the direction of signed gradient c of the dissimilarity function:

Original: [Equation]
Revised: [Equation]

Original: If needed, clip the perturbed input to satisfy the norm constraint:
Revised: If needed, clip the perturbed input to satisfy the norm constraint:

Original: || x p − x t || ∞ ≤ (cid:15) end for
Revised: || x p − x t || ∞ ≤ (cid:15) end for

Original: Among { x 1 , . . .
Revised: Among { x 1 , . . .

Original: , x P } , return the element with the largest value for the dissimilarity function and the same prediction as the original test image.
Revised: , x P } , return the element with the largest value for the dissimilarity function and the same prediction as the original test image.

Original: a The goal is to damp the saliency scores of the k features originally identiﬁed as the most important.
Revised: a The goal is to damp the saliency scores of the k features originally identiﬁed as the most important.

Original: b
Revised: b

Original: The center of mass is deﬁned for a W × H image as:
Revised: The center of mass is deﬁned for a W × H image as:

Original: [Equation]
Revised: [Equation]

Original: c In some networks, such as those with ReLUs, this gradient is always 0.
Revised: c In some networks, such as those with ReLUs, this gradient is always 0.

Original: To attack interpretability in such networks, we replace the ReLU activations with their smooth approximation (softplus) when calculating the gradient and generate the perturbed image using this approximation.
Revised: To attack interpretability in such networks, we replace the ReLU activations with their smooth approximation (softplus) when calculating the gradient and generate the perturbed image using this approximation.

Original: The perturbed images that result are effective adversarial attacks against the original ReLU network, as discussed in Section 4.
Revised: The perturbed images that result are effective adversarial attacks against the original ReLU network, as discussed in Section 4.

Original: We follow the same setup for computing the inﬂuence function as was done by the authors of Koh & Liang (2017).
Revised: We follow the same setup for computing the inﬂuence function as was done by the authors of Koh & Liang (2017).

Original: Because the inﬂuence is only calculated with respect to the parameters that change during training, we calculate the gradients only with respect to parameters in the ﬁnal layer of our network (InceptionNet, see Section 4).
Revised: Because the inﬂuence is only calculated with respect to the parameters that change during training, we calculate the gradients only with respect to parameters in the ﬁnal layer of our network (InceptionNet, see Section 4).

Original: This makes it feasible for us to compute (4) exactly, but it gives us the perturbation of the input into the ﬁnal layer , not the ﬁrst layer.
Revised: This makes it feasible for us to compute (4) exactly, but it gives us the perturbation of the input into the ﬁnal layer , not the ﬁrst layer.

Original: So, we use standard back-propagation to calculate the corresponding gradient for the input test image.
Revised: So, we use standard back-propagation to calculate the corresponding gradient for the input test image.

Original: We then take the sign of this gradient as the perturbation and clip the image to produce the adversarial test image.
Revised: We then take the sign of this gradient as the perturbation and clip the image to produce the adversarial test image.

Original:  E XPERIMENTS & R ESULTS
Revised:  E XPERIMENTS & R ESULTS

Original: Data sets and models To evaluate the robustness of feature-importance methods, we used two image classiﬁcation data sets: ILSVRC2012 (ImageNet classiﬁcation challenge data set)
Revised: Data sets and models To evaluate the robustness of feature-importance methods, we used two image classiﬁcation data sets: ILSVRC2012 (ImageNet classiﬁcation challenge data set)

Original: (Russakovsky et al., 2015) and CIFAR-10 (Krizhevsky, 2009).
Revised: (Russakovsky et al., 2015) and CIFAR-10 (Krizhevsky, 2009).

Original: For the ImageNet classiﬁcation data set, we used a pre-trained SqueezeNet 2 model introduced by Iandola et al.
Revised: For the ImageNet classiﬁcation data set, we used a pre-trained SqueezeNet 2 model introduced by Iandola et al.

Original: For the CIFARdata set we trained our own convolutional network, whose architecture is presented in Appendix A.
Revised: For the CIFARdata set we trained our own convolutional network, whose architecture is presented in Appendix A.

Original: For both data sets, the results are examined using simple gradient, integrated gradients, and
Revised: For both data sets, the results are examined using simple gradient, integrated gradients, and

Original: DeepLIFT feature importance methods.
Revised: DeepLIFT feature importance methods.

Original: For DeepLIFT, we used the pixel-wise and the channelwise mean images as the CIFAR-10 and ImageNet reference points respectively.
Revised: For DeepLIFT, we used the pixel-wise and the channelwise mean images as the CIFAR-10 and ImageNet reference points respectively.

Original: For the integrated gradients method, the same references were used with parameter M=100.
Revised: For the integrated gradients method, the same references were used with parameter M=100.

Original: We ran all iterative attack algorithms for P = 300 iterations with step size α = 0 .
Revised: We ran all iterative attack algorithms for P = 300 iterations with step size α = 0 .

Original: [Figure]
Revised: [Figure]

Original: Simple Gradient
Revised: Simple Gradient

Original: [Figure]
Revised: [Figure]

Original: DeepLIFT
Revised: DeepLIFT

Original: [Figure]
Revised: [Figure]

Original: Integrated Gradients
Revised: Integrated Gradients

Original: [Figure]
Revised: [Figure]

Original: To evaluate the robustness of inﬂuence functions, we followed a similar experimental setup to that of the original authors: we trained an InceptionNet v3 with all but the last layer frozen (the weights were pre-trained on ImageNet and obtained from Keras 3 ).
Revised: To evaluate the robustness of inﬂuence functions, we followed a similar experimental setup to that of the original authors: we trained an InceptionNet v3 with all but the last layer frozen (the weights were pre-trained on ImageNet and obtained from Keras 3 ).

Original: The last layer was trained on a binary ﬂower classiﬁcation task ( roses vs. sunﬂowers ), using a data set consisting of 1,000 training images 4 .
Revised: The last layer was trained on a binary ﬂower classiﬁcation task ( roses vs. sunﬂowers ), using a data set consisting of 1,000 training images 4 .

Original: This data set was chosen because it consisted of images that the network had not seen during pre-training on ImageNet.
Revised: This data set was chosen because it consisted of images that the network had not seen during pre-training on ImageNet.

Original: The network achieved a validation accuracy of 97.5% on this task.
Revised: The network achieved a validation accuracy of 97.5% on this task.

Original: Results for feature-importance methods From the ImageNet test set, 512 correctly-classiﬁed images were randomly sampled for evaluation purposes.
Revised: Results for feature-importance methods From the ImageNet test set, 512 correctly-classiﬁed images were randomly sampled for evaluation purposes.

Original: Examples of the center-shift attack against three feature importance methods were presented in Fig.
Revised: Examples of the center-shift attack against three feature importance methods were presented in Fig.

Original: Further representative examples of different attacks on additional images are found in Appendix B.
Revised: Further representative examples of different attacks on additional images are found in Appendix B.

Original: In Fig.
Revised: In Fig.

Original: 3, we present results aggregated over all 512 images.
Revised: 3, we present results aggregated over all 512 images.

Original: We compare different attack methods using top-1000 intersection and rank correlation methods.
Revised: We compare different attack methods using top-1000 intersection and rank correlation methods.

Original: Random sign perturbation already causes signiﬁcant changes in both top-1000 intersection and rank order correlation.
Revised: Random sign perturbation already causes signiﬁcant changes in both top-1000 intersection and rank order correlation.

Original: For example, with
Revised: For example, with

Original: L ∞ = 8 , on average, there is less than 30% overlap in the top 1000 most salient pixels between the original and the randomly perturbed images across all three of interpretation methods.
Revised: L ∞ = 8 , on average, there is less than 30% overlap in the top 1000 most salient pixels between the original and the randomly perturbed images across all three of interpretation methods.

Original: This suggests that the saliency of individual or small groups of pixels can be extremely fragile to the input and should be interpreted with caution.
Revised: This suggests that the saliency of individual or small groups of pixels can be extremely fragile to the input and should be interpreted with caution.

Original: With targeted perturbations, we observe more dramatic fragility.
Revised: With targeted perturbations, we observe more dramatic fragility.

Original: Even with a perturbation of L ∞ = 2 , the interpretations change signiﬁcantly.
Revised: Even with a perturbation of L ∞ = 2 , the interpretations change signiﬁcantly.

Original: Both iterative attack algorithms have similar effects on feature importance of test images when measured on the basis of rank correlation or top-1000 intersection.
Revised: Both iterative attack algorithms have similar effects on feature importance of test images when measured on the basis of rank correlation or top-1000 intersection.

Original: In Appendix C, we show an additional metric: the displacement of the center of mass between the original and perturbed saliency maps.
Revised: In Appendix C, we show an additional metric: the displacement of the center of mass between the original and perturbed saliency maps.

Original: Empirically, we ﬁnd this metric to correspond most strongly with intuitive perceptions of the similarity between two saliency maps.
Revised: Empirically, we ﬁnd this metric to correspond most strongly with intuitive perceptions of the similarity between two saliency maps.

Original: Not surprisingly, we found that the center attack method was more effective than the top- k attack at moving the center of mass of the saliency maps.
Revised: Not surprisingly, we found that the center attack method was more effective than the top- k attack at moving the center of mass of the saliency maps.

Original: Comparing interpretation methods, we found that the integrated gradients method was the most robust to both random and adversarial attacks.
Revised: Comparing interpretation methods, we found that the integrated gradients method was the most robust to both random and adversarial attacks.

Original: Similar results for CIFAR-10 can be found in Appendix C.
Revised: Similar results for CIFAR-10 can be found in Appendix C.

Original: Results for inﬂuence functions
Revised: Results for inﬂuence functions

Original: We evaluate the robustness of inﬂuence functions on a test data set consisting of 200 images of roses and sunﬂowers.
Revised: We evaluate the robustness of inﬂuence functions on a test data set consisting of 200 images of roses and sunﬂowers.

Original: Fig.
Revised: Fig.

Original: 4 shows a representative test image to which we have applied the gradient sign attack.
Revised: 4 shows a representative test image to which we have applied the gradient sign attack.

Original: Although the prediction of the image does not change, the most inﬂuential training examples selected according to (3), as explanation for the prediction, change entirely from images of sunﬂowers and yellow petals that resemble the input image to those of red and pink roses that do not.
Revised: Although the prediction of the image does not change, the most inﬂuential training examples selected according to (3), as explanation for the prediction, change entirely from images of sunﬂowers and yellow petals that resemble the input image to those of red and pink roses that do not.

Original: Additional examples can be found in Appendix D.
Revised: Additional examples can be found in Appendix D.

Original: In Fig.
Revised: In Fig.

Original: 5, we compare the random perturbations and gradient sign attacks across all of the test images.
Revised: 5, we compare the random perturbations and gradient sign attacks across all of the test images.

Original: We ﬁnd that the gradient sign-based attacks are signiﬁcantly more effective at decreasing the rank correlation of the inﬂuence of the training images, as well as distorting the top-5 inﬂuential images.
Revised: We ﬁnd that the gradient sign-based attacks are signiﬁcantly more effective at decreasing the rank correlation of the inﬂuence of the training images, as well as distorting the top-5 inﬂuential images.

Original: For example, on average, with a targeted perturbation of magnitude (cid:15) = 8 , only 2 of the top 5 most inﬂuential training images remain as the top 5 most inﬂuential images after the visually imperceptible perturbation.
Revised: For example, on average, with a targeted perturbation of magnitude (cid:15) = 8 , only 2 of the top 5 most inﬂuential training images remain as the top 5 most inﬂuential images after the visually imperceptible perturbation.

Original: The inﬂuences of the training images before and after an adversarial attack are essentially uncorrelated.
Revised: The inﬂuences of the training images before and after an adversarial attack are essentially uncorrelated.

Original: However, we ﬁnd that even random attacks can have a nonnegligible effect on inﬂuence functions, on average reducing the rank correlation to 0.8 ( (cid:15)
Revised: However, we ﬁnd that even random attacks can have a nonnegligible effect on inﬂuence functions, on average reducing the rank correlation to 0.8 ( (cid:15)

Original: ≈ 10 ).
Revised: ≈ 10 ).

Original: [Figure]
Revised: [Figure]

Original:  H ESSIAN ANALYSIS
Revised:  H ESSIAN ANALYSIS

Original: In this section, we try to understand the source of interpretation fragility.
Revised: In this section, we try to understand the source of interpretation fragility.

Original: The question is whether fragility a consequence of the complex non-linearities of a deep network or a characteristic present even in high-dimensional linear models, as is the case for adversarial examples for prediction (Goodfellow et al., 2014).
Revised: The question is whether fragility a consequence of the complex non-linearities of a deep network or a characteristic present even in high-dimensional linear models, as is the case for adversarial examples for prediction (Goodfellow et al., 2014).

Original: To gain more insight into the fragility of gradient based interpretations, let
Revised: To gain more insight into the fragility of gradient based interpretations, let

Original: S ( x ; W ) denote the score function of interest; x ∈ R d is an input vector and W is the weights of the neural network, which is ﬁxed since the network has ﬁnished training.
Revised: S ( x ; W ) denote the score function of interest; x ∈ R d is an input vector and W is the weights of the neural network, which is ﬁxed since the network has ﬁnished training.

Original: We are interested in the
Revised: We are interested in the

Original: Hessian H whose entries are H i,j = ∂S ∂x i ∂x j .
Revised: Hessian H whose entries are H i,j = ∂S ∂x i ∂x j .

Original: The reason is that the ﬁrst order approximation of gradient for some input perturbation direction δ ∈ R d is: ∇ x S ( x + δ ) − ∇ x S ( x ) ≈ H δ .
Revised: The reason is that the ﬁrst order approximation of gradient for some input perturbation direction δ ∈ R d is: ∇ x S ( x + δ ) − ∇ x S ( x ) ≈ H δ .

Original: First, consider a linear model whose score for an input x is S = w (cid:62) x .
Revised: First, consider a linear model whose score for an input x is S = w (cid:62) x .

Original: Here, ∇ x S = w and ∇ 2 x S
Revised: Here, ∇ x S = w and ∇ 2 x S

Original: = 0 ; the feature-importance vector w is robust, because it is completely independent of x .
Revised: = 0 ; the feature-importance vector w is robust, because it is completely independent of x .

Original: Thus, some non-linearity is required for interpretation fragility.
Revised: Thus, some non-linearity is required for interpretation fragility.

Original: A simple network that is susceptible
Revised: A simple network that is susceptible

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: R an k c o rr e l a t i on
Revised: an k c o rr e l a t i on

Original: Rank Correlation
Revised: [Figure] perturbation

Original: Random perturbation Gradient sign attack (a)  10 L norm of perturbation
Revised: [Figure] sign [Figure] (a)  10 L norm of perturbation

Original: T op - 5 I n t e r s e c t i on
Revised: T op - 5 I n t e r s e c t i on

Original: Top-5 Intersection
Revised: Top-5 Intersection

Original: [Figure]
Revised: [Figure]

Original: to adversarial attacks on interpretations consists of a set of weights connecting the input to a single neuron followed by a non-linearity (e.g. logistic regression): S = g ( w (cid:62) x ) .
Revised: to adversarial attacks on interpretations consists of a set of weights connecting the input to a single neuron followed by a non-linearity (e.g. logistic regression): S = g ( w (cid:62) x ) .

Original: We can calculate the change in saliency map due to a small perturbation in x → x + δ .
Revised: We can calculate the change in saliency map due to a small perturbation in x → x + δ .

Original: The ﬁrst-order approximation for the change in saliency map will be equal to : H · δ = ∇ 2 x S · δ .
Revised: The ﬁrst-order approximation for the change in saliency map will be equal to : H · δ = ∇ 2 x S · δ .

Original: In particular, the saliency of the i th feature changes by ( ∇ 2 x S · δ ) i and furthermore, the relative change is ( ∇ 2 x S · δ ) i / ( ∇ x S ) i .
Revised: In particular, the saliency of the i th feature changes by ( ∇ 2 x S · δ ) i and furthermore, the relative change is ( ∇ 2 x S · δ ) i / ( ∇ x S ) i .

Original: For the simple network, this relative change is:
Revised: For the simple network, this relative change is:

Original: [Equation]
Revised: [Equation]

Original: ) where we have used g (cid:48) ( · ) and g (cid:48)(cid:48) ( · ) to refer to the ﬁrst and second derivatives of g ( · ) .
Revised: ) where we have used g (cid:48) ( · ) and g (cid:48)(cid:48) ( · ) to refer to the ﬁrst and second derivatives of g ( · ) .

Original: Note that g (cid:48) ( w (cid:62) x ) and g (cid:48)(cid:48) ( w (cid:62) x ) do not scale with the dimensionality of x because in general, independent from the dimensionality, x and w are (cid:96) 2 -normalized or have ﬁxed (cid:96) 2 -norm due to data preprocessing and weight decay regularization.
Revised: Note that g (cid:48) ( w (cid:62) x ) and g (cid:48)(cid:48) ( w (cid:62) x ) do not scale with the dimensionality of x because in general, independent from the dimensionality, x and w are (cid:96) 2 -normalized or have ﬁxed (cid:96) 2 -norm due to data preprocessing and weight decay regularization.

Original: However, if we choose δ = (cid:15) sign ( w ) , then the relative change in the saliency grows with the dimension, since it is proportional to the (cid:96) 1 -norm of w .
Revised: However, if we choose δ = (cid:15) sign ( w ) , then the relative change in the saliency grows with the dimension, since it is proportional to the (cid:96) 1 -norm of w .

Original: When the input is high-dimensional—which is the case with images—the relative effect of the perturbation can be substantial.
Revised: When the input is high-dimensional—which is the case with images—the relative effect of the perturbation can be substantial.

Original: Note also that this perturbation is exactly the sign of the ﬁrst right singular vector of the Hessian ∇ 2 x S , which is appropriate since that is the vector that has the maximum effect on the gradient of S .
Revised: Note also that this perturbation is exactly the sign of the ﬁrst right singular vector of the Hessian ∇ 2 x S , which is appropriate since that is the vector that has the maximum effect on the gradient of S .

Original: A similar analysis can be carried out for inﬂuence functions (see Appendix E).
Revised: A similar analysis can be carried out for inﬂuence functions (see Appendix E).

Original: For this simple network, the direction of adversarial attack on interpretability, sign ( w ) is the same as the adversarial attack on prediction.
Revised: For this simple network, the direction of adversarial attack on interpretability, sign ( w ) is the same as the adversarial attack on prediction.

Original: This means that we cannot perturb interpretability independently of prediction.
Revised: This means that we cannot perturb interpretability independently of prediction.

Original: For more complex networks, this is not the case and in Appendix F we show this analytically for a simple case of a two-layer network.
Revised: For more complex networks, this is not the case and in Appendix F we show this analytically for a simple case of a two-layer network.

Original: As an empirical test, in Fig.
Revised: As an empirical test, in Fig.

Original: 6(a), we plot the distribution of the angle between ∇ x S and v 1 (the ﬁrst right singular vector of H which is the most fragile direction of feature importance) for 1000 CIFAR10 images (Details of the network in Appendix A).
Revised: 6(a), we plot the distribution of the angle between ∇ x S and v 1 (the ﬁrst right singular vector of H which is the most fragile direction of feature importance) for 1000 CIFAR10 images (Details of the network in Appendix A).

Original: In Fig.
Revised: In Fig.

Original: 6(b), we plot the equivalent distribution for inﬂuence functions, computed across all 200 test images.
Revised: 6(b), we plot the equivalent distribution for inﬂuence functions, computed across all 200 test images.

Original: The result conﬁrms that the steepest direction of change in interpretation and prediction are generally orthogonal, justifying how the perturbations can change the interpretation without changing the prediction.
Revised: The result conﬁrms that the steepest direction of change in interpretation and prediction are generally orthogonal, justifying how the perturbations can change the interpretation without changing the prediction.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Related works To the best of our knowledge, the notion of adversarial attacks has not previously been studied in the context of interpretation of neural networks.
Revised: Related works To the best of our knowledge, the notion of adversarial attacks has not previously been studied in the context of interpretation of neural networks.

Original: Adversarial attacks to the input that changes the prediction of a network has been actively studied.
Revised: Adversarial attacks to the input that changes the prediction of a network have been actively studied.

Original: Szegedy et al.
Revised: Szegedy et al.

Original: (2013) demonstrated that it is relatively easy to fool neural networks into making very different predictions for test images
Revised: (2013) demonstrated that it is relatively easy to fool neural networks into making very different predictions for test images

Original: [Figure]
Revised: [Figure]

Original: that are visually very similar to each other.
Revised: that are visually very similar to each other.

Original: Goodfellow et al.
Revised: Goodfellow et al.

Original: (2014) introduced the Fast Gradient
Revised: (2014) introduced the Fast Gradient

Original: Sign Method (FGSM) as a one-step prediction attack.
Revised: Sign Method (FGSM) as a one-step prediction attack.

Original: This was followed by more effective iterative attacks (Kurakin et al., 2016) seeking to change the prediction of network by a small perturbation.
Revised: This was followed by more effective iterative attacks (Kurakin et al., 2016) seeking to change the prediction of network by a small perturbation.

Original: Different metrics for quantifying the size of the perturbation have been used.
Revised: Different metrics for quantifying the size of the perturbation have been used.

Original: Moosavi-Dezfooli et al.
Revised: Moosavi-Dezfooli et al.

Original: ; Szegedy et al.
Revised: ; Szegedy et al.

Original: (2013) used (cid:96) 2 ; Papernot et al.
Revised: (2013) used (cid:96) 2 ; Papernot et al.

Original: (2016) considered the number of perturbed pixels ( (cid:96) 0 ); and Goodfellow et al.
Revised: (2016) considered the number of perturbed pixels ( (cid:96) 0 ); and Goodfellow et al.

Original: (2014) suggest using (cid:96) ∞ , because this tightly controls how much individual feature can change.
Revised: (2014) suggest using (cid:96) ∞ , because this tightly controls how much individual feature can change.

Original: We followed the popular practice and evaluate with (cid:96) ∞ .
Revised: We followed the popular practice and evaluate with (cid:96) ∞ .

Original: Interpretation of neural network predictions is also an active research area.
Revised: Interpretation of neural network predictions is also an active research area.

Original: Post-hoc interpretability (Lipton, 2016) is one family of methods that seek to ”explain” the prediction without talking about the details of black-box model’s hidden mechanisms.
Revised: Post-hoc interpretability (Lipton, 2016) is one family of methods that seek to ”explain” the prediction without talking about the details of black-box model’s hidden mechanisms.

Original: These included tools to explain predictions by networks in terms of the features of the test example (Simonyan et al., 2013; Shrikumar et al., 2017; Sundararajan et al., 2017; Zhou et al., 2016), as well as in terms of contribution of training examples to the prediction at test time (Koh & Liang, 2017).
Revised: These included tools to explain predictions by networks in terms of the features of the test example (Simonyan et al., 2013; Shrikumar et al., 2017; Sundararajan et al., 2017; Zhou et al., 2016), as well as in terms of contribution of training examples to the prediction at test time (Koh & Liang, 2017).

Original: These interpretations have gained increasing popularity, as they confer a degree of insight to human users of what the neural network might be doing (Lipton, 2016).
Revised: These interpretations have gained increasing popularity, as they confer a degree of insight to human users of what the neural network might be doing (Lipton, 2016).

Original: Conclusion This paper demonstrates that interpretation of neural networks can be fragile.
Revised: Conclusion This paper demonstrates that interpretation of neural networks can be fragile.

Original: We develop new perturbations to illustrate this fragility and propose evaluation metrics as well as insights on why fragility occurs.
Revised: We develop new perturbations to illustrate this fragility and propose evaluation metrics as well as insights on why fragility occurs.

Original: Fragility of interpretation is orthogonal to fragility of the prediction—we demonstrate how perturbations can substantially change the interpretation without changing the predicted label.
Revised: Fragility of interpretation is orthogonal to fragility of the prediction—we demonstrate how perturbations can substantially change the interpretation without changing the predicted label.

Original: The two types of fragility do arise from similar factors, as we discuss in Section 5.
Revised: The two types of fragility do arise from similar factors, as we discuss in Section 5.

Original: Our focus is on the interpretation method, rather than on the original network, and as such we do not explore how interpretable is the original predictor.
Revised: Our focus is on the interpretation method, rather than on the original network, and as such we do not explore how interpretable is the original predictor.

Original: There is a separately line of research that tries to design simpler and more interpretable prediction models (Ba & Caruana, 2014).
Revised: There is a separately line of research that tries to design simpler and more interpretable prediction models (Ba & Caruana, 2014).

Original: Our main message is that robustness of the interpretation of a prediction is an important and challenging problem, especially as in many applications (e.g. many biomedical and social settings) users are as interested in the interpretation as in the prediction itself.
Revised: Our main message is that robustness of the interpretation of a prediction is an important and challenging problem, especially as in many applications (e.g. many biomedical and social settings) users are as interested in the interpretation as in the prediction itself.

Original: Our results raise concerns on how interpretation is sensitive to noise and can be manipulated.
Revised: Our results raise concerns on how interpretation is sensitive to noise and can be manipulated.

Original: We do not suggest that interpretations are meaningless, just as adversarial attacks on predictions do not imply that neural networks are useless.
Revised: We do not suggest that interpretations are meaningless, just as adversarial attacks on predictions do not imply that neural networks are useless.

Original: Interpretations do need to be used and evaluated with caution.
Revised: Interpretations do need to be used and evaluated with caution.

Original: Especially in settings where the importance of individual or a small subset of features are interpreted, we show that these importance scores can be sensitive to even random perturbation.
Revised: Especially in settings where the importance of individual or a small subset of features are interpreted, we show that these importance scores can be sensitive to even random perturbation.

Original: More dramatic manipulations of interpretations can be achieved with our targeted perturbations, which raise security concerns.
Revised: More dramatic manipulations of interpretations can be achieved with our targeted perturbations, which raise security concerns.

Original: While we focus on image data (ImageNet and CIFAR-10), because these are the standard benchmarks for popular interpretation tools, this fragility issue can be wide-spread in biomedical, economic and other settings where neural networks are increasingly used.
Revised: While we focus on image data (ImageNet and CIFAR-10), because these are the standard benchmarks for popular interpretation tools, this fragility issue can be wide-spread in biomedical, economic and other settings where neural networks are increasingly used.

Original: Understanding interpretation fragility in these applications and develop more robust methods are important agendas of research
Revised: Understanding interpretation fragility in these applications and develop more robust methods are important agendas of research

Original: A PPENDICES A D ESCRIPTION OF THE CIFAR 10 CLASSIFYING NETWORK
Revised: A PPENDICES A D ESCRIPTION OF THE CIFAR-10 CLASSIFICATION NETWORK

Original: We trained the following structure using ADAM optimizer (Kingma & Ba, 2014) with default parameters.
Revised: We trained the following structure using ADAM optimizer (Kingma & Ba, 2014) with default parameters.

Original: The resulting test accuracy using ReLU activation was 73%.
Revised: The resulting test accuracy using ReLU activation was 73%.

Original: For the experiment in section 6, we replaced ReLU activation with Softplus and retrained the network (with the ReLU network weights as initial weights).
Revised: For the experiment in section 6, we replaced ReLU activation with Softplus and retrained the network (with the ReLU network weights as initial weights).

Original: The resulting accuracy was 73%.
Revised: The resulting accuracy was 73%.

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Revised: Relu

Original: [Figure]
Revised: [Figure]

Original: B A DDITIONAL EXAMPLES OF FEATURE IMPORTANCE PERTURBATIONS
Revised: B A DDITIONAL EXAMPLES OF FEATURE IMPORTANCE PERTURBATIONS

Original: Here we provide three more examples from ImageNet.
Revised: Here we provide three more examples from ImageNet.

Original: For each example, all three methods of feature imoprtance are attacked by random sign noise and our two targeted adversarial algorithms.
Revised: For each example, all three methods of feature imoprtance are attacked by random sign noise and our two targeted adversarial algorithms.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ando
Revised: ando

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: r i g i na l
Revised: r i g i na l

Original: T opA tt a c k
Revised: T opA tt a c k

Original: C en t e r
Revised: C en t e r

Original: A tt a
Revised: A tt a

Original: [Figure]
Revised: [Figure]

Original: C C OMPLEMENTARY RESULTS
Revised: C M EASURING CENTER OF MASS MOVEMENT

Original: C ENTER S HIFT R ESULTS FOR I MAGENET

Original: [Figure]

Original: t e r-

Original: [Figure]

Original: h i ft

Original: [Figure]

Original: p i

Original: [Figure]

Original: e l

Original: [Figure]
Revised: [Figure]

Original: Simple Gradient
Revised: Simple Gradient

Original: [Figure]
Revised: [Figure]

Original: R ESULTS FOR ADVERSARIAL ATTACKSAGAINST CIFAR10 FEATURE IMPORTANCE METHODS
Revised: R ESULTS FOR ADVERSARIAL ATTACKS

Revised: AGAINST CIFAR-10 FEATURE IMPORTANCE METHODS

Original: [Figure]
Revised: [Figure]

Original: Simple Gradient
Revised: Simple Gradient

Original: [Figure]
Revised: [Figure]

Original: DeepLIFT
Revised: DeepLIFT

Original: [Figure]
Revised: [Figure]

Original: Integrated Gradients
Revised: Integrated Gradients

Original: [Figure]
Revised: [Figure]

Original: Center-shift attack moves the center of mass more than two other perturbations.
Revised: Center-shift attack moves the center of mass more than two other perturbations.

Original: Among different feature importance methods, integrated gradients is more robust than the two other methods.
Revised: Among different feature importance methods, integrated gradients is more robust than the two other methods.

Original: Additionally, results for CIFAR10 show that images in this data set are more robust against adversarial attack compared to ImageNet images which agrees with our analysis that higher dimensional inputs are tend to be more fragile.
Revised: Additionally, results for CIFAR10 show that images in this data set are more robust against adversarial attack compared to ImageNet images which agrees with our analysis that higher dimensional inputs are tend to be more fragile.

Original: D A DDITIONAL E XAMPLES OF A DVERSARIAL A TTACKS ON I NFLUENCE F UNCTIONS
Revised: D A DDITIONAL E XAMPLES OF A DVERSARIAL A TTACKS ON I NFLUENCE F UNCTIONS

Original: In this appendix, we provide additional examples of the fragility of inﬂuence functions, analogous to Fig. 4.
Revised: In this appendix, we provide additional examples of the fragility of inﬂuence functions, analogous to Fig. 4.

Original: The effect of gradient sign-based perturbations on inﬂuence varies.

Original: In some cases, the inﬂuences of the test image barely changes, but in most cases, the inﬂuence of the top images drops dramatically, and in some cases, as shown below, the inﬂuences even reverse!

Original: [Figure]
Revised: [Figure]

Original: Overall, the inﬂuences of the training images before and after the attack are uncorrelated.
Revised: Overall, the inﬂuences of the training images before and after the attack are uncorrelated.

Original: In this example, the perturbation has remarkably caused the training images to almost completely reverse in inﬂuence.
Revised: In this example, the perturbation has remarkably caused the training images to almost completely reverse in inﬂuence.

Original: Training images that had the most positive effect on prediction now have the most negative effects and the other way round.
Revised: Training images that had the most positive effect on prediction now have the most negative effects and the other way round.

Original: E D IMENSIONALITY -B ASED E XPLANATION FOR F RAGILITY OF I NFLUENCE F UNCTIONS
Revised: E D IMENSIONALITY -B ASED E XPLANATION FOR F RAGILITY OF I NFLUENCE F UNCTIONS

Original: Here, we demonstrate that increasing the dimension of the input of a simple neural network increases the fragility of that network with respect to inﬂuence functions, analogous to the calculations carried out for importance-feature methods in Section 5.
Revised: Here, we demonstrate that increasing the dimension of the input of a simple neural network increases the fragility of that network with respect to inﬂuence functions, analogous to the calculations carried out for importance-feature methods in Section 5.

Original: Recall that the inﬂuence of a training image z i =
Revised: Recall that the inﬂuence of a training image z i =

Original: ( x i , y i ) on a test image z
Revised: ( x i , y i ) on a test image z

Original: = ( x , y ) is given by:
Revised: = ( x , y ) is given by:

Original: [Equation]
Revised: [Equation]

Original: We restrict our attention to the term in (6) that is dependent on x , and denote it by J def = ∇ θ L .
Revised: We restrict our attention to the term in (6) that is dependent on x , and denote it by J def = ∇ θ L .

Original: J represents the inﬁnitesimal effect of each of the parameters in the network on the loss function evaluated at the test image.
Revised: J represents the inﬁnitesimal effect of each of the parameters in the network on the loss function evaluated at the test image.

Original: Now, let us calculate the change in this term due to a small perturbation in x → x + δ .
Revised: Now, let us calculate the change in this term due to a small perturbation in x → x + δ .

Original: The ﬁrstorder approximation for the change in J is equal to: ∇ x J · δ = ∇ θ ∇ x L · δ .
Revised: The ﬁrstorder approximation for the change in J is equal to: ∇ x J · δ = ∇ θ ∇ x L · δ .

Original: In particular, for the i th parameter, J i changes by ( ∇ θ ∇ x L · δ ) i and furthermore, the relative change is ( ∇ θ ∇ x L · δ ) i / ( ∇ θ L ) i .
Revised: In particular, for the i th parameter, J i changes by ( ∇ θ ∇ x L · δ ) i and furthermore, the relative change is ( ∇ θ ∇ x L · δ ) i / ( ∇ θ L ) i .

Original: For the simple network deﬁned in Section 5, this evaluates to (replacing θ with w for consistency of notation):
Revised: For the simple network deﬁned in Section 5, this evaluates to (replacing θ with w for consistency of notation):

Original: [Equation]
Revised: [Equation]

Original: where for simplicity, we have let deﬁned the loss to be L = g ( w (cid:62) x ) , the output of the network (e.g. this can be thought of as the error for data point that belongs to class 0).
Revised: where for simplicity, we have let deﬁned the loss to be L = g ( w (cid:62) x ) , the output of the network (e.g. this can be thought of as the error for data point that belongs to class 0).

Original: Furthermore, we have used g (cid:48) ( · ) and g (cid:48)(cid:48) ( · ) to refer to the ﬁrst and second derivatives of g ( · ) .
Revised: Furthermore, we have used g (cid:48) ( · ) and g (cid:48)(cid:48) ( · ) to refer to the ﬁrst and second derivatives of g ( · ) .

Original: Note that g (cid:48) ( w (cid:62) x ) and g (cid:48)(cid:48) ( w (cid:62) x ) do not scale with the dimensionality of x because x and w are generalized L 2 -normalized due to data preprocessing and weight decay regularization.
Revised: Note that g (cid:48) ( w (cid:62) x ) and g (cid:48)(cid:48) ( w (cid:62) x ) do not scale with the dimensionality of x because x and w are generalized L 2 -normalized due to data preprocessing and weight decay regularization.

Original: However, if we choose δ = (cid:15) sign ( w ) , then the relative change in the saliency grows with the dimension, since it is proportional to the L 1 -norm of w .
Revised: However, if we choose δ = (cid:15) sign ( w ) , then the relative change in the saliency grows with the dimension, since it is proportional to the L 1 -norm of w .

Original: F O RTHOGONALITY OF STEEPEST DIRECTIONS OF CHANGE IN SCORE AND FEATURE IMPORTANCE FUNCTIONS IN A S IMPLE T WO - LAYER NETWORK
Revised: F O RTHOGONALITY OF STEEPEST DIRECTIONS OF CHANGE IN SCORE AND FEATURE IMPORTANCE FUNCTIONS IN A S IMPLE T WO - LAYER NETWORK

Original: Consider a two layer neural network with activation function g ( . ) , input x ∈ R d , hidden vector u ∈ R h , and score function S ) , we have:
Revised: Consider a two layer neural network with activation function g ( . ) , input x ∈ R d , hidden vector u ∈ R h , and score function S ) , we have:

Original: [Equation]
Revised: [Equation]

Original: where w j = || w j || 2 ˆ w j .
Revised: where w j = || w j || 2 ˆ w j .

Original: We have:
Revised: We have:

Original: [Equation]
Revised: [Equation]

Original: Now for an input sample x perturbation δ , for the change in feature importance:
Revised: Now for an input sample x perturbation δ , for the change in feature importance:

Original: [Equation]
Revised: [Equation]

Original: which is equal to:
Revised: which is equal to:

Original: [Equation]
Revised: [Equation]

Original: We further assume that the input is high-dimensional so that h < d and for i (cid:54) =
Revised: We further assume that the input is high-dimensional so that h < d and for i (cid:54) =

Original: j we have w j · w i = 0 .
Revised: j we have w j · w i = 0 .

Original: For maximizing the (cid:96) 2 norm of saliency difference we have the following perturbation direction:
Revised: For maximizing the (cid:96) 2 norm of saliency difference we have the following perturbation direction:

Original: [Equation]
Revised: [Equation]

Original: where:
Revised: where:

Original: [Equation]
Revised: [Equation]

Original: comparing which to the direction of feature importance:
Revised: comparing which to the direction of feature importance:

Original: [Equation]
Revised: [Equation]

Original: we conclude that the two directions are not parallel unless g (cid:48) ( . )
Revised: we conclude that the two directions are not parallel unless g (cid:48) ( . )

Original: =
Revised: =

Original: g (cid:48)(cid:48) ( . ) which is not the case for many activation functions like Softplus, Sigmoid, etc.
Revised: g (cid:48)(cid:48) ( . ) which is not the case for many activation functions like Softplus, Sigmoid, etc.

Original: G D ESIGNING I NTERPRETABILITY -R OBUST N ETWORKS
Revised: G D ESIGNING I NTERPRETABILITY -R OBUST N ETWORKS

Original: The analyses and experiments in this paper have demonstrated that small perturbations in the input layers of deep neural networks can have large changes in the interpretations.
Revised: The analyses and experiments in this paper have demonstrated that small perturbations in the input layers of deep neural networks can have large changes in the interpretations.

Original: This is analogous to classical adversarial examples, whereby small perturbations in the input produce large changes in the prediction .
Revised: This is analogous to classical adversarial examples, whereby small perturbations in the input produce large changes in the prediction .

Original: In that setting, it has been proposed that the Lipschitz constant of the network be constrained during training to limit the effect of adversarial perturbations (Szegedy et al., 2013).
Revised: In that setting, it has been proposed that the Lipschitz constant of the network be constrained during training to limit the effect of adversarial perturbations (Szegedy et al., 2013).

Original: This has found some empirical success (Cisse et al., 2017).
Revised: This has found some empirical success (Cisse et al., 2017).

Original: Here, we propose an analogous method to upper-bound the change in interpretability of a neural network as a result of perturbations to the input.
Revised: Here, we propose an analogous method to upper-bound the change in interpretability of a neural network as a result of perturbations to the input.

Original: Speciﬁcally, consider a network with K layers, which takes as input a data point we denote as y 0 .
Revised: Speciﬁcally, consider a network with K layers, which takes as input a data point we denote as y 0 .

Original: The output of the i th layer is given by y
Revised: The output of the i th layer is given by y

Original: i +

Revised: i +

Original: =
Revised: =

Original: f i ( y i ) for i = 0 , 1 . . .
Revised: f i ( y i ) for i = 0 , 1 . . .

Original: K − 1 .
Revised: K − 1 .

Original: We deﬁne S def =
Revised: We deﬁne S def =

Original: f K − 1 ( f K − 2 ( . . .
Revised: f K − 1 ( f K − 2 ( . . .

Original: f 

Revised: f

Original: ( y 0 ) . . . ))
Revised: ( y 0 ) . . . ))

Original: to be the output (e.g. score for the correct class) of our network, and we are interested in designing a network whose gradient S (cid:48)
Revised: to be the output (e.g. score for the correct class) of our network, and we are interested in designing a network whose gradient S (cid:48)

Original: = ∇ y 0 S is relatively insensitive to perturbations in the input, as this corresponds to a network whose feature importances are robust.
Revised: = ∇ y 0 S is relatively insensitive to perturbations in the input, as this corresponds to a network whose feature importances are robust.

Original: A natural quantity to consider is the Lipschitz constant of S (cid:48) with respect to y 0 .
Revised: A natural quantity to consider is the Lipschitz constant of S (cid:48) with respect to y 0 .

Original: By the chain rule, the Lipschitz constant of S (cid:48) is
Revised: By the chain rule, the Lipschitz constant of S (cid:48) is

Original: [Equation]
Revised: [Equation]

Original: Now consider the function f i ( · ) , which maps y i to y i +1 .
Revised: Now consider the function f i ( · ) , which maps y i to y i +1 .

Original: In the simple case of the fully-connected network, which we consider here, f i ( y i ) =
Revised: In the simple case of the fully-connected network, which we consider here, f i ( y i ) =

Original: g i
Revised: g i

Original: ( W i y i ) , where g i is a non-linearity and W i are the trained weights for that layer.
Revised: ( W i y i ) , where g i is a non-linearity and W i are the trained weights for that layer.

Original: Thus, the Lipschitz constant of the i th partial derivative in (8) is the
Revised: Thus, the Lipschitz constant of the i th partial derivative in (8) is the

Original: Lipschitz constant of
Revised: Lipschitz constant of

Original: [Equation]
Revised: [Equation]

Original: which is upper-bounded by || W
Revised: which is upper-bounded by || W

Original: i || 2 · L ( g (cid:48)
Revised: i || 2 · L ( g (cid:48)

Original: i ( · )) , where || W || denotes the operator norm of W (its largest singular value) 5 .
Revised: i ( · )) , where || W || denotes the operator norm of W (its largest singular value) 5 .

Original: This suggests that a conservative upper ceiling for (8) is
Revised: This suggests that a conservative upper ceiling for (8) is

Original: [Equation]
Revised: [Equation]

Original: Because the Lipschitz constant of the non-linearities g (cid:48) i ( · ) are ﬁxed, this result suggests that a regularization based on the operator norms of the weights W i may allow us to train networks that are robust to attacks on feature importance.
Revised: Because the Lipschitz constant of the non-linearities g (cid:48) i ( · ) are ﬁxed, this result suggests that a regularization based on the operator norms of the weights W i may allow us to train networks that are robust to attacks on feature importance.

Revised: The calculations in this Appendix section is meant to be suggestive rather than conclusive, since in practice the Lipschitz bounds are rarely tight.