Original: R ETHINKING THE R OLE OF G RADIENT - BASED A TTRIBUTION M ETHODS FOR M ODEL I NTERPRETABILITY
Revised: R ETHINKING THE R OLE OF G RADIENT - BASED A TTRIBUTION M ETHODS FOR M ODEL I NTERPRETABILITY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Current methods for the interpretability of discriminative deep neural networks commonly rely on the model’s input-gradients, i.e., the gradients of the output logits w.r.t.
Revised: Current methods for the interpretability of discriminative deep neural networks commonly rely on the model’s input-gradients, i.e., the gradients of the output logits w.r.t.

Original: the inputs.
Revised: the inputs.

Original: The common assumption is that these input-gradients contain information regarding p θ ( y | x ) , the model’s discriminative capabilities, thus justifying their use for interpretability.
Revised: The common assumption is that these input-gradients contain information regarding p θ ( y | x ) , the model’s discriminative capabilities, thus justifying their use for interpretability.

Original: However, in this work we show that these input-gradients can be arbitrarily manipulated as a consequence of the shiftinvariance of softmax without changing the discriminative function.
Revised: However, in this work we show that these input-gradients can be arbitrarily manipulated as a consequence of the shiftinvariance of softmax without changing the discriminative function.

Original: This leaves an open question: if input-gradients can be arbitrary, why are they highly structured and explanatory in standard models?
Revised: This leaves an open question: if input-gradients can be arbitrary, why are they highly structured and explanatory in standard models?

Original: We investigate this by re-interpreting the logits of standard softmax-based classiﬁers as unnormalized log-densities of the data distribution and show that input-gradients can be viewed as gradients of a class-conditional density model p θ ( x | y )
Revised: We investigate this by re-interpreting the logits of standard softmax-based classiﬁers as unnormalized log-densities of the data distribution and show that input-gradients can be viewed as gradients of a class-conditional density model p θ ( x | y )

Original: implicit within the discriminative model.
Revised: implicit within the discriminative model.

Original: This leads us to hypothesize that the highly structured and explanatory nature of input-gradients may be due to the alignment of this class-conditional model p θ ( x | y ) with that of the ground truth data distribution p data ( x | y ) .
Revised: This leads us to hypothesize that the highly structured and explanatory nature of input-gradients may be due to the alignment of this class-conditional model p θ ( x | y ) with that of the ground truth data distribution p data ( x | y ) .

Original: We test this hypothesis by studying the effect of density alignment on gradient explanations.
Revised: We test this hypothesis by studying the effect of density alignment on gradient explanations.

Original: To achieve this density alignment, we use an algorithm called score-matching, and propose novel approximations to this algorithm to enable training large-scale models.
Revised: To achieve this density alignment, we use an algorithm called score-matching, and propose novel approximations to this algorithm to enable training large-scale models.

Original: Our experiments show that improving the alignment of the implicit density model with the data distribution enhances gradient structure and explanatory power while reducing this alignment has the opposite effect.
Revised: Our experiments show that improving the alignment of the implicit density model with the data distribution enhances gradient structure and explanatory power while reducing this alignment has the opposite effect.

Original: This also leads us to conjecture that unintended density alignment in standard neural network training may explain the highly structured nature of input-gradients observed in practice.
Revised: This also leads us to conjecture that unintended density alignment in standard neural network training may explain the highly structured nature of input-gradients observed in practice.

Original: Overall, our ﬁnding that input-gradients capture information regarding an implicit generative model implies that we need to re-think their use for interpreting discriminative models.
Revised: Overall, our ﬁnding that input-gradients capture information regarding an implicit generative model implies that we need to re-think their use for interpreting discriminative models.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Input-gradients, or gradients of outputs w.r.t.
Revised: Input-gradients, or gradients of outputs w.r.t.

Original: inputs, are commonly used for the interpretation of deep neural networks (Simonyan et al., 2013).
Revised: inputs, are commonly used for the interpretation of deep neural networks (Simonyan et al., 2013).

Original: For image classiﬁcation tasks, an input pixel with a larger input-gradient magnitude is attributed a higher ‘importance’ value, and the resulting maps are observed to agree with human intuition regarding which input pixels are important for the task at hand (Adebayo et al., 2018).
Revised: For image classiﬁcation tasks, an input pixel with a larger input-gradient magnitude is attributed a higher ‘importance’ value, and the resulting maps are observed to agree with human intuition regarding which input pixels are important for the task at hand (Adebayo et al., 2018).

Original: Quantitative studies (Samek et al., 2016; Shrikumar et al., 2017) also show that these importance estimates are meaningful in predicting model response to larger structured perturbations.
Revised: Quantitative studies (Samek et al., 2016; Shrikumar et al., 2017) also show that these importance estimates are meaningful in predicting model response to larger structured perturbations.

Original: These results suggest that input-gradients do indeed capture relevant information regarding the underlying model.
Revised: These results suggest that input-gradients do indeed capture relevant information regarding the underlying model.

Original: However in this work, we show that input-gradients can be arbitrarily manipulated using the shift-invariance of softmax without changing the underlying discriminative model, which calls into question the reliability of input-gradient based attribution methods for interpreting arbitrary black-box models.
Revised: However in this work, we show that input-gradients can be arbitrarily manipulated using the shift-invariance of softmax without changing the underlying discriminative model, which calls into question the reliability of input-gradient based attribution methods for interpreting arbitrary black-box models.

Original: Given that input-gradients can be arbitrarily structured, the reason for their highly structured and explanatory nature in standard pre-trained models is puzzling.
Revised: Given that input-gradients can be arbitrarily structured, the reason for their highly structured and explanatory nature in standard pre-trained models is puzzling.

Original: Why are input-gradients relatively wellbehaved when they can just as easily be arbitrarily structured, without affecting discriminative model performance?
Revised: Why are input-gradients relatively wellbehaved when they can just as easily be arbitrarily structured, without affecting discriminative model performance?

Original: What factors inﬂuence input-gradient structure in standard deep neural networks?
Revised: What factors inﬂuence input-gradient structure in standard deep neural networks?

Original: To answer these, we consider the connections made between softmax-based discriminative classiﬁers and generative models (Bridle, 1990; Grathwohl et al., 2020), made by viewing the logits of standard classiﬁers as un-normalized log-densities.
Revised: To answer these, we consider the connections made between softmax-based discriminative classiﬁers and generative models (Bridle, 1990; Grathwohl et al., 2020), made by viewing the logits of standard classiﬁers as un-normalized log-densities.

Original: This connection reveals an alternate interpretation of input-gradients, as representing the log-gradients of a class-conditional density model which is implicit within standard softmax-based deep models, which we shall call the implicit density model .
Revised: This connection reveals an alternate interpretation of input-gradients, as representing the log-gradients of a class-conditional density model which is implicit within standard softmax-based deep models, which we shall call the implicit density model .

Original: This connection compels us to consider the following hypothesis: perhaps input-gradients are highly structured because this implicit density model is aligned with the ‘ground truth’ class-conditional data distribution?
Revised: This connection compels us to consider the following hypothesis: perhaps input-gradients are highly structured because this implicit density model is aligned with the ‘ground truth’ class-conditional data distribution?

Original: The core of this paper is dedicated to testing the validity of this hypothesis, whether or not input-gradients do become more structured and explanatory if this alignment increases and vice versa.
Revised: The core of this paper is dedicated to testing the validity of this hypothesis, whether or not input-gradients do become more structured and explanatory if this alignment increases and vice versa.

Original: For the purpose of validating this hypothesis, we require mechanisms to increase or decrease the alignment between the implicit density model and the data distribution.
Revised: For the purpose of validating this hypothesis, we require mechanisms to increase or decrease the alignment between the implicit density model and the data distribution.

Original: To this end, we consider a generative modelling approach called score-matching, which reduces the density modelling problem to that of local geometric regularization.
Revised: To this end, we consider a generative modelling approach called score-matching, which reduces the density modelling problem to that of local geometric regularization.

Original: Hence by using score-matching, we are able to view commonly used geometric regularizers in deep learning as density modelling methods.
Revised: Hence by using score-matching, we are able to view commonly used geometric regularizers in deep learning as density modelling methods.

Original: In practice, the score-matching objective is known for being computationally expensive and unstable to train (Song & Ermon, 2019; Kingma & LeCun, 2010).
Revised: In practice, the score-matching objective is known for being computationally expensive and unstable to train (Song & Ermon, 2019; Kingma & LeCun, 2010).

Original: To this end, we also introduce approximations and regularizers which allow us to use score-matching on practical large-scale discriminative models.
Revised: To this end, we also introduce approximations and regularizers which allow us to use score-matching on practical large-scale discriminative models.

Original: The rest of the paper is organized as follows.
Revised: The rest of the paper is organized as follows.

Original: We show in § 2 that it is trivial to manipulate input-gradients of standard classiﬁers using the shift-invariance of softmax without affecting the discriminative model.
Revised: We show in § 2 that it is trivial to manipulate input-gradients of standard classiﬁers using the shift-invariance of softmax without affecting the discriminative model.

Original: In § 3 we state our main hypothesis and describe the details of score-matching, present a tractable approximation for the same that eliminates the need for expensive Hessian computations.
Revised: In § 3 we state our main hypothesis and describe the details of score-matching, present a tractable approximation for the same that eliminates the need for expensive Hessian computations.

Original: § 4 revisits other interpretability tools from a density modelling perspective.
Revised: § 4 revisits other interpretability tools from a density modelling perspective.

Original: Finally, § 5 presents experimental evidence for the validity of the hypothesis that improved alignment between the implicit density model and the data distribution can improve the structure and explanatory nature of input-gradients.
Revised: Finally, § 5 presents experimental evidence for the validity of the hypothesis that improved alignment between the implicit density model and the data distribution can improve the structure and explanatory nature of input-gradients.

Original:  I NPUT -G RADIENTS ARE NOT U NIQUE
Revised:  I NPUT -G RADIENTS ARE NOT U NIQUE

Original: In this section, we show that it is trivial to manipulate input-gradients of discriminative deep networks, using the well-known shift-invariance property of softmax.
Revised: In this section, we show that it is trivial to manipulate input-gradients of discriminative deep networks, using the well-known shift-invariance property of softmax.

Original: Here we shall make a distinction between two types of input-gradients: logit-gradients and loss-gradients .
Revised: Here we shall make a distinction between two types of input-gradients: logit-gradients and loss-gradients .

Original: While logit-gradients are gradients of the pre-softmax output of a given class w.r.t.
Revised: While logit-gradients are gradients of the pre-softmax output of a given class w.r.t.

Original: the input, loss-gradients are the gradients of the loss w.r.t.
Revised: the input, loss-gradients are the gradients of the loss w.r.t.

Original: the input.
Revised: the input.

Original: In both cases, we only consider outputs of a single class, usually the target class.
Revised: In both cases, we only consider outputs of a single class, usually the target class.

Original: Let x ∈ R D be a data point, which is the input for a neural network model f : R D → R C intended for classiﬁcation, which produces pre-softmax logits for C classes.
Revised: Let x ∈ R D be a data point, which is the input for a neural network model f : R D → R C intended for classiﬁcation, which produces pre-softmax logits for C classes.

Original: The cross-entropy loss function for some class 1 ≤ i ≤ C, i ∈ N corresponding to an input x is given by (cid:96) ( f ( x ) , i ) ∈ R + , which is shortened to (cid:96) i ( x ) for convenience.
Revised: The cross-entropy loss function for some class 1 ≤ i ≤ C, i ∈ N corresponding to an input x is given by (cid:96) ( f ( x ) , i ) ∈ R + , which is shortened to (cid:96) i ( x ) for convenience.

Original: Note that here the loss function subsumes the softmax function as well.
Revised: Note that here the loss function subsumes the softmax function as well.

Original: The logit-gradients are given by ∇ x f i ( x )
Revised: The logit-gradients are given by ∇ x f i ( x )

Original: ∈
Revised: ∈

Original: R D for class i , while loss-gradients are ∇ x (cid:96) i ( x ) ∈ R D .
Revised: R D for class i , while loss-gradients are ∇ x (cid:96) i ( x ) ∈ R D .

Original: Let the softmax function be p ( y = i | x ) =
Revised: Let the softmax function be p ( y = i | x ) =

Original: exp
Revised: exp

Original: ( f i ( x )) /
Revised: ( f i ( x )) /

Original: (cid:80) Cj =1 exp( f j ( x )) , which we denote as p i for simplicity.
Revised: (cid:80) Cj =1 exp( f j ( x )) , which we denote as p i for simplicity.

Original: Here, we make the observation that upon adding the same scalar function g to all logits, the logit-gradients can arbitrarily change but the loss values do not.
Revised: Here, we make the observation that upon adding the same scalar function g to all logits, the logit-gradients can arbitrarily change but the loss values do not.

Original: Observation.
Revised: Observation.

Original: Assume an arbitrary function g : R D
Revised: Assume an arbitrary function g : R D

Original: → R .
Revised: → R .

Original: Consider another neural network function given by ˜ f
Revised: Consider another neural network function given by ˜ f

Original: i ( · ) =
Revised: i ( · ) =

Original: f i ( · ) + g ( · ) , for 0 ≤ i ≤ C , for which we obtain ∇ x ˜ f
Revised: f i ( · ) + g ( · ) , for 0 ≤ i ≤ C , for which we obtain ∇ x ˜ f

Original: i
Revised: i

Original: · ) = ∇ x f i ( · ) + ∇ x g ( · ) .
Revised: · ) = ∇ x f i ( · ) + ∇ x g ( · ) .

Original: For this, the corresponding loss values and loss-gradients are unchanged, i.e.; ˜ (cid:96) i ( · ) =
Revised: For this, the corresponding loss values and loss-gradients are unchanged, i.e.; ˜ (cid:96) i ( · ) =

Original: (cid:96) i ( · ) and ∇ x ˜ (cid:96) i ( · ) =
Revised: (cid:96) i ( · ) and ∇ x ˜ (cid:96) i ( · ) =

Original: ∇ x (cid:96) i ( · ) as a consequence of the shift-invariance of softmax.
Revised: ∇ x (cid:96) i ( · ) as a consequence of the shift-invariance of softmax.

Original: This explains how the structure of logit-gradients can be arbitrarily changed: one simply needs to add an arbitrary function g to all logits.
Revised: This explains how the structure of logit-gradients can be arbitrarily changed: one simply needs to add an arbitrary function g to all logits.

Original: This implies that individual logit-gradients ∇ x f i ( x ) and logits f i ( x ) are meaningless on their own, and their structure may be uninformative regarding the underlying discriminative model.
Revised: This implies that individual logit-gradients ∇ x f i ( x ) and logits f i ( x ) are meaningless on their own, and their structure may be uninformative regarding the underlying discriminative model.

Original: Despite this, a large fraction of work in interpretable deep learning (Simonyan et al., 2013; Selvaraju et al., 2017; Smilkov et al., 2017; Fong & Vedaldi, 2017; Srinivas & Fleuret, 2019) uses individual logits and logit-gradients for saliency map computation.
Revised: Despite this, a large fraction of work in interpretable deep learning (Simonyan et al., 2013; Selvaraju et al., 2017; Smilkov et al., 2017; Fong et al., 2019; Srinivas & Fleuret, 2019) uses individual logits and logit-gradients for saliency map computation.

Original: We also provide a similar illustration in the supplementary material for the case of loss-gradients, where we show that it is possible for loss-gradients to diverge signiﬁcantly even when the loss values themselves do not.
Revised: We also provide a similar illustration in the supplementary material for the case of loss-gradients, where we show that it is possible for loss-gradients to diverge signiﬁcantly even when the loss values themselves do not.

Original: These simple observations leave an open question: why are input-gradients highly structured and explanatory when they can just as easily be arbitrarily structured, without affecting discriminative model performance?
Revised: These simple observations leave an open question: why are input-gradients highly structured and explanatory when they can just as easily be arbitrarily structured, without affecting discriminative model performance?

Original: Further, if input-gradients do not depend strongly on the underlying discriminative function, what aspect of the model do they depend on instead?
Revised: Further, if input-gradients do not depend strongly on the underlying discriminative function, what aspect of the model do they depend on instead?

Original: In the section that follows, we shall consider a generative modelling view of discriminative neural networks that offers insight into the information encoded by logit-gradients.
Revised: In the section that follows, we shall consider a generative modelling view of discriminative neural networks that offers insight into the information encoded by logit-gradients.

Original:  I MPLICIT D ENSITY M ODELS W ITHIN D ISCRIMINATIVE C LASSIFIERS
Revised:  I MPLICIT D ENSITY M ODELS W ITHIN D ISCRIMINATIVE C LASSIFIERS

Original: Let us consider the following link between generative models and the softmax function.
Revised: Let us consider the following link between generative models and the softmax function.

Original: We ﬁrst deﬁne the following joint density on the logits f i of classiﬁers: p θ ( x , y = i ) =
Revised: We ﬁrst deﬁne the following joint density on the logits f i of classiﬁers: p θ ( x , y = i ) =

Original: exp( f i ( x ; θ ))
Revised: exp( f i ( x ; θ ))

Original: Z ( θ ) , where Z ( θ ) is the partition function.
Revised: Z ( θ ) , where Z ( θ ) is the partition function.

Original: We shall henceforth suppress the dependence of f on θ for brevity.
Revised: We shall henceforth suppress the dependence of f on θ for brevity.

Original: Upon using
Revised: Upon using

Original: Bayes’ rule to obtain p θ ( y = i | x ) , we observe that we recover the standard softmax function.
Revised: Bayes’ rule to obtain p θ ( y = i | x ) , we observe that we recover the standard softmax function.

Original: Thus the logits of discriminative classiﬁers can alternately be viewed as un-normalized log-densities of the joint distribution.
Revised: Thus the logits of discriminative classiﬁers can alternately be viewed as un-normalized log-densities of the joint distribution.

Original: Assuming equiprobable classes, we have p θ ( x | y
Revised: Assuming equiprobable classes, we have p θ ( x | y

Original: = i ) =
Revised: = i ) =

Original: exp( f i ( x ))
Revised: exp( f i ( x ))

Original: Z ( θ ) /C , which is the quantity of interest for us.
Revised: Z ( θ ) /C , which is the quantity of interest for us.

Original: Thus while the logits represent un-normalized log-densities, logit-gradients represent the score function, i.e.; ∇ x log p θ ( x | y
Revised: Thus while the logits represent un-normalized log-densities, logit-gradients represent the score function, i.e.; ∇ x log p θ ( x | y

Original: = i ) =
Revised: = i ) =

Original: ∇ x f i ( x ) , which avoids dependence on the partition function Z ( θ ) as it is independent of x .
Revised: ∇ x f i ( x ) , which avoids dependence on the partition function Z ( θ ) as it is independent of x .

Original: This viewpoint naturally leads to the following hypothesis, that perhaps the reason for the highly structured and explanatory nature of input-gradients is that the implicit density model p θ ( x | y ) is close to that of the ground truth class-conditional data distribution p data ( x | y ) ?
Revised: This viewpoint naturally leads to the following hypothesis, that perhaps the reason for the highly structured and explanatory nature of input-gradients is that the implicit density model p θ ( x | y ) is close to that of the ground truth class-conditional data distribution p data ( x | y ) ?

Original: We propose to test this hypothesis explicitly using score-matching as a density modelling tool.
Revised: We propose to test this hypothesis explicitly using score-matching as a density modelling tool.

Original: Hypothesis.
Revised: Hypothesis.

Original: (Informal) Improved alignment of the implicit density model to the ground truth classconditional density model improves input-gradient interpretability via both qualitative and quantitative measures, whereas deteriorating this alignment has the opposite effect.
Revised: (Informal) Improved alignment of the implicit density model to the ground truth classconditional density model improves input-gradient interpretability via both qualitative and quantitative measures, whereas deteriorating this alignment has the opposite effect.

Original: 3. S CORE -M ATCHING
Revised: 3. S CORE -M ATCHING

Original: Score-matching (Hyv ¨ arinen, 2005) is a generative modelling objective that focusses solely on the derivatives of the log density instead of the density itself, and thus does not require access to the partition function Z ( θ ) .
Revised: Score-matching (Hyv ¨ arinen, 2005) is a generative modelling objective that focusses solely on the derivatives of the log density instead of the density itself, and thus does not require access to the partition function Z ( θ ) .

Original: Speciﬁcally, for our case we have ∇ x log p θ ( x | y
Revised: Speciﬁcally, for our case we have ∇ x log p θ ( x | y

Original: = i ) =
Revised: = i ) =

Original: ∇ x f i ( x ) , which are the logit-gradients.
Revised: ∇ x f i ( x ) , which are the logit-gradients.

Original: Given i.i.d.
Revised: Given i.i.d.

Original: samples X = { x i ∈ R D } from a latent data distribution p data ( x ) , the objective of generative modelling is to recover this latent distribution using only samples X .
Revised: samples X = { x i ∈ R D } from a latent data distribution p data ( x ) , the objective of generative modelling is to recover this latent distribution using only samples X .

Original: This is often done by training a parameterized distribution p θ ( x ) to align with the latent data distribution p data ( x ) .
Revised: This is often done by training a parameterized distribution p θ ( x ) to align with the latent data distribution p data ( x ) .

Original: The score-matching objective instead aligns the gradients of log densities, as given below.
Revised: The score-matching objective instead aligns the gradients of log densities, as given below.

Original: [Equation]
Revised: [Equation]

Original: The above relationship is proved (Hyv¨arinen, 2005) using integration by parts.
Revised: The above relationship is proved (Hyv¨arinen, 2005) using integration by parts.

Original: This is a consistent objective, i.e, J ( θ ) =
Revised: This is a consistent objective, i.e, J ( θ ) =

Original: ⇐⇒ p data = p θ .
Revised: ⇐⇒ p data = p θ .

Original: This approch is appealing also because this reduces the problem of generative modelling to that of regularization of the local geometry of functions, i.e.; the resulting terms only depend on the point-wise gradients and Hessian-trace.
Revised: This approch is appealing also because this reduces the problem of generative modelling to that of regularization of the local geometry of functions, i.e.; the resulting terms only depend on the point-wise gradients and Hessian-trace.

Original: 3. E FFICIENT ESTIMATION OF H ESSIAN - TRACE
Revised: 3. E FFICIENT ESTIMATION OF H ESSIAN - TRACE

Original: In general, equation 2 is intractable for high-dimensional data due to the Hessian trace term.
Revised: In general, equation 2 is intractable for high-dimensional data due to the Hessian trace term.

Original: To address this, we can use the Hutchinson’s trace estimator (Hutchinson, 1990) to efﬁciently compute an estimate of the trace by using random projections, which is given by: trace ( ∇ 2 x log p θ ( x ))
Revised: To address this, we can use the Hutchinson’s trace estimator (Hutchinson, 1990) to efﬁciently compute an estimate of the trace by using random projections, which is given by: trace ( ∇ 2 x log p θ ( x ))

Original: =
Revised: =

Original: E v ∼N (0 , I ) v T ∇ 2 x log p θ ( x ) v .
Revised: E v ∼N (0 , I ) v T ∇ 2 x log p θ ( x ) v .

Original: This estimator has been previously applied to score-matching (Song et al., 2019), and can be computed efﬁciently using Pearlmutter’s trick (Pearlmutter, 1994).
Revised: This estimator has been previously applied to score-matching (Song et al., 2019), and can be computed efﬁciently using Pearlmutter’s trick (Pearlmutter, 1994).

Original: However, this trick still requires two backward passes for a single monte-carlo sample, which is computationally expensive.
Revised: However, this trick still requires two backward passes for a single monte-carlo sample, which is computationally expensive.

Original: To further improve computational efﬁciency, we introduce the following approximation to Hutchinson’s estimator using a Taylor series expansion, which applies to small values of σ ∈ R .
Revised: To further improve computational efﬁciency, we introduce the following approximation to Hutchinson’s estimator using a Taylor series expansion, which applies to small values of σ ∈ R .

Original: [Equation]
Revised: [Equation]

Original: Note that equation 7 involves a difference of log probabilities, which is independent of the partition function.
Revised: Note that equation 7 involves a difference of log probabilities, which is independent of the partition function.

Original: For our case, log p θ ( x + v | y=
Revised: For our case, log p θ ( x + v | y

Original: [Equation]

Original:  − log p θ ( x | y=
Revised: = i ) − log p θ ( x | y

Original: [Equation] =
Revised: = i ) =

Original: f i ( x + v ) − f i ( x ) .
Revised: f i ( x + v ) − f i ( x ) .

Original: We have thus considerably simpliﬁed and speeded-up the computation of the Hessian trace term, which now can be approximated with no backward passes , but using only a single additional forward pass.
Revised: We have thus considerably simpliﬁed and speeded-up the computation of the Hessian trace term, which now can be approximated with no backward passes , but using only a single additional forward pass.

Original: We present details regarding the variance of this estimator in the supplementary material.
Revised: We present details regarding the variance of this estimator in the supplementary material.

Revised: A concurrent approach (Pang et al., 2020) also presents a similar algorithm, however it is applied primarily to

Revised: Noise Contrastive Score Networks (Song & Ermon, 2019) and Denoising Score Matching (Vincent, 2011), whereas we apply it to vanilla score-matching on discriminative models.

Original: 3. S TABILIZED S CORE - MATCHING
Revised: 3. S TABILIZED S CORE - MATCHING

Original: In practice, a naive application of score-matching objective is unstable, causing the Hessian-trace to collapse to negative inﬁnity.
Revised: In practice, a naive application of score-matching objective is unstable, causing the Hessian-trace to collapse to negative inﬁnity.

Original: This occurs because the ﬁnite-sample variant of equation 1 causes the model to ‘overﬁt’ to a mixture-of-diracs density, which places a dirac-delta distribution at every data point.
Revised: This occurs because the ﬁnite-sample variant of equation 1 causes the model to ‘overﬁt’ to a mixture-of-diracs density, which places a dirac-delta distribution at every data point.

Original: Gradients of such a distribution are undeﬁned, causing training to collapse.
Revised: Gradients of such a distribution are undeﬁned, causing training to collapse.

Original: To overcome this, regularized score-matching (Kingma & LeCun, 2010) and noise conditional score networks (Song &
Revised: To overcome this, regularized score-matching (Kingma & LeCun, 2010) and noise conditional score networks (Song &

Original: Ermon, 2019) propose to add noise to inputs for score-matching to make the problem well-deﬁned.
Revised: Ermon, 2019) propose to add noise to inputs for score-matching to make the problem well-deﬁned.

Original: However, this did not help for our case.
Revised: However, this did not help for our case.

Original: Instead, we use a heuristic where we add a small penalty term proportional to the square of the Hessian-trace.
Revised: Instead, we use a heuristic where we add a small penalty term proportional to the square of the Hessian-trace.

Original: This discourages the Hessian-trace becoming too large, and thus stabilizes training.
Revised: This discourages the Hessian-trace becoming too large, and thus stabilizes training.

Original:  I NTERPRETABILITY THROUGH THE L ENS OF  D ENSITY M ODELLING 
Revised:  I MPLICATIONS OF THE D ENSITY M ODELLING V IEWPOINT

Original: In the previous section we related input-gradients to the implicit density model, thus linking gradient interpretability to density modelling through our hypothesis.
Revised: In the previous section we related input-gradients to the implicit density model, thus linking gradient interpretability to density modelling through our hypothesis.

Original: In this section, we consider two other interpretability tools: activity maximization and the pixel perturbation test, and show how these can interpreted from a density modelling perspective.
Revised: In this section, we consider two other interpretability tools: activity maximization and the pixel perturbation test, and show how these can interpreted from a density modelling perspective.

Original: These perspectives also enable us to draw parallels between score-matching and adversarial training.
Revised: These perspectives also enable us to draw parallels between score-matching and adversarial training.

Original: 4. A CTIVITY M AXIMIZATION AS S AMPLING FROM THE I MPLICIT D ENSITY M ODEL
Revised: 4. A CTIVITY M AXIMIZATION AS S AMPLING FROM THE I MPLICIT D ENSITY M ODEL

Original: The canonical method to obtain samples from score-based generative models is via Langevin sampling (Welling & Teh, 2011; Song & Ermon, 2019), which involves performing gradient ascent on the density model with noise added to the gradients.
Revised: The canonical method to obtain samples from score-based generative models is via Langevin sampling (Welling & Teh, 2011; Song & Ermon, 2019), which involves performing gradient ascent on the density model with noise added to the gradients.

Original: Without this added noise, the algorithm recovers the modes of the density model.
Revised: Without this added noise, the algorithm recovers the modes of the density model.

Original: We observe that activity maximization algorithms used for neural network visualizations are remarkably similar to this scheme.
Revised: We observe that activity maximization algorithms used for neural network visualizations are remarkably similar to this scheme.

Original: For instance, Simonyan et al.
Revised: For instance, Simonyan et al.

Original: (2013) recover inputs which maximize the logits of neural networks, thus exactly recovering the modes of the implicit density model.
Revised: (2013) recover inputs which maximize the logits of neural networks, thus exactly recovering the modes of the implicit density model.

Original: Similarly, deep-dream-like methods (Mahendran & Vedaldi, 2016; Nguyen et al., 2016; Mordvintsev et al., 2015) extend this by using “image priors” to ensure that the resulting samples are closer to the distribution of natural images, and by adding structured noise to the gradients in the form of jitter, to obtain more visually pleasing samples.
Revised: Similarly, deep-dream-like methods (Mahendran & Vedaldi, 2016; Nguyen et al., 2016; Mordvintsev et al., 2015) extend this by using “image priors” to ensure that the resulting samples are closer to the distribution of natural images, and by adding structured noise to the gradients in the form of jitter, to obtain more visually pleasing samples.

Original: From the density modelling perspective, we can alternately view these visualization techniques as biased sampling methods for score-based density models trained on natural images.
Revised: From the density modelling perspective, we can alternately view these visualization techniques as biased sampling methods for score-based density models trained on natural images.

Original: However, given the fact that they draw samples from the implicit density model, their utility in interpreting discriminative models may be limited.
Revised: However, given the fact that they draw samples from the implicit density model, their utility in interpreting discriminative models may be limited.

Original: 4. P IXEL P ERTURBATION AS A D ENSITY R ATIO T EST
Revised: 4. P IXEL P ERTURBATION AS A D ENSITY R ATIO T EST

Original: A popular test for saliency map evaluation is based on pixel perturbation (Samek et al., 2016).
Revised: A popular test for saliency map evaluation is based on pixel perturbation (Samek et al., 2016).

Original: This involves ﬁrst selecting the least-relevant (or most-relevant) pixels according to a saliency map representation, ‘deleting’ those pixels and measuring the resulting change in output value.
Revised: This involves ﬁrst selecting the least-relevant (or most-relevant) pixels according to a saliency map representation, ‘deleting’ those pixels and measuring the resulting change in output value.

Original: Here, deleting a pixel usually involves replacing the pixel with a non-informative value such as a random or a ﬁxed constant value.
Revised: Here, deleting a pixel usually involves replacing the pixel with a non-informative value such as a random or a ﬁxed constant value.

Original: A good saliency method identiﬁes those pixels as less relevant whose deletion does not cause a large change in output value.
Revised: A good saliency method identiﬁes those pixels as less relevant whose deletion does not cause a large change in output value.

Original: We observe that this change in outputs criterion is identical to the density ratio, i.e. , log ( p θ ( x + v | y
Revised: We observe that this change in outputs criterion is identical to the density ratio, i.e. , log ( p θ ( x + v | y

Original: = i )
Revised: = i )

Original: /p
Revised: /p

Original: θ ( x | y = i ))
Revised: θ ( x | y = i ))

Original: =
Revised: =

Original: f i ( x + v ) − f i ( x ) .
Revised: f i ( x + v ) − f i ( x ) .

Original: Thus when logits are used for evaluating the change in outputs (Samek et al., 2016; Ancona et al., 2018), the pixel perturbation test exactly measures the density ratio between the perturbed image and the original image.
Revised: Thus when logits are used for evaluating the change in outputs (Samek et al., 2016; Ancona et al., 2018), the pixel perturbation test exactly measures the density ratio between the perturbed image and the original image.

Original: Thus if a perturbed image has a similar density to that of the original image under the implicit density model, then the saliency method that generated these perturbations is considered to be explanatory.
Revised: Thus if a perturbed image has a similar density to that of the original image under the implicit density model, then the saliency method that generated these perturbations is considered to be explanatory.

Original: Similarly,
Revised: Similarly,

Original: Fong & Vedaldi (2017) optimize over this criterion to identify pixels whose removal causes minimal change in logit activity, thus obtaining perturbed images with a high implicit density value similar to that of activity maximization.
Revised: Fong et al. (2019) optimize over this criterion to identify pixels whose removal causes minimal change in logit activity, thus obtaining perturbed images with a high implicit density value similar to that of activity maximization.

Original: Overall, this test captures sensitivity of the implicit density model, and not the underlying discriminative model which we wish to interpret.
Revised: Overall, this test captures sensitivity of the implicit density model, and not the underlying discriminative model which we wish to interpret.

Original: We thus recommend that the pixel perturbation test always be used in conjunction with either the change in output probabilities, or the change in the accuracy of classiﬁcation, rather than the change in logits.
Revised: We thus recommend that the pixel perturbation test always be used in conjunction with either the change in output probabilities, or the change in the accuracy of classiﬁcation, rather than the change in logits.

Original: 4. C ONNECTING S CORE -M ATCHING TO A DVERSARIAL T RAINING
Revised: 4. C ONNECTING S CORE -M ATCHING TO A DVERSARIAL T RAINING

Original: Recent works in adversarial machine learning (Etmann et al., 2019; Engstrom et al., 2019; Santurkar et al., 2019; Kaur et al., 2019; Ross & Doshi-Velez, 2017) have observed that saliency map structure and samples from activation maximization are more perceptually aligned for adversarially trained models than for standard models.
Revised: Recent works in adversarial machine learning (Etmann et al., 2019; Engstrom et al., 2019; Santurkar et al., 2019; Kaur et al., 2019; Ross & Doshi-Velez, 2017) have observed that saliency map structure and samples from activation maximization are more perceptually aligned for adversarially trained models than for standard models.

Original: However it is unclear from these works why this occurs.
Revised: However it is unclear from these works why this occurs.

Revised: Separate from this line of work, Chalasani et al.

Revised: (2018) also connect regularization of a variant of integrated gradients with adversarial training, suggesting a close interplay between the two.

Original: We notice that these properties are shared with score-matched models, or models trained such that the implicit density model is aligned with the ground truth.
Revised: We notice that these properties are shared with score-matched models, or models trained such that the implicit density model is aligned with the ground truth.

Original: Further, we note that both score-matching and adversarial training are often based on local geometric regularization, usually involving regularization of the gradient-norm (Ross & Doshi-Velez, 2017; Jakubovitz & Giryes, 2018), and training both the discriminative model and the implicit density model (Grathwohl et al., 2020) has been shown to improve adversarial robustness.
Revised: Further, we note that both score-matching and adversarial training are often based on local geometric regularization, usually involving regularization of the gradient-norm (Ross & Doshi-Velez, 2017; Jakubovitz & Giryes, 2018), and training both the discriminative model and the implicit density model (Grathwohl et al., 2020) has been shown to improve adversarial robustness.

Original: From these results, we can conjecture that training the implicit density model via score-matching may have similar outcomes as adversarial training, with score-matching imposing a stronger constraint.
Revised: From these results, we can conjecture that training the implicit density model via score-matching may have similar outcomes as adversarial training.

Original: We leave the veriﬁcation and proof of this conjecture to future work.
Revised: We leave the veriﬁcation and proof of this conjecture to future work.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we present experimental results to show the efﬁcacy of score-matching and the validation of the hypothesis that density alignment inﬂuences the gradient explanation quality.
Revised: In this section, we present experimental results to show the efﬁcacy of score-matching and the validation of the hypothesis that density alignment inﬂuences the gradient explanation quality.

Original: For experiments, we shall consider the CIFAR100 dataset.
Revised: For experiments, we shall consider the CIFAR100 dataset.

Original: We present experiments with CIFAR10 in the supplementary section.
Revised: We present experiments with CIFAR10 in the supplementary section.

Original: Unless stated otherwise, the network structure we use shall be a 18-layer
Revised: Unless stated otherwise, the network structure we use shall be a 18-layer

Original: ResNet that achieves 78.01% accuracy on CIFAR100, and the optimizer used shall be SGD with momentum.
Revised: ResNet that achieves 78.01% accuracy on CIFAR100, and the optimizer used shall be SGD with momentum.

Original: All models use the softplus non-linearity with β = 10 , which is necessary to ensure that the Hessian is non-zero for score-matching.
Revised: All models use the softplus non-linearity with β = 10 , which is necessary to ensure that the Hessian is non-zero for score-matching.

Original: Before proceeding with our experiments, we shall brieﬂy introduce the score-matching variants we shall be using for comparisons.
Revised: Before proceeding with our experiments, we shall brieﬂy introduce the score-matching variants we shall be using for comparisons.

Original: Score-Matching
Revised: Score-Matching

Original: We propose to use the score-matching objective as a regularizer in neural network training to increase the alignment of the implicit density model to the ground truth, as shown in equation 4, with the stability regularizer discussed in §3.3.
Revised: We propose to use the score-matching objective as a regularizer in neural network training to increase the alignment of the implicit density model to the ground truth, as shown in equation 4, with the stability regularizer discussed in §3.3.

Original: For this, we use a regularization constant λ = 1 e − 3 .
Revised: For this, we use a regularization constant λ = 1 e − 3 .

Original: This model achieves 72 .
Revised: This model achieves 72 .

Original: 20% accuracy on the test set, which is a drop of about 5 . 8% compared to the original model.
Revised: 20% accuracy on the test set, which is a drop of about 5 . 8% compared to the original model.

Original: [Equation]

Original: f ( x ) , i ) [Table] [Equation] [Table]
Revised: In the supplementary material, we perform a thorough hyper-parameter sweep and show that it is possible to obtain better performing models.

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation]
Revised: [Table] [Equation]

Original: [Table]
Revised: [Table]

Revised: 

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]

Original: Anti-score-matching We would like to have a tool that can decrease the alignment between the implicit density model and the ground truth.
Revised: Anti-score-matching We would like to have a tool that can decrease the alignment between the implicit density model and the ground truth.

Original: To enable this, we propose to maximize the hessiantrace, in an objective we call anti-score-matching .
Revised: To enable this, we propose to maximize the hessiantrace, in an objective we call anti-score-matching .

Original: For this, we shall use a the clamping function on hessian-trace, which ensures that its maximization stops after a threshold is reached.
Revised: For this, we shall use a the clamping function on hessian-trace, which ensures that its maximization stops after a threshold is reached.

Original: We use a threshold of τ = 1000 , and regularization constant λ = 1 e − 4 .
Revised: We use a threshold of τ = 1000 , and regularization constant λ = 1 e − 4 .

Original: This model achieves an accuracy of 74 .
Revised: This model achieves an accuracy of 74 .

Original: Gradient-Norm regularization We propose to use gradient-norm regularized models as another baseline for comparison, using a regularization constant of λ = 

Revised: Gradient-Norm regularization We propose to use gradient-norm regularized models as another baseline for comparison, using a regularization constant of λ =

Original: e − 3 .
Revised: e − 3 .

Original: This model achieves an accuracy of 76 .
Revised: This model achieves an accuracy of 76 .

Original: 5. E VALUATING THE E FFICACY OF S CORE -M ATCHING AND A NTI -S CORE -M ATCHING
Revised: 5. E VALUATING THE E FFICACY OF S CORE -M ATCHING AND A NTI -S CORE -M ATCHING

Original: Here we demonstrate that training with score-matching / anti-score-matching is possible, and that such training improves / deteriorates the quality of the implicit density models respectively as expected.
Revised: Here we demonstrate that training with score-matching / anti-score-matching is possible, and that such training improves / deteriorates the quality of the implicit density models respectively as expected.

Original: 5.1. D ENSITY R ATIOS
Revised: 5.1. D ENSITY R ATIOS

Original: One way to characterize the generative behaviour of models is to compute likelihoods on data points.
Revised: One way to characterize the generative behaviour of models is to compute likelihoods on data points.

Original: However this is intractable for high-dimensional problems, especially for un-normalized models.
Revised: However this is intractable for high-dimensional problems, especially for un-normalized models.

Original: We observe although that the densities p ( x | y
Revised: We observe although that the densities p ( x | y

Original: = i ) themselves are intractable, we can easily compute density ratios p ( x + η | y
Revised: = i ) themselves are intractable, we can easily compute density ratios p ( x + η | y

Original: = i ) /p
Revised: = i ) /p

Original: ( x | y
Revised: ( x | y

Original: = i ) =
Revised: = i ) =

Original: exp( f i ( x + η ) − f i ( x )) for a random noise variable η .
Revised: exp( f i ( x + η ) − f i ( x )) for a random noise variable η .

Original: Thus, we propose to plot the graph of density ratios locally along random directions.
Revised: Thus, we propose to plot the graph of density ratios locally along random directions.

Original: These can be thought of as local cross-sections of the density sliced at random directions.
Revised: These can be thought of as local cross-sections of the density sliced at random directions.

Original: We plot these values for gaussian noise η for different standard deviations, which are averaged across points in the entire dataset.[Figure] [Table]
Revised: We plot these values for gaussian noise η for different standard deviations, which are averaged across points in the entire dataset.

Original: [Figure]
Revised: [Figure]

Original: noise added to the input is better).

Original: We observe that score-matched model is robust to a larger range of noise values, while antiscore-matching is very sensitive even to small amounts of noise.

Original: In Figure 1, we plot the density ratios upon training on the CIFAR100 dataset.
Revised: In Figure 1, we plot the density ratios upon training on the CIFAR100 dataset.

Original: We observe that the baseline model assigns higher density values to noisy inputs than real inputs.
Revised: We observe that the baseline model assigns higher density values to noisy inputs than real inputs.

Original: With anti-score-matching, we observe that the density proﬁle grows still steeper, assigning higher densities to inputs with smaller noise.
Revised: With anti-score-matching, we observe that the density proﬁle grows still steeper, assigning higher densities to inputs with smaller noise.

Original: Gradient-norm regularized models and scorematched models improve on this behaviour, and are robust to larger amounts of noise added.
Revised: Gradient-norm regularized models and scorematched models improve on this behaviour, and are robust to larger amounts of noise added.

Original: Thus we are able to obtain penalty terms that can both improve and deteriorate the density modelling behaviour within models
Revised: Thus we are able to obtain penalty terms that can both improve and deteriorate the density modelling behaviour within discriminative models.

Original: 5.1. S AMPLE Q UALITY
Revised: 5.1. S AMPLE Q UALITY

Original: We are interested in recovering modes of our density models while having access to only the gradients of the log density.
Revised: We are interested in recovering modes of our density models while having access to only the gradients of the log density.

Original: For this purpose, we apply gradient ascent on the log probability log p ( x | y
Revised: For this purpose, we apply gradient ascent on the log probability log p ( x | y

Original: = i ) =
Revised: = i ) =

Original: f i ( x ) , similar to activity maximization.
Revised: f i ( x ) , similar to activity maximization.

Original: Our results are shown in Figure 2.
Revised: Our results are shown in Figure 2.

Original: We observe that samples from the score-matched and gradient-norm regularized models are signiﬁcantly less noisy than other models.
Revised: We observe that samples from the score-matched and gradient-norm regularized models are signiﬁcantly less noisy than other models.

Original:  We also propose to qualitatively measure the sample quality using the GAN-test approach (Shmelkov et al., 2018).
Revised: [Table] [Figure] We also propose to qualitatively measure the sample quality using the GAN-test approach (Shmelkov et al., 2018).

Original: This test proposes to measure the discriminative accuracy of generated samples via an independently trained discriminative model.
Revised: This test proposes to measure the discriminative accuracy of generated samples via an independently trained discriminative model.

Original: In contrast with more popular metrics such as the inception-score, this captures sample quality rather than diversity, which is what we are interested in.
Revised: In contrast with more popular metrics such as the inception-score, this captures sample quality rather than diversity, which is what we are interested in.

Original: We show the results in table 1, which conﬁrms the qualitative trend seen in samples above.
Revised: We show the results in table 1, which conﬁrms the qualitative trend seen in samples above.

Original: Surprisingly, we ﬁnd that gradient-norm regularized models perform better than score-matched models.
Revised: Surprisingly, we ﬁnd that gradient-norm regularized models perform better than score-matched models.

Original: This implies that such models are able to implicitly perform density modelling without being explicitly trained to do so.
Revised: This implies that such models are able to implicitly perform density modelling without being explicitly trained to do so.

Original: We leave further investigation of this phenomenon to future work.
Revised: We leave further investigation of this phenomenon to future work.

Original: [Table]

Original: [Figure]

Original: 5. E VALUATING THE E FFECT OF D ENSITY A LIGNMENT ON G RADIENT E XPLANATIONS
Revised: 5. E VALUATING THE E FFECT OF D ENSITY A LIGNMENT ON G RADIENT E XPLANATIONS

Original: Here we shall evaluate the gradient explanations of various models.
Revised: Here we shall evaluate the gradient explanations of various models.

Original: First, we shall look at quantitative results on a discriminative variant of the pixel perturbation test.
Revised: First, we shall look at quantitative results on a discriminative variant of the pixel perturbation test.

Original: Second, we visualize the gradient maps to assess qualitative differences between them.
Revised: Second, we visualize the gradient maps to assess qualitative differences between them.

Original: 5.2. Q UANTITATIVE R ESULTS ON D ISCRIMINATIVE P IXEL P ERTURBATION
Revised: 5.2. Q UANTITATIVE R ESULTS ON D ISCRIMINATIVE P IXEL P ERTURBATION

Original: As noted in 4.2, it is recommended to use the pixel perturbation test using accuracy changes, and we call this variant as discriminative pixel perturbation .
Revised: As noted in 4.2, it is recommended to use the pixel perturbation test using accuracy changes, and we call this variant as discriminative pixel perturbation .

Original: We select the least relevant pixels and replace them with the mean pixel value of the image, note down the accuracy of the model on the resulting samples.
Revised: We select the least relevant pixels and replace them with the mean pixel value of the image, note down the accuracy of the model on the resulting samples.

Original: We note that this test is only used so far to compare different saliency methods for the same underlying model.
Revised: We note that this test is only used so far to compare different saliency methods for the same underlying model.

Original: However, we here seek to compare saliency methods across models.
Revised: However, we here seek to compare saliency methods across models.

Original: For this we consider two experiments.
Revised: For this we consider two experiments.

Original: First, we perform the pixel perturbation experiment with each of the four trained models on their own input-gradients and plot the results in Figure 3a.
Revised: First, we perform the pixel perturbation experiment with each of the four trained models on their own input-gradients and plot the results in Figure 3a.

Original: These results indicate that the input-gradients of score-matched and gradient-norm regularized models are better equipped to identify least relevant pixels in this model.
Revised: These results indicate that the input-gradients of score-matched and gradient-norm regularized models are better equipped to identify least relevant pixels in this model.

Original: However, it is difﬁcult to completely disentangle the robustness beneﬁts of such score-matched models against improved identiﬁcation of less relevant pixels through such a plot.
Revised: However, it is difﬁcult to completely disentangle the robustness beneﬁts of such score-matched models against improved identiﬁcation of less relevant pixels through such a plot.

Original: To this end, we conduct a second experiment in Figure 3b, where we use input-gradients obtained from these four trained models to explain the same standard baseline ResNet model.
Revised: To this end, we conduct a second experiment in Figure 3b, where we use input-gradients obtained from these four trained models to explain the same standard baseline ResNet model.

Original: This disentangles the robustness of different models as inputs to the same model is perturbed in all cases.
Revised: This disentangles the robustness of different models as inputs to the same model is perturbed in all cases.

Original: Here also  we ﬁnd that gradients from score-matched and gradient-norm regularized models explain behavior of standard baseline models better than the gradients of the baseline model itself.
Revised: Here also [Figure] we ﬁnd that gradients from score-matched and gradient-norm regularized models explain behavior of standard baseline models better than the gradients of the baseline model itself.

Original: Together, these tests show that training with score-matching indeed produces input-gradients that quantitatively more explanatory than baseline models.
Revised: Together, these tests show that training with score-matching indeed produces input-gradients that quantitatively more explanatory than baseline models.

Original: 5.2. Q UALITATIVE G RADIENT V ISUALIZATIONS
Revised: 5.2. Q UALITATIVE G RADIENT V ISUALIZATIONS

Original: We visualize the structure of logit-gradients of different models in Figure 4.
Revised: We visualize the structure of logit-gradients of different models in Figure 4.

Original: We observe that gradientnorm regularized model and score-matched model have highly perceptually aligned gradients, when compared to the baseline and anti-score-matched gradients, corroborating the quantitative results.
Revised: We observe that gradientnorm regularized model and score-matched model have highly perceptually aligned gradients, when compared to the baseline and anti-score-matched gradients, corroborating the quantitative results.

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper [Figure] we investigated the cause for the highly structured and explanatory nature of inputgradients in standard pre-trained models, and showed that alignment of the implicit density model with the ground truth data density improves gradients explanations.
Revised: In this paper, we investigated the cause for the highly structured and explanatory nature of inputgradients in standard pre-trained models, and showed that alignment of the implicit density model with the ground truth data density improves gradients explanations.

Original: This density modelling interpretation enabled us to view canonical approaches in interpretability such as gradient-based saliency methods, activity maximization and the pixel perturbation test through a density modelling perspective, showing that these capture information relating to the implicit density model, not the underlying discriminative model which we wish to interpret.
Revised: This density modelling interpretation enabled us to view canonical approaches in interpretability such as gradient-based saliency methods, activity maximization and the pixel perturbation test through a density modelling perspective, showing that these capture information relating to the implicit density model, not the underlying discriminative model which we wish to interpret.

Original: This calls for a need to re-think the role of these tools in  interpretation of discriminative models.
Revised: This calls for a need to re-think the role of these tools in the interpretation of discriminative models.

Revised: For practitioners, we believe it is best to avoid usage of gradient-based tools for interpretability.

Revised: If unavoidable, it is recommended to use only gradient-norm regularized or score-matched models, as input-gradients of these models produce more reliable estimates of the gradient of the underlying distribution.

Revised: As our experiments show, these may be a useful tool even though they are not directly related to the discriminative model.

Original: However, our work still does not answer the question of why pre-trained models may have their implicit density models aligned with ground truth in the ﬁrst place.
Revised: However, our work still does not answer the question of why pre-trained models may have their implicit density models aligned with ground truth in the ﬁrst place.

Original: To this end, recent work (Barrett & Dherin, 2020) has independently shown that there is an implicit gradient regularization due to the training dynamics of gradient descent.
Revised: To this end, recent work (Barrett & Dherin, 2020) has independently shown that there is an implicit gradient regularization due to the training dynamics of gradient descent.

Original: If such implicit gradient regularization also holds for large models and stochastic optimizers used in practice, then this offers an explanation for the phenomenon.
Revised: If such implicit gradient regularization also holds for large models and stochastic optimizers used in practice, then this offers an explanation for the phenomenon.

Original: Another open question is to understand why gradient-norm regularized models are able to perform implicit density modelling as observed in our experiments in § 5.1.2, which lead to improved gradient explanations.
Revised: Another open question is to understand why gradient-norm regularized models are able to perform implicit density modelling as observed in our experiments in § 5.1.2, which lead to improved gradient explanations

Original: Appendix A A NOTE ABOUT OTHER F OOLING METHODS
Revised: Appendix A A NOTE ABOUT OTHER F OOLING METHODS

Original: Our work is connected to the literature around unreliability of saliency methods.
Revised: Our work is connected to the literature around unreliability of saliency methods.

Original: While most such works consider how the explanations for nearly identical images can be arbitrarily different (Dombrowski et al., 2019; Subramanya et al., 2019; Zhang et al., 2020; Ghorbani et al., 2019), our work considers how one may change the model itself to yield arbitrary explanations without affecting discriminative performance, similar to Heo et al.
Revised: While most such works consider how the explanations for nearly identical images can be arbitrarily different (Dombrowski et al., 2019; Subramanya et al., 2019; Zhang et al., 2020; Ghorbani et al., 2019), our work considers how one may change the model itself to yield arbitrary explanations without affecting discriminative performance, similar to Heo et al.

Original: While Heo et al.
Revised: While Heo et al.

Original: (2019) show this experimentally, we provide an analytical reason for why this happens relating to the shift-invariance of softmax.
Revised: (2019) show this experimentally, we provide an analytical reason for why this happens relating to the shift-invariance of softmax.

Original: B F OOLING G RADIENTS IS SIMPLE
Revised: B F OOLING G RADIENTS IS SIMPLE

Original: Observation.
Revised: Observation.

Original: Assume an arbitrary function g : R D
Revised: Assume an arbitrary function g : R D

Original: → R .
Revised: → R .

Original: Consider another neural network function given by ˜ f
Revised: Consider another neural network function given by ˜ f

Original: i ( · ) =
Revised: i ( · ) =

Original: f i ( · ) + g ( · ) , for 0 ≤ i ≤ C , for which we obtain ∇ x ˜ f
Revised: f i ( · ) + g ( · ) , for 0 ≤ i ≤ C , for which we obtain ∇ x ˜ f

Original: i
Revised: i

Original: · ) = ∇ x f i ( · ) + ∇ x g ( · ) .
Revised: · ) = ∇ x f i ( · ) + ∇ x g ( · ) .

Original: For this, the corresponding loss values and loss-gradients are unchanged, i.e.; ˜ (cid:96) i ( · ) =
Revised: For this, the corresponding loss values and loss-gradients are unchanged, i.e.; ˜ (cid:96) i ( · ) =

Original: (cid:96) i ( · ) and ∇ x ˜ (cid:96) i ( · ) =
Revised: (cid:96) i ( · ) and ∇ x ˜ (cid:96) i ( · ) =

Original: ∇ x (cid:96) i ( · ) .
Revised: ∇ x (cid:96) i ( · ) .

Original: Proof.
Revised: Proof.

Original: The following expressions relate the loss and neural network function outputs, for the case of cross-entropy loss and usage of the softmax function.
Revised: The following expressions relate the loss and neural network function outputs, for the case of cross-entropy loss and usage of the softmax function.

Original: [Equation]
Revised: [Equation]

Original: Upon replacing f i with ˜ f
Revised: Upon replacing f i with ˜ f

Original: i = f
Revised: i = f

Original: i + g , the proof follows.
Revised: i + g , the proof follows.

Original: B. M ANIPULATING L OSS -G RADIENTS
Revised: B. M ANIPULATING L OSS -G RADIENTS

Original: Here, we show how we can also change loss-gradients arbitrarily without signiﬁcantly changing the loss values themselves.
Revised: Here, we show how we can also change loss-gradients arbitrarily without signiﬁcantly changing the loss values themselves.

Original: In this case, the trick is to add a high frequency low amplitude sine function to the loss.
Revised: In this case, the trick is to add a high frequency low amplitude sine function to the loss.

Original: Observation.
Revised: Observation.

Original: Consider g ( x ) = (cid:15) sin( m x ) , and ˜ (cid:96) i ( x ) =
Revised: Consider g ( x ) = (cid:15) sin( m x ) , and ˜ (cid:96) i ( x ) =

Original: (cid:96) i ( x ) + g ( x ) , for (cid:15), m ≥ R + and x ∈ R D .
Revised: (cid:96) i ( x ) + g ( x ) , for (cid:15), m ≥ R + and x ∈ R D .

Original: Then, it is easy to see that | ˜ (cid:96) i ( x ) − (cid:96) i ( x ) | ≤ (cid:15) , and (cid:107)∇ x ˜ (cid:96) i ( x ) − ∇ x (cid:96) i ( x ) (cid:107) 1 ≤ m × (cid:15) × D .
Revised: Then, it is easy to see that | ˜ (cid:96) i ( x ) − (cid:96) i ( x ) | ≤ (cid:15) , and (cid:107)∇ x ˜ (cid:96) i ( x ) − ∇ x (cid:96) i ( x ) (cid:107) 1 ≤ m × (cid:15) × D .

Original: Thus two models with losses differing by some small (cid:15) can have gradients differing by m × (cid:15) × D .
Revised: Thus two models with losses differing by some small (cid:15) can have gradients differing by m × (cid:15) × D .

Original: For m → ∞ and a ﬁxed (cid:15) , the gradients can diverge signiﬁcantly.
Revised: For m → ∞ and a ﬁxed (cid:15) , the gradients can diverge signiﬁcantly.

Original: Thus, loss-gradients are also unreliable, as two models with very similar loss landscapes and hence discriminative abilities, can have drastically different loss-gradients.
Revised: Thus, loss-gradients are also unreliable, as two models with very similar loss landscapes and hence discriminative abilities, can have drastically different loss-gradients.

Original: This simple illustration highlights the fact that gradients of high-dimensional black-box models are not well-behaved in general, and this depends on both the model smoothness and the high-dimensionality of the inputs.
Revised: This simple illustration highlights the fact that gradients of high-dimensional black-box models are not well-behaved in general, and this depends on both the model smoothness and the high-dimensionality of the inputs.

Original: Further, loss values and loss-gradients for highly conﬁdent samples are close to zero.
Revised: Further, loss values and loss-gradients for highly conﬁdent samples are close to zero.

Original: Thus any external noise added (due to stochastic training, for instance) can easily dominate the loss-gradient terms even when smoothness conditions (small m ) are enforced.
Revised: Thus any external noise added (due to stochastic training, for instance) can easily dominate the loss-gradient terms even when smoothness conditions (small m ) are enforced.

Original: C S CORE -M ATCHING A PPROXIMATION
Revised: C S CORE -M ATCHING A PPROXIMATION

Original: We consider the approximation derived for the estimator of the Hessian trace, which is ﬁrst derived from Hutchinson’s trace estimator Hutchinson (1990).
Revised: We consider the approximation derived for the estimator of the Hessian trace, which is ﬁrst derived from Hutchinson’s trace estimator Hutchinson (1990).

Original: We replace log p θ ( x ) terms used in the main text with f ( x ) terms here for clarity.
Revised: We replace log p θ ( x ) terms used in the main text with f ( x ) terms here for clarity.

Original: The Taylor series trick for approximating the Hessian-trace is given below.
Revised: The Taylor series trick for approximating the Hessian-trace is given below.

Original: [Equation]
Revised: [Equation]

Original: As expected, the approximation error vanishes in the limit of small σ .
Revised: As expected, the approximation error vanishes in the limit of small σ .

Original: Let us now consider the ﬁnite sample variants of this estimator, with N samples.
Revised: Let us now consider the ﬁnite sample variants of this estimator, with N samples.

Original: We shall call this the Taylor Trace Estimator .
Revised: We shall call this the Taylor Trace Estimator .

Original: [Equation]
Revised: [Equation]

Original: We shall henceforth suppress the dependence on i for brevity.
Revised: We shall henceforth suppress the dependence on i for brevity.

Original: For this estimator, we can compute its variance for quadratic functions f , where higher-order Taylor expansion terms are zero.
Revised: For this estimator, we can compute its variance for quadratic functions f , where higher-order Taylor expansion terms are zero.

Original: We make the following observation.
Revised: We make the following observation.

Original: Observation.
Revised: Observation.

Original: For quadratic functions f , the variance of the Taylor Trace Estimator is greater than the variance of the Hutchinson estimator by an amount at most equal to 4 σ − 2 (cid:107)∇ x f ( x ) (cid:107) 2 .
Revised: For quadratic functions f , the variance of the Taylor Trace Estimator is greater than the variance of the Hutchinson estimator by an amount at most equal to 4 σ − 2 (cid:107)∇ x f ( x ) (cid:107) 2 .

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Thus we have decomposed the variance of the overall estimator into two terms: the ﬁrst captures the variance of the Taylor approximation
Revised: Thus we have decomposed the variance of the overall estimator into two terms: the ﬁrst captures the variance of the Taylor approximation

Original: [Equation]
Revised: [Equation]

Original: and the second captures the variance of the Hutchinson estimator
Revised: and the second captures the variance of the Hutchinson estimator

Original: [Equation]
Revised: [Equation]

Original: Considering only the ﬁrst term
Revised: Considering only the ﬁrst term

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e.; the variance of the Taylor approximation
Revised: e.; the variance of the Taylor approximation

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: The intermediate steps involve expanding the summation, noticing that pairwise terms cancel, and applying the
Revised: The intermediate steps involve expanding the summation, noticing that pairwise terms cancel, and applying the

Original: Thus we have a trade-off: a large σ results in lower estimator variance but a large Taylor approximation error, whereas the opposite is true for small σ .
Revised: Thus we have a trade-off: a large σ results in lower estimator variance but a large Taylor approximation error, whereas the opposite is true for small σ .

Original: However for functions with small gradient norm, both the estimator variance and Taylor approximation error is small for small σ .
Revised: However for functions with small gradient norm, both the estimator variance and Taylor approximation error is small for small σ .

Original: We note that when applied to score-matching Hyv¨arinen (2005), the gradient norm of the function is also minimized.
Revised: We note that when applied to score-matching Hyv¨arinen (2005), the gradient norm of the function is also minimized.

Original: This implies that in practice, the gradient norm of the function is likely to be low, thus resulting in a small estimator variance even for small σ .
Revised: This implies that in practice, the gradient norm of the function is likely to be low, thus resulting in a small estimator variance even for small σ .

Original: The variance of the Hutchinson estimator is given below for reference Hutchinson (1990); Avron & Toledo (2011):
Revised: The variance of the Hutchinson estimator is given below for reference Hutchinson (1990); Avron & Toledo (2011):

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: D E VALUATING E FFECT OF S CORE -M ATCHING ON G RADIENT E XPLANATIONS ( ON CIFAR10)
Revised: D E VALUATING E FFECT OF S CORE -M ATCHING ON G RADIENT E XPLANATIONS ( ON CIFAR10)

Original: We repeat the pixel perturbation experiments on the CIFAR10 dataset and we observe similar qualitative trends.
Revised: We repeat the pixel perturbation experiments on the CIFAR10 dataset and we observe similar qualitative trends.

Original: In both cases, we observe that score-matched and gradient norm regularized models have more explanatory gradients, while anti-score-matched model contains the least explanatory gradients.
Revised: In both cases, we observe that score-matched and gradient norm regularized models have more explanatory gradients, while anti-score-matched model contains the least explanatory gradients.

Original: We also present visualization results of input-gradients of various models for reference.
Revised: We also present visualization results of input-gradients of various models for reference.

Original: [Figure]
Revised: [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure]
Revised: We present results on a hyper-parameter sweep on the λ and µ parameters of score-matching, where we provide both test-set accuracy on CIFAR100 and the corresponding GAN-test scores. We ﬁnd upon performing a hyper-parameter sweep that λ = 1 e − 5 and µ = 1 e − 3 seems to perform the best, whereas in the main paper we present results for λ = 1 e − 3 and µ = 1 e − 4 . It is possible that changing the training schedule by increasing the number of epochs or learning rate may further improve these results

Original: [Figure]
Revised: [Table]

Revised: but we did not explore that here.

Original: [Figure]
Revised: [Table]