Original: S TATISTICALLY C ONSISTENT S ALIENCY E STIMATION
Revised: S TATISTICALLY C ONSISTENT S ALIENCY E STIMATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The use of deep learning for a wide range of data problems has increased the need for understanding and diagnosing these models, and deep learning interpretation techniques have become an essential tool for data analysts.
Revised: The use of deep learning for a wide range of data problems has increased the need for understanding and diagnosing these models, and deep learning interpretation techniques have become an essential tool for data analysts.

Original: Although numerous model interpretation methods have been proposed in recent years, most of these procedures are based on heuristics with little or no theoretical guarantees.
Revised: Although numerous model interpretation methods have been proposed in recent years, most of these procedures are based on heuristics with little or no theoretical guarantees.

Original: In this work, we propose a statistical framework for saliency estimation for black box computer vision models.
Revised: In this work, we propose a statistical framework for saliency estimation for black box computer vision models.

Original: We build a model-agnostic estimation procedure that is statistically consistent and passes the saliency checks of Adebayo et al.
Revised: We build a model-agnostic estimation procedure that is statistically consistent and passes the saliency checks of Adebayo et al.

Original: (2018b).
Revised: (2018b).

Original: Our method requires solving a linear program, whose solution can be efﬁciently computed in polynomial time.
Revised: Our method requires solving a linear program, whose solution can be efﬁciently computed in polynomial time.

Original: Through our theoretical analysis, we establish an upper bound on the number of model evaluations needed to recover the region of importance with high probability, and build a new perturbation scheme for estimation of local gradients that is shown to be more efﬁcient than the commonly used random perturbation schemes.
Revised: Through our theoretical analysis, we establish an upper bound on the number of model evaluations needed to recover the region of importance with high probability, and build a new perturbation scheme for estimation of local gradients that is shown to be more efﬁcient than the commonly used random perturbation schemes.

Original: Validity of the new method is demonstrated through sensitivity analysis.
Revised: Validity of the new method is demonstrated through sensitivity analysis.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep learning models have achieved great predictive performance in many tasks.
Revised: Deep learning models have achieved great predictive performance in many tasks.

Original: However, these complex, often un-tractable models are difﬁcult to interpret and understand.
Revised: However, these complex, often un-tractable models are difﬁcult to interpret and understand.

Original: This lack of interpretability is a major barrier for their wide adoption, especially in domains (e.g., medicine) where models need to be qualitatively understood and/or veriﬁed for robustness.
Revised: This lack of interpretability is a major barrier for their wide adoption, especially in domains (e.g., medicine) where models need to be qualitatively understood and/or veriﬁed for robustness.

Original: In order to address these issues, several interpretation approaches have been proposed in the last few years.
Revised: In order to address these issues, several interpretation approaches have been proposed in the last few years.

Original: A group of methods are based on visualizations, either by quantifying the effect of particular neurons or features, or by creating new images that maximize the target score for speciﬁc classes (Erhan et al., 2009; Simonyan et al., 2013; Zeiler & Fergus, 2014).
Revised: A group of methods are based on visualizations, either by quantifying the effect of particular neurons or features, or by creating new images that maximize the target score for speciﬁc classes (Erhan et al., 2009; Simonyan et al., 2013; Zeiler & Fergus, 2014).

Original: A large collection of the techniques build saliency maps by attributing the gradients of the neural network to the input image through various procedures or by ﬁnding perturbations that signiﬁcantly change the output(Springenberg et al., 2014; Bach et al., 2015; Montavon et al., 2017; Shrikumar et al., 2017;
Revised: A large collection of the techniques build saliency maps by attributing the gradients of the neural network to the input image through various procedures or by ﬁnding perturbations that signiﬁcantly change the output(Springenberg et al., 2014; Bach et al., 2015; Montavon et al., 2017; Shrikumar et al., 2017;

Original: Zhou et al., 2016;
Revised: Zhou et al., 2016;

Original: Selvaraju et al., 2017; Smilkov et al., 2017; Fong & Vedaldi, 2017; Adebayo et al., 2018a; Dumitru et al., 2018; Singla et al., 2019).
Revised: Selvaraju et al., 2017; Smilkov et al., 2017; Fong & Vedaldi, 2017; Adebayo et al., 2018a; Dumitru et al., 2018; Singla et al., 2019).

Original: Another class of approaches treat the deep learner as a black-box.
Revised: Another class of approaches treat the deep learner as a black-box.

Original: In this domain, Baehrens et al.
Revised: In this domain, Baehrens et al.

Original: (2010) use a Parzen window classiﬁer to approximate the target classiﬁer locally.
Revised: (2010) use a Parzen window classiﬁer to approximate the target classiﬁer locally.

Original: Ribeiro et al.
Revised: Ribeiro et al.

Original: (2016) propose the LIME procedure, where small perturbations on the instance are used to obtain additional samples with which a sparse linear model is ﬁt.
Revised: (2016) propose the LIME procedure, where small perturbations on the instance are used to obtain additional samples with which a sparse linear model is ﬁt.

Original: Lundberg & Lee (2017) propose SHapley
Revised: Lundberg & Lee (2017) propose SHapley

Original: Additive exPlanation(SHAP), which combines the Shapley value from the game theory with the additive feature attribution methods.
Revised: Additive exPlanation(SHAP), which combines the Shapley value from the game theory with the additive feature attribution methods.

Original: They also make connections of the SHAP procedure with various existing methods including LRP, LIME and DeepLIFT.
Revised: They also make connections of the SHAP procedure with various existing methods including LRP, LIME and DeepLIFT.

Original: Chen et al.
Revised: Chen et al.

Original: propose L- and
Revised: propose L- and

Original: C-Shapley procedures which can reliably approximate the Shapley values in linear time with respect to the number of features.
Revised: C-Shapley procedures which can reliably approximate the Shapley values in linear time with respect to the number of features.

Original: Majority of the listed methods are heuristics which are constructed according to certain desirable qualities.
Revised: Majority of the listed methods are heuristics which are constructed according to certain desirable qualities.

Original: For these methods, it is not clear what the main estimand is, if it can be consistently estimated or if (and how) the estimand can be computed more efﬁciently.
Revised: For these methods, it is not clear what the main estimand is, if it can be consistently estimated or if (and how) the estimand can be computed more efﬁciently.

Original: In fact, according to the recent research by Adebayo et al. (2018b), most methods with great visual inspection lack sensitivity to the model and the data generating process.
Revised: In fact, according to the recent research by Adebayo et al. (2018b), most methods with great visual inspection lack sensitivity to the model and the data generating process.

Original: Theoretical explanation for why guided back-propagation and deconvolutional methods perform image recovery is provided by Nie et al.
Revised: Theoretical explanation for why guided back-propagation and deconvolutional methods perform image recovery is provided by Nie et al.

Original: In this work, we propose a statistically valid technique for model-agnostic saliency estimation, and prove its consistency under reasonable assumptions.
Revised: In this work, we propose a statistically valid technique for model-agnostic saliency estimation, and prove its consistency under reasonable assumptions.

Original: Furthermore, our method passes the sanity checks given by Adebayo et al.
Revised: Furthermore, our method passes the sanity checks given by Adebayo et al.

Original: (2018b).
Revised: (2018b).

Original: Through our analysis, we obtain insights into how to improve the accuracy and reliability of our approach.
Revised: Through our analysis, we obtain insights into how to improve the accuracy and reliability of our approach.

Original: We note that there is recent work by Burns et al.
Revised: We note that there is recent work by Burns et al.

Original: (2019) where they provide a saliency estimation technique with theoretical guarantees – more speciﬁcally, FDR control.
Revised: (2019) where they provide a saliency estimation technique with theoretical guarantees – more speciﬁcally, FDR control.

Original: Although their procedure is very promising from a statistical perspective, and theoretically valid under a very general set of assumptions, their technique requires human input and has a signiﬁcant computational load as it uses a generative model for ﬁlling in certain regions of the target image.
Revised: Although their procedure is very promising from a statistical perspective, and theoretically valid under a very general set of assumptions, their technique requires human input and has a signiﬁcant computational load as it uses a generative model for ﬁlling in certain regions of the target image.

Original: Our main contributions are as follows:
Revised: Our main contributions are as follows:

Original: • We introduce a new saliency estimation framework for CNNs and propose a new method based on input perturbation.
Revised: • We introduce a new saliency estimation framework for CNNs and propose a new method based on input perturbation.

Original: Our procedure requires solving a linear program, and hence the estimates can be computed very efﬁciently.
Revised: Our procedure requires solving a linear program, and hence the estimates can be computed very efﬁciently.

Original: Furthermore, the optimization problem can be recast as a “parametric simplex” (Vanderbei, 2014), which allows the computation of the full solution path in an expedient manner.
Revised: Furthermore, the optimization problem can be recast as a “parametric simplex” (Vanderbei, 2014), which allows the computation of the full solution path in an expedient manner.

Original: • We establish conditions under which the signiﬁcant pixels in the input can be identiﬁed with high probability.
Revised: • We establish conditions under which the signiﬁcant pixels in the input can be identiﬁed with high probability.

Original: We present ﬁnite-sample convergence rates that can be used to determine the number of necessary model evaluations.
Revised: We present ﬁnite-sample convergence rates that can be used to determine the number of necessary model evaluations.

Original: We ﬁnd that the noise distribution for the perturbation has a substantial effect on the convergence rate.
Revised: We ﬁnd that the noise distribution for the perturbation has a substantial effect on the convergence rate.

Original: We propose a new perturbation scheme which uses a highly correlated Gaussian, instead of the widely used independent Gaussian distribution.
Revised: We propose a new perturbation scheme which uses a highly correlated Gaussian, instead of the widely used independent Gaussian distribution.

Original: In the following section, we deﬁne the linearly estimated gradient (LEG), which is the saliency parameter of interest (i.e. the estimand), and introduce our statistical framework.
Revised: In the following section, we deﬁne the linearly estimated gradient (LEG), which is the saliency parameter of interest (i.e. the estimand), and introduce our statistical framework.

Original: In section 3, we propose a regularized estimation procedure for LEG that penalizes the anisotropic total-variation.
Revised: In section 3, we propose a regularized estimation procedure for LEG that penalizes the anisotropic total-variation.

Original: We provide our theoretical results in Section 4 and the result of our numerical comparisons in Section 5.
Revised: We provide our theoretical results in Section 4 and the result of our numerical comparisons in Section 5.

Original: 1. N OTATION
Revised: 1. N OTATION

Original: For a matrix B , we use vec ( B ) and vec − 1 ( B ) to denote its vectorization and inverse vectorization, respectively.
Revised: For a matrix B , we use vec ( B ) and vec − 1 ( B ) to denote its vectorization and inverse vectorization, respectively.

Original: The transpose of a matrix B is given by B T and we use B + for its pseudo-inverse .
Revised: The transpose of a matrix B is given by B T and we use B + for its pseudo-inverse .

Original: The largest and smallest eigenvalue of a symmetric matrix B are denoted by λ max ( B ) and λ min ( B ) .
Revised: The largest and smallest eigenvalue of a symmetric matrix B are denoted by λ max ( B ) and λ min ( B ) .

Original: For a set S , we use S C to denote its complement.
Revised: For a set S , we use S C to denote its complement.

Original: For a vector u ∈ R p and a set S ⊆ [1 , . . .
Revised: For a vector u ∈ R p and a set S ⊆ [1 , . . .

Original: , p ] , we use u S to refer to its components indexed by elements in S .
Revised: , p ] , we use u S to refer to its components indexed by elements in S .

Original: The q -norm for a vector u is given by (cid:107) u (cid:107) q
Revised: The q -norm for a vector u is given by (cid:107) u (cid:107) q

Original: and we use (cid:107) B (cid:107)
Revised: and we use (cid:107) B (cid:107)

Original: Fr for the Frobenius norm of a matrix B .
Revised: Fr for the Frobenius norm of a matrix B .

Original: The vector of size p whose values are all equal to 1 is denoted by 1 p .
Revised: The vector of size p whose values are all equal to 1 is denoted by 1 p .

Original: Finally, for a continuous distribution F , we use F + x 0 to denote a distribution that is mean-shifted by x 0 , i.e. F ( z ) =
Revised: Finally, for a continuous distribution F , we use F + x 0 to denote a distribution that is mean-shifted by x 0 , i.e. F ( z ) =

Original: G ( z − x 0 ) for all z , where G = F + x 0 .
Revised: G ( z − x 0 ) for all z , where G = F + x 0 .

Original:  L INEARLY E STIMATED G RADIENT
Revised:  L INEARLY E STIMATED G RADIENT

Original: In gradient based saliency approaches, the main goal is to recover the gradient of the deep learner with respect to the input.
Revised: In gradient based saliency approaches, the main goal is to recover the gradient of the deep learner with respect to the input.

Original: More speciﬁcally, let f ( x ) be a deep learner, f : X → [0 , 1] , where
Revised: More speciﬁcally, let f ( x ) be a deep learner, f : X → [0 , 1] , where

Original: X is the input space, e.g., [0 , 255] 28 × 28 for the MNIST dataset, where the input are given asby 28 sized images.
Revised: X is the input space, e.g., [0 , 255] 28 × 28 for the MNIST dataset, where the input are given asby 28 sized images.

Original: In this notation, the output is the probability of a speciﬁc class, for instance
Revised: In this notation, the output is the probability of a speciﬁc class, for instance

Original: P model ( x is a 9 ) ; although this can be modiﬁed to check for comparative quantities by setting the output as
Revised: P model ( x is a 9 ) ; although this can be modiﬁed to check for comparative quantities by setting the output as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Then, local saliency is deﬁned as the derivative of f ( · ) with respect to the input, evaluated at a point of interest x 0 ∈ X , i.e. ∇ f ( x ) | x = x 0 .
Revised: Then, local saliency is deﬁned as the derivative of f ( · ) with respect to the input, evaluated at a point of interest x 0 ∈ X , i.e. ∇ f ( x ) | x = x 0 .

Original: However, in practice, local saliency is often too noisy and one instead uses an average of the gradient around x 0 (Shrikumar et al., 2017; Smilkov et al., 2017).
Revised: However, in practice, local saliency is often too noisy and one instead uses an average of the gradient around x 0 (Shrikumar et al., 2017; Smilkov et al., 2017).

Original: In order to study the saliency procedure from a statistical perspective, we start by deﬁning an estimand, whose deﬁnition is motivated by the LIME procedure (Ribeiro et al., 2016).
Revised: In order to study the saliency procedure from a statistical perspective, we start by deﬁning an estimand, whose deﬁnition is motivated by the LIME procedure (Ribeiro et al., 2016).

Original: Deﬁnition 1 (LEG) .
Revised: Deﬁnition 1 (LEG) .

Original: For a continuous distribution F , an initial point x 0 ∈ X with X ⊂ R p 1 × p 2 , and a function f : X → [ − 1 , 1] , the linearly estimated gradient (LEG), γ ∈ R p 1 × p 2 is given by
Revised: For a continuous distribution F , an initial point x 0 ∈ X with X ⊂ R p 1 × p 2 , and a function f : X → [ − 1 , 1] , the linearly estimated gradient (LEG), γ ∈ R p 1 × p 2 is given by

Original: [Equation]
Revised: [Equation]

Original: LEG is based on a ﬁrst order Taylor series expansion of the function f ( x ) around the point of interest x 0 .
Revised: LEG is based on a ﬁrst order Taylor series expansion of the function f ( x ) around the point of interest x 0 .

Original: The estimand is a proxy for the local gradient, and is the coefﬁcient that gives the best linear approximation, in terms of the squared error, among all possible choices.
Revised: The estimand is a proxy for the local gradient, and is the coefﬁcient that gives the best linear approximation, in terms of the squared error, among all possible choices.

Original: The distribution F determines the range of points the analyst wants to consider.
Revised: The distribution F determines the range of points the analyst wants to consider.

Original: We visually demonstrate LEG on two toy examples with a single pixel (i.e. p 1 = p 2 = 1 ) in Figure 1.
Revised: We visually demonstrate LEG on two toy examples with a single pixel (i.e. p 1 = p 2 = 1 ) in Figure 1.

Original: [Figure]
Revised: [Figure]

Original: We note that the variance of F has a large effect on LEG.
Revised: We note that the variance of F has a large effect on LEG.

Original: As F converges to a point mass at 0, if f ( x ) is twice continuously differentiable in the neighborhood of x 0 , then γ → ∇ x f ( x ) .
Revised: As F converges to a point mass at 0, if f ( x ) is twice continuously differentiable in the neighborhood of x 0 , then γ → ∇ x f ( x ) .

Original: On the other hand, if F has high variance, then samples from x 0 + F are very different from x 0 and the gradients might no longer be useful for  model interpretation.
Revised: On the other hand, if F has high variance, then samples from x 0 + F are substantially different from xand LEG might no longer be useful for interpreting the model at x 0 .

Original: This phenomenon can also described in terms of local vs global interpretation: for F with a small variance, LEG provides a very local interpretation, i.e. a gradient that is valid in a small neighborhood around x 0 , and as the variance of F increases, LEG produces a more global interpretation, since a larger neighborhood around x 0 is considered in the calculation.
Revised: This phenomenon can also described in terms of local vs global interpretation: for F with a small variance, LEG provides a very local interpretation, i.e. a gradient that is valid in a small neighborhood around x 0 , and as the variance of F increases, LEG produces a more global interpretation, since a larger neighborhood around x 0 is considered in the calculation.

Original: LEG has an analytical solution as the next lemma shows.
Revised: LEG has an analytical solution as the next lemma shows.

Original: Lemma 1.
Revised: Lemma 1.

Original: Let Z be the random variable with a centered distribution F , i.e. Z ∼ F and E [ Z ] = 0  .
Revised: Let Z be the random variable with a centered distribution F , i.e. Z ∼ F and E [ Z ] = 0 p 1 × p 2 .

Original: Assume that covariance of vec ( Z ) exists, and is positive-deﬁnite.
Revised: Assume that covariance of vec ( Z ) exists, and is positive-deﬁnite.

Original: Let Σ =
Revised: Let Σ =

Original: Cov ( vec ( Z )) , then
Revised: Cov ( vec ( Z )) , then

Original: [Equation]
Revised: [Equation]

Original: Proof of the lemma is provided in the Appendix.
Revised: Proof of the lemma is provided in the Appendix.

Original: Lemma 1 shows that the LEG can be written as an afﬁne transformation of a high dimensional integral where the integrand is (cid:82) ( f ( x 0 + z ) − f ( x 0 ))
Revised: Lemma 1 shows that the LEG can be written as an afﬁne transformation of a high dimensional integral where the integrand is (cid:82) ( f ( x 0 + z ) − f ( x 0 ))

Original: zF ( z ) dz .
Revised: zF ( z ) dz .

Original: This analysis also suggests an empirical estimate for the LEG, by replacing the expectation with the empirical mean.
Revised: This analysis also suggests an empirical estimate for the LEG, by replacing the expectation with the empirical mean.

Original: The empirical mean can be obtained by sampling x from F + x 0 , calculating f ( x ) , and then applying Lemma 1.
Revised: The empirical mean can be obtained by sampling x from F + x 0 , calculating f ( x ) , and then applying Lemma 1.

Original: More formally, let x 1 , . . .
Revised: More formally, let x 1 , . . .

Original: , x n be random samples from F + x 0 , and let y 1 , . .
Revised: , x n be random samples from F + x 0 , and let y 1 , . .

Original: , y n be the function evaluations with y i =
Revised: , y n be the function evaluations with y i =

Original: f
Revised: f

Original: ( x i ) .
Revised: ( x i ) .

Original: Further, let ˜ y
Revised: Further, let ˜ y

Original: i =
Revised: i =

Original: f ( x i ) − f ( x 0 ) and z i =
Revised: f ( x i ) − f ( x 0 ) and z i =

Original: x i − x 0 .
Revised: x i − x 0 .

Original: Then, the empirical
Revised: Then, the empirical

Original: LEG estimate is given by
Revised: LEG estimate is given by

Original: [Equation]
Revised: [Equation]

Original: As the function f ( x ) is bounded and F has a positive-deﬁnite covariance matrix, then it follows that as n → ∞ , ˆ γ → γ .
Revised: As the function f ( x ) is bounded and F has a positive-deﬁnite covariance matrix, then it follows that as n → ∞ , ˆ γ → γ .

Original: However, classical linear model theory (Ravishanker & Dey, 2001) shows that rate of the convergence is very slow, on the order of 1 λ min (Σ) (cid:112) p 1 p 2 /n , where p 1 and p 2 are the dimensions of X .
Revised: However, classical linear model theory (Ravishanker & Dey, 2001) shows that rate of the convergence is very slow, on the order of 1 λ min (Σ) (cid:112) p 1 p 2 /n , where p 1 and p 2 are the dimensions of X .

Original: This severely limits the practicality of the empirical approach.
Revised: This severely limits the practicality of the empirical approach.

Original: In the next section we propose to use regularization in order to obtain faster convergence rates.
Revised: In the next section we propose to use regularization in order to obtain faster convergence rates.

Original:  E FFICIENT E STIMATION OF LEG
Revised:  E FFICIENT E STIMATION OF LEG

Original: For interpretation of image classiﬁers, one expects that the saliency scores are located at a certain region, i.e. a contiguous body or a union of such bodies.
Revised: For interpretation of image classiﬁers, one expects that the saliency scores are located at a certain region, i.e. a contiguous body or a union of such bodies.

Original: This idea has lead to various procedures that estimate saliency scores by penalizing the local differences of the solution, often utilizing some form of the total variation (TV) penalty (Fong & Vedaldi, 2017).
Revised: This idea has lead to various procedures that estimate saliency scores by penalizing the local differences of the solution, often utilizing some form of the total variation (TV) penalty (Fong & Vedaldi, 2017).

Original: The approach is very sensible from a practical point of view: Firstly, it produces estimates that are easy to interpret as the important regions can be easily identiﬁed; secondly, penalization signiﬁcantly shrinks the variance of the estimate and helps produce reliable solutions with less model evaluations.
Revised: The approach is very sensible from a practical point of view: Firstly, it produces estimates that are easy to interpret as the important regions can be easily identiﬁed; secondly, penalization signiﬁcantly shrinks the variance of the estimate and helps produce reliable solutions with less model evaluations.

Original: In the light of the above, we propose to estimate the LEG coefﬁcient with an anisotropic L 1 TV penalty.
Revised: In the light of the above, we propose to estimate the LEG coefﬁcient with an anisotropic L 1 TV penalty.

Original: Deﬁnition 2 (LEG-TV) .
Revised: Deﬁnition 2 (LEG-TV) .

Original: For a hyperparameter, L ≥ 0 , the TV-penalized LEG estimate is given as ˜ γ = vec − 1 ( g ) where g is the solution of the following linear program 
Revised: For a hyperparameter, L ≥ 0 , the TV-penalized LEG estimate is given as ˜ γ = vec − 1 ( g ) where g is the solution of the following linear program [Equation] s.t [Equation]

Revised: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: where D ∈ R (2 p 1 p 2 − p 1 − p 2 ) × ( p 1 p 2 ) is the differencing matrix with D i,j = 1 , D i,k = − 1 if the j th and the k th component of g are connected on the two dimensional grid.
Revised: where D ∈ R (2 p 1 p 2 − p 1 − p 2 ) × ( p 1 p 2 ) is the differencing matrix with D i,j = 1 , D i,k = − 1 if the j th and the k th component of g are connected on the two dimensional grid.

Original: Our approach is based on the “high conﬁdence set” approach which has been successful in numerous applications in high dimensional statistics (Candes et al., 2007;
Revised: Our method is based on the “high conﬁdence set” approach which has been successful in numerous applications in high dimensional statistics (Candes et al., 2007;

Original: Cai et al., 2011; Fan, 2013).
Revised: Cai et al., 2011; Fan, 2013).

Original: The set of g that satisfy the constraint in the formulation is our high conﬁdence set; if L is chosen properly, this set contains the true LEG coefﬁcient, γ ( f, x 0 , F ) , with high probability.
Revised: The set of g that satisfy the constraint in the formulation is our high conﬁdence set; if L is chosen properly, this set contains the true LEG coefﬁcient, γ ( f, x 0 , F ) , with high probability 1 .

Original: This setup ensures that the distance between γ and ˜ γ is small.
Revised: This setup ensures that the distance between γ and ˜ γ is small.

Original: When combined with the TV penalty in the objective function, the procedure seeks to ﬁnd a solution that both belongs to the conﬁdence set and has sparse differences on the grid.
Revised: When combined with the TV penalty in the objective function, the procedure seeks to ﬁnd a solution that both belongs to the conﬁdence set and has sparse differences on the grid.

Original: Thus, the estimator is extremely effective at recovering γ that have small total variation.
Revised: Thus, the estimator is extremely effective at recovering γ that have small total variation.

Original: The proposed method enjoys low computational complexity as it is a linear program.
Revised: The proposed method enjoys low computational complexity.

Original: The problem in equation 4  can be solved in polynomial time, for instance by using a primal-dual interior-point method, for which the time complexity is O (cid:0) ( p 1 p 2 ) 3 . 5 (cid:1) (Nocedal & Wright, 2006).
Revised: The problem in equation 4 is a linear program and can be solved in polynomial time, for instance by using a primal-dual interior-point method, for which the time complexity is O (cid:0) ( p 1 p 2 ) 3 . 5 (cid:1) (Nocedal & Wright, 2006).

Original: However, in practice, solutions can be obtained much faster using simplex solvers.
Revised: However, in practice, solutions can be obtained much faster using simplex solvers.

Original: For instance, in our implementations, we use MOSEK, a commercial grade simplex solver by ApS (2019), and are able to obtain a solution in less than 2 seconds on a standard 8-core PC for a problem of size p 1 = p 2 = 28 .
Revised: For instance, in our implementations, we use MOSEK, a commercial grade simplex solver by ApS (2019), and are able to obtain a solution in less than 2 seconds on a standard 8-core PC for a problem of size p 1 = p 2 = 28 .

Original: Additionally, the alternative formulation (provided in the Appendix) can be solved using parametric simplex approaches which yield the whole solution path in L (Vanderbei, 2014).
Revised: Additionally, the alternative formulation (provided in the Appendix) can be solved using parametric simplex approaches which yield the whole solution path in L (Vanderbei, 2014).

Original: The last point is often a necessity in deployment when L needs to be tuned according to some criteria.
Revised: The last point is often a necessity in deployment when L needs to be tuned according to some criteria.

Original: We note that the procedure does not require any knowledge about the underlying neural network and is completely model-agnostic.
Revised: We note that the procedure does not require any knowledge about the underlying neural network and is completely model-agnostic.

Original: In fact, in applications where security or privacy could be a concern and returning multiple prediction values needs to be avoided, the term given by (cid:80) ni =1 vec (˜ y
Revised: In fact, in applications where security or privacy could be a concern and returning multiple prediction values needs to be avoided, the term given by (cid:80) ni =1 vec (˜ y

Original: i z i ) can be computed on the side and supplied alongside the prediction.
Revised: i z i ) can be computed on the side and supplied alongside the prediction.

Original: In Figure 2, we show the resulting estimates of the method with n = 500 model evaluations for a VGG-19 (Simonyan & Zisserman, 2014) network.
Revised: In Figure 2, we show the resulting estimates of the method with n = 500 model evaluations for a VGG-19 (Simonyan & Zisserman, 2014) network.

Original: For the distribution F , we use a multivariate
Revised: For the distribution F , we use a multivariate

Original: Gaussian distribution with the proposed perturbation scheme in Section 4.2.
Revised: Gaussian distribution with the proposed perturbation scheme in Section 4.2.

Original: We compute ˜ γ separately for each channel, and then sum the absolute values of the different channels to obtain the ﬁnal saliency score.
Revised: We compute ˜ γ separately for each channel, and then sum the absolute values of the different channels to obtain the ﬁnal saliency score.

Original: [Figure]
Revised: [Figure]

Original:  T HEORETICAL A NALYSIS AND I MPLEMENTATION
Revised:  T HEORETICAL A NALYSIS AND I MPLEMENTATION

Original: In this section, we analyze the procedure from a theoretical perspective and derive ﬁnite sample convergence rates of the proposed LEG-TV estimator.
Revised: In this section, we analyze the procedure from a theoretical perspective and derive ﬁnite sample convergence rates of the proposed LEG-TV estimator.

Original: As we noted earlier, this analysis also gives us insight on the properties of the ideal perturbation distribution.
Revised: As we noted earlier, this analysis also gives us insight on the properties of the ideal perturbation distribution.

Original: 4. C ONSISTENCY
Revised: 4. C ONSISTENCY

Original: We ﬁrst present our condition, which has a major role in the convergence rate of our estimator.
Revised: We ﬁrst present our condition, which has a major role in the convergence rate of our estimator.

Original: The condition is akin to the restricted eigenvalue condition (Bickel et al., 2009) with adjustments speciﬁc to our problem.
Revised: The condition is akin to the restricted eigenvalue condition (Bickel et al., 2009) with adjustments speciﬁc to our problem.

Original: Assumption 1.
Revised: Assumption 1.

Original: Let D + be the pseudo-inverse of the differencing matrix D , and denote the elements of singular value decomposition of D as U, Θ , V where D = U Θ V T .
Revised: Let D + be the pseudo-inverse of the differencing matrix D , and denote the elements of singular value decomposition of D as U, Θ , V where D = U Θ V T .

Original: Furthermore, denote the last p 1 p 2 − p 1 − p 2 columns of U that correspond to zero singular values as U 2 .
Revised: Furthermore, denote the last p 1 p 2 − p 1 − p 2 columns of U that correspond to zero singular values as U 2 .

Original: For the covariance matrix Σ , and any set S with size s , it holds that κ > 0 , where
Revised: For the covariance matrix Σ , and any set S with size s , it holds that κ > 0 , where

Original: [Equation]
Revised: [Equation]

Original: The following theorem is our main result.
Revised: The following theorem is our main result.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let γ ∗ = γ ( f, x 0 , F ) and Σ = Cov ( vec ( Z )) with Z ∼ F .
Revised: Let γ ∗ = γ ( f, x 0 , F ) and Σ = Cov ( vec ( Z )) with Z ∼ F .

Original: Let ˜ γ be the LEG-TV estimate with L = (cid:112) [Equation] D + (cid:107) 1 log ( p 1 p 2 /(cid:15) ) /n .
Revised: Let ˜ γ be the LEG-TV estimate with L = (cid:112) 2 (cid:107) D + (cid:107) 1 log ( p 1 p 2 /(cid:15) ) /n .

Original: If Assumption 1 holds for the covariance matrix Σ with constant κ , then with probability 1 − (cid:15) ,
Revised: If Assumption 1 holds for the covariance matrix Σ with constant κ , then with probability 1 − (cid:15) ,

Original: [Equation]
Revised: [Equation]

Original: n [Equation] where m ∈ R is a mean shift parameter, s is the number of non-zero elements in Dγ ∗ , C p = 4 (cid:112) 2 (cid:107) D + (cid:107) 

Revised: where m ∈ R is a mean shift parameter, s is the number of non-zero elements in Dγ ∗ , C p = 4 (cid:112) 2 (cid:107) D + (cid:107)

Original: ∝ p 1 / 4 1 p 1 / 4and C
Revised: ∝ p 1 / 4 1 p 1 / 4and C

Original: d is the minimal positive singular value of D .
Revised: d is the minimal positive singular value of D .

Original: The proof is built on top of the “high conﬁdence set” approach of Fan (2013).
Revised: The proof is built on top of the “high conﬁdence set” approach of Fan (2013).

Original: In the proof, we ﬁrst establish that, for an appropriately chosen value of L , γ ∗ = γ ( f, x 0 , F ) satisﬁes the constraint in our formulation with high probability.
Revised: In the proof, we ﬁrst establish that, for an appropriately chosen value of L , γ ∗ = γ ( f, x 0 , F ) satisﬁes the constraint in equation 4 with high probability.

Original: Then, we make use of TV sparsity of ˜ γ and γ ∗ to argue that the two quantities cannot be too far away from each other, since both are in the constraint set.
Revised: Then, we make use of TV sparsity of ˜ γ and γ ∗ to argue that the two quantities cannot be too far away from each other, since both are in the constraint set.

Original: The full proof is provided in the Appendix.
Revised: The full proof is provided in the Appendix.

Original: Our theorem has two major implications:
Revised: Our theorem has two major implications:

Original: We can recover the true parameter as the number of model evaluations increase.
Revised: We can recover the true parameter as the number of model evaluations increase.

Original: That is, TV penalized LEG is a statistically consistent model interpretation scheme.
Revised: That is, TV penalized LEG is a statistically consistent model interpretation scheme.

Original: Furthermore, our result states that, ignoring the log terms, one needs n =
Revised: Furthermore, our result states that, ignoring the log terms, one needs n =

Original: O ( s ( p 1 p 2 ) 1 / 2 ) many model evaluations to reliably recover γ ∗ .
Revised: O ( s ( p 1 p 2 ) 1 / 2 ) many model evaluations to reliably recover γ ∗ .

Original: Our bound depends on the constant κ , which further depends on the choice of Σ for the perturbation scheme.
Revised: Our bound depends on the constant κ , which further depends on the choice of Σ for the perturbation scheme.

Original: It is possible to obtain faster rates of convergence with a carefully tuned choice of Σ .
Revised: It is possible to obtain faster rates of convergence with a carefully tuned choice of Σ .

Original: As a side note, since γ ∗ also depends on Σ , the estimand changes when Σ is adjusted.
Revised: As a side note, since γ ∗ also depends on Σ , the estimand changes when Σ is adjusted.

Original: In other words, our result states that certain estimands require less samples.
Revised: In other words, our result states that certain estimands require less samples.

Original: We note that our procedure identiﬁes the LEG coefﬁcient up to a mean shift parameter, m , which is the average of the true LEG coefﬁcient γ .
Revised: We note that our procedure identiﬁes the LEG coefﬁcient up to a mean shift parameter, m , which is the average of the true LEG coefﬁcient γ .

Original: In practice, the average can be consistently estimated (for instance, using the empirical version of LEG in equation 3), and the mean can be subtracted to yield consistent estimates for γ .
Revised: In practice, the average can be consistently estimated (for instance, using the empirical version of LEG in equation 3), and the mean can be subtracted to yield consistent estimates for γ .

Original: However, in our numerical studies, we see that this mean shift is almost non-existent: LEG-TV yields solutions that has no mean differences with the LEG coefﬁcient, which we deﬁne as the solution of the empirical version as n → ∞ .
Revised: However, in our numerical studies, we see that this mean shift is almost non-existent: LEG-TV yields solutions that has no mean differences with the LEG coefﬁcient, which we deﬁne as the solution of the empirical version as n → ∞ .

Original: 4. P ERTURBATION S CHEME
Revised: 4. P ERTURBATION S CHEME

Original: In our main result, we established that the convergence of our estimator depends on the quantity κ which is related to the spectral properties of Σ .
Revised: In our main result, we established that the convergence of our estimator depends on the quantity κ which is related to the spectral properties of Σ .

Original: In this subsection we explore the ramiﬁcations of the assumption.
Revised: In this subsection we explore the ramiﬁcations of the assumption.

Original: Our main result in Theorem 1 states that the rate of convergence to the true LEG coefﬁcient is inversely proportional to the term κ .
Revised: Our main result in Theorem 1 states that the rate of convergence to the true LEG coefﬁcient is inversely proportional to the term κ .

Original: Thus, perturbation schemes for which the restricted eigenvalues are large, as deﬁned in Deﬁnition 1, yield saliency maps that require less samples to estimate the
Revised: Thus, perturbation schemes for which the restricted eigenvalues are large, as deﬁned in Deﬁnition 1, yield saliency maps that require less samples to estimate the

Original: LEG.
Revised: LEG.

Original: We note that most of the saliency estimation procedures that make use of perturbations take these perturbations to be independent, which results in a covariance matrix that is equal to the identity matrix, Σ
Revised: We note that most of the saliency estimation procedures that make use of perturbations take these perturbations to be independent, which results in a covariance matrix that is equal to the identity matrix, Σ

Original: = σ 2 I ( p 1 p 2 ) × ( p 1 p 2 ) for some σ 2 > 0 .
Revised: = σ 2 I ( p 1 p 2 ) × ( p 1 p 2 ) for some σ 2 > 0 .

Original: For LEG estimation without penalization, i.e. using equation 1, this choice is also optimal as the convergence rates under the normal setup depend on 1 /λ min (Σ) .
Revised: For LEG estimation without penalization, i.e. using equation 1, this choice is also optimal as the convergence rates under the normal setup depend on 1 /λ min (Σ) .

Original: However, when one seeks to ﬁnd an estimate for which the solution is sparse in the TV norm, this choice is no longer ideal as demonstrated by our theorem.
Revised: However, when one seeks to ﬁnd an estimate for which the solution is sparse in the TV norm, this choice is no longer ideal as demonstrated by our theorem.

Original: In order to choose the covariance matrix of our perturbation scheme in a manner that maximizes the bound in equation 5, one also needs some prior information about the size of S , s .
Revised: In order to choose the covariance matrix of our perturbation scheme in a manner that maximizes the bound in equation 5, one also needs some prior information about the size of S , s .

Original: As that requires estimation of s , and a complex optimization procedure, we instead propose a heuristic: we choose Σ so that its eigenvectors match D + ∆ for vectors ∆ with unit-norm and U T 2 ∆ = 0 .
Revised: As that requires estimation of s , and a complex optimization procedure, we instead propose a heuristic: we choose Σ so that its eigenvectors match D + ∆ for vectors ∆ with unit-norm and U T 2 ∆ = 0 .

Original: This choice ﬁxes p 1 p 2 − 1 many of the eigenvectors of Σ .
Revised: This choice ﬁxes p 1 p 2 − 1 many of the eigenvectors of Σ .

Original: For the last eigenvector, we use the one vector as it is orthogonal to the rest of the eigenvectors.
Revised: For the last eigenvector, we use the one vector as it is orthogonal to the rest of the eigenvectors.

Original: Our proposed perturbation scheme is as follows:
Revised: Our proposed perturbation scheme is as follows:

Original: Compute the singular value decomposition of D , and let D = U Θ V T .
Revised: Compute the singular value decomposition of D , and let D = U Θ V T .

Original: Let Σ = σ 2 (cid:16) V Θ 2 V T + 1 p 1 p 2 1 p 1 p 2 1 Tp 1 p 2 (cid:17) for some choice of σ 2 > 0 .
Revised: Let Σ = σ 2 (cid:16) V Θ 2 V T + 1 p 1 p 2 1 p 1 p 2 1 Tp 1 p 2 (cid:17) for some choice of σ 2 > 0 .

Original: As D + = V Θ + U T , with the proposed Σ , the numerator in equation 5 reduces to σ 2 ∆ T ∆ and hence κ = σ 2 .
Revised: As D + = V Θ + U T , with the proposed Σ , the numerator in equation 5 reduces to σ 2 ∆ T ∆ and hence κ = σ 2 .

Original: Without any additional assumptions on S , this is the maximal value for κ .
Revised: Without any additional assumptions on S , this is the maximal value for κ .

Original: We plot some of the eigenvectors for our proposed Σ with p 1 = p 2 = 28 in Figure 3.
Revised: We plot some of the eigenvectors for our proposed Σ with p 1 = p 2 = 28 in Figure 3.

Original: These eigenvectors are the principal directions of the perturbation distribution F , and the samples drawn from F contain a combination of these directions.
Revised: These eigenvectors are the principal directions of the perturbation distribution F , and the samples drawn from F contain a combination of these directions.

Original: We see these samples will have sharp contrasts at certain locations.
Revised: We see these samples will have sharp contrasts at certain locations.

Original: This result is very intuitive: The perturbation scheme is created for a speciﬁc problem where boundaries for objects are assumed to exist, and large jumps in the magnitude of the distribution help our method recover these boundaries efﬁciently.
Revised: This result is very intuitive: The perturbation scheme is created for a speciﬁc problem where boundaries for objects are assumed to exist, and large jumps in the magnitude of the distribution help our method recover these boundaries efﬁciently.

Original: We conclude this section with a demonstration of the perturbation scheme using Gaussian noise.
Revised: We conclude this section with a demonstration of the perturbation scheme using Gaussian noise.

Original: In
Revised: In

Original: Figure 4, we plot a digit from the MNIST dataset (LeCun et al., 1998), along with instances obtained by independent perturbation and by our suggested distribution.
Revised: Figure 4, we plot a digit from the MNIST dataset (LeCun et al., 1998), along with instances obtained by independent perturbation and by our suggested distribution.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] [Figure]
Revised: uniformly

Original: 4. I MPLEMENTATION D ETAILS
Revised: 4. I MPLEMENTATION D ETAILS

Original: LEG-TV procedure has two tuning parameters: (i) F , which determines the structure of the perturbation; and (ii) L , which controls the sparsity of the chosen interpretation.
Revised: LEG-TV procedure has two tuning parameters: (i) F , which determines the structure of the perturbation; and (ii) L , which controls the sparsity of the chosen interpretation.

Original: Regarding F , we propose to use a multivariate Gaussian distribution as it is easy to sample from.
Revised: Regarding F , we propose to use a multivariate Gaussian distribution as it is easy to sample from.

Original: For Σ , we propose a theoretically driven heuristic for determining the correlation structure of Σ in
Revised: For Σ , we propose a theoretically driven heuristic for determining the correlation structure of Σ in

Original: Section 4.2.
Revised: Section 4.2.

Original: However, the choice of the magnitude of Σ , i.e. σ 2 , is left to the user.
Revised: However, the choice of the magnitude of Σ , i.e. σ 2 , is left to the user.

Original: If this quantity is chosen too low, then the added perturbations are small in magnitude, and the predictions of the neural network do not change, resulting in a LEG near zero.
Revised: If this quantity is chosen too low, then the added perturbations are small in magnitude, and the predictions of the neural network do not change, resulting in a LEG near zero.

Original: On the other hand, with a very large value of σ 2 , the results have too much variance as some of the pixel values are set to the minimum or the maximum pixel intensity.
Revised: On the other hand, with a very large value of σ 2 , the results have too much variance as some of the pixel values are set to the minimum or the maximum pixel intensity.

Original: In our implementations, we ﬁnd that setting σ 2 to be between 0.
Revised: In our implementations, we ﬁnd that setting σ 2 to be between 0.

Original: and 0.30 results in reasonable solutions.
Revised: and 0.30 results in reasonable solutions.

Original: We determine this range by computing perturbations of various sizes on numerous images using the VGG-19 classiﬁer.
Revised: We determine this range by computing perturbations of various sizes on numerous images using the VGG-19 classiﬁer.

Original: The provided range is found to create perturbations large enough to change the prediction probabilities but small enough to avoid major changes in the image.
Revised: The provided range is found to create perturbations large enough to change the prediction probabilities but small enough to avoid major changes in the image.

Original: Most of our presented results are given for σ 2 = 0 .
Revised: Most of our presented results are given for σ 2 = 0 .

Original: For the choice of L , we propose two solutions: The ﬁrst is the theoretically suggested quantity given in Theorem 1, although this often results in estimates that are too conservative.
Revised: For the choice of L , we propose two solutions: The ﬁrst is the theoretically suggested quantity given in Theorem 1, although this often results in estimates that are too conservative.

Original: Our second method is a heuristic based on some of the quantities in the optimization problem and we use this for our demonstrations.
Revised: Our second method is a heuristic based on some of the quantities in the optimization problem and we use this for our demonstrations.

Original: We set L = K L L max where K is a constant between 0 and 1 and L max is the smallest value of L for which the solution in equation 4 would result with g = 0 ; i.e. L max = n − 1 (cid:107) D + T ( (cid:80) ni
Revised: We set L = K L L max where K is a constant between 0 and 1 and L max is the smallest value of L for which the solution in equation 4 would result with g = 0 ; i.e. L max = n − 1 (cid:107) D + T ( (cid:80) ni

Original: =1 vec (˜ y
Revised: =1 vec (˜ y

Original: i z i ))
Revised: i z i ))

Original: (cid:107) .
Revised: (cid:107) .

Original: We use K L = 0 .
Revised: We use K L = 0 .

Original: 05 or K L = 0 .
Revised: 05 or K L = 0 .

Original: 10 in our implementations.
Revised: 10 in our implementations.

Original: We note that is possible to obtain the solution for all L by using a parametric simplex solver (Vanderbei, 2014), or by starting with a large initial L , then using the solution of the program as a warm-start for a smaller choice of L .
Revised: We note that is possible to obtain the solution for all L by using a parametric simplex solver (Vanderbei, 2014), or by starting with a large initial L , then using the solution of the program as a warm-start for a smaller choice of L .

Original: Both approaches return the solution path for all L , and might be more desirable in practice than relying on heuristics.
Revised: Both approaches return the solution path for all L , and might be more desirable in practice than relying on heuristics.

Original:  E XAMPLES
Revised:  E XAMPLES

Original: In this section, we demonstrate the robustness and validity of our procedure by two numerical experiments.
Revised: In this section, we demonstrate the robustness and validity of our procedure by two numerical experiments.

Original: In Section 5.1, we perform sanity checks as laid out by Adebayo et al.
Revised: In Section 5.1, we perform sanity checks as laid out by Adebayo et al.

Original: (2018b), and show that the LEG-TV estimator fails to detect objects when the weights of the neural network are chosen randomly.
Revised: (2018b), and show that the LEG-TV estimator fails to detect objects when the weights of the neural network are chosen randomly.

Original: In Section 5.2, we implement a sensitivity analysis in which we use various saliency methods to compute regions of importance, and then perturb these regions in order to see their effect on the prediction.
Revised: In Section 5.2, we implement a sensitivity analysis in which we use various saliency methods to compute regions of importance, and then perturb these regions in order to see their effect on the prediction.

Original: For the deep learner, we use VGG-19 (Simonyan & Zisserman, 2014).
Revised: For the deep learner, we use VGG-19 (Simonyan & Zisserman, 2014).

Original: For computational efﬁciency, we compute saliency maps on a 28 by 28 grid (i.e. ˜ γ ∈ R 28 × 28 ) although the standard input for VGG-19 is 224 by 224.
Revised: For computational efﬁciency, we compute saliency maps on a 28 by 28 grid (i.e. ˜ γ ∈ R 28 × 28 ) although the standard input for VGG-19 is 224 by 224.

Original: The perturbations on the image are scaled up by 8 via upsampling in order for the dimensions to match.
Revised: The perturbations on the image are scaled up by 8 via upsampling in order for the dimensions to match.

Original: 5. S ANITY C HECKS
Revised: 5. S ANITY C HECKS

Original: In Adebayo et al.
Revised: In Adebayo et al.

Original: (2018b)
Revised: (2018b)

Original: , the validity of saliency estimation procedures are tested by varying the weights of the neural network.
Revised: , the validity of saliency estimation procedures are tested by varying the weights of the neural network.

Original: In a technique named, “cascading randomization”, authors propose to replace the ﬁtted weights of a CNN layer by layer, and compute the saliency scores with each change.
Revised: In a technique named, “cascading randomization”, authors propose to replace the ﬁtted weights of a CNN layer by layer, and compute the saliency scores with each change.

Original: As a deep learner with randomly chosen weights should have no prediction power, one expects to see the same effect in the resulting saliency scores: namely, as more of the weights are perturbed, the explanation offered by interpretability methods should become more and more meaningless.
Revised: As a deep learner with randomly chosen weights should have no prediction power, one expects to see the same effect in the resulting saliency scores: namely, as more of the weights are perturbed, the explanation offered by interpretability methods should become more and more meaningless.

Original: Surprisingly, Adebayo et al.
Revised: Surprisingly, Adebayo et al.

Original: (2018b) show that most commonly adopted interpretation procedures provide some saliency even after full randomization, and conclude that these methods act as edge detectors.
Revised: (2018b) show that most commonly adopted interpretation procedures provide some saliency even after full randomization, and conclude that these methods act as edge detectors.

Original: Our procedure treats the classiﬁer as a black-box and the explanations offered by LEG-TV are based solely on the predictions made by the neural network.
Revised: Our procedure treats the classiﬁer as a black-box and the explanations offered by LEG-TV are based solely on the predictions made by the neural network.

Original: During the sanity check, when the weights of the neural network are randomly perturbed, the predictions change signiﬁcantly and no longer depend on the input.
Revised: During the sanity check, when the weights of the neural network are randomly perturbed, the predictions change signiﬁcantly and no longer depend on the input.

Original: Thus, we expect the local linear approximations of the underlying function to be ﬂat, which would result in saliency scores of zero for all of the pixels.
Revised: Thus, we expect the local linear approximations of the underlying function to be ﬂat, which would result in saliency scores of zero for all of the pixels.

Original: Finally, small artifacts that might arise in this process, such as positive or negative saliency scores with no spatial structure, should be smoothed over due to the TV penalty, further robustifying our procedure.
Revised: Finally, small artifacts that might arise in this process, such as positive or negative saliency scores with no spatial structure, should be smoothed over due to the TV penalty, further robustifying our procedure.

Original: In order to verify our intuition, we perform cascading randomization on the weights of a VGGnetwork.
Revised: In order to verify our intuition, we perform cascading randomization on the weights of a VGGnetwork.

Original: For all of the images in our analysis, we ﬁnd that the LEG estimate, ˜ γ , is reduced to zero after randomization of either the top (i.e. logits) or the second top layer (i.e. second fully connected layer).
Revised: For all of the images in our analysis, we ﬁnd that the LEG estimate, ˜ γ , is reduced to zero after randomization of either the top (i.e. logits) or the second top layer (i.e. second fully connected layer).

Original: The results of our experiment for two images are given in Figure 5.
Revised: The results of our experiment for two images are given in Figure 5.

Original: It is seen that after the weights are perturbed, the LEG-TV method fails to detect any signal that could be used for interpretation.
Revised: It is seen that after the weights are perturbed, the LEG-TV method fails to detect any signal that could be used for interpretation.

Original: In fact, due to penalization, the estimate is set to zero.
Revised: In fact, due to penalization, the estimate is set to zero.

Original: These results show that the interpretation given by our proposed method is reliable and is dependent on the classiﬁer
Revised: These results show that the interpretation given by our proposed method is reliable and is dependent on the classiﬁer.

Original: [Figure]
Revised: [Figure]

Original: 5. S ENSITIVITY A NALYSIS
Revised: 5. S ENSITIVITY A NALYSIS

Original: For our second validity test, we use various interpretation models to compute regions of high importance.
Revised: For our second validity test, we use various interpretation models to compute regions of high importance.

Original: We then mask these regions by decreasing the value of the pixels to zero which is equivalent to painting them black.
Revised: We then mask these regions by decreasing the value of the pixels to zero which is equivalent to painting them black.

Original: We compute and assess the difference of the predictions for the target class with each perturbation.
Revised: We compute and assess the difference of the predictions for the target class with each perturbation.

Original: We compare our method against three alternatives: GradCAM (Selvaraju et al., 2017), LIME (Ribeiro et al., 2016) and SHAP (Lundberg & Lee, 2017).
Revised: We compare our method against four alternatives: GradCAM (Selvaraju et al., 2017), LIME (Ribeiro et al., 2016), SHAP (Lundberg & Lee, 2017) and C-Shapley (Chen et al., 2019).

Original: The last two methods are chosen as they are model-agnostic, like LEG, and do not make use of the architecture of the neural network.
Revised: The last three methods are chosen as they are model-agnostic, like LEG, and do not make use of the architecture of the neural network.

Original: GradCAM is chosen due to its popularity.
Revised: GradCAM is chosen due to its popularity.

Original: We present the probability changes given by the perturbations for three randomly chosen images in Figure 6.
Revised: The saliency maps using C-Shapley and LEG are computed for a 28 by 28 grid.

Original: In order to make the comparison between the methods more fair, we downsize the saliency maps resulting from GradCAM, LIME and SHAP to a 28 by 28 grid.
Revised: In order to make the comparison between the methods more fair, we downsize the saliency maps resulting from GradCAM, LIME and SHAP to the same size.

Original: Interestingly, we ﬁnd that this step improves the performance of these estimators.
Revised: Interestingly, we ﬁnd that this step improves the performance of these estimators; that is, the perturbations identiﬁed using the low resolution saliency maps result in faster drops in the predicted score. For LEG, LIME and SHAP, the saliency scores are computed using 3000 model evaluations, where as 3136 ( 28 × 28 × 4) evaluations are used for C-Shapley.

Original: For LEG, we provide two solutions, a sparse solution which corresponds to a larger choice of the penalty parameter L and a noisy solution which is obtained with a smaller choice of L , denoted by LEG and LEG0, respectively.
Revised: For LEG, we provide two solutions, a sparse solution which corresponds to a larger choice of the penalty parameter L and a noisy solution which is obtained with a smaller choice of L , denoted by LEG and LEG0, respectively.

Revised: We present the results for 500 images that are randomly chosen from a subsample of the ImageNet dataset (Deng et al., 2009) 2 .

Revised: The average of the log odds ratios across the 500 images are provided in Figure 6.

Original: [Figure]
Revised: [Figure]

Revised: to be the most accurate in determining the critical pixels, followed by LEG, GradCAM, C-Shapley and LIME.

Original: We see that as the size of the perturbation increases, the predictions for the target class drop for all of the methods.
Revised: We see that as the size of the perturbation increases, the predictions for the target class drop for all of the methods.

Original: The slope is sharpest for SHAP and LEG0, suggesting that these two methods identify pixels that are crucial for the predictions.
Revised: The slope is sharpest for SHAP and LEG0, suggesting that these two methods identify pixels that are crucial for the predictions.

Original: In Figure 7, we plot the chosen pixels for SHAP, LEG0,
Revised: In Figure 7, we plot the chosen pixels for SHAP, LEG0,

Original: LEG and GradCAM  for the pineapple image.
Revised: LEG, GradCAM and C-Shapley for an image classiﬁed as “pineapple”.

Original: The pixels chosen by SHAP appear to correspond to speciﬁc a convolution pattern and the chosen region is not contiguous.
Revised: The pixels chosen by SHAP appear to correspond to speciﬁc a convolution pattern and the chosen region is not contiguous.

Original: On the other hand, the pixels identiﬁed by LEG are visually meaningful to the human eye and contain pixels that are more likely to be relevant for the prediction.
Revised: On the other hand, the pixels identiﬁed by LEG are visually meaningful to the human eye and contain pixels that are more likely to be relevant for the prediction.

Original: [Figure]
Revised: [Figure]

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We have proposed a statistical framework for saliency estimation that relies on local linear approximations.
Revised: We have proposed a statistical framework for saliency estimation that relies on local linear approximations.

Original: Using this framework, we have built a computationally efﬁcient saliency estimator that has theoretical guarantees.
Revised: Using this framework, we have built a computationally efﬁcient saliency estimator that has theoretical guarantees.

Original: Through our theoretical analysis, we have identiﬁed how the sample complexity of the estimator can be improved by altering the perturbation scheme.
Revised: Through our theoretical analysis, we have identiﬁed how the sample complexity of the estimator can be improved by altering the perturbation scheme.

Original: Finally, we have established the practical validity of our approach through empirical studies.
Revised: Finally, we have established the practical validity of our approach through empirical studies.

Original: More speciﬁcally, we have shown that (i) unlike most of its competitors, our method passes the recently proposed sanity checks for saliency estimation; and (ii) pixels identiﬁed through our approach are highly relevant for the predictions, and our method often chooses regions with higher importance thanthe regions suggested by its alternatives.
Revised: More speciﬁcally, we have shown that (i) unlike most of its competitors, our method passes the recently proposed sanity checks for saliency estimation; and (ii) pixels identiﬁed through our approach are highly relevant for the predictions, and our method often chooses regions with higher importance than regions suggested by its alternatives

Original: A A PPENDIX A. A LTERNATIVE F ORMULATION
Revised: A A PPENDIX A. A LTERNATIVE F ORMULATION

Original: Our linear program can also be recast by a change of variables and setting α = Dg .
Revised: Our linear program can also be recast by a change of variables and setting α = Dg .

Original: In this case, the elements of α correspond to differences between adjoint pixels.
Revised: In this case, the elements of α correspond to differences between adjoint pixels.

Original: This program can be written as:
Revised: This program can be written as:

Original: [Equation]
Revised: [Equation]

Original: where D + is the pseudo-inverse of D and U 2 is related to the left singular vectors of D .
Revised: where D + is the pseudo-inverse of D and U 2 is related to the left singular vectors of D .

Original: More precisely, letting D = U Θ V T denote the singular value decomposition of D , U 2 is the submatrix that corresponds to the columns of U for which Θ j is zero.
Revised: More precisely, letting D = U Θ V T denote the singular value decomposition of D , U 2 is the submatrix that corresponds to the columns of U for which Θ j is zero.

Original: The linearity constraint ensures that the differences between the adjoint pixels is proper.
Revised: The linearity constraint ensures that the differences between the adjoint pixels is proper.

Original: Derivation of the alternative formulation follows from Theorem 1 in Gaines et al.
Revised: Derivation of the alternative formulation follows from Theorem 1 in Gaines et al.

Original: and is omitted.
Revised: and is omitted.

Original: This formulation can be expressed in the standard augmented form, i.e. min Ax = b,x ≥ 

Revised: This formulation can be expressed in the standard augmented form, i.e. min Ax = b,x ≥

Original: c T x , by writing x =
Revised: c T x , by writing x =

Original: [ α + , α − , s + , s − ] T ,A =   [Equation] U
Revised: [ α + , α − , s + , s − ] T ,

Original: [Equation]

Original: [Equation]

Original: I [Equation] [Equation] − I
Revised: A =

Original: [Equation]
Revised: [Equation]

Original:   , b =  

Original: L 1 m − D

Original: + y − L 1 m − D

Original: [Equation]
Revised: [Equation]

Original:  y   , c =   1 m 1 m 
Revised: + y   , c =   1 m 1 m 

Original:  , where y = 1 n (cid:80) ni
Revised:  , where y = 1 n (cid:80) ni

Original: =1 ˜ f
Revised: =1 ˜ f

Original: (˜ x i ) ˜
Revised: (˜ x i ) ˜

Original: x
Revised: x

Original: i and m = 2 p 1 p 2 − p 1 − p 2 .
Revised: i and m = 2 p 1 p 2 − p 1 − p 2 .

Original: The γ coefﬁcient in the original formulation can be obtained by setting γ = D + ( α + − α − ) .
Revised: The γ coefﬁcient in the original formulation can be obtained by setting γ = D + ( α + − α − ) .

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Our proof depends on the following lemma.
Revised: Our proof depends on the following lemma.

Original: Lemma 2.
Revised: Lemma 2.

Original: For L ≥ (cid:112) 2 (cid:107) D + (cid:107) 1 log ( p 1 p 2 /(cid:15) ) /n , γ ∗ is in the feasibility set with probability 1 − (cid:15) , that is(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) D + (cid:32)n n (cid:88)
Revised: For L ≥ (cid:112) 2 (cid:107) D + (cid:107) 1 log ( p 1 p 2 /(cid:15) ) /n , γ ∗ is in the feasibility set with probability 1 − (cid:15) , that is

Original: i = ˜ f
Revised: [Equation]

Original: (˜ x i ) ˜ x i (cid:33) − D + Σ γ ∗ (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ∞ ≤ L.

Original: Proof.
Revised: Proof.

Original: For ease of notation, let G
Revised: For ease of notation, let G

Original: = D + E (cid:104)
Revised: = D + E (cid:104)

Original: 1 n (cid:80) ni
Revised: 1 n (cid:80) ni

Original: =1 ˜ f
Revised: =1 ˜ f

Original: (˜ x i ) ˜ x i (cid:105) , and note that G = D + Σ γ ∗ .
Revised: (˜ x i ) ˜ x i (cid:105) , and note that G = D + Σ γ ∗ .

Original: Furthermore, let z i = ˜
Revised: Furthermore, let z i = ˜

Original: f
Revised: f

Original: (˜ x i ) D + ˜ x i .
Revised: (˜ x i ) D + ˜ x i .

Original: We also assume that the images have been rescaled so that the maximum value of ˜ x i is 1 (without rescaling, the maximum would be given as the largest intensity, i.e. 255).
Revised: We also assume that the images have been rescaled so that the maximum value of ˜ x i is 1 (without rescaling, the maximum would be given as the largest intensity, i.e. 255).

Original: Since, the function values are also in the range given by [-2,2], we can bound |
Revised: Since, the function values are also in the range given by [-2,2], we can bound |

Original: z i,j | , that is
Revised: z i,j | , that is

Original: [Equation]
Revised: [Equation]

Original: The proof follows by applying the McDiarmid’s inequality (Vershynin, 2018) for each row of the difference and then taking the supremum over the terms.
Revised: The proof follows by applying the McDiarmid’s inequality (Vershynin, 2018) for each row of the difference and then taking the supremum over the terms.

Original: By application of McDiarmid’s inequality, we have that
Revised: By application of McDiarmid’s inequality, we have that

Original: [Equation]
Revised: [Equation]

Original: Let L = (cid:112) 2 (cid:107) D
Revised: Let L = (cid:112) 2 (cid:107) D

Original: + (cid:107) 1 log ( p 1 p 2 / 2 (cid:15) )
Revised: + (cid:107) 1 log ( p 1 p 2 / 2 (cid:15) )

Original: /n .
Revised: /n .

Original: Then, taking a union bound over all variables, we have
Revised: Then, taking a union bound over all variables, we have

Original: [Table]
Revised: [Equation]

Original: Now note that that the feasibility set for any L (cid:48) ≥ L contains that of L and thus γ ∗ is automatically included.
Revised: Now note that that the feasibility set for any L (cid:48) ≥ L contains that of L and thus γ ∗ is automatically included.

Original: We now present the proof of the theorem.
Revised: We now present the proof of the theorem.

Original: Note that the technique is based on the Conﬁdence Set approach by Fan (2013).
Revised: Note that the technique is based on the Conﬁdence Set approach by Fan (2013).

Original: In the proof, we use γ to refer to vec ( γ ) for ease of presentation.
Revised: In the proof, we use γ to refer to vec ( γ ) for ease of presentation.

Original: Proof.
Revised: Proof.

Original: First, let the high probability set for which Lemma 2 holds by A .
Revised: First, let the high probability set for which Lemma 2 holds by A .

Original: All of the following statements hold true for A .
Revised: All of the following statements hold true for A .

Original: We let ∆ =
Revised: We let ∆ =

Original: D (ˆ γ − γ ∗ ) .
Revised: D (ˆ γ − γ ∗ ) .

Original: We know that (cid:107) D ˆ γ (cid:107) 1 ≤ (cid:107)
Revised: We know that (cid:107) D ˆ γ (cid:107) 1 ≤ (cid:107)

Original: Dγ ∗ (cid:107) 1 since both are in the feasibility set, as stated in Lemma 2.
Revised: Dγ ∗ (cid:107) 1 since both are in the feasibility set, as stated in Lemma 2.

Original: Let α ∗ =
Revised: Let α ∗ =

Original: Dγ ∗ , ˆ α = D ˆ γ and deﬁne S =
Revised: Dγ ∗ , ˆ α = D ˆ γ and deﬁne S =

Original: { j : α ∗ j (cid:54) = 0 } , and the complement of S as S C .
Revised: { j : α ∗ j (cid:54) = 0 } , and the complement of S as S C .

Original: By assumption of the Theorem, we have that the cardinality of S is s , i.e. | S | =
Revised: By assumption of the Theorem, we have that the cardinality of S is s , i.e. | S | =

Original: s .
Revised: s .

Original: Now let ∆ S as the elements of ∆ in S .
Revised: Now let ∆ S as the elements of ∆ in S .

Original: Then, using the above statement, one can show that (cid:107) ∆ S (cid:107) 1 ≥ {+(cid:107) ∆ S C (cid:107) 1 . Note, (cid:107) ˆ α (cid:107) 
 = (cid:107) α ∗+}
Revised: Then, using the above statement, one can show that (cid:107) ∆ S (cid:107) 1 ≥ {+(cid:107) ∆ S C (cid:107) 1 . Note, (cid:107) ˆ α (cid:107) 
 = (cid:107) α ∗+}

Original: [Equation]
Revised: [Equation]

Original: ∆ S C
Revised: ∆ S C

Original: [Equation]
Revised: [Equation]

Original: Note, [Equation] and (cid:107) ∆ S (cid:107) 1 ≥ (cid:107) ∆ S C (cid:107) 1 follows immediately.
Revised: 1 , and (cid:107) ∆ S (cid:107) 1 ≥ (cid:107) ∆ S C (cid:107) 1 follows immediately.

Original: Furthermore
Revised: Furthermore

Original: [Equation]
Revised: [Equation]

Original: where the last line uses the previous result.
Revised: where the last line uses the previous result.

Original: Additionally, note that
Revised: Additionally, note that

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality follows by Holder’s inequality and the second follows from Lemma 2 and the fact that both ˆ γ and γ ∗ are in the feasibility set for L = (cid:112) 2 (cid:107) D + (cid:107) 1 log ( p 1 p 2 /(cid:15) )
Revised: where the ﬁrst inequality follows by Holder’s inequality and the second follows from Lemma 2 and the fact that both ˆ γ and γ ∗ are in the feasibility set for L = (cid:112) 2 (cid:107) D + (cid:107) 1 log ( p 1 p 2 /(cid:15) )

Original: /n .
Revised: /n .

Original: We further bound the right hand side of the inequality by using the previous result, which gives
Revised: We further bound the right hand side of the inequality by using the previous result, which gives

Original: [Equation]
Revised: [Equation]

Original: Next, we bound (cid:107) ∆ (cid:107) 2 by combining the previous results.
Revised: Next, we bound (cid:107) ∆ (cid:107) 2 by combining the previous results.

Original: Now, by assumption of the Theorem, we have that
Revised: Now, by assumption of the Theorem, we have that

Original: [Equation]
Revised: [Equation]

Original: Dividing both sides by (cid:107) ∆ (cid:107) 2 , we obtain that
Revised: Dividing both sides by (cid:107) ∆ (cid:107) 2 , we obtain that

Original: [Equation]
Revised: [Equation]

Original: Finally, we note that
Revised: Finally, we note that

Original: [Equation]
Revised: [Equation]

Original: where D is the smallest singular value of D that is positive.
Revised: where D is the smallest singular value of D that is positive.

Original: This follows from the fact that D has only one zero right singular value, whose eigenvector is given by a vector of ones multiplied by 1 / √ p 1 p 2 .
Revised: This follows from the fact that D has only one zero right singular value, whose eigenvector is given by a vector of ones multiplied by 1 / √ p 1 p 2 .

Original: Letting m
Revised: Letting m

Original: = ( p 1 p 2 ) − 1 (cid:16)(cid:80)
Revised: = ( p 1 p 2 ) − 1 (cid:16)(cid:80)

Original: j γ ∗ j − (cid:80)
Revised: j γ ∗ j − (cid:80)

Original: j ˜ γ j (cid:17) concludes the proof.
Revised: j ˜ γ j (cid:17) concludes the proof.

Original: A. P ROOF OF L EMMA
Revised: A. P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: Note that h ( g ) is quadratic and convex in g .
Revised: Note that h ( g ) is quadratic and convex in g .

Original: Taking the derivative with respect to vec ( g ) , and setting it to zero we obtain
Revised: Taking the derivative with respect to vec ( g ) , and setting it to zero we obtain

Original: [Equation]
Revised: [Equation]

Original: where g ∗ is the minimizer.
Revised: where g ∗ is the minimizer.

Original: After reorganizing the terms and setting z = x − x 0 , we get
Revised: After reorganizing the terms and setting z = x − x 0 , we get

Original: [Equation]
Revised: [Equation]

Original: where we use that Σ =
Revised: where we use that Σ =

Original: Cov ( vec ( z )) in the last equation.
Revised: Cov ( vec ( z )) in the last equation.

Original: The result follows trivially.
Revised: The result follows trivially.

Original: A. E QUIVALENCY OF LEG-TV WITH E MPIRICAL LEG IF L =
Revised: A. E QUIVALENCY OF LEG-TV WITH E MPIRICAL LEG IF L =

Original: Lemma 3.
Revised: Lemma 3.

Original: For the LEG-TV estimate with L = 0 , if the one vector is an eigenvector of Σ , i.e.
Revised: For the LEG-TV estimate with L = 0 , if the one vector is an eigenvector of Σ , i.e.

Original: Σ1 p 1 p 2 = λ 1 p 1 p 2 , then the solution is equal to the empirical LEG estimate up to a location shift.
Revised: Σ1 p 1 p 2 = λ 1 p 1 p 2 , then the solution is equal to the empirical LEG estimate up to a location shift.

Original: That is, ˜ γ
Revised: That is, ˜ γ

Original: = ˆ γ + a 1 p 1 p 2 , for some a ∈ R .
Revised: = ˆ γ + a 1 p 1 p 2 , for some a ∈ R .

Original: Before the proof, we note that the eigenvector condition on Σ can satisﬁed either with independent noise or our suggested scheme in Section 4.2.
Revised: Before the proof, we note that the eigenvector condition on Σ can satisﬁed either with independent noise or our suggested scheme in Section 4.2.

Original: Proof.
Revised: Proof.

Original: Note that, if L = 0 , then we have that
Revised: Note that, if L = 0 , then we have that

Original: [Equation]
Revised: [Equation]

Original: As the only right singular vector of D + T with zero singular value is the one vector, the above statement is true iff
Revised: As the only right singular vector of D + T with zero singular value is the one vector, the above statement is true iff

Original: [Equation]
Revised: [Equation]

Original: for some c ∈ R .
Revised: for some c ∈ R .

Original: Solving for g , we obtain,
Revised: Solving for g , we obtain,

Original: [Equation]
Revised: [Equation]

Original: where we use the fact that the one vector is an eigenvector of Σ − 1 with eigenvalue λ − 1 .
Revised: where we use the fact that the one vector is an eigenvector of Σ − 1 with eigenvalue λ − 1 .

Original: Setting a = − cλ concludes the proof.
Revised: Setting a = − cλ concludes the proof.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]