Which Explanation Should I Choose? A Function Approximation Perspective to Characterizing Post Hoc Explanations
Abstract
To explain the predictions of black-box models, multiple post hoc explanation methods have been proposed. However, different methods are motivated by different objectives (e.g., function approximation, game theory notions such as ShapleyAumann values, and ad hoc goals of obtaining clean visualizations).
The lack of a common foundational goal among methods not only prevents a coherent conceptual view of and systematic way to study post hoc explainability but also causes the practical challenge of not knowing which method to use when, thereby making systemic progress in the field difficult to achieve. In this work, we take the first steps towards bridging these gaps and unify eight popular post hoc explanation methods (LIME, C-LIME, SHAP, Vanilla Gradients,
Gradients × Input, Integrated Gradients, SmoothGrad, Occlusion) by showing that they can all be viewed as performing local function approximation of the black-box model, with methods differing only by the local neighbourhoods and loss functions used to perform the approximation.
This unification enables us to (1) state a no free lunch theorem for explanation methods which demonstrates that no single method can perform optimally across all neighbourhoods, and (2) provide a guiding principle to choose among methods based on the faithfulness of an explanation to the black-box model.
We empirically validate our theoretical results using various real-world datasets, model classes, and prediction tasks.
By formalizing a mathematical framework which unifies diverse explanation methods, our work (1) advances the conceptual understanding of these methods, revealing their shared local function approximation objective and characterizing their relation to one another, and (2) guides the use of these methods in practice, providing a principled approach to choose among methods and paving the way for the creation of new ones.
Introduction
Machine learning models are increasingly being deployed in critical domains such as healthcare, law, and finance [47, 44, 8].
Consequently, there is a growing emphasis on explaining the predictions of these models to decision makers (e.g. doctors, judges) so that they can understand the rationale behind model predictions and determine when to rely on them.
To this end, several methods have been proposed in the literature to explain model predictions in a post hoc local fashion.
While these local explanation methods differ in a variety of ways, they can be broadly classified as perturbation-based or gradient-based methods based on their approach to estimating each feature’s influence on the prediction.
Popular perturbation-based methods include LIME [31], SHAP [27] and Occlusion [48], and gradient-based ones include Vanilla Gradients [34], Gradient x Input [33], SmoothGrad [36], and Integrated Gradients [41].
Although these methods are all local post hoc explanation methods, they have different objectives: for example, some are motivated by function approximation [31], some use game theoretic notions of Shapley-Aumann values [27], while others are driven by the goal of obtaining clean visualizations [36].
Such fragmentation of goals prevents a coherent conceptual understanding of post hoc explainability. Furthermore, the basic properties of each method and the conditions under which each method is effective are not well understood, leading to the practical challenge of not knowing which method to use when.
The proper use of explanation methods in practice is particularly critical as different explanation methods have been shown to generate disagreeing, and sometimes even contradictory, explanations [22].
Prior works have taken the first steps towards characterizing explanation methods and establishing connections between them.
For example, Agarwal et al.
[2] proved that C-LIME (a continuous variant of LIME [31]) and SmoothGrad [36] converge to the same explanation in expectation.
In addition, Lundberg and Lee [27] proposed a framework based on Shapley values to unify binary perturbation-based explanations and Covert et al.
[11] found that many perturbation-based methods share the property of estimating feature importance based on the change in model behavior upon feature removal.
Ancona et al.
[4] also analyzed four gradient-based explanation methods and the conditions under which they produce similar explanations.
However, these analyses of explanation methods are based on mechanistic properties (such as Shapley values and feature removal), are limited in scope (connecting only two methods, only perturbation-based methods, or only gradient-based methods), and do not inform when one method is preferable to another.
To address the aforementioned gaps, we study explanation methods from a function approximation perspective and formalize an overarching mathematical framework that characterizes a broad set of methods (spanning both perturbation-based and gradient-based methods), and provides a principled approach to choose among these methods.
Our work makes the following contributions:
We show that eight popular explanation methods (such as LIME , C-LIME , KernelSHAP , Occlusion , Vanilla Gradients , Gradient x Input , SmoothGrad , and Integrated Gradients ) spanning both perturbation-based and gradient-based methods, all perform local function approximation of the underlying model over different neighbourhoods using different loss functions.
We introduce a no free lunch theorem for explanation methods which demonstrates that no single explanation method can perform local function approximation faithfully across all local neighbourhoods, thereby underscoring the need for a strategy to choose among various explanation methods. 3.
To choose among methods, we set forth a guiding principle based on the function approximation perspective, deeming a method to be effective if its explanation model recovers the underlying model when the former is in the same model class as the latter (i.e. if the explanation model perfectly approximates the underlying model when it is able to do so).
We validate the theoretical results above through extensive empirical evaluation with various real-world datasets, predictive model classes, and prediction tasks.
Related Work
Post hoc explanations.
Post hoc explanation techniques of pre-trained models can be classified based on their access to the complex model (i.e. black box vs. access to model internals), scope of approximation (i.e. global vs. local), search technique (i.e. perturbation-based vs. gradient-based), and basic units of explanation (i.e. feature importance vs. rule-based).
Examples are provided in the previous section.
Recent work has also developed counterfactual explanations which capture the changes that need to be made to a given instance in order to flip its prediction [43, 42, 17, 30, 26, 6, 18, 19].
Such explanations can be leveraged to provide recourse to individuals negatively impacted by algorithmic decisions.
An alternate approach is to construct global explanations , summarizing the complete behavior of a black-box model by approximating it using an interpretable model [23, 7, 20].
Analyzing post hoc explanations.
Recent work has shed light on the limitations of post hoc explanation methods.
For instance, prior works [15, 35, 12, 1, 3] empirically demonstrated that perturbation-based methods, such as LIME and SHAP , may not be robust, i.e. small changes to the input may lead to drastic changes in the explanations.
Adebayo et al.
[1] and Srinivas and Fleuret [38] also demonstrated that gradient-based methods do not always generate explanations that are faithful to the underlying model.
In addition, recent research has also theoretically analyzed the robustness [24, 9] and other properties [14] of popular post hoc explanation methods. Finally, Watson et al. [45] show that logical concepts of sufficiency and necessity play a role in several post hoc explanations.
However, none of these works provide a framework to characterize post hoc explanation methods and the connections among them or guide the selection of methods, all of which are the focus of our work.
Explanations as Local Function Approximations
In this section, we formalize the local function approximation framework and describe its connection to existing explanation methods.
We proceed by defining the notation used in the rest of the paper.
Notation : Let f : X → Y be the black-box function we seek to explain in a post hoc manner, with input domain X (e.g. X = R d or { 0 , 1 } d ) and output domain Y (e.g. Y = R or [0 , 1] ).
Let
G = { g : X → Y} be the class of interpretable models used to generate a local explanation for f by selecting a suitable interpretable model g ∈ G .
We characterize locality around a point x 0 ∈ X using a noise random variable ξ which is sampled from distribution Z .
Let x ξ = x 0 ⊕ ξ be a perturbation of x 0 generated by combining x 0 and ξ using a binary operator ⊕ (e.g. addition, multiplication).
Lastly, let ℓ ( f, g, x 0 , ξ ) ∈ R + be the loss function (e.g. squared error, cross-entropy) measuring the distance between f and g on a noise variable ξ around x 0 .
We now define the local function approximation framework.
Definition 1.
Local function approximation (LFA) of a black-box model f on a neighbourhood distribution Z around x 0 by an interpretable model family G and a loss function ℓ is given by
[Equation]
where a valid loss ℓ is such that E ξ ∼Z ℓ ( f, g, x 0 , ξ ) =[-
-]
⇐⇒ f ( x ξ ) =
g ( x [Equation] Z
The LFA framework is a formalization of the function approximation perspective first introduced by LIME [31] to motivate local explanations.
Note that this conceptual framework is distinct from the algorithm introduced by LIME .
We elaborate on this distinction below.
(1) The LFA framework requires that f and g share the same input domain X and output domain Y , a fundamental prerequisite for function approximation.
This implies, for example, that using an interpretable model g with binary inputs ( X = { 0 , 1 } d ) to approximate a black-box model f with continuous inputs ( X = R d ), as proposed in LIME , is not true function approximation.
(2) By imposing a condition on the loss function, the LFA framework ensures model recovery under specific conditions: g ∗ recovers f
(i.e. g ∗ = f ) through LFA when f itself is of the interpretable model class G (i.e. f ∈ G ) and perturbations span the input domain of f (i.e. domain ( x ) = X ).
This is a key distinction between the LFA framework and LIME (which has no such requirement) and guides the characterization of explanation methods in §4.
(3) Efficiently minimizing Equation 1 requires following standard machine learning methodology of splitting the perturbation data into train / validation / test sets and tuning hyper-parameters on the validation set to ensure generalization.
To our knowledge, implementations of LIME do not adopt this procedure, making it possible to overfit to a small number of perturbations.
The LFA framework is generic enough to accommodate a variety of explanation methods.
In fact, we will show that specific instances of this framework converge to existing methods, as summarized in
Table 1.
At a high level, existing methods use a linear model g to locally approximate the black-box model f in different input domains (binary or continuous) over different local neighbourhoods specified by noise random variable ξ (where ξ is binary or continuous, drawn from a specified distribution, and combined additively or multiplicatively with point x 0 ) using different loss functions (squared-error or gradient-matching loss).
We discuss the details of these connections in the following sections. [Table] Local Neighbourhood Z around x
Loss Function ℓ [Table] 3.1 LFA with Continuous Noise: Gradient-Based Explanation Methods
To connect gradient-based explanation methods to the LFA framework, we leverage the gradientmatching loss function ℓ gm .
We define ℓ gm and show that it is a valid loss function for LFA.
[Equation]
This loss function has been previously used in the contexts of generative modeling (where it is dubbed score-matching)
[16] and model distillation [38].
However, to our knowledge, its use in interpretability is novel.
Proposition 1.
The gradient-matching loss function ℓ gm is a valid loss function for LFA up to {+a constant, i.e. E ξ ∼Z ℓ gm ( f, g, x 0 , ξ ) = 
+}
[-i [Table] e. E ⇠ ⇠ Z ` gm ( f, g, x 0 , ⇠ ) = 
-]⇐⇒ f
( x ξ ) =
g ( x ξ ) + C ∀ ξ ∼ Z , where C ∈ R .
Proof.
If f ( x ξ ) =
g ( x ξ ) , then ∇ ξ f ( x ξ ) = ∇ ξ g ( x ξ ) and it follows from the definition of ℓ gm that ℓ gm = 0 .
Integrating ∇ ξ f ( x ξ ) = ∇ ξ g ( x ξ ) gives f ( x ξ )
=
g ( x ξ ) + C .
Proposition 1 implies that, when using the linear model class G parameterized by g ( x ) = w ⊤ x + b to approximate f , g ∗ recovers w but not b .
This can be fixed by setting b = f (0) .
Theorem 1.
LFA with gradient-matching loss is equivalent to (1) SmoothGrad for additive continuous Gaussian noise, which converges to Vanilla Gradients in the limit of a small standard deviation for the Gaussian distribution; and (2) Integrated Gradients for multiplicative continuous Uniform noise, which converges to Gradient x Input in the limit of a small support for the
Uniform distribution.
Proof Sketch.
For SmoothGrad and Integrated Gradients , the idea is that these methods are exactly the first-order stationary points of the gradient-matching loss function under their respective noise distributions. In other words, the weights of the interpretable model g that minimize the loss function is the explanation returned by each method.
For Vanilla Gradients and Gradient x Input , the result is derived by taking the specified limits and using the Dirac delta function to calculate the limit. In the limit, the weights of the interpretable model g converge to the explanation of each method.
Full proofs are in Appendix A.1.
Along with gradient-based methods, C-LIME [2] (a perturbation-based method) is an instance of the LFA framework by definition, using the squared-error loss function.
The analysis in this section characterizes methods that use continuous noise.
It does not extend to binary nor discrete noise methods as gradients nor continuous random variables apply in these domains.
In the next section, we discuss binary noise methods.
3.2 LFA with Binary Noise: LIME, KernelSHAP and Occlusion maps
Theorem 2.
LFA with multiplicative binary noise and squared-error loss is equivalent to (1) LIME for noise sampled from an unnormalized exponential kernel over binary vectors; (2) KernelSHAP for noise sampled from an unnormalized Shapley kernel; and (3) Occlusion for noise in the form of one-hot vectors.
Proof Sketch.
For LIME and KernelSHAP , the equivalence is mostly by definition: these methods have components that correspond to the interpretable model g and the loss function ℓ of the LFA framework and we need only to determine the local neighbourhood Z .
We define the local neighbourhood Z using each method’s weighting kernel. In this setup, the LFA framework yields the respective explanation method in expectation via importance sampling.
For Occlusion , the equivalence involves enumerating all perturbations, specifying an appropriate loss function, and computing the resulting stationary points of the loss function. Full proofs are in Appendix A.1. 3.3 Which Methods Do Not Perform LFA?
Some popular explanation methods are not instances of the LFA framework due to properties of these methods.
These methods include guided backpropagation, DeepLIFT [33], Grad-CAM.
For a more detailed discussion, please see Appendix A.2.
When Do Explanations Perform Model Recovery?
Having described the LFA framework and its connections to existing explanation methods, we now leverage this framework to analyze the performance of methods under different conditions.
We introduce a no free lunch theorem for explanation methods , inspired by classical no free lunch theorems in learning theory and optimization.
Then, we assess the ability of existing methods to perform model recovery based on which we provide recommendations for choosing among methods.
4.1 No Free Lunch Theorem for Explanation Methods
An important implication of the function approximation perspective is that no explanation can be optimal across all neighbourhoods because each explanation is designed to perform LFA in a specific neighbourhood. This is especially true for explanations of non-linear models.
We distill this intuitive observation into the following theorem.
Theorem 3 (No Free Lunch for Explanation Methods) .
Consider explaining a black-box model f around point x 0 using an interpretable model g from class G and a valid loss function ℓ where the distance between f and G is given by d ( f, G ) =
min g ∈G max x ∈X ℓ ( f, g, 0 , x ) .
Then, for any explanation g ∗ on a neighbourhood distribution ξ∼ Z 1 such that max ξ 1 ℓ ( f, g ∗ , x 0 , ξ 1 ) ≤ ϵ , we can always find another neighbourhood ξ 2 ∼ Z 2 such that max ξ 2 ℓ ( f, g ∗ , x 0 , ξ 2 ) ≥ d ( f, G ) .
Proof Sketch.
The idea is that, given an explanation obtained by using g to approximate f over a specific local neighbourhood Z , it is always possible to find a local neighbourhood over which this explanation does not perform well (i.e. does not perform faithful LFA).
Thus, no single explanation method can perform well over all local neighbourhoods. The proof entails constructing an “adversarial” input for an explanation g ∗ such that g ∗ has a large loss for this input and then creating a neighbourhood that contains this adversarial input which will provably have a large loss.
The magnitude of this loss is d ( f, G ) , the distance between f and the model class G , inspired by the Haussdorf distance. The proof is generic and makes no assumptions regarding the forms of ℓ , G or Z 1 .
The full proof is in Appendix A.3.
Thus, an explanation on a finite Z 1 necessarily cannot approximate function behaviour at all other points, especially when G is less expressive than f , which is indicated by a large value of d ( f, G ) .
Thus, in the general case, one cannot perform model recovery as G is less expressive than f .
An important implication of Theorem 3 is that seeking to find the “best” explanation without specifying a corresponding neighbourhood is futile as no universal “best” explanation exists.
Furthermore, once the neighbourhood is specified, the best explanation is exactly given by the corresponding instance of the LFA framework.
In the next section, we consider the special case when d ( f, G ) =
(i.e. when f ∈ G ), where
Theorem 3 does not apply as the same explanation can be optimal for multiple neighbourhoods and model recovery is thus possible.
4.2 Characterizing Explanation Methods via Model Recovery
We proceed by formally stating the model recovery condition for explanation methods and then use this condition as a guiding principle to choose among methods.
Definition 2 (Model Recovery: Guiding Principle) .
Given an instance of the LFA framework with a black-box model f such that f ∈ G and a specific noise type (e.g. Gaussian, Uniform), an explanation method performs model recovery if there exists some noise distribution Z such that LFA returns g ∗ = f .
In other words, when the black-box model f itself is of the interpretable model class G , there must exist some setting of the noise distribution (within the noise type specified in the instance of the LFA framework) that is able to recover the black-box model.
Thus, in this special case, we require local function approximation to lead to global model recovery over all inputs.
This criterion can be thought of as a “sanity check” for explanation methods to ensure that they remain faithful to the black-box model.
Next, we analyze the impact of the choice of perturbation neighbourhood Z , the binary operator ⊕ , and the interpretable model class G on an explanation method’s ability to satisfy the model recovery guiding principle in different input domains X .
Note that while we can choose Z , ⊕ , and G , we cannot choose X , the input domain.
Which explanation should I choose for continuous X ?
We now analyze the model recovery properties of existing explanation methods when the input domain is continuous.
We consider methods based on additive continuous noise ( SmoothGrad , Vanilla Gradients , and C-LIME ), multiplicative continuous noise ( Integrated Gradients and Gradient x Input ), and multiplicative binary noise ( LIME , KernelSHAP , and Occlusion ).
For these methods, we make the following remark regarding model recovery for the class of linear models.
Remark 1.
For X = R d and linear models f and g where f ( x ) = w ⊤ f x and g ( x ) =
w ⊤ g x , additive continuous noise methods recover f (i.e. w g = w f ) while multiplicative continuous and multiplicative binary noise methods do not and instead recover w g = w f ⊙ x .
This remark can be verified by directly evaluating the explanations (weights) of linear models, where the gradient exactly corresponds to the weights.
We point out that the inability of multiplicative continuous noise methods to recover the blackbox model is not due to the multiplicative nature of the noise, but rather due to the parameterization of the loss function.
Specifically, these methods (implicitly) use the loss function ℓ ( f, g, x 0 , ξ ) = ∥∇ ξ f ( x ξ ) − ∇ ξ g ( ξ ) ∥ 22 .
Slightly changing the loss function to ℓ ( f, g, x 0 , ξ ) = ∥∇ ξ f ( x ξ ) − ∇ ξ g ( x ξ ) ∥ 22 , i.e. replacing g ( ξ ) with g ( x ξ ) , would enable g ∗ to recover f .
This would change Integrated Gradients to (cid:82) 1 α =0 ∇ α x f ( α x ) (omitting the input multiplication term) and
Gradient x Input to Vanilla Gradients .
A similar argument can be made for binary noise methods which parameterize the loss function as ℓ ( f, g, x 0 , ξ ) =
∥ f ( x ξ ) − g ( ξ ) ∥ 2 .
By changing the loss function to ℓ ( f, g, x 0 , ξ ) =
∥ f ( x ξ ) − g ( x ξ ) ∥ 2 , binary noise methods can recover f for the case described in Remark 1.
However, binary noise methods for continuous domains are unreliable, as there are cases where, despite the modification to ℓ , model recovery is not guaranteed.
The following is an example of such a case.
Remark 2.
For X = R d , periodic functions f and g where f ( x ) = (cid:80) di =1 sin( w f i ⊙ x i ) and g ( x ) = (cid:80) di =1 sin( w g i ⊙ x i ) , and an integer n , binary noise methods do not perform model recovery for | w f i | ≥ nπ x 0 i .
This is because, for the conditions specified, sin( w f i x 0 i ) =
sin( ± nπ ) = sin(0) = 0 , i.e. sin( w f i x 0 i ) outputs zero for all binary perturbations, thereby preventing model recovery.
In this case, the discrete nature of the noise makes model recovery impossible.
In general, discrete noise is inadequate for the recovery of models with large frequency components.
Which explanation should I choose for binary X ?
In the binary domain, continuous noise methods are invalid, restricting us to binary noise methods.
By the discussion above, methods with perturbation neighbourhoods characterized by multiplicative binary perturbations, such as LIME , KernelSHAP , and Occlusion , only enable g ∗ to recover f in the binary domain.
Note that the sinusoidal example in Remark 2 does not apply in this regime due to the continuous nature of its domain.
Which explanation should I choose for discrete X ?
In the discrete domain, continuous noise methods are also invalid.
In addition, binary noise methods, such as LIME , KernelSHAP and Occlusion , also cannot be used as model recovery is not guaranteed in the sinusoidal case (Remark 2), following a logic similar to that presented for continuous noise.
We notice that none of the existing methods in
Table 1 perform general discrete perturbations, suggesting that these methods are not suitable for the discrete domain.
Thus, in the discrete domain, a user can apply the LFA framework to define a new explanation method, specifying an appropriate discrete noise type.
In the next section, we discuss more broadly about how one can use the LFA framework to create novel explanation methods.
4.3 Designing Novel Explanations with LFA
The LFA framework not only unifies existing explanation methods but also guides the creation of new ones.
To explain a given black-box model prediction using the LFA framework, a user must specify the (1) interpretable model class G , (2) neighbourhood distribution Z , (3) loss function ℓ , and (4) binary operator ⊕ to combine the input and the noise.
Specifying these completely specifies an instance of the LFA framework, thereby generating an explanation method tailored to a given context.
To illustrate this, consider a scenario in which a user seeks to create a sparse variant of SmoothGrad that yields non-zero gradients for only a small number of features (“ SparseSmoothGrad ”).
Designing SparseSmoothGrad only requires the addition of a regularization term to the loss function used in the SmoothGrad instance of the LFA framework (e.g. ℓ = ℓ SmoothGrad + ∥∇ ξ g ( x ξ ) ∥ 0 ), at which point, sparse solvers may be employed to solve the problem.
Note that, unlike SmoothGrad ,
SparseSmoothGrad does not have a closed form solution, but that is not an issue for the LFA framework.
More generally, by allowing for the customization of (1), (2), (3), and (4), the LFA framework creates new explanation methods through “variations on a theme”.
We summarize Section §4 as a table in Appendix A.4 and discuss the practical implications of Section §4 by providing the following recommendation for choosing among explanation methods.
Recommendation for choosing among explanation methods.
In general, choose methods that satisfy the guiding principle of model recovery in the input domain in question.
For continuous data, use additive continuous noise methods (e.g. SmoothGrad , Vanilla Gradients , C-LIME ) or modified multiplicative continuous noise methods (e.g. Integrated Gradients , Gradient x Input ) as described in Section §4.2.
For binary data, use binary noise methods (e.g. LIME ,
KernelSHAP , Occlusion ).
Given that methods that use discrete noise do not exist, in case of discrete data, design novel explanation methods using the LFA framework with discrete noise neighbourhoods.
Within each input domain, choosing among appropriate methods boils down to determining the perturbation neighbourhood most suitable in the given context.
Empirical Evaluation
In this section, we present an empirical evaluation of the LFA framework.
We first describe the experimental setup and then discuss three experiments and their findings.
5.1 Datasets, Models, and Metrics
Datasets.
We experiment with two real-world datasets for two prediction tasks.
The first dataset is the life expectancy dataset from the World Health Organization (WHO) [46].
It consists of countries’ demographic, economic, and health factors from 2000 to 2015, with 2,938 observations forcontinuous features.
We use this dataset to perform regression, predicting life expectancy.
The other dataset is the home equity line of credit (HELOC) dataset from FICO [13].
It consists of information on HELOC applications, with 9,871 observations for 24 continuous features.
We use this dataset to perform classification, predicting whether an applicant made payments without being 90 days overdue. Additional dataset details are described in Appendix A.5.
Models.
For each dataset, we train four models: a simple model (linear regression for the WHO dataset and logistic regression for the HELOC dataset) that can satisfy conditions of the guiding principle and three more complex models (neural networks of varying complexity) that are more reflective of real-world applications.
Model architectures and performance are described in Appendix A.5. [Figure]
Metrics.
To measure the similarity between two vectors (e.g. between two sets of explanations or between an explanation and the true model weights), we use L1 distance and cosine distance.
L distance ranges between [0, ∞ ) and is 0 when two vectors are the same.
Cosine distance measures the angle between two vectors.
It ranges between [0, 2] and is 0 when the angle between two vectors is 0 ◦ (or 360 ◦ ).
For both metrics, the lower the value, the more similar two given vectors are.
5.2 Experiments
Here, we describe the set-up of the experiments, present experimental results, and discuss their implications.
Experiment 1: Existing explanation methods are instances of the LFA framework.
First, we compare existing methods with corresponding instances of the LFA framework to assess whether they generate the same explanations.
To this end, we use seven methods to explain the predictions of black-box models on 100 randomly-selected test set points.
For each method, explanations are computed using either the existing method (implemented by Meta’s Captum library [21]) or the corresponding instance of the LFA framework (Table 1).
The similarity of a given pair of explanations is measured using L1 distance or cosine distance.
The L1 distance values for a neural network with three hidden layers trained on the WHO dataset are shown in Figure 1.
In Figure 1a, lowest L1 distance values appear in the diagonal of the heatmap, indicating that explanations generated by existing methods and corresponding instances of the LFA framework are very similar.
Figures 1b and 1c show that explanations generated by instances of the LFA framework corresponding to SmoothGrad and Integrated Gradients converge to those of Vanilla Gradients and Gradient x Input , respectively.
Together, these results demonstrate that, consistent with the theoretical results derived in Section §3, existing methods are instances of the LFA framework.
In addition, the clustering of the methods in Figure 1a indicates that, consistent with the theoretical analysis in Section §4, for continuous data, SmoothGrad and Vanilla Gradients generate similar explanations while LIME , KernelSHAP , Occlusion , Integrated Gradients , and Gradient x Input generate similar explanations.
We observe similar results across various datasets, models, and metrics (Appendix A.6.1).
Experiment 2: Some methods recover the underlying model while others do not (guiding principle).
Next, we empirically assess which existing methods satisfy the guiding principle, i.e. which methods recover the black-box model f when f is of the interpretable model class G .
We specify a setting in which f and g are of the same model class, generate explanations using each [Figure] [Figure] method, and assess whether g recovers f for each explanation.
For the WHO dataset, we set f and g to be linear regression models and generate explanations for 100 randomly-selected test set points.
Then, for each point, we compare g ’s weights with f ’s gradients or with f ’s gradients multiplied by the input because, based on Section §4, some methods generate explanations on the scale of gradients while others on the scale of gradient-times-input.
Note that, for linear regression, f ’s gradients are f ’s weights.
Results are shown in Figure 2.
Consistent with Section §4, for continuous data, SmoothGrad and Vanilla Gradients recover the black-box model, thereby satisfying the guiding principle, while LIME , KernelSHAP , Occlusion , Integrated Gradients , and Gradient x Input do not.
We observe similar results for the HELOC dataset using logistic regression models for f and g (Appendix A.6.2).
Experiment 3: No single method performs best across all neighbourhoods (no free lunch theorem).
Lastly, we perform a set of experiments to illustrate the no free lunch theorem in Section §4.
We generate explanations for black-box model predictions for 100 randomly-selected test set points and evaluate the explanations using perturbation tests based on top- k or bottom- k features.
For perturbation tests based on top- k features, the setup is as follows.
For a given data point, k , and explanation, we identify the top- k features and either replace them with zero (binary perturbation) or add Gaussian noise to them (continuous perturbation).
Then, we calculate the absolute difference in model prediction before and after perturbation.
For each point, we generate one binary perturbation (since such perturbations are deterministic) and 100 continuous perturbations (since such perturbations are random), computing the average absolute difference in model prediction for the latter. In this setup, methods that better identify important features yield larger changes in model prediction. For perturbation tests based on bottom- k features, we follow the same procedure but perturb the bottom- k features instead.
In this setup, methods that better identify unimportant features yield smaller changes in model prediction.
Results of perturbation tests based on bottom- k features performed on explanations for a neural network with three hidden layers trained on the WHO dataset are displayed in Figure 3.
Consistent with the no free lunch theorem, LIME , KernelSHAP , Occlusion , Integrated Gradients , and Gradient x Input perform best on binary perturbation neighbourhoods (Figure 3a) while SmoothGrad and Vanilla Gradients perform best on continuous perturbation neighborhoods (Figure 3b).
We observe consistent results across perturbation test types (top- k and bottom- k ), datasets, and models (Appendix A.6.3).
These findings have important implications: one should carefully consider the perturbation neighborhood not only when selecting a method to generate explanations but also when selecting a method to evaluate explanations.
In fact, the type of perturbations used to evaluate explanations directly determines explanation method performance.
Conclusions and Future Work
In this work, we formalized the local function approximation (LFA) framework and demonstrated that eight popular explanation methods can be characterized as instances of this framework with different local neighbourhoods and loss functions.
We also introduced the no free lunch theorem for explanation methods , showing that no single method can perform optimally across all neighbourhoods, and provided a guiding principle for choosing among methods.
The function approximation perspective captures the essence of an explanation – a simplification of the real world (i.e. a black box model) that is nonetheless accurate enough to be useful (i.e. predict outcomes of a set of perturbations).
When the real world is “simple”, an explanation should completely capture its behaviour, a hallmark expressed precisely by the guiding principle.
When the requirements from two explanations are distinct (i.e. they are trained to predict different sets of perturbations), then such explanations disagree and yet are accurate in their own domains, a principle precisely expressed by the no free lunch theorem.
Our work makes several fundamental contributions to the field.
The LFA framework enables the unification of popular explanation methods, bringing diverse methods into a common framework for the first time.
As a result, the LFA framework brings conceptual coherence and clarity : diverse explanation algorithms, even those seemingly unrelated to function approximation, perform LFA but differ in the way they perform it.
In addition, it enables theoretical simplicity : to study diverse explanation methods, instead of analyzing each method individually, one can simply analyze the
LFA framework and apply the findings to each method.
An example of this is the no free lunch theorem which analyzes the LFA framework and holds true for all instances of the LFA framework (including existing methods).
Furthermore, the LFA framework provides practical guidance by offering a principled approach to select among methods and design new ones (i.e., select or design the algorithm that performs LFA most faithfully in a given situation).
Our work also addresses key open questions in the field. In response to criticism about the lack of agreement in the field regarding the overall goals of post hoc explainability [25], our work points to function approximation as a principled goal.
It also addresses the disagreement problem [22] and explains why different methods generate different explanations for the same model prediction.
According to the LFA framework, this disagreement occurs because different methods approximate the black box model over different neighborhoods using different loss functions.
To build on this work, future research could consider the following directions.
First, we analyzed eight popular post hoc explanation methods and this analysis could be extended to other methods.
Second, our work focuses on the faithfulness (fidelity) rather than on the interpretability of explanations.
The latter is encapsulated in the “interpretable” model class G , which includes all the information about human preferences with regards to interpretability.
However, it is unclear what constitutes an interpretable explanation and elucidating this takes not only conceptual understanding but also human-computer interaction research such as user studies.
These are important directions for future research. Acknowledgements
The authors would like to thank the anonymous reviewers for their helpful feedback and all the funding agencies listed below for supporting this work.
This work is supported in part by the NSF awards IIS-2008461 and IIS-2040989, and research awards from Google, JP Morgan, Amazon,
Harvard Data Science Initiative and D 3 Institute at Harvard.
H.L. would like to thank Sujatha and Mohan Lakkaraju for their continued support and encouragement. T.H. is supported in part by an NSF GRFP fellowship.
The views expressed here are those of the authors and do not reflect the official policy or position of the funding agencies.
Checklist
For
all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] See Abstract, Section §1.
(b) Did you describe the limitations of your work?
[Yes] See Section §6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Section §6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Sections §3, §4, and Appendix.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Sections §3, §4, and Appendix. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We plan to include a zip file with the code in the supplementary materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Figure 3 and Appendix.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 5.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] We do not directly obtain data from individuals.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] To our knowledge, the data contains no such information nor content.
See Appendix. 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
We did not use crowdsourcing nor did we conduct research with human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A]
We did not use crowdsourcing nor did we conduct research with human subjects.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
We did not use crowdsourcing nor did we conduct research with human subjects.
A Appendix A.1 Proofs: Existing Methods are Instances of the LFA Framework (Section 3) A.1.1 LIME
The instance of the LFA framework with (1) interpretable model class G as the class of linear models where g ( x ) =
w ⊤ x , (2) perturbations of the form x ξ = x 0 ⊙ ξ where ξ ( ∈ { 0 , 1 } d ) ∼ π x 0 with π xbeing the exponential kernel (defined below), and (3) loss function as squared-error loss given by ℓ ( f, g, x 0 , ξ ) = ( f ( x ξ ) − g ( ξ )) 2 is equivalent to LIME . As defined in [31] (Section 3.4), the exponential kernel π x 0 ( ξ ) ∝ exp {− D ( x 0 , x ξ ) σ} with distance function D (such as cosine distance or L2 distance) and width σ . Proof.
For this instance of the LFA framework, by definition, the interpretable model g is given by:
[Equation]
ξ )] where p is the Bernouilli
[Equation]
[Equation]
5) distribution
Through importance sampling using a Bernouilli(0.5) proposal distribution (i.e. a Uniform(0,1) distribution over the space of binary inputs), the optimization setting of the LFA framework is that described for LIME by Ribeiro et al.
[31] (Equations 1 and 2).
A.1.2 KernelSHAP
The instance of the LFA framework with (1) interpretable model class G as the class of linear models where g ( x ) =
w ⊤ x , (2) perturbations of the form x ξ = x 0 ⊙ ξ where ξ ( ∈ { 0 , 1 } d ) ∼ π with π being the Shapley kernel (defined below), and (3) loss function as squared-error loss given by ℓ ( f, g, x 0 , ξ ) =
( f ( x ξ ) − g ( ξ )) 2 is equivalent to KernelSHAP .
As defined in [27] (Theorem 2), the Shapley kernel π ( ξ ) ∝
M − 1 ( Mk ) · k · ( M − k ) where M is the total number of elements in ξ and k is the number of ones in ξ .
Proof.
For this instance of the LFA framework, by definition, the interpretable model g is given by:
[Equation]
E
[Equation]
g, x
[Equation]
, ξ ) ·
π ( ξ )] where p is the Bernouilli(0.5) distribution
Through importance sampling using a Bernouilli(0.5) proposal distribution (i.e. a Uniform(0,1) distribution over the space of binary inputs), the optimization setting of the LFA framework is that described for KernelSHAP by Lundberg and Lee [27] (Equation 2 and Theorem 2).
A.1.3 Occlusion
The instance of the LFA framework with (1) interpretable model class G as the class of linear models where g ( x ) =
w ⊤ x , (2) perturbations of the form x ξ = x 0 ⊙ ξ where ξ ( ∈ { 0 , 1 } d ) is a random one-hot vector, and (3) loss function as squared-error loss given by ℓ ( f, g, x 0 , ξ ) = (∆ f − g ( ξ ))
where
∆ f = f ( x 0 ) − f ( x 0 (1 − ξ )) converges to Occlusion .
Proof.
This instance of the LFA framework optimizes g ( ξ ) to approximate ∆ f .
For ξ i (a one-hot vector with element i equal to 1), g ( ξ i ) =
w i and ∆ f
i is the difference in the model prediction when feature i takes the original value versus when feature i is set to zero.
∆ f is the definition of explanations generated by Occlusion .
Thus, in this instance of the LFA framework, the weights of g recover the explanations of Occlusion .
A.1.4 C-LIME
The instance of the LFA framework with (1) interpretable model class G as the class of linear models where g ( x ) =
w ⊤ x , (2) perturbations of the form x ξ = x 0 + ξ where ξ ( ∈ R d ) ∼ Normal (0 , σ 2 ) , and (3) loss function as squared-error loss given by ℓ ( f, g, x 0 , ξ ) =
( f ( x ξ ) − g ( ξ )) 2 is equivalent to
C-LIME .
Proof.
This instance of the LFA framework is equivalent to C-LIME by definition of C-LIME .
A.1.5 SmoothGrad
In this section, we provide two derivations showing the connection between the LFA framework and
SmoothGrad .
When using gradient-matching loss, the instance of the LFA framework is exactly equivalent to SmoothGrad given the same n perturbations.
When using squared-error loss, the instance of the LFA framework is equivalent to SmoothGrad asymptotically for a large number of perturbations.
Gradient-matching loss function
This instance of the LFA framework with (1) interpretable model class G as the class of linear models where g ( x ) =
w ⊤ x , (2) perturbations of the form x ξ = x 0 + ξ where ξ ( ∈ R d ) ∼ Normal (0 , σ 2 ) , and (3) loss function as gradient-matching loss given by ℓ gm ( f, g, x 0 , ξ )
= ∥∇ ξ f ( x ξ ) − ∇ ξ g ( ξ )
∥is equivalent to SmoothGrad .
In other words, for the same n perturbations, this instance of the LFA framework and SmoothGrad yield the same explanation.
Proof.
For this instance of the LFA framework, by definition, the interpretable model g is given by g ∗ =
arg min g ∈G L where:
[Equation]
To derive the solution for w , take the partial derivative of L w.r.t.
to w , set the partial derivative to zero, and solve for w .
[Equation]
Therefore, for the same n perturbations, the weights w of the interpretable model g are equivalent to the SmoothGrad explanations.
Squared-error loss function
Consider the instance of the LFA framework corresponding to SmoothGrad described above, except with loss function as squared-error loss given by ℓ ( f, g, x 0 , ξ )
=
( f ( x ξ ) − g ( ξ )) 2 .
This instance of the LFA framework converges to SmoothGrad in expectation.
Note that this instance of the LFA framework is C-LIME and its convergence to SmoothGrad in expectation is consistent with the results of Agarwal et al.
[2] who previously proved the same convergence (using a different approach).
Proof.
For this instance of the LFA framework, by definition, the interpretable model g is given by g ∗ =
arg min g ∈G L where:
[Equation]
To derive the solution for w , take the partial derivative of L w.r.t.
to w , set the partial derivative to zero, and solve for w .
∇ w L =−
[Equation]
E
[Equation]
E
[Equation]
[Equation]
( x
[Equation]
⊤ ]
[Equation]
σ 2 E ξ [ ∇ x ξ f ( x ξ ) ⊤ ] − σ
[Equation]
w ⊤ = 0 by Stein’s Lemma
[Equation]
Therefore, the weights w of the interpretable model g converge to SmoothGrad explanations in expectation.
A.1.6 Vanilla gradients
Consider the instance of the LFA framework corresponding to SmoothGrad described above (with loss function as either squared-error loss or gradient-matching loss).
As σ → 0 , this instance of the
LFA framework converges to Vanilla Gradients .
Proof.
Starting with the solution for w derived for SmoothGrad , take the limit of w as σ → 0 + .
[Equation]
η ξ
[Equation]
ξ )
=
p
[Equation]
ξ
[Equation]
[Equation]
σ ) =
[Equation]
x
[Equation]
f ( x 0 ) by property of the Dirac delta distribution
To derive the third line from the second line, we view the Normal density function p ( ξ ; 0 , σ ) as a nascent delta function η ξ ( ξ ) (which is defined such that lim σ → 0 + (cid:82) ∞
−∞ p ( ξ ; 0 , σ ) = δ ( ξ ) , where δ is the Dirac delta distribution) and by assuming that ∇ x 0 f ( x ξ ) has a compact support.
Therefore, the weights w of the interpretable model g converge to Vanilla Gradients explanations.
A.1.7 Integrated Gradients
This instance of the LFA framework with (1) interpretable model class G as the class of linear models where g ( x ) =
w ⊤ x , (2) perturbations of the form x ξ = x 0 ⊙ ξ where ξ ( ∈ R d ) ∼ Uniform (0 , 1) , and (3) loss function as gradient-matching loss given by ℓ gm ( f, g, x 0 , ξ ) = ∥∇ ξ f ( x ξ ) − ∇ ξ g ( ξ )
∥is equivalent to Integrated Gradients .
In other words, for the same n perturbations, this instance of the LFA framework and Integrated Gradients yield the same explanation.
Proof.
For this instance of the LFA framework, by definition, the interpretable model g is given by g ∗ =
arg min g ∈G L where:
[Equation]
Note that, by the chain rule, ∇ ξ f ( x
[Equation]
=
[Equation]
To derive the solution for w , take the partial derivative of L w.r.t.
to w , set the partial derivative to zero, and solve for w .
[Equation]
Therefore, the weights w of the interpretable model g converge to Integrated Gradients explanations in expectation.
A.1.8 Gradient × Input
Consider the instance of the LFA framework corresponding to Integrated Gradients described above, except with ξ ( ∈ R d ) ∼ Uniform ( a, 1) .
As a → 1 , this instance of the LFA framework converges to Gradient x Input .
Proof.
As a → 1 , ξ → ⃗ 1 , and w → x 0 ⊙ ∇ x 0 f ( x 0 ) .
Therefore, the weights w of the interpretable model g converge to Gradient x Input explanations.
A.2 Which Explanations Are Not Function Approximators?
In this section, we briefly discuss explanation methods that cannot be viewed as instances of the LFA framework.
In the cases listed below, the lack of connection to the LFA framework is mainly due to a property of the explanation method.
Model-independent methods : Some explanation methods are known to produce attributions that are independent from the model they intend to explain.
These methods cannot be cast in the LFA framework in any meaningful way due to the model recovery conditions we impose.
Such modelindependent methods include guided backpropagation [37] and DeconvNet [29], following theory by
Nie et al.
[28], as well as logit-gradient based methods [40] such as Grad-CAM [32], Grad-CAM++ [10], and FullGrad [39].
Modified-backpropagation methods : Some explanation methods such as DeepLIFT [33], guided backpropagation [ ? ], DeconvNet [29], and layer-wise relevance propagation [5] work by modifying the backpropagation equations and propagating attributions using finite-difference-like methods.
Such methods break an important property called “implementation invariance”, first identified by
Sundararajan et al.
[41], which states that two functionally identical models can have different attributions due to the lack of a chain rule for modified backpropagation methods.
This property ensures that such methods cannot be function approximators, as the attribution changes based on the function implementation.
Unsigned-gradient methods
: Some gradient-based methods return unsigned attribution values instead of the full signed values.
Such methods can be written in the LFA framework using the following loss function ℓ ( f, g, x 0 , ξ ) =
∥|∇
ξ f ( x 0 ⊕ ξ ) | − w g ∥ 2 where w g consists of the weights of the interpretable model g .
Using this loss function with different choices for neighborhoods gives unsigned versions of different gradient methods.
However, this loss function is not a valid loss function, i.e., ℓ ≠ = ⇒ f
= g .
Using this loss function, w g is always positive and thus cannot recover an underlying model’s negative weights.
A.3 Proof: No Free Lunch Theorem ((Section 4))
Theorem 4.
Assume a black-box model f and an interpretable model class G , and distance between them given by d ( f, G )
=
min g ∈G max x ∈X ℓ ( f, g, 0 , x ) , and that we are required to explain f around input x 0 , and the loss ℓ is a valid distance metric.
Then, for any explanation g ∗ on a neighborhood distribution ξ∼
Z 1 such that max ξ 1 ℓ ( f, g ∗ 1 , x 0 , ξ 1 ) ≤ ϵ , we can always find another neighborhood ξ 2 ∼ Z 2 such that max ξ 2 ℓ ( f, g ∗ 1 , x 0 , ξ 2 ) ≥ d ( f, G ) .
Proof.
Given an explanation g ∗ , we can find an "adversarial" input x adv such that x adv = arg max x ∈X ℓ ( f, g ∗ , 0 , x ) has a large error ℓ .
Construct perturbation x 2 = x 0 + ξ 2 such that p ( ξ 2 ) =
Uniform (0 , x adv − x 0 ) , which implies p ( x 2 ) =
Uniform ( x 0 , x adv ) .
By definition max ξ 2 ℓ ( f, g ∗ , x 0 , ξ 2 ) =
ℓ ( f, g ∗ , x 0 , x adv − x 0 ) =
max x ∈X ℓ ( f, g ∗ , 0 , x ) ≥ min g ∈G max x ∈X ℓ ( f, g, 0 , x ) = d ( f, G ) A salient feature of this proof is that it makes no assumptions about the form of model, input or output domains. This implies that the result applies equally to discrete and continuous domains, regression and classification tasks, and for any model type.
A.4 Summary of Properties of Existing Explanation Methods
[Table]
In this table, we consider the scale of g ’s weights when X ∈ R d .
A.5 Setup of Experiments
Datasets.
The first dataset is the life expectancy dataset from the Global Health Observatory data repository of the World Health Organization (WHO) [46].
The WHO dataset consists of demographic, economic, and health factors of 193 countries from 2000 to 2015 such as a country’s population, gross domestic product, health expenditure, human development index, infant mortality rate, hepatitis B immunization rate, and life expectancy.
The other dataset is the home equity line of credit (HELOC) dataset from the Explainable Machine Learning Challenge organized by FICO [13].
The HELOC dataset contains information on HELOC applications made by homeowners, such as an applicant’s installment balance, number of trades, longest delinquency period, and risk category (whether an applicant made payments without being 90 days overdue).
To our knowledge, these datasets do not contain personally identifiable information nor offensive content.
For the WHO dataset, missing values were imputed using kNN imputation with k = 5 .
For the
HELOC dataset, missing values were dropped.
For both datasets, continuous features were meancentered and then normalized to [0, 1] range.
Models.
{+For the WHO dataset, we train four models: a linear regression model (train MSE: 9 . 
+}
× 10 − 5 ; test MSE: 9 . 80 × 10 − 5 ) and three feed-forward neural networks.
The neural networks have 8-node hidden layers with tanh activation and a linear output layer.
The first neural network has 3 hidden layers (train MSE: 7 . 83 × 10 − 5 ; test MSE: 8 .
23 × 10 − 5 ), the second has 5 hidden layers (train MSE: 7 .
76 × 10 − 5 ; test MSE: 8 .
11 × 10 − 5 ), and the third has 8 hidden layers (train MSE: 7 . 78 × 10 − 5 ; test MSE: 8 . 20 × 10 − 5 ).
The neural networks will be referred to as NN1, NN2, and
NN3, respectively.
For the HELOC dataset, we train four models: a logistic regression model (train accuracy: 0.73; test accuracy: 0.74) and three feed-forward neural networks.
The neural networks have 8-node hidden layers with relu activation and an output layer with sigmoid activation.
The first neural network has 3 hidden layers (train accuracy: 0.75; test accuracy: 0.75), the second has 5 hidden layers (train accuracy: 0.75; test accuracy: 0.75), and the third has 8 hidden layers (train accuracy: 0.75; test accuracy: 0.75).
The neural networks will be referred to as NNA, NNB, and NNC, respectively.
Models were trained based on an 80/20 train/test split using stochastic gradient descent.
Hyperparameters were selected to reach decent model performance.
The emphasis is on generating explanations for individual model predictions, not on high model performance.
Thus, we do not focus on tuning model hyperparameters.
Linear and logistic regression models trained for 100 epochs while neural network models trained for 300 epochs.
All models used a batch size of 64 and a cosine annealing scheduler for the learning rate.
Hyperparameters
Explanation Methods.
Each explanation method is implemented using (1) the existing method and (2) the LFA framework.
For (1), we used Meta’s Captum library [21].
When using Captum, methods with number of perturbations as a parameter (i.e. LIME, KernelSHAP, SmoothGrad, and
Integrated Gradients) used 1000 perturbations, a number of perturbations at which explanations for the method converged.
For (2), we implemented the LFA framework, instantiating each method based on Table 1.
For each method, the number of perturbations is set to 1000 for the same reason above.
The interpretable model g is optimized using stochastic gradient descent.
The perturbations are split into a train and test set (80/20 split) and g ∗ is optimized based on test set performance.
Analyses were performed on GPUs.
The total amount of compute is approximately 54 GPU-hours.
A.6 Full Results for Experiments A.6.1 Experiment 1: Existing Methods Are Instances of the LFA Framework