F AST SHAP: R EAL -T IME S HAPLEY V ALUE E STIMATION
A BSTRACT
Although Shapley values are theoretically appealing for explaining black-box models, they are costly to calculate and thus impractical in settings that involve large, high-dimensional models.
To remedy this issue, we introduce FastSHAP, a new method for estimating Shapley values in a single forward pass using a learned explainer model.
To enable efﬁcient training without requiring ground truth Shapley values, we develop an approach to train FastSHAP via stochastic gradient descent using a weighted least squares objective function.
In our experiments with tabular and image datasets, we compare FastSHAP to existing estimation approaches and ﬁnd that it generates accurate explanations with an orders-ofmagnitude speedup.
I NTRODUCTION
With the proliferation of black-box models, Shapley values (Shapley, 1953) have emerged as a popular explanation approach due to their strong theoretical properties (Lipovetsky and Conklin, 2001; Štrumbelj and Kononenko, 2014; Datta et al., 2016; Lundberg and Lee, 2017).
In practice, however, Shapley value-based explanations are known to have high computational complexity, with an exact calculation requiring an exponential number of model evaluations (Van den Broeck et al., 2021).
Speed becomes a critical issue as models increase in size and dimensionality, and for the largest models in ﬁelds such as computer vision and natural language processing, there is an unmet need for signiﬁcantly faster Shapley value approximations that maintain high accuracy.
Recent work has addressed the computational challenges with Shapley values using two main approaches.
First, many works have proposed stochastic estimators (Castro et al., 2009; Štrumbelj and Kononenko, 2014; Lundberg and Lee, 2017; Covert et al., 2020) that rely on sampling either feature subsets or permutations; though often consistent, these estimators require many model evaluations and impose an undesirable trade-off between run-time and accuracy.
Second, some works have proposed model-speciﬁc approximations , e.g., for trees (Lundberg et al., 2020) or neural networks (Shrikumar et al., 2017; Chen et al., 2018b; Ancona et al., 2019; Wang et al., 2021); while generally faster, these approaches can still require many model evaluations, often induce bias, and typically lack ﬂexibility regarding the handling held-out features—a subject of ongoing debate in the ﬁeld (Aas et al., 2019;
Janzing et al., 2020; Frye et al., 2020; Covert et al., 2021).
Here, we introduce a new approach for efﬁcient Shapley value estimation: to achieve the fastest possible run-time, we propose learning a separate explainer model that outputs precise Shapley value estimates in a single forward pass.
Naïvely, such a learning-based approach would seem to require a large training set of ground truth Shapley values, which would be computationally intractable.
Instead, our approach trains an explainer model by minimizing an objective function inspired by the Shapley value’s weighted least squares characterization (Charnes et al., 1988), which enables efﬁcient gradient-based optimization.
Our contributions.
We introduce FastSHAP, an amortized approach for generating real-time Shapley value explanations.
1 We derive an objective function from the Shapley value’s weighted least
E n g li s h Sp r
i n g e r
FastSHAP
[Figure]
CXPlain
G a s P u m p
T e n c h squares characterization and investigate several ways to reduce gradient variance during training.
Our experiments show that FastSHAP provides accurate Shapley value estimates with an ordersof-magnitude speedup relative to non-amortized estimation approaches.
Finally, we also ﬁnd that
FastSHAP generates high-quality image explanations (ﬁg. 1) that outperform gradient-based methods (e.g., IntGrad and GradCAM) on quantitative inclusion and exclusion metrics.
B ACKGROUND
In this section, we introduce notation used throughout the paper and provide an overview of Shapley values and their weighted least squares characterization.
Let x 2 X be a random vector consisting of d features, or x = ( x 1 , . . .
, x d ) , and let y 2 Y = f 1 , . .
, K g be the response variable for a classiﬁcation problem.
We use s 2 f 0 , 1 g d to denote subsets of the indices f 1 , . . .
, d g and deﬁne x s :=
f x i g i : s i =1 .
The symbols x , y , s are random variables and x, y, s denote possible values.
We use 1 and 0 to denote vectors of ones and zeros in R d , so that 1 ⊤ s is a subset’s cardinality, and we use e i to denote the i th standard basis vector.
Finally, f ( x ; η ) : X 7!
∆ K − 1 is a model that outputs a probability distribution over y given x , and f y ( x ; η ) is the probability for the y th class.
2. S HAPLEY VALUES
Shapley values were originally developed as a credit allocation technique in cooperative game theory (Shapley, 1953), but they have since been adopted to explain predictions from black-box machine learning models (Štrumbelj and Kononenko, 2014; Datta et al., 2016; Lundberg and Lee, 2017).
For any value function (or set function) v : 2 d 7!
R , the Shapley values ϕ ( v ) 2 R d , or ϕ i ( v ) 2 R for each feature i = 1 , . . .
, d , are given by the formula
[Equation]
The difference v ( s + e i ) (cid:0) v ( s ) represents the i th feature’s contribution to the subset s , and the summation represents a weighted average across all subsets that do not include i .
In the model explanation context, the value function is chosen to represent how an individual prediction varies as different subsets of features are removed.
For example, given an input-output pair ( x, y ) , the prediction for the y th class can be represented by a value function v x,y deﬁned as
[Equation]
where the held out features x 1 − s are marginalized out using their joint marginal distribution p ( x 1 − s ) , and a link function (e.g., logit) is applied to the model output.
Recent work has debated the properties of different value function formulations, particularly the choice of how to remove features (Aas et al., 2019;
Janzing et al., 2020; Frye et al., 2020; Covert et al., 2021).
However, regardless of the formulation, this approach to model explanation enjoys several useful theoretical properties due to its use of Shapley values: for example, the attributions are zero for irrelevant features, and they are guaranteed to sum to the model’s prediction.
We direct readers to prior work for a detailed discussion of these properties (Lundberg and Lee, 2017; Covert et al., 2021).
Unfortunately, Shapley values also introduce computational challenges: the summation in eq.
(1) involves an exponential number of subsets, which makes it infeasible to calculate for large d .
Fast approximations are therefore required in practice, as we discuss next.
2. K ERNEL SHAP
KernelSHAP (Lundberg and Lee, 2017) is a popular Shapley value implementation that relies on an alternative Shapley value interpretation.
Given a value function v x,y ( s ) , eq.
(1) shows that the values ϕ ( v x,y ) are the features’ weighted average contributions; equivalently, their weighted least squares characterization says that they are the solution to an optimization problem over ϕ x,y 2 R d ,
[Equation]
Efﬁciency
[Equation]
where the distribution p ( s ) is deﬁned as
[Equation]
Shapley kernel
[Equation]
for s such that 0 < 1 ⊤ s < d (Charnes et al., 1988).
Based on this view of the Shapley value,
Lundberg and Lee (2017) introduced KernelSHAP, a stochastic estimator that solves an approximate version of eq.
(3) given some number of subsets sampled from p ( s ) .
Although the estimator is consistent and empirically unbiased (Covert and Lee, 2021), KernelSHAP often requires many samples to achieve an accurate estimate, and it must solve eq.
(3) separately for each input-output pair ( x, y ) .
As a result, it is unacceptably slow for some use cases, particularly in settings with large, high-dimensional models.
Our approach builds on KernelSHAP, leveraging the Shapley value’s weighted least squares characterization to design a faster, amortized estimation approach.
F AST SHAP
We now introduce FastSHAP, a method that amortizes the cost of generating Shapley values across many data samples.
FastSHAP has two main advantages over existing approaches: (1) it avoids solving separate optimization problems for each input to be explained, and (2) it can use similar data points to efﬁciently learn the Shapley value function ϕ ( v x,y ) .
3. A MORTIZING S HAPLEY VALUES
In our approach, we propose generating Shapley value explanations using a learned parametric function ϕ fast ( x , y ; θ ) : X (cid:2)
Y 7!
R d .
Once trained, the parametric function can generate explanations in a single forward pass, providing a signiﬁcant speedup over methods that approximate Shapley values separately for each sample ( x, y ) .
Rather than using a dataset of ground truth Shapley values for training, we train ϕ fast ( x , y ; θ ) by penalizing its predictions according to the weighted least squares objective in eq.
(3), or by minimizing the following loss,
[Equation]
where Unif ( y ) represents a uniform distribution over classes.
If the model’s predictions are forced to satisfy the Efﬁciency constraint, then given a large enough dataset and a sufﬁciently expressive model class for ϕ fast , the global optimizer ϕ fast ( x , y ; θ ∗ ) is a function that outputs exact Shapley values (see proof in appendix A).
Formally, the global optimizer satisﬁes the following:
[Equation]
We explore two approaches to address the efﬁciency requirement.
First, we can enforce efﬁciency by adjusting the Shapley value predictions using their additive efﬁcient normalization (Ruiz et al., 1998), which applies the following operation to the model’s outputs:
[Equation]
The normalization step can be applied at inference time and optionally during training; in appendix B, we show that this step is guaranteed to make the estimates closer to the true Shapley values.
Second, we can relax the efﬁciency property by augmenting L ( θ ) with a penalty on the efﬁciency gap (see eq.
; the penalty requires a parameter γ > 0 , and as we set γ !
1 we can guarantee that efﬁciency holds (see appendix A).
Algorithm 1 summarizes our training approach.
Empirical considerations.
Optimizing L ( θ ) using a single set of samples ( x, y, s ) is problematic because of high variance in the gradients, which can lead to poor optimization.
We therefore consider several steps to reduce gradient variance.
First, as is conventional in deep learning, we minibatch across multiple samples from p ( x ) .
Next, when possible, we calculate the loss jointly across all classes y 2 f 1 , . .
K g .
Then, we experiment with using multiple samples s (cid:24) p ( s ) for each input sample x .
Finally, we explore paired sampling , where each sample s is paired with its complement 1 (cid:0) s , which has been shown to reduce KernelSHAP’s variance (Covert and Lee, 2021).
Appendix C shows proofs that these steps are guaranteed to reduce gradient variance, and ablation experiments in appendix D demonstrate their improvement on FastSHAP’s accuracy.
3. A DEFAULT VALUE FUNCTION FOR F AST SHAP
Input: Value function v x,y , learning rate α
Output: FastSHAP explainer ϕ
fast ( x , y ; θ ) initialize ϕ fast ( x , y ; θ ) while not converged do sample x (cid:24) p ( x ) , y (cid:24) Unif ( y ) , s (cid:24) p ( s ) predict ˆ ϕ ϕ fast ( x, y ; θ ) if normalize then set ˆ ϕ ˆ ϕ + d −(cid:16) v x,y ( 1 ) (cid:0) v x,y ( 0 ) (cid:0) 1 T ˆ ϕ (cid:17) end calculate
L (cid:16) v x,y ( s ) (cid:0) v x,y ( 0 ) (cid:0) s T ˆ ϕ (cid:17)update θ θ (cid:0)
α r θ L end
FastSHAP has the ﬂexibility to work with any value function v x,y ( s ) .
Here, we describe a default value function that is useful for explaining predictions from a classiﬁcation model.
The value function’s aim is to assess, for each subset s , the classiﬁcation probability when only the features x s are observed.
Because most models f ( x ; η ) do not support making predictions without all the features, we require an approximation that simulates the inclusion of only x s (Covert et al., 2021).
To this end, we use a supervised surrogate model (Frye et al., 2020; Jethani et al., 2021) to approximate marginalizing out the remaining features x 1 − s using their conditional distribution.
Separate from the original model f
( x ; η ) , the surrogate model p surr ( y j m ( x , s ); β ) takes as input a vector of masked features m ( x, s ) , where the masking function m replaces features x i such that s i = 0 with a [mask] value that is not in the support of X .
Similar to prior work (Frye et al., 2020;
Jethani et al., 2021), the parameters β are learned by minimizing the following loss function:
[Equation]
It has been shown that the global optimizer to eq.
(7), or p surr ( y j m ( x , s ); β ∗ ) , is equivalent to marginalizing out features from f ( x ; η ) with their conditional distribution (Covert et al., 2021):
[Equation]
The choice of distribution over p ( s ) does not affect the global optimizer of eq.
but we use the
Shapley kernel to put more weight on subsets likely to be encountered when training FastSHAP.
We use the surrogate model as a default choice for two reasons.
First, it requires a single prediction for each evaluation of v x,y ( s ) , which permits faster training than the common approach of averaging across many background samples (Lundberg and Lee, 2017; Janzing et al., 2020).
Second, it yields explanations that reﬂect the model’s dependence on the information communicated by each feature, rather than its algebraic dependence (Frye et al., 2020; Covert et al., 2021).
R ELATED W ORK
Recent work on Shapley value explanations has largely focused on how to remove features (Aas et al., 2019; Frye et al., 2020; Covert et al., 2021) and how to approximate Shapley values efﬁciently (Chen et al., 2018b; Ancona et al., 2019; Lundberg et al., 2020; Covert and Lee, 2021).
Model-speciﬁc approximations are relatively fast, but they often introduce bias and are entangled with speciﬁc feature removal approaches (Shrikumar et al., 2017; Ancona et al., 2019; Lundberg et al., 2020).
In contrast, model-agnostic stochastic approximations are more ﬂexible, but they must trade off run-time and accuracy in the explanation.
For example, KernelSHAP samples subsets to approximate the solution to a weighted least squares problem (Lundberg and Lee, 2017), while other
[Figure]
approaches sample marginal contributions (Castro et al., 2009; Štrumbelj and Kononenko, 2014) or feature permutations (Illés and Kerényi, 2019; Mitchell et al., 2021).
FastSHAP trains an explainer model to output an estimate that would otherwise require orders of magnitude more model evaluations, and, unlike other fast approximations, it is agnostic to the model class and feature removal approach.
Other methods have been proposed to generate explanations using learned explainer models.
These are referred to as amortized explanation methods (Covert et al., 2021; Jethani et al., 2021), and they include several approaches that are comparable to gradient-based methods in terms of compute time (Dabkowski and Gal, 2017;
Chen et al., 2018a; Yoon et al., 2018; Schwab and Karlen, 2019;
Schulz et al., 2020; Jethani et al., 2021).
Notably, one approach generates a training dataset of ground truth explanations and then learns an explainer model to output explanations directly (Schwab and
Karlen, 2019)—a principle that can be applied with any attribution method, at least in theory.
However, for Shapley values, generating a large training set would be very costly, so FastSHAP sidesteps the need for a training set using a custom loss function based on the Shapley value’s weighted least squares characterization (Charnes et al., 1988).
S TRUCTURED D ATA E XPERIMENTS
We analyze FastSHAP’s performance by comparing it to several well-understood baselines.
First, we evaluate its accuracy on tabular (structured) datasets by comparing its outputs to the ground truth Shapley values.
Then, to disentangle the beneﬁts of amortization from the in-distribution value function, we make the same comparisons using different value function formulations v x,y ( s ) .
Unless otherwise stated, we use the surrogate model value function introduced in section 3.2.
Later, in section 6, we test FastSHAP’s ability to generate image explanations.
Baseline methods.
To contextualize FastSHAP’s accuracy, we compare it to several nonamortized stochastic estimators.
First, we compare to KernelSHAP (Lundberg and Lee, 2017) and its acceleration that uses paired sampling (Covert and Lee, 2021).
Next, we compare to a permutation sampling approach and its acceleration that uses antithetical sampling (Mitchell et al., 2021).
As a performance metric, we calculate the proximity to Shapley values that were obtained by running KernelSHAP to convergence; we use these values as our ground truth because KernelSHAP is known to converge to the true Shapley values given inﬁnite samples (Covert and Lee, 2021).
These baselines were all run using an open-source implementation.
Implementation details.
We use either neural networks or tree-based models for each of f ( x ; η ) and p surr ( y j m ( x , s ); β ) .
The FastSHAP explainer model ϕ fast ( x , y ; θ ) is implemented with a network g ( x ; θ ) :
X !
R d (cid:2)Y that outputs a vector of Shapley values for every y 2 Y ; deep neural networks are ideal for FastSHAP because they have high representation capacity, they can provide many-to-many mappings, and they can be trained by stochastic gradient descent.
Appendix D contains more details about our implementation, including model classes, network architectures and training hyperparameters.
[Figure]
We also perform a series of experiments to determine several training hyperparameters for FastSHAP, exploring (1) whether or not to use paired sampling, (2) the number of subset samples to use, and (3) how to best enforce the efﬁciency constraint.
Based on the results (see appendix D), we use the following settings for our tabular data experiments: we use paired sampling, between 32 andsamples of s per x sample, additive efﬁcient normalization during both training and inference, and we set γ = 0 (since the normalization step is sufﬁcient to enforce efﬁciency).
5. A CCURACY OF F AST SHAP EXPLANATIONS
Here, we test whether FastSHAP’s estimates are close to the ground truth Shapley values.
Our experiments use data from a 1994 United States census , a bank marketing campaign, bankruptcy statistics, and online news articles (Dua and Graff, 2017).
The census data contains 12 input features, and the binary label indicates whether a person makes over $ 50 K a year (Kohavi et al., 1996).
The marketing dataset contains 17 input features, and the label indicates whether the customer subscribed to a term deposit (Moro et al., 2014).
The bankruptcy dataset contains 96 features describing various companies and whether they went bankrupt (Liang et al., 2016).
The news dataset contains 60 numerical features about articles published on Mashable, and our label indicates whether the share count exceeds the median number (1400) (Fernandes et al., 2015).
The datasets were each split 80/10/10 for training, validation and testing.
In ﬁg. 2, we show the distance of each method’s estimates to the ground truth as a function of the number of model evaluations for the news , census and bankruptcy datasets.
Figure 3 shows results for the marketing dataset with three different value functions (see section 5.2).
For the baselines, each sample s requires evaluating the model given a subset of features, but since FastSHAP requires only a single forward pass of ϕ fast ( x , y ; θ ) , we show it as a horizontal line.
To reach FastSHAP’s level of accuracy on the news , census and bankruptcy datasets, KernelSHAP requires between 1,200-2,000 model evaluations; like prior work (Covert and Lee, 2021), we ﬁnd that paired sampling improves KernelSHAP’s rate of convergence, helping reach FastSHAP’s accuracy in 250-1,000 model evaluations.
The permutation sampling baselines tend to be faster: the original version requires between 300-1,000 evaluations, and antithetical sampling takes 200-500 evaluations to reach an accuracy equivalent to FastSHAP.
Across all four datasets, however,
FastSHAP achieves its level of accuracy at least at least 600 (cid:2) faster than the original version of
KernelSHAP, and 200 (cid:2) faster than the best non-amortized baseline.
5. D ISENTANGLING AMORTIZATION AND THE CHOICE OF VALUE FUNCTION
In this experiment, we verify that FastSHAP produces accurate Shapley value estimates regardless of the choice of value function.
We use the marketing dataset for this experiment and test the following value functions:
(Surrogate/In-distribution
(Marginal/Out-of-distribution
η )] 3.
(Baseline removal) v x,y ( s ) =
f y ( x s , x b 1 − s ; η ) , where x b 2 X are ﬁxed baseline values (the mean for continuous features and mode for discrete ones)
H
o r s e
FastSHAP
KernelSHAP
KernelSHAP-S
GradCAM
Integrated Gradients
SmoothGrad
DeepSHAP
CXPlain
B i r d
A i r p l a n e
In ﬁg. 3 we compare FastSHAP to the same non-amortized baseline methods, where each method generates Shapley value estimates using the value functions listed above.
The results show that
FastSHAP maintains the same computational advantage across all three cases: to achieve the same accuracy as FastSHAP’s single forward pass, the baseline methods require at least 200 model evaluations, but in some cases up to nearly 1,000.
I MAGE E XPERIMENTS
Images represent a challenging setting for Shapley values due to their high dimensionality and the computational cost of model evaluation.
We therefore compare FastSHAP to KernelSHAP on two image datasets.
We also consider several widely used gradient-based explanation methods, because they are the most commonly used methods for explaining image classiﬁers.
6. D ATASETS
We consider two popular image datasets for our experiments.
CIFAR-10 (Krizhevsky et al., 2009) contains 60 , 000 32 (cid:2) 32 images across 10 classes, and we use 50 , 000 samples for training and 5 , 000 samples each for validation and testing.
Each image is resized to 224 (cid:2) 224 using bilinear interpolation to interface with the ResNet-50 architecture (He et al., 2016).
Figure 4 shows example
CIFAR-10 explanations generated by each method.
The Imagenette dataset (Howard and Gugger, 2020), a subset of 10 classes from the ImageNet dataset, contains 13 , 394 total images.
Each image is cropped to keep the 224 (cid:2) 224 central region, and the data is split 9 , 469 / 1 , 963 / 1 , 962 .
Example
Imagenette explanations are shown in ﬁg. 1.
6. E XPLANATION METHODS
We test three Shapley value estimators, FastSHAP, KernelSHAP, and DeepSHAP (Lundberg and
Lee, 2017), where the last is an existing approximation designed for neural networks.
We test KernelSHAP with the zeros baseline value function, which we refer to simply as KernelSHAP, and with the in-distribution surrogate value function, which we refer to as KernelSHAP-S. We also compare these methods to the gradient-based explanation methods GradCAM (Selvaraju et al., 2017),
SmoothGrad (Smilkov et al., 2017) and IntGrad (Sundararajan et al., 2017).
Gradient-based methods are relatively fast and have therefore been widely adopted for explaining image classiﬁers.
Finally, we also compare to CXPlain (Schwab and Karlen, 2019), an amortized explanation method that generates attributions that are not based on Shapley values.
Implementation details.
The models f ( x ; η ) and p surr ( y j m ( x , s ); β ) are both ResNet-50 networks (He et al., 2016) pretrained on ImageNet and ﬁne-tuned on the corresponding imaging dataset.
FastSHAP, CXPlain, and KernelSHAP are all implemented to output 14 (cid:2) 14 superpixel attributions for each class.
For FastSHAP, we parameterize ϕ fast ( x , y ; θ ) to output superpixel attributions: we use an identical pretrained ResNet-50 but replace the ﬁnal layers with a 1 (cid:2) 1 convolutional layer so that the output is 14 (cid:2) 14 (cid:2) K (see details appendix D).
We use an identical network to produce attributions for CXPlain.
For FastSHAP, we do not use additive efﬁcient normalization, and we set γ = 0 ; we ﬁnd that this relaxation of the Shapley value’s efﬁciency property does not inhibit
FastSHAP’s ability to produce high-quality image explanations.
KernelSHAP and KernelSHAP-S are implemented using the shap 3 package’s default parameters, and GradCAM, SmoothGrad, and
IntGrad are implemented using
6. Q UALITATIVE REMARKS
Explanations generated by each method are shown in ﬁg. 4 for CIFAR-10 and ﬁg. 1 for Imagenette (see appendix E for more examples).
While a qualitative evaluation is insufﬁcient to draw conclusions about each method, we offer several remarks on these examples.
FastSHAP, and to some extent
GradCAM, appear to reliably highlight the important objects, while the KernelSHAP explanations are noisy and fail to localize important regions.
To a lesser extent, CXPlain occasionally highlights important regions.
In comparison, the remaining methods (SmoothGrad, IntGrad and DeepSHAP) are granulated and highlight only small parts of the key objects.
Next, we consider quantitative metrics that test these observations more systematically
[Figure]
6. Q UANTITATIVE EVALUATION
[Figure]
Evaluating the quality of Shapley value estimates requires access to ground truth Shapley values, which is computationally infeasible for images.
Instead, we use two metrics that evaluate an explanation’s ability to identify informative image regions.
These metrics build on several recent proposals ( Petsiuk et al., 2018; Hooker et al., 2018;
Jethani et al., 2021) and evaluate the model’s classiﬁcation accuracy after including or excluding pixels according to their estimated importance.
Similar to Jethani et al.
(2021), we begin by training a single evaluation model p eval to approximate the f ( x ; η ) model’s output given a subset of features; this serves as an alternative to training separate models on each set of features (Hooker et al., 2018) and offers a more realistic option than masking features with zeros (Schwab and Karlen, 2019).
This procedure is analogous to the p surr training procedure in section 3.2, except it sets the subset distribution to p ( s )
=
Uniform( f 0 , 1 g d ) to ensure all subsets are equally weighted.
Next, we analyze how the model’s predictions change as we remove either important or unimportant features according to each explanation.
Using a set of 1,000 images, each image is ﬁrst labeled by the original model f
( x ; η ) using the most likely predicted class.
We then use explanations generated by each method to produce feature rankings and compute the top-1 accuracy (a measure of agreement with the original model) as we either include or exclude the most important features, ranging from 0100%.
The area under each curve (AUC) is termed the Inclusion AUC or Exclusion AUC .
These metrics match the idea that an accurate image explanation should (1) maximally degrade the performance of p eval when important features are excluded, and (2) maximally improve the performance of p eval when important features are included (Petsiuk et al., 2018; Hooker et al., 2018).
The explanations are evaluated by removing superpixels; for gradient-based methods, we coarsen the explanations using the sum total importance within each superpixel.
In appendix E, we replicate these metrics using log-odds rather than top-1 accuracy, ﬁnding a similar ordering among methods.
Results.
Table 1 shows the Inclusion and Exclusion AUC achieved by each method for both
CIFAR-10 and Imagenette.
In ﬁg. 5, we also present the curves used to generate these AUCs for
Imagenette
Lower Exclusion AUCs and higher Inclusion AUCs are better.
These results show that
FastSHAP outperforms all baseline methods when evaluated with Exclusion AUC
: when the pixels identiﬁed as important by FastSHAP are removed from the images
, the sharpest decline in top ac curacy is observed.
Additionally, FastSHAP performs well when evaluated on the basis of Inclusion
AUC, second only to KernelSHAP-S.
[Table]
For Imagenette, GradCAM performs competitively with FastSHAP on Exclusion AUC and
KernelSHAP-S marginally beats FastSHAP on Inclusion AUC.
KernelSHAP-S also outperforms on Inclusion AUC with CIFAR-10, which is perhaps surprising given its high level of noise (ﬁg. 4).
However, KernelSHAP-S does not do as well when evaluated using Exclusion AUC, and GradCAM does not do as well on Inclusion AUC.
The remaining methods are, by and large, not competitive on either metric (except DeepSHAP on CIFAR-10 Inclusion AUC).
An accurate explanation should perform well on both metrics
[Table]
so these results show that FastSHAP provides the most versatile explanations, because it is the only approach to excel at both Inclusion and Exclusion AUC.
Finally
[Table]
we also test FastSHAP’s robustness to limited training data.
In appendix E, we ﬁnd that
FastSHAP outperforms most baseline methods on Inclusion and Exclusion AUC when using just 25% of the Imagenette data
[Table]
and that it remains competitive when using just 10%.
6. S PEED EVALUATION
[Table]
The image experiments were run using 8 cores of an Intel Xeon Gold 6148 processor and a single NVIDIA Tesla V100.
Table 2 records the time required to explain 1,000 images.
For FastSHAP, KernelSHAP-S and CXPlain, we also report the time required to train the surrogate and/or explainer models.
The amortized explanation methods, FastSHAP and CXPlain, incur a ﬁxed training cost but very low marginal cost for each explanation.
The gradient-based methods are slightly slower, but
KernelSHAP requires signiﬁcantly more time.
These results suggest that FastSHAP is well suited for real-time applications where it is crucial to keep explanation times as low as possible.
Further, when users need to explain a large quantity of data, FastSHAP’s low explanation cost can quickly compensate for its training time.
D ISCUSSION
In this work, we introduced FastSHAP, a method for estimating Shapley values in a single forward pass using a learned explainer model.
To enable efﬁcient training, we sidestepped the need for a training set and derived a learning approach from the Shapley value’s weighted least squares characterization.
Our experiments demonstrate that FastSHAP can produce accurate Shapley value estimates while achieving a signiﬁcant speedup over non-amortized approaches, as well as more accurate image explanations than popular gradient-based methods.
While Shapley values provide a strong theoretical grounding for model explanation, they have not been widely adopted for explaining large-scale models due to their high computational cost.
FastSHAP can solve this problem, making fast and high-quality explanations possible in ﬁelds such as computer vision and natural language processing.
By casting model explanation as a learning problem, FastSHAP stands to beneﬁt as the state of deep learning advances, and it opens a new direction of research for efﬁcient Shapley value estimation.
R EPRODUCIBILITY
Code to implement FastSHAP is available online in two separate repositories: https://github. com/iancovert/fastshap contains a PyTorch implementation and https://github. com/neiljethani/fastshap/ a TensorFlow implementation, both with examples of tabular and image data experiments.
The complete code for our experiments is available at https: //github.com/iclr1814/fastshap , and details are described throughout section 5 and section 6, with model architectures and hyperparameters reported in appendix D. Proofs for our theoretical claims are provided in appendix A, appendix B, and appendix C.
A CKNOWLEDGEMENTS
We thank the reviewers for their thoughtful feedback, and we thank the Lee Lab for helpful discussions.
Neil Jethani was partially supported by NIH T32 GM136573.
Mukund Sudarshan was partially supported by a PhRMA Foundation Predoctoral Fellowship.
Mukund Sudarshan and Rajesh Ranganath were partly supported by NIH/NHLBI Award R01HL148248, and by NSF
Award 1922658 NRT-HDR: FUTURE Foundations, Translation, and Responsibility for Data Science.
Ian Covert and Su-In Lee were supported by the NSF Awards CAREER DBI-1552309 and
DBI-1759487; the NIH Awards R35GM128638 and R01NIAAG061132; and the American Cancer
Society Award 127332-RSG-15-097-01-TBG.
A F AST SHAP GLOBAL OPTIMIZER
Here, we prove that FastSHAP is trained using an objective function whose global optimizer outputs the true Shapley values.
Recall that the loss function for the explainer model ϕ fast ( x , y ; θ ) is
[Equation]
As mentioned in the main text, it is necessary to force the model to satisfy the Efﬁciency constraint, or the property that the predictions from ϕ fast ( x , y ; θ ) satisfy
[Equation]
One option for guaranteeing the efﬁciency property is to adjust the model outputs using their additive efﬁcient normalization (see section 3.1).
Incorporating this constraint on the predictions, we can then view the loss function as an expectation across ( x , y ) and write the expected loss for each sample ( x, y ) as a separate optimization problem over the variable ϕ x,y 2 R d :
[Equation]
x
[Equation]
This is a constrained weighted least squares problem with a unique global minimizer, and it is precisely the Shapley value’s weighted least squares characterization (see eq.
We can therefore conclude that the optimal prediction for each pair ( x, y ) is the true Shapley values, or that ϕ ∗ x,y = ϕ
( v x,y ) .
As a result, the global optimizer for our objective is a model ϕ fast ( x , y ; θ ∗ ) that outputs the true Shapley values almost everywhere in the data distribution p ( x , y ) .
Achieving the global optimum requires the ability to sample from p ( x ) (or a sufﬁciently large dataset), perfect optimization, and a function class for ϕ fast ( x , y ; θ ) that is expressive enough to contain the global optimizer.
The universal approximation theorem (Cybenko, 1989; Hornik, 1991) implies that a sufﬁciently large neural network can represent the Shapley value function to arbitrary accuracy as long as it is a continuous function.
Speciﬁcally, we require the function h y ( x )
=
ϕ ( v x,y ) to be continuous in x for all y .
This holds in practice when we use a surrogate model parameterized by a continuous neural network, because v x,y ( s ) is continuous in x for all ( s, y ) , and the Shapley value is a linear combination of v x,y ( s ) across different values of s (see eq. Another approach to enforce the efﬁciency property is by using efﬁciency regularization, or penalizing the efﬁciency gap in the explainer model’s predictions (section 3.1). If we incorporate this regularization term with parameter γ > 0 , then our objective yields the following optimization problem for each ( x, y ) pair: [Equation]
For ﬁnite hyperparameter values γ 2 [0 , 1 ) , this problem relaxes the Shapley value’s efﬁciency property and eliminates the requirement that predictions must sum to the grand coalition’s value. However, as we let γ ! 1 , the penalty term becomes closer to the hard constraint in eq.
Note that in practice, we use ﬁnite values for γ and observe sufﬁciently accurate results, whereas using excessively large γ values would render gradient-based optimization ineffective. B A DDITIVE EFFICIENT NORMALIZATION
Here, we provide a geometric interpretation for the additive efﬁcient normalization step and prove that it is guaranteed to yield Shapley value estimates closer to their true values.
Consider a game v with Shapley values ϕ ( v ) 2 R d , and assume that we have Shapley values estimates ˆ ϕ 2 R d that do not satisfy the efﬁciency property.
To force this property to hold, we can project these estimates onto the efﬁcient hyperplane , or the subset of R d where the efﬁciency property is satisﬁed.
This corresponds to solving the following optimization problem over ϕ eﬀ 2 R d :
[Equation]
We can solve the problem via its Lagrangian, denoted by L ( ϕ eﬀ , ν ) , with the Lagrange multiplier ν 2 R as follows:
[Equation]
This transformation, where the efﬁciency gap is split evenly and added to each estimate, is known as additive efﬁcient normalization (Ruiz et al., 1998).
We implement it as an output layer for FastSHAP’s predictions to ensure that they satisfy the efﬁciency property (section 3).
This step can therefore be understood as a projection of the network’s output onto the efﬁcient hyperplane.
The normalization step is guaranteed to produce corrected estimates ϕ ∗ eﬀ that are closer to the true
Shapley values ϕ ( v ) than the original estimates ˆ ϕ .
To see this, note that the projection step guarantees that ˆ ϕ (cid:0)
ϕ ∗ eﬀ and ϕ ∗ eﬀ (cid:0)
ϕ ( v ) are orthogonal vectors, so the Pythagorean theorem yields the following inequality:
[Equation]
C R EDUCING GRADIENT VARIANCE
Recall that our objective function L ( θ ) is deﬁned as follows:
[Equation]
The objective’s gradient is given by
[Equation]
where we deﬁne
[Equation]
When FastSHAP is trained with a single sample ( x, y, s ) , the gradient covariance is given by
Cov (cid:0) r ( x , y , s ; θ ) (cid:1) , which may be too large for effective optimization.
We use several strategies to reduce gradient variance.
First, given a model that outputs estimates for all classes y 2 f 1 , . . .
, K g , we calculate the loss jointly for all classes.
This yields gradients that we denote as r ( x , s ; θ ) , deﬁned as
[Equation]
where we have the relationship
[Equation]
due to the law of total covariance.
Next, we consider minibatches of b independent x samples, which yields gradients r b ( x , s ; θ ) with covariance given by
[Equation]
We then consider sampling m independent coalitions s for each input x , resulting in the gradients r mb ( x , s ; θ ) with covariance given by
[Equation]
Finally, we consider a paired sampling approach, where each sample s (cid:24) p ( s ) is paired with its complement 1 (cid:0) s .
Paired sampling has been shown to reduce KernelSHAP’s variance (Covert and Lee, 2021), and our experiments show that it helps improve FastSHAP’s accuracy (appendix D).
The training algorithm in the main text is simpliﬁed by omitting these gradient variance reduction techniques, so we also provide algorithm 2 below, which includes minibatching, multiple coalition samples, paired sampling, efﬁciency regularization and parallelization over all output classes.
Input: Value function v x,y , learning rate α , batch size b , samples m , penalty parameter γ
Output: FastSHAP explainer ϕ
fast ( x , y ; θ ) initialize ϕ fast ( x , y ; θ ) while not converged do set R 0 , L for i = 1 , . . .
, b do sample x (cid:24) p ( x ) for y = 1 , . . .
, K do predict ˆ ϕ ϕ fast ( x, y ; θ ) calculate R
R + (cid:16) v x,y ( 1 ) (cid:0) v x,y ( 0 ) (cid:0)
⊤ ˆ ϕ (cid:17)// Pre-normalization if normalize then set ˆ ϕ ˆ ϕ + d −(cid:16) v x,y ( 1 ) (cid:0) v x,y ( 0 ) (cid:0) 1 ⊤ ˆ ϕ (cid:17) endfor j = 1 , . .
, m do if paired sampling
and i mod 2 = 0 then set s 1 (cid:0) s //
Invert previous subset else sample s (cid:24) p ( s ) end calculate L
L + (cid:16) v x,y ( s ) (cid:0) v x,y ( 0 ) (cid:0) s ⊤ ˆ ϕ (cid:17)end end end update θ θ (cid:0)
α r θ (cid:0) L bmK [Equation] (cid:1) end
[Figure]
D F AST SHAP MODELS AND HYPERPARAMETERS
In this section, we describe the models and architectures used for each dataset, as well as the hyperparameters used when training FastSHAP.
D. M ODELS
Tabular datasets.
For the original model f ( x ; η ) , we use neural networks for the news and marketing datasets and gradient boosted trees for the census (LightGBM (Ke et al., 2017)) and bankruptcy (XGBoost (Chen and Guestrin, 2016)) datasets.
The FastSHAP model ϕ fast ( x , y ; θ ) and the surrogate model p surr ( y j m ( x , s ); β ) are implemented using neural networks that consist of 2-3 fully connected layers with 128 units and ReLU activations.
The p surr models use a softmax output layer, while ϕ fast has no output activation.
The models are trained using Adam with a learning rate of 10 − 3 , and we use a learning rate scheduler that multiplies the learning rate by a factor of 0 .after 3 epochs of no validation loss improvement.
Early stopping was triggered after the validation loss ceased to improve for 10 epochs.
Image datasets.
The models f ( x ; η ) and p surr are ResNet-50 models pretrained on Imagenet.
We use these without modiﬁcation to the architecture and ﬁne-tune them on each image dataset.
To create the ϕ fast ( x , y ; θ ) model, we modify the architecture to return a tensor of size 14 (cid:2) 14 (cid:2) K .
First, the layers after the 4th convolutional block are removed; the output of this block is 14 (cid:2) 14 (cid:2) 256 .
We then append a 2D convolutional layer with K ﬁlters, each of size 1 (cid:2) 1 , so that the output is 14 (cid:2) 14 (cid:2) K and the y th 14 (cid:2) 14 slice corresponds to the superpixel-level Shapley values for each class y 2 Y .
Each model is trained using Adam with a learning rate of 10 − 3 , and we use a learning rate scheduler that multiplies the learning rate by a factor of 0 .
8 after 3 epochs of no validation loss improvement.
Early stopping was triggered after the validation loss ceased to improve forepochs.
D. F AST SHAP HYPERPARAMETERS
We now explore various settings of FastSHAP’s hyperparameters and observe their impact on FastSHAP’s performance.
There are two types of hyperparameters: sampling hyperparameters, which affect the number of samples of s taken during training, and efﬁciency hyperparameters, which control how we enforce the Efﬁciency constraint.
Sampling hyperparameters include: (1) whether to use paired sampling, and (2) the number of samples of s per x to take during training.
Efﬁciency hyperparameters include: (1) the choice of γ in eq.
(4), and (2) whether to perform the additive efﬁcient normalization during training, inference or both.
To understand the effect of sampling hyperparameters, we perform experiments using the same tabular datasets from the main text.
We use the in-distribution value function p surr and compute the ground truth SHAP values the same way as in our previous experiments (i.e., by running KernelSHAP to convergence).
Figure 6 shows the mean ℓ 2 distance between FastSHAP’s estimates and the ground truth.
We ﬁnd that across all four datasets, increasing the number of training samples of s generally improves the mean ℓ 2 distance to ground truth.
We also ﬁnd that for any ﬁxed number of samples (greater than 1), using paired sampling improves FastSHAP’s accuracy.
Table 3 shows the results of an ablation study for the efﬁciency hyperparameters.
Normalization (or Norm. ) refers to the additive efﬁcient normalization step (applied during training and inference,
[Table]
or only during inference), and penalty refers to the efﬁciency regularization technique with the parameter set to γ = 0 .
We ﬁnd that using normalization during training uniformly achieves better results than without normalization or with normalization only during inference.
The efﬁciency regularization approach proves to be less effective, generally leading to less accurate Shapley value estimates.
Based on these results, we opt to use additive efﬁcient normalization in our tabular data experiments.
E A DDITIONAL RESULTS FOR IMAGE EXPERIMENTS
In this section, we provide additional results for the FastSHAP image experiments.
E. I NCLUSION AND EXCLUSION METRICS
Table 4 shows our inclusion and exclusion metrics when replicated using log-odds rather than accuracy.
Similar to our metrics described in the main text, we choose the class predicted by the original model for each image, and we measure the average log-odds for that class as we include or exclude important features according to the explanations generated by each method.
The results conﬁrm roughly the same ordering between methods, with FastSHAP being the only method to achieve strong results on both metrics for both datasets.
Figure 7 shows the raw inclusion and exclusion curves for both the accuracy and log-odds-derived metrics.
[Table]
[Figure]
Exclusion
[Figure]
Inclusion
[Figure]
cc
[Figure]
a c y
Exclusion Curve
[Figure]
Inclusion
[Figure] [Figure]
[Figure]
cc
[Figure]
a c y
Exclusion Curve
[Figure]
Inclusion
[Figure] [Figure]
[Figure]
Exclusion AUC
[Figure]
Inclusion AUC Learning Curve
[Figure]
E. F AST SHAP ROBUSTNESS TO LIMITED DATA
To test FastSHAP’s robustness to the size of the training data
[Figure]
we compare its performance when trained with varying amounts of the Imagenette dataset.
Figure 8 plots the change in inclusion and exclusion AUC, calculated using top-1 accuracy, achieved when training FastSHAP with 95%, 85%, 75%, 50%, 25%, 15%, 10%
[Figure]
and 5% of the training dataset.
We ﬁnd that FastSHAP remains competitive when using just 10% of the data, and that it outperforms most baseline methods by a large margin when using just 25%.
E. E XAMPLE F AST SHAP IMAGE EXPLANATIONS
Finally, we show additional explanations generated by FastSHAP and the baseline methods for both
CIFAR-10 and Imagenette.
Airplane -16.
[Table]
Automobile -11.
Bird 11.
Cat -16.
Deer -
[Table] [Table]
Dog -
[Table]
Frog
[Table]
Horse -
[Table]
Ship
[Table]
Truck
[Table] [Table]
[Table] [Table]
[Table] [Table]
C a t
FastSHAP
KernelSHAP
KernelSHAP-S
GradCAM
Integrated Gradients
SmoothGrad
DeepSHAP
CXPlain
B i r d
F r o g
S h i p
F r o g
C a t
F r o g
B i r d
T r u c k
D o g
S h i p
A i r p l a n e
C a t
H
o r s e
A u t o m o b il e
E n g li s h Sp r
i n g e r
FastSHAP
KernelSHAP KernelSHAP-S GradCAM
Integrated Gradients SmoothGrad DeepSHAP
CXPlain
C hu r c h
G a s P u m p
C hu r c h
C a ss e tt e
P l a y e r
F r e n c h
H o r n
C hu r c h
G a s P u m p
C h a i n S a w
F r e n c h
H o r n
G o l f B a ll
G a s P u m p
T e n c h T e n c h
G o l f B a ll