S HAPLEY E XPLANATION N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Shapley values have become one of the most popular feature attribution explanation methods.
However, most prior work has focused on post-hoc Shapley explanations, which can be computationally demanding (exponential time complexity) and preclude model regularization based on Shapley explanations during training.
Thus, we propose to incorporate Shapley values themselves as latent representations in deep models—thereby making Shapley explanations ﬁrst-class citizens in the modeling paradigm.
This intrinsic explanation approach enables layer-wise explanations, explanation regularization of the model during training, and fast explanation computation at test time.
We deﬁne the Shapley transform that transforms the input into a Shapley representation given one or more scalar functions.
We operationalize the Shapley transform as a neural network module and construct both shallow and deep networks, called S HAP N ET s, by composing Shapley modules.
We prove that our Shallow S HAP N ET s compute the exact Shapley values and our Deep S HAP N ET s maintain the missingness and accuracy properties of
Shapley values.
We demonstrate on synthetic and real world datasets that
our
S HAP N ET s enable layer-wise Shapley explanations, novel Shapley regularizations during training, and fast computation while maintaining reasonable classiﬁcation accuracy.
I NTRODUCTION
Explaining the predictions of machine learning models has become increasingly important for many crucial applications such as healthcare, recidivism prediction, or loan assessment.
Explanations based on feature importance are one key approach to explaining a model prediction.
More speciﬁcally, additive feature importance explanations have become popular, and in Lundberg & Lee (2017), the authors argue for theoretically-grounded additive explanation method called SHAP based on Shapley values—a way to assign credit to members of a group developed in cooperative game theory (Shapley, 1953).
Lundberg & Lee (2017) deﬁned three intuitive theoretical properties called local accuracy, missingness, and consistency, and proved that only SHAP explanations satisfy all three properties.
Despite these elegant theoretically-grounded properties, exact Shapley value computation has exponential time complexity in the general case.
To alleviate the computational issue, several methods have been proposed to approximate Shapley values via sampling (Strumbelj & Kononenko, 2010) and weighted regression (Kernel SHAP), a modiﬁed backpropagation step (Deep SHAP) (Lundberg &
Lee, 2017), utilization of the expectation of summations (Ancona et al., 2019), or making assumptions on underlying data structures (Chen et al., 2019).
To avoid approximation, the model class could be restricted to allow for simpler computation.
Along this line, Lundberg et al.
(2020) propose a method for computing exact Shapley values for tree-based models such as random forests or gradient boosted trees.
However, even if this drawback is overcome, prior Shapley work has focused on post-hoc explanations, and thus, the explanation approach cannot aid in model design or training.
On the other hand, Generalized Additive Models (GAM) as explored in Lou et al.
Caruana et al.
(2015) (via tree boosting) and Wang et al.
(2018) (via neural networks) can be seen as interpretable model class that exposes the exact Shapley explanation directly.
In particular, the output of a GAM model is simply the summation of interaction-free functions:
f GAM ( x ) = (cid:80) s f s ( x s ) , where f s ( · ) are univariate functions that can be arbitrarily complex.
Interestingly, the Shapley explanation values, often denoted φ s ( x , f ) , for a GAM are exactly the values of the independent function, i.e., ∀ s, φ s ( x , f GAM ) =
f s ( x s ) .
Hence, the prediction and the corresponding exact SHAP explanation can be computed simultaneously for GAM models.
However, GAM models are inherently
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]
[Table]
[Figure]
° 2 °°   c
i r c l e
Input 0.5 1.0 1.
ShapleyRepresentation ° 2 °°
L i n e a r £ s i n
Input °  °
ShapleyRepresentation ° 0.90 ° 0.
° 0.8 ° 0.4 0.
° 1.8 ° 1.2 ° 0.6 0.
° 3.2 °
[Table]
°
[Table]
°
[Table] [Table]
<latexit sha
[Table] [Table]
/WaWdusbRWm83Ml5gXdeGrtvwisPfw=</latexit>
Ref.
values : ( limited in their representational power, particularly for perceptual data such as images in which deep networks are state-of-the-art.
Thus, prior work is either post-hoc (which precludes leveraging the method during training) or limited in its representational power (e.g., GAMs).
To overcome these drawbacks, we propose to incorporate Shapley values themselves as learned latent representations (as opposed to post-hoc) in deep models—thereby making Shapley explanations ﬁrst-class citizens in the modeling paradigm.
Intuitive illustrations of such representation are provided in Fig.
1 and detailed discussion in subsection 2.1.
We summarize our core contributions as follows:
• We formally deﬁne the Shapley transform and prove a simple but useful linearity property for constructing networks.
We also develop important extensions of the base case to enhance the representational power of the transform.
• We develop a novel network architecture S HAP N ET , which include Shallow and Deep S HAP N ET , that intrinsically provides layer-wise explanations (i
e., explanations at every layer of the network) in the same forward pass as the prediction.
• We prove that a Shallow S HAP N ET explanations are the exact Shapley values—thus satisfying all three SHAP properties—and prove that Deep S HAP N ET explanations maintain the missingness and local accuracy properties.
• We propose a novel explanation regularization on Shapley values of the model during training enabled by the fact that the explanation is a latent representation in our model.
• We demonstrate empirically that our S HAP N ET s can
provide new capabilities such as layerwise explanations and novel explanation regularizations based on Shapley values while maintaining comparable performance to other deep models.
We also show that our method compares favorably with Shapley-based post-hoc explanation methods.
Dedicated Related Works Section We present related works that motivated our work above and in the text where appropriate.
Due to space limit, for dedicated literature review, we refer to Appendix C.
S HAPLEY EXPLANATION NETWORKS
Background We give a short introduction to SHAP explanations and their properties as originally introduced in Lundberg & Lee (2017).
Given a model v : R d (cid:55)→ R that is not inherently interpretable (e.g., neural nets), additive feature-attribution methods form a linear approximation of the function over simpliﬁed binary inputs, denoted z ∈ { 0 , 1 } d , indicating the “presence” and “absence” of each feature, respectively: i.e., a local linear approximation u ( z ) = a 0 + (cid:80) di
=1 a i z i .
While there are different ways to model “absence” and “presence”, in this work, we take a simpliﬁed viewpoint: “presence” means that we keep the original value whereas “absence” means we replace the original value with a reference value, which has been validated in Sundararajan & Najmi (2020) as Baseline
Shapley.
If we denote the reference vector for all features by r , then we can deﬁne a simple mapping function between z and x as Ψ x , r ( z ) =
z (cid:12) x
+ (1 − z ) (cid:12) r , where (cid:12) denotes element-wise product (eg, Ψ x , r ([0 , 1 , 0 , 1 , 1]) =
[ r 1 , x 2 , r 3 , x 4 , x 5 ] ).
A simple generalization is to group certain features together and consider including or removing all features in the group.
Lundberg & Lee
(2017) propose three properties that additive feature attribution methods should intuitively satisfy.
The ﬁrst property called local accuracy states that the approximate model u at z = 1 should match the output of the model v at the corresponding x , i.e., v ( x ) =
u ( 1 ) =
(cid:80) di =
a i .
The second property called missingness formalizes the idea that features that are “missing” from the input x (or correspondingly the zeros in z ) should have zero attributed effect on the output of the approximation, i.e., z i =
⇒ a i = 0 .
Finally, the third property called consistency formalizes the idea that if one model always sees a larger effect when removing a feature, the attribution value should be larger (see (Lundberg & Lee, 2017) for full deﬁnition).
Deﬁnition 1 (SHAP Values (Lundberg & Lee, 2017)) .
SHAP values are deﬁned as:
[Equation]
where | ˜ z | is the number of non-zero entries, z \ i means z j = 1 for all j (cid:54)
=
i and z i = 0 , ˜ z ⊆
z \ i represents all ˜ z whose non-zeros are subsets of the non-zero entries of z \ i .
2. S HAPLEY TRANSFORM AND S HAPLEY REPRESENTATION
For the sake of generality, we will deﬁne the Shapley transform in terms of tensors (or multidimensional arrays).
However, we ﬁrst make a distinction between tensor dimensions that need to be explained and other tensor dimensions.
For example, in image classiﬁcation, usually an explanation (or attribution) value is required for each pixel (i.e., the spatial dimensions) but all channels are grouped together (i.e., an attribution is provided for each pixel but not for each channel of each pixel).
We generalize this idea in the following deﬁnition of explainable and channel dimensions.
Deﬁnition 2 (Explainable Dimensions and Channel Dimensions) .
Given a tensor representation x ∈ R D × C ≡ R ( d 1 × d 2 ×··· ) × ( c 1 × c 2 ×··· ) , the tensor dimensions D ≡ d 1 × d 2 × · · · that require attribution will be called explainable dimensions and the other tensor dimensions C ≡ c 1 × c 2 × · · · will be called the channel dimensions .
As a simplest example, if the input is a vector x ∈ R d × 1 , then D = d and C = 1 , i.e., we have one (since C = 1 ) importance value assigned to each features.
For images in the space R h × w × c where h , w denote the height and width, respectively, the explainable dimensions correspond to the spatial dimensions (i.e., D = h × w ) and the channel dimensions correspond to the single channel dimension (i.e., C = c ).
While the previous examples discuss tensor dimensions of the input,
our
Shapley transforms can also operate on latent representations.
For example, our latent representations for image models could be in the space R w × h × c 1 × c 2 , where the explainable dimensions correspond to spatial dimensions (i.e., D = h × w ) but there are two channel dimensions (i.e., C = c
1 × c 2 ).
Given this distinction between explainable and channel dimensions, we can now deﬁne the Shapley transform and Shapley representation.
Deﬁnition 3 (Shapley Transform) .
Given an arbitrary function f ∈ F : R D × C (cid:55)→ R C (cid:48) where
C (cid:48) (cid:44) c (cid:48) 1 × c (cid:48) 2 × · · · , the Shapley transform Ω: R D × C × F
(cid:55)→ R D × C (cid:48) is deﬁned as:
[Equation]
where I D denotes the set of all possible indices for values captured by the explainable dimensions D (and similarly for I C (cid:48) ) and f j denotes the scalar function corresponding to the j -th output of f .
Deﬁnition 4 (Shapley Representation) .
Given a function f ∈ F as in Def. 3 and a tensor x ∈ R D × C , we simply deﬁne the Shapley representation to be: Z (cid:44) Ω( x , f ) ∈ R D × C (cid:48) .
Notice that in the Shapley transform, we always keep the explainable dimensions D unchanged.
However, the channel dimensions of the output representation space, i.e., C (cid:48) , is determined by the co-domain of the function f .
For example, if f is a scalar function (i.e., C (cid:48) = 1 ), then the attribution for the explainable dimensions is a scalar.
However, if f is a vector-valued function (i.e., C (cid:48) =
c 1 ), then the attribution is a vector (corresponding to the vector output of f ).
A multi-class classiﬁer is a simple example of an f that is a vector-valued function.
In summary, the Shapley transform maintains the explainable tensor dimensions, but each explainable element of D may be associated to a tensor of attribution values corresponding to each output of f .
[Table] [Table]
[Table] [Table]
>
[Table]
<latexit sha1_base
[Table] [Table]
[Table] [Table]
+2oB1NjrqcRJ7ueb5CuTm4KlknT/
PFd0CQI8017qwHeF4q9IaqtA5Jy31VnuM3EL+ct1FFhtf8/msTAZvIfHK3zObmuWWZT3hOQ9y/o0tWzRb2q+VKrWx93Cxt7+Q38TK9oJf0CnFt0Tbt0C66iDP0lb7Tj6U/hVLhdWE9M11cyDHP6MpTqP0Ft5o7Ng==</latexit>
Edge
[Figure]
We now present a simple but useful property of Shapley transforms (proof in Appendix A).
Without loss of generality, we can assume that the explainable dimensions and the channel dimensions have been factorized so that D = d 1 d 2 d 3 · · · and C = c 1 c 2 c 3 · · · in the following lemma.
Lemma 5 (Linear transformations of Shapley transforms are Shapley transforms) .
The linear transform, denoted by a matrix A ∈ R C (cid:48) × C , of a Shapley representation, Z (cid:44) Ω( x
, f ) ∈ R D × C , is itself a Shapley transform for modiﬁed functions ˜ f , i.e.,
[Equation]
Tk
[Equation]
x
[Equation] [Equation]
where a k ∈ R C is the k -th column of A , and
˜ f k ( x ) is the corresponding k -th value of the output (see j for f j in Def. 3) of the modiﬁed function.
Remark 6 (Alignment of channel dimensions and number of classes) .
By the local accuracy property and our construction, the output of the model is simply the summation of the Shapley values.
This is natural if the output of a model is a scalar (e.g., binary classiﬁcation).
To make it work with multi-class models, we can force the dimensionality of the co-domain, C (cid:48) , of a function f to be able to decompose as C (cid:48)
=
c (cid:48) 1 × c (cid:48) 2 × c (cid:48) 3 × · · · , where c (cid:48) 1 is the number of classes, and for the sake of simplicity, we will use c as the number of output classes henceforth.
We perform summation for the dimensions captured by c 2 × c 3 × · · · to obtain the logit for each class.
In this formulation, we can have attribution values for each class assigned to each element in R D .
Note that the summation operation described above, denoted by sum [ − 1] : R m × C (cid:48) (cid:55)→ R n × R c where n can be any positive integer, is a simple linear transformation so we can use it to create various simpliﬁed Shapley transforms via Lemma 5.
2. S HALLOW S HAP N ET
We now concretely instantiate our S HAP N ET s by presenting our Shallow S HAP N ET and its accompanying Theorem 8 which is an extension of Lemma 5 with complete proof in Appendix A.
Deﬁnition 7 (Shallow S HAP N ET ) .
A Shallow S HAP N ET G : R D (cid:55)→ R c is deﬁned as
[Equation]
where sum [ D ] denotes the summation over explainable dimensions D , which satisﬁes local accuracy, and g ( x , f ) (cid:44) sum [ − 1] ◦ Ω( x
, f ) ∈ R c
× D is the Shallow S HAP N ET explanation.
Theorem 8.
Shallow S HAP N ET s compute the exact Shapley values, i.e., g i,j ( x ; f )
=
φ i ( G j , x ) .
Remark 9 (Sparsity in active sets curbs computation) .
The bottleneck of computing Shapley values is the exponential complexity in the number of input features that renders large-scale computing of Shapley values infeasible.
To alleviate such issue, we would like to promote sparsity in the computation.
Consider a function f depending on a subset of input features, (e.g. f j ( x 1 , x 2 , x 3 )
= 5 x 2 + x 3 only depends on x 2 and x 3 ), we will refer to such a subset active set , A ( f ) ( A ( f j ) =
{ x 2 , x 3 } and for the example above).
In principle, we would like to keep the cardinality of the active set low: |A
( f j ) | (cid:28) d for j ∈ I C (cid:48) , s.t.
we can limit the computational overhead, as features in the complementary set of
A ( f j ) will be trivially assigned Shapley values of 0’s.
To enforce the sparsity as discussed in Remark 9, we propose a simple but novel wrapper around an arbitrarily complex function f
j derived from f called a Shapley module F that explicitly computes the Shapley values locally but only for the active set of inputs.
One of the examples is in Fig.
Deﬁnition 10 (Shapley Module) .
Given an arbitrarily complex function f : R D × C (cid:55)→ R C (cid:48) parameterized by a neural network, from which we can derive f j : R D × C (cid:55)→ R where j ∈ I C (cid:48) , and a reference vector r ∈ R D × C , a Shapley module F : R D × C (cid:55)→ R D × 1 is deﬁned as:
F ( x | f j , r ) (cid:44) [ φ 1 ( f j , x ) , . .
. , φ d ( f j , x )] T , where φ i ( f j , x ) =
if x i / ∈ A ( f j ) .
We apply Shapley Modules inside Shallow S HAP N ET s, and as shown in subsection 2.3 Deep
S HAP N ET s to make our model scalable and efﬁcient.
Notice that this is similar in spirit with LShapley in Chen et al.
(2019), though L-Shapley computes local Shapley values based on the Markov assumption on data while we build such locally computed Shapley values directly into the model.
2. D EEP S HAP N ET
While Shallow S HAP N ET is quite powerful, it does not enable inter-layer Shapley representations within a deep model.
To enable this, we construct deep S HAP N ET s by cascading the output of
Shallow S HAP N ET s from Def.
7 with different co-domain R C (cid:48) of different underlying functions in different stages except the last co-domain which has to align with the desired output space per
Remark 6.
Note that effectively the Shapley representation Z in Def.
4 from a Shallow S HAP N ET can be seen as input x to the next Shallow S HAP N ET , making it possible to cascade the representations.
Deﬁnition 11 (Deep S HAP N ET ) .
A Deep S HAP N ET H is a cascade of pre-summation Shallow
S HAP N ET s:
[Equation]
where h ( x ) ∈ R D × c is the Deep S HAP N ET explanation , Ω ( l ) is the l -th Shapley transform with its own underlying function f ( l ) , and all the reference values are set to 0 except for the ﬁrst Shapley transform Ω (whose reference values will depend on the application).
To ground our Deep S HAP N ET s, we present the following theoretical properties about the explanations of Deep S HAP N ET s (proof in Appendix B).
Theorem 12.
The Deep S HAP N ET explanation h ( x ) deﬁned in Def.
11 provides an explanation that satisﬁes both local accuracy and missingness with respect to H .
Deep S HAP N ET with Disjoint Pairs and Butterﬂy Connections It is immediately obvious that the computational graph determined by the choice of active sets in Shapley Modules dictates how the features interact with each other in a in Shallow S HAP N ET s and hence a Deep S HAP N ET .
For our experiments, we focus on one particular construction of a Deep S HAP N ET based on disjoint pairs (active sets of Shapley modules do not overlap: ∩ cj =1 A ( f j ) = ∅ ) in each layer and a butterﬂy permutation between each layer to allow interactions between many different pairs of features— similar to the Fast Fourier Transform (FFT) butterﬂy construction.
This also means that the cardinality of the active set of all the Shapley modules is set to 2, making the overhead for computing the Shapley values roughly 4 × that of the underlying function.
An example of this construction can be seen on the right of Fig.
We emphasize that this permutation is one of the choices that enable fast feature interactions for constructing deep S HAP N ET s from Shallow S HAP N ET s. We do not claim it is necessarily the best but believe it is a reasonable choice if no other assumptions are made.
One could also construct S HAP N ET s based on prior belief (an example for images below).
Deep S HAP N ET for Images
Here we describe one of the many possibilities to work on image dataset with S HAP N ET s. This is largely inspired by works including Desjardins et al.
; Dinh et al.
; Kingma & Dhariwal (2018).
Still, the Deep S HAP N ET here consists of different layers of Shallow S HAP N ET s. We begin by describing the operations in each of the consecutive Shallow
S HAP N ET s, and then cascade from one stage to the next.
Canonical convolution operation is composed with three consecutive operations: sliding-window (unfolding the image representation tensor to acquire small patches matching the the ﬁlter), matrix multiplication between ﬁlters and the small patches from the images, and folding the resulting representation into the usual image representation tensor.
We merely 1) replace the matrix multiplication operation with Shapley modules (similar to that in Desjardins et al.
(2015)) and 2), since we have the same number of pixels, we can put the corresponding output vector representation in space R c (cid:48) back in the location of the original pixel (and performed summation of the values overlapping at a pixel location just as canonical convolution operation would).
To create a hierarchical representation of images, we use à-trous convolution with increasing dilation as the Shapley transform goes from one stage to the next, similar to Chen et al.
; Chen et al.
To reduce computation, we can choose a subset of the channel dimensions of each pixels to take as inputs, similar to Dinh et al.
; Kingma & Dhariwal (2018).
For the ﬁnal prediction, we do a global summation pooling, given that the ﬁnal Shapley representation comform to Remark 6.
More detailed discussion on this entire model can be found in subsection G.3.
2. E XPLANATION REGULARIZATION DURING TRAINING
One of the important beneﬁts to S HAP N ET s is that we can regularize the explanations during training .
The main idea is to regularize the last layer explanation so that the model learns to attribute features in ways aligned with human priors–e.g. sparse or smooth.
This is quite different from smoothing the explanation post-hoc as in saliency map smoothing methods (Ross et al., 2017; Smilkov et al., 2017;
Sundararajan et al., 2017; Yeh et al., 2019), but falls more into the lines of usage of interpretations of models (Liu & Avci, 2019; Erion et al., 2019; Rieger et al., 2020).
For (cid:96) 1 regularization, the main idea is similar to sparse autoencoders in which we assume the latent representation is sparse.
This is related to the sparsity regularization in LIME (Ribeiro et al., 2016) but is fundamentally different because it actually changes the learned model rather than just the explanation.
For (cid:96) ∞ regularization, the idea is to smooth the Shapley values so that none of the input features become too important individually, be useful for example in a security setting where the model should not be too sensitive to any one sensor because the sensors could be adversarial attacked.
Finally, with ad-hoc knowledge on the attribution appropriate for the task at hand, the user could specify different regularizations.
2. C ONDITIONAL DYNAMIC P RUNING DURING INFERENCE TIME
We present a lemma that allows us to perform pruning during inference time.
Lemma 13 (Missingness in Deep S HAP N ET s) .
In any of the Shapley representations, Z ( l ) i ∈ R C , from each Shallow S HAP N ET inside a Deep S HAP N ET , Z ( l ) , if the Shapley values corresponding to a certain input feature i ∈ I D is equal to the reference vector r i ∈ R C
(zeros for Shallow S HAP N ET s other than the ﬁrst) for the same feature, we have Z ( m ) i = 0 for any m > l .
Lemma 13 is a simple extension of Theorem 12, which allows us to skip the computation for any input features whose importance values become zeros during any stage, as their importance values in all the subsequent representations will be zeros.
In particular, the computation of a Shapley Module could be avoided if its features in active sets are all set to zeros.
In practice, we set a threshold close to zeros, and skip the computations involving the input features with importance values lower than the threshold.
Note that the importance values for the features are conditioned on the input instances instead of a static pruned model (Ancona et al., 2020), allowing more ﬂexibility.
Moreover, with the help of (cid:96) 1 regularization discussed in subsection 2.4 (which can applied on each Z ( l ) ), we can avoid even more computation as shown in Fig.
An illustration can be found in Fig.
We draw similarity to prior works including Han et al.
; Bengio et al. (2015); Theodorakopoulos et al.
E XPERIMENTS & V ISUALIZATIONS
We will (1) validate that our S HAP N ET models can be quite expressive despite the intrinsic explanation design, (2) demonstrate that our intrinsic S HAP N ET explanations perform comparably or better
[Figure]
than posthoc explanations, and (3) highlight the novel capabilities of layer-wise explanations and explanation regularization enabled by the design of S HAP N ET .
More details in Appendix G.
Datasets First, we create a synthetic regression dataset by sampling the input v ∈ [ − 0 .
5]from the uniform distribution where the true (synthetic) regression model is f ( v ) =
(cid:81)
i v i
+ (cid:80) i v 2 i
+ 0 .
05 (cid:15) , where (cid:15) is sampled from a standard normal distribution.
Second, we choose ﬁve real-world datasets from Dua & Graff (2017):
Yeast ( d = 8 ) and Breast Cancer Wisconsin (Diagnostic) ( d = 30 ),
MNIST (LeCun & Cortes, 2010), FashionMNIST (Xiao et al., 2017), and Cifar-10 (Krizhevsky, 2009) datasets to validate S HAP N ET on higher-dimensional datasets.
S HAP N ET model performance We would ﬁrst like to validate that our Deep and Shallow S HAPN ET models can be comparable in performance to other models—i.e., that our Shapley module structure does not signiﬁcantly limit the representational power of our model.
Thus, we deﬁne DNN models that are roughly comparable in terms of computation or the number of parameters—since our networks require roughly four times as much computation as a vanilla DNN but share many parameters.
For our comparison DNNs, we set up a general feedforward network with residual connections (He et al., 2016).
Note that for the vision tasks, we use two other models for comparison with our Deep S HAP N ET s: a comparable convolutional-layer based model which shares the same amount of channels, layers, and hence parameters with only LeakyReLU activation (Xu et al., 2015) with no other tricks other than normalizing the input and random cropping after padding (same as our
Deep S HAP N ET s), and the state-of-the-art on each dataset (Byerly et al., 2020; Tanveer et al., 2020;
Anonymous, 2021).
The performance of the models are shown in Table 1 and Table 2 in which the loss is shown for the synthetic dataset (lower is better) and classiﬁcation accuracy for the other datasets (higher is better).
While there is a very slight degradation in performance, our lower-dimensional models are comparable in performance even with the structural restriction.
While there is some gap between state-of-the-art for image data, our image-based deep S HAP N ET models can indeed do reasonably well even on these high-dimensional non-linear classiﬁcation problems, but we emphasize that they also provide fundamentally new capabilities as explored later.
See Appendix G.
Intrinsic S HAP N ET explanations compared to post-hoc explanations
We now compare the intrinsic S HAP N ET explanations with other post-hoc explanation methods.
First, because our intrinsic S HAP N ET explanations are simultaneously produced with the prediction, the explanation time is merely the cost of a single forward pass (we provide a wall-clock time experiment in Appendix D).
For the lower-dimensional datasets, we validate our intrinsic explanations compared to other SHAPbased post-hoc explanations by computing the difference to the true SHAP values (which can be computed exactly or approximated well in low dimensions).
Our results show that Shallow S HAP N ET indeed gives the true SHAP values up to numerical precision as proved by Theorem 8, and Deep
[Table]
[Table] [Table]
[Table]
S HAP N ET explanations are comparable to other post-hoc explanations (result in Table 3 and more discussion in Appendix E).
For image-based data, we will evaluate our high-dimensional explanations in the MNIST dataset based on 1) digit ﬂipping experiments as introduced in Shrikumar et al. (2017), where higher relative log-odds score is better, 2) removing the least- k salient features
[Table]
where lower fraction of change is better, 3) dropping the top- k features as identiﬁed by the explanation and measuring the remaining output activation of the original predicted class
[Table]
where ideally, the top features would drastically reduce the output logits—showing that our explanations do highlight the important features.
We compare Deep S HAP N ET explanations with DeepLIFT (Shrikumar et al., 2017), DeepSHAP (Lundberg & Lee, 2017), Integrated Gradients (Sundararajan et al., 2017),
Input × Gradients (Shrikumar et al., 2016) and vanilla gradients adjusted for signs.
As shown in
Fig.
4, Deep S HAP N ET offers high-quality explanations on different benchmarks (Samek et al., 2017;
Shrikumar et al., 2017; Ancona et al., 2018; Srinivas & Fleuret, 2019).
Layer-wise explanations With the layer-wise explanation structure
, we can probe into the Shapley representation at each stage of a Deep S HAP N ET as in Fig.
3 and perform dynamic pruning.
Hence we also provide a set of pruning experiment with retaining accuracy shown in Fig.
5 with discussion in Appendix F, where we show how pruning the values in the Shapley representations of different layer changes model performance.
This showcases this unique ability to perform pruning dynamically conditioned input instances.
We note that though this has been done before, our models is, to the best of our knowledge, the only model capable of learning the importance of different computational components in tandem with the importance of each feature, performing conditional dynamic pruning during inference time, and explanation regularization.
[Figure] [Figure]
Regular 20 40 60 80 100 % of pixels pruned 0.
Output ‘ ∞
Layer
Layer Layer
Layer Layer
Layer Layer
[Figure]
Explanation regularization during training We experiment with (cid:96) 1 and (cid:96) ∞ explanation regularizations during training to see if the regularizations signiﬁcantly affect the underlying model behavior.
For the lower-dimensional datasets, we compute the accuracy, the sparsity, and the coefﬁcient of variation (CV, standard deviation divided by mean) for our Deep S HAP N ET .
We see in Table 4 that our (cid:96) ∞ regularization spreads the feature importance more evenly over features (i.e., low CV), (cid:96)regularization increases the sparsity (i.e., higher sparsity), and both either improve or marginally degrade the accuracy.
For MNIST data, we visualize the explanations from different regularization models in Fig.
6, and we perform the top- k feature removal experiment with different regularizers in
Fig.
7, which shows that our regularizers produce some model robustness towards missing features.
Moreover, Fig. 5 also shows that (cid:96) 1 regularized model achieves the best result under value pruning across different layers.
A P ROOF OF T HEOREM 8 & L EMMA
Proof of Lemma 5.
For each row of W (cid:44) AZ , we can show that it is the Shapley value for ˜ f
( k ) using the linearity of Shapley values (the original motivating axiom of Shapley values, see Shapley (1953, Axiom 4 & Axiom 5)):
[Equation]
Combining this result for all rows with Def. 3, we arrive at the result.
We deﬁne a feature selector function π ( w ) as follows:
Deﬁnition 14 (Feature selector function) .
π ( w ) : R d (cid:55)→ R | w | (for any w ∈ W , where W again indexes the set of all component Shapley modules in a given Shallow S HAP N ET as in Def. 7) is deﬁned to select the input features by the index w , which contains the indices of the supposed input features of F ( w ) :
[Equation]
where i ∈ w.
[Equation]
Lemma 15.
The composite function F (cid:48) ( w ) for a given Shapley module F ( w ) , deﬁned as
[Equation]
has the following Shapley values for its input features
[Equation]
Proof.
This holds since the absence (set to reference values) of a feature does not make a difference to the ﬁnal output of F (cid:48)
( w ) .
Now we are ready to present proof for Theorem 8:
Proof.
We consider a simpler, and yet more general case: consider where each of the modules deﬁned in Def.
10, we ﬁrst transform any Shapley module F ( w ) in the given Shallow S HAP N ET G to F (cid:48) ( w ) .
Then, by Def. 7, the zeros in Lemma 15 do not make any difference whether to the output of G or
its associated β
i ’s, and yet β i ’s are nothing but summations of Shapley values of different F ( w ) ’s and by
Lemma 5,
[Equation]
(10) which completes the proof.
B P ROOF OF T HEOREM
Proof.
The proof of local accuracy is trivial by construction since H ( x ) (cid:44) sum
[ d ] ( h ( x )) from
Def. 11.
The proof of missingness is straightforward via induction on the number of layers, where the claim is that if x i =
r i (i.e., the input for one feature equals the reference value), then ∀ (cid:96), h ( (cid:96) )
i ( x ) =(i.e., the pre-summation outputs of the Shapley modules in all layers corresponding to feature i will all be zero).
For the base case of the ﬁrst layer, we know that if x i
= r
i , then we know that Z (1) i
=by the construction of Shapley modules.
For the future layers, the inductive hypothesis is then that if previous layer representation Z ( (cid:96) − 1) i of a feature i is 0 , then the representation from the current layer (cid:96) for this feature remains the same, i.e., Z ( (cid:96) )
i = 0 .
Because the reference values for all layers except the ﬁrst layer are zero, then we can again apply the property of our Shapley modules and thus the inductive hypothesis will hold for (cid:96) > 1 .
Thus, the the claim holds for all layers (cid:96) , and h i ( x ) = sum
[ c ] Z ( (cid:96) )
i will also be zero—which proves missingness for Deep S HAP N ET s.
C D EDICATED L ITERATURE R EVIEW
Post-hoc feature-attribution explanation methods, gained traction with the introduction of LIME explanations
(Ribeiro et al., 2016) and its variants (Ribeiro et al., 2018; Guidotti et al., 2018a;b), which formulate an explanation as a local approximation to the model near the target instance.
LIME attempts to highlight which features were important for the prediction.
Saliency map explanations for image predictions based on gradients also assign importance to each feature (or pixel)
(Simonyan et al., 2014; Shrikumar et al., 2017).
SHAP explanations attempt to unify many previous feature attribution methods under a common additive feature attribution framework (Lundberg & Lee,
[Table]
Intrinsically interpretable models are an alternative to post-hoc explanation methods.
In particular,
GAM (Lou et al., 2012) and its extension (Lou et al., 2013; Caruana et al., 2015) construct models that can be displayed as series of line graphs or heatmaps.
Because the entire model can be displayed and showed to users, it is possible for a user to directly edit the model based on their domain knowledge.
In addition, Alvarez Melis & Jaakkola (2018) proposed an architecture that are complex and explicit if trained with a regularizer tailored to that architecture.
Our methodology sits in between post-hoc and intrinsic approaches by constructing an explainable model but using SHAP post-hoc explanation concepts and allowing more complex feature interactions.
Recent works also focus on Shapley value approximations in deep neural nets, where Deep Approximate Shapley Propagation (DASP) approximates in polynomial time (Ancona et al., 2019), L-Shapley and C-Shapley do in linear (Chen et al., 2019) with the Markov assumption, and our method performs approximation with a constant complexity, which additionally allows regularizing the explanations according to human priors and hence modifying the underlying model during training.
This falls in line with previous works (Ross et al., 2017; Liu & Avci, 2019; Erion et al., 2019; Rieger et al., 2020) each of which adds an extra term in the loss function measuring the disagreement between the explanations and human priors.
Ross et al.
(2017) proposed to regularize the input gradients as explanations, which creates the need for second-order gradient.
Liu & Avci (2019) uses Integrated
Gradients (IG) (Sundararajan et al., 2017), which is an extension to Shapley values (Sundararajan &
Najmi, 2020).
Due to the use of IG, several samples (for one explanation) are additionally warranted.
Erion et al.
(2019) proposed Expected Gradient to rid of reference values in IG and suggested the usage of a single sample.
Rieger et al.
(2020) adds to this line of work with Contextual Decomposition (Murdoch et al., 2018;
Singh et al., 2019) which avoids the need of computation of second-order gradients and extra samples.
Our work differs with these four in that we do not need a second order gradient and that we are focusing on the more theoretically-grounded Shapley values.
The explanation-related regularization in Alvarez Melis & Jaakkola (2018) differs with ours in that the regularizer is used to promote interpretability and ours uses regularization to modify the underlying model to conform to prior belief.
We also draw close similarity to works that that attempt to smooth saliency map explanations (Smilkov et al., 2017; Sundararajan et al., 2017; Yeh et al., 2019) which acts on explanation methods to induce certain properties in explanations but not the actual models.
D W ALL - CLOCK TIME FOR EXPLANATION EXPERIMENT
To validate the efﬁciency of computing an explanation, we compare the wall-clock time of our
S HAP N ET explanations to other approximation methods related to Shapley-values including Deep
SHAP and Kernel SHAP with different numbers of samples—40, 77, and default ( 2 × d + 2 11 ).
The wall-clock computation time in seconds can be seen in Table 5.
We note importantly that this comparison is a bit odd because other Shapley-value explanation methods are post-hoc explanations, whereas S HAP N ET is both a model and an intrinsic explanation method.
Thus, the most fair comparison in terms of time is to have the post-hoc explanation methods explain our S HAP N ET model as seen in the ﬁrst column of Table 5.
Nevertheless, we also show the wall-clock times for the post-hoc methods on our comparable DNNs in columns two and three of Table 5—note that
S HAP N ET cannot explain other DNNs hence the N/A’s.
From the ﬁrst column, we can see that our method is inherently faster than the others when applied to our model.
However, smaller model (i.e.,
DNN w. equivalent parameters) may come with faster explanations.
E L OWER - DIMENSIONAL COMPARISON BETWEEN S HAP N ET EXPLANATIONS AND POST - HOC EXPLANATIONS
For lower-dimensional datasets, we seek validation for the explanations h ( x ) provided by Deep
S HAP N ET s by comparing to the true Shapley values of our models (which can be computed exactly or aproximated well in low dimensions).
Our metric is the normalized (cid:96) 1 norm between the true
Shapley values denoted φ and the explanation denoted γ deﬁned as:
[Equation]
Note that we measure the average difference from vector-output classiﬁers by means discussed in subsection E.1.
The experimental setup and results are presented in subsection E.2.
E. C OMPARING SHAP APPROXIMATION ERRORS WITH VECTOR OUTPUT MODELS
To compare the approximation errors for every output of the model, we use a weighted summation of the explanations for different outputs, where the weights are the models’ output after passing into the soft-max function.
Thus, we up weight the explanations that have a higher probability rather than taking a simple average.
Thus, in the extremes, if the predicted probability for a label is zero, then the difference in explanation is ignored, while if the predicted probability for a label is one, then we put all the weight on that single explanation.
Concretely, for a classiﬁcation model C : R d (cid:55)→ R n , where d is the number of input features and n is the number of classes, we have the output vector of such model with an input instance x ∈ R d as C ( x ) ∈ R n .
For each output scalar value [ C ( x )]
j , we compute the approximation of the SHAP values w.r.t.
that particular scalar for feature indexed i , denoted by γ ( j ) i , and hence its corresponding errors as measured in normalized (cid:96) 1 distance as discussed in Eqn.
11: (cid:96) ( j ) 1 .
The ﬁnal error measure that we compare between classiﬁers (cid:96) ∗ 1 is simply a weighted sum version of the normalized (cid:96) 1 :
[Equation]
where the softmax is taken on the non-normalized output of the classiﬁers.
Note that (cid:96) j 1 is also taken on the raw output.
E. S ETUP AND RESULTS
Because the explanation difference from SHAP is independent of the training procedure but solely depends on the methods, we compare the performance of the explanation methods on untrained models that have randomly initialized parameters (i.e., they have the model structure but the parameters are random)—this is a valid model it is just not a model trained on a dataset.
Additionally, we consider the metric after training the models on the synthetic, yeast, and breast cancer datasets.
Note that for even a modest number of dimensions, computing the exact Shapley values is infeasible so we use a huge number of samples ( 2 16 ) with the kernel SHAP method to approximate the true Shapley values as the ground truth for all but the Yeast dataset, which only requires 2 8 samples for exact computation.
Note that the extended Shapley value (ext.) requires 2 14 times of model evaluation for a single explanation which is already extremely expensive, and thus included merely for comparison.
Results of the average difference can be seen in Table 3.
From the results in Table 3, we can see that indeed our Shallow S HAP N ET model produces explanations that are the exact Shapley values (error by ﬂoating point computation) as Theorem 8 states.
Additionally, we can see that even though our
Deep S HAP N ET model does not compute exact Shapley values, our explanations are actually similar to the true Shapley values.
We also provide in Table 6 the standard deviation of the approximation errors as in Table 3.
These are the same runs as those in Table 3.
[Table]
F P RUNING EXPERIMENT
We also conduct a set of pruning (set to reference values) experiment.
This is, again, enabled by the layer-wise explainable property of Deep S HAP N ET .
Recent work has investigated the usage of
Shapley values for pruning (Ancona et al., 2020), but our work means to showcase the ability even further.
Results are shown in Fig.
We perform the pruning experiment with all four MNIST-traiend models as discussed in subsection 2. and subsection 2.5 with no, output (cid:96) 1 , layer (cid:96) 1 , and (cid:96) ∞ regularizer, respectively.
The values are removed (set to reference) by the order of their magnitude (from smallest to largest).
We can see that most of the values might be dropped and the model performance stays unhinged.
Moreover, (cid:96) 1 seems to have the best robustness against pruning, which makes sense as it induces sparsity in explanation values.
Future works are to be done on the computational cost reduction under such pruning technique.
G M ORE EXPERIMENTAL DETAILS
For all the experiments described below, the optimizers are Adam (Kingma & Ba, 2014) with default parameters in PyTorch (Paszke et al., 2019).
G. T IMING EXPERIMENTS
For Table 5, the timing is for the untrained (randomly initialized) models with 16 features.
The numbers after Kernel SHAP indicates the number of samples allowed for Kernel SHAP’s sampling method.
Time is measured in seconds on CPU with 1000 independent trials per cell as Kernel SHAP supports solely CPU from the implementation of the authors of Lundberg & Lee (2017).
The model of the CPU used for testing is Intel(R)
Xeon(R)
Silver 4114 CPU @
2.20GHz.
We perform explanation on randomly initialized 16-feature Deep S HAP N ET model for 1000 rounds.
The 16-feature setup means we will have (16 / 2) × log 2 16 = 32 Shapley modules inside the entire model spanned out in 4 stages according to the Butterﬂy mechanism.
The inner function f for every module in this setting is set as a fully-connected neural network of 1 hidden layer with 100 hidden units.
For the two reference models, the model equivalent in computation time has 8 layers ofhidden units in each layer, while the model equivalent in parameters has 11 layers of 123 hidden units in each layer.
The output of the models are all scalars.
G. Y EAST AND B REAST C ANCER DATASETS EXPERIMENT SETUPS
This setup applies for the experiments involving both datasets, including those experiments for model performance and those that showcase the ability for explanation regularization.
The experiments on the two datasets were run for 10 rounds each with different random initialization and the end results were obtained by averaging.
For each round, we perform 5-fold cross validation on the training set, and arrive at 5 different models, giving us 50 different models to explain for each datasets.
For the two datasets, we do 9 training epochs with a batch size of 32.
For the preprocessing procedures, we ﬁrst randomly split the data into training and testing sets, with 75% and 25% of the data respectively.
Then we normalize the training set by setting the mean of each feature to 0 and the standard deviation to 1.
We use the training sets’ mean and standard deviation to normalize the testing set, as is standard practice.
The inner functions of the Shapley modules are fully connected multilayer neural networks.
The ﬁrst and second stages of two models on the two datasets are identical in structure (not in parameter values).
The ﬁrst stage modules have 2 inputs and 25 outputs, with a single hidden layer of size 50.
The modules in the second stage have 100 input units and 50 output units with two hidden layer of size 100.
The non-linearity is introduced by ReLU (Nair & Hinton, 2010) in the inner functions.
The rest of the speciﬁcs about the inner function are discussed below.
Note that the dimensionality of the output from the last stage differs with that of the input to the next stage, as is expected by Shapley module where the input has twice the dimensionality of that of the output.
Model for Yeast dataset The Yeast dataset has 8 input features and hence 4 Shapley modules at each of the log 2 8 = 3 stages.
Within each stage, the structure of the inner function is identical except for the parameters.
For the Yeast dataset, the third (last) stage comprises four-layer fully-connected model with inputs from 50 × 2 = 100 units to the number of classes, which is 10, with two hidden layers of 150 units.
Model for Breast Cancer Wisconsin (Diagnostic)
This dataset has 30 input features, which is not a exponent of 2, but we can still construct a Deep S HAP N ET by setting two extra variables to zero.
In fact, in doing so we can simplify the Shapley modules since we know which of the modules in the network always have one or both of its input being 0 at all times.
To construct the model, we realize that the number of stages is 5 and the number of modules at each stage is 16.
For the Breast
Cancer Wisconsin (Diagnostic) dataset, in the the third stage of the model, we use a model withinputs units and 75 output units with two hidden layers of 150 units.
The fourth stage has input units of 75 × 2 = 150 and output of 100 units with two hidden layers of size 200 .
The ﬁfth (last) stage has a input dimension of 200 , two hidden layers of 250 units and output of size 10 .
G. S PECIFICS FOR VISION TASKS G.3. S TRUCTURE USED FOR VISION EXPERIMENTS
For all of the vision tasks, we use the same structure with the exception of input channel counts.
We use a simple Linear-LeakyReLU
(Xu et al., 2015) inside each Shapley Module, and the following are the output dimension of the linear layer: 1, 128, 128, 128, 128, 128, 128, 10, with 1 being the input channel of images in the case of MNIST & FashionMNIST or 3 with Cifar-10.
À-trous convolution À-trous convolution is used for learning hierarchical representations of images.
In our model, the dilations are set as follows: 1, 3, 5, 7, 9, 11, 13.
G.3. T RAINING RECIPE
Learning process As discussed before, all the models are trained with Adam in PyTorch with the default parameters.
The batch size is set to 64 for all the vision tasks.
No warm-up or any other scheduling techniques are applied.
The FashionMNIST & MNIST models in Table 1 are trained for 10 epochs while the MNIST models with which we investigate the new capabilities (layer-wise explanations & Shapley explanation regularizations) in Section 3 are trained with just 5 epochs.
The
Cifar-10 models are trained for 60 epochs.
Preprocessing For both MNIST dataset and FashionMNIST dataset, we normalize the training and testing data with mean of 0.1307 and standard deviation of 0.3081, this gives us non-zero background and we would therefore set the reference values to 0 ’s, allowing negative or positive contribution from the background.
Both FashionMNIST and MNIST share the same preprocessing pipeline: 1) for training dataset, we extend the resolution from 28 × 28 to 36 × 36 and then randomly crop to 32 × 32 , and then normalize. 2) for testing data, we simply pad the image to 32 × 32 and normalize them.
For Cifar-10 dataset, during training, we ﬁrst pad 4 pixels on four sides of the image and then randomly crop back to 32 × 32 , perform a horizontal ﬂip with probability 0.5, and normalize the data with means 0 . 4914 , 0 . 4822 , 0 . 4465 and standard deviaiton of 0 . 2023 , 0 . 1994 , 0 . 2010 for each channel, respectively.
For testing, we simply normalize the data with the same mean and standard deviation.
G. I LLUSTRATIONS OF EXPLANATION REGULARIZATION EXPERIMENTS
We present visual representations of our models’ explanation on MNIST (LeCun & Cortes, 2010) dataset in Fig.
8, Fig. 9, and Fig.
In all three ﬁgures, the gray images from left to right are explanations laid out for all 10 classes from 0 to 9, where the red pixels indicate positive contribution and blue pixels indicate negative contribution, with magnitudes presented in the color bar below.
From an intuitive visual interpretation, we argue that our explanation regularizations are indeed working in the sense that (cid:96) ∞ regularization is indeed smoothing the explanations (to put more weight on a large number of pixels) and that (cid:96) 1 regularization limits the number of pixels contributing to the ﬁnal classiﬁcation.
One can notice that, by comparing the color bars below the ﬁgures, the contribution magnitudes for (cid:96) ∞ and (cid:96) 1 regularized models are lower than those of the unregularized counterpart.
We have expected this to happen as both regularization techniques require to make one or more of the attribution values to be smaller.
However, we note that this change in scale of the attribution values should not impact the accuracy of the prediction as the softmax operation at the end will normalize across the outputs for all of the classes.
More interesting discussions are provided in the caption of Fig.
8, Fig. 9, and Fig.
and we strongly encourage readers to have a read.
[Figure]
∞ regularization improves robustness when removing
G. M ORE RESULTS ON D IGIT FLIPPING
More results on digit ﬂipping are in Fig.
[Figure] [Figure]
[Figure] [Figure]