S HAPLEY E XPLANATION N ETWORKS
A BSTRACT
Shapley values have become one of the most popular feature attribution explanation methods.
However, most prior work has focused on post-hoc Shapley explanations, which can be computationally demanding due to exponential time complexity and preclude model regularization based on Shapley explanations during training.
Thus, we propose to incorporate Shapley values themselves as latent representations in deep models—thereby making Shapley explanations ﬁrst-class citizens in the modeling paradigm.
This intrinsic explanation approach enables layer-wise explanations, explanation regularization of the model during training, and fast explanation computation at test time.
We deﬁne the Shapley transform that transforms the input into a Shapley representation given a speciﬁc function.
We operationalize the Shapley transform as a neural network module and construct both shallow and deep networks, called S HAP N ET s, by composing Shapley modules.
We prove that our Shallow S HAP N ET s compute the exact Shapley values and our Deep S HAP N ET s maintain the missingness and accuracy properties of Shapley values.
We demonstrate on synthetic and real-world datasets that our S HAP N ET s enable layer-wise Shapley explanations, novel Shapley regularizations during training, and fast computation while maintaining reasonable performance. Code is available at https://www.github.com/
RuiWang
ShapleyExplanationNetworks
I NTRODUCTION
Explaining the predictions of machine learning models has become increasingly important for many crucial applications such as healthcare, recidivism prediction, or loan assessment.
Explanations based on feature importance are one key approach to explaining a model prediction.
More speciﬁcally, additive feature importance explanations have become popular and in Lundberg & Lee (2017), the authors argue for theoretically-grounded additive explanation method called SHAP based on Shapley values—a way to assign credit to members of a group developed in cooperative game theory (Shapley, 1953).
Lundberg & Lee (2017) deﬁned three intuitive theoretical properties called local accuracy, missingness, and consistency, and proved that only SHAP explanations satisfy all three properties.
Despite these elegant theoretically-grounded properties, exact Shapley value computation has exponential time complexity in the general case.
To alleviate the computational issue, several methods have been proposed to approximate Shapley values via sampling (Strumbelj & Kononenko, 2010) and weighted regression (Kernel SHAP), a modiﬁed backpropagation step (Deep SHAP) (Lundberg &
Lee, 2017), utilization of the expectation of summations (Ancona et al., 2019), or making assumptions on underlying data structures (Chen et al., 2019).
To avoid approximation, the model class could be restricted to allow for simpler computation.
Along this line, Lundberg et al.
(2020) propose a method for computing exact Shapley values for tree-based models such as random forests or gradient boosted trees.
However, even if this drawback is overcome, prior Shapley work has focused on post-hoc explanations, and thus, the explanation approach cannot aid in model design or training.
On the other hand, Generalized Additive Models (GAM) as explored in Lou et al.
Caruana et al.
(2015) (via tree boosting) and Wang et al.
(2018) (via neural networks) can be seen as interpretable model class that exposes the exact Shapley explanation directly.
In particular, the output of a GAM model is simply the summation of interaction-free functions:
f GAM ( x ) = (cid:80) s f s ( x s ) ,
[Table]
[Figure]
[Table]
[Figure]
[Table] [Table]
[Figure]
[Table] [Table] [Table]
[Figure]
° 2 °°   c
i r c l e
Input 0.5 1.0 1.
ShapleyRepresentation ° 2 °°
L i n e a r £ s i n
Input °  °
ShapleyRepresentation ° 0.90 ° 0.
° 0.8 ° 0.4 0.
° 1.8 ° 1.2 ° 0.6 0.
° 3.2 ° 2.4 ° 1.6 ° 0.8 0.00. <latexit sha1_base
[Table]
[Table]
[Table] [Figure]
where f s ( · ) are univariate functions that can be arbitrarily complex. Interestingly, the Shapley explanation values, often denoted φ s ( x , f ) , for a GAM are exactly the values of the independent function, i.e., ∀ s, φ s ( x , f GAM ) =
f s ( x s ) .
Hence, the prediction and the corresponding exact SHAP explanation can be computed simultaneously for GAM models.
However, GAM models are inherently limited in their representational power, particularly for perceptual data such as images in which deep networks are state-of-the-art.
Thus, prior work is either post-hoc (which precludes leveraging the method during training) or limited in its representational power (e.g., GAMs).
To overcome these drawbacks, we propose to incorporate Shapley values themselves as learned latent representations (as opposed to post-hoc) in deep models—thereby making Shapley explanations ﬁrst-class citizens in the modeling paradigm.
Intuitive illustrations of such representation are provided in Fig.
1 and detailed discussion in subsection 2.1.
We summarize our core contributions as follows:
• We formally deﬁne the Shapley transform and prove a simple but useful linearity property for constructing networks.
We also develop important extensions of the base case to enhance the representational power of the transform.
• We develop a novel network architecture, S HAP N ET , which include Shallow and Deep S HAP N ET , that intrinsically provides layer-wise explanations (i
e., explanations at every layer of the network) in the same forward pass as the prediction.
• We prove that a Shallow S HAP N ET explanations are the exact Shapley values—thus satisfying all three SHAP properties—and prove that Deep S HAP N ET explanations maintain the missingness and local accuracy properties.
• We show that it is possible to perform learned conditional pruning in Deep S HAP N ET s. •
We propose a novel explanation regularization on Shapley values of the model during training enabled by the fact that the explanation is a latent representation in our model.
• We demonstrate empirically that our S HAP N ET s can provide new capabilities such as layerwise explanations and novel explanation regularizations based on Shapley values while maintaining comparable performance to other deep models.
We also show that our method compares favorably with Shapley-based post-hoc explanation methods.
Dedicated Related Works Section We present related works above and in the text where appropriate.
Due to space limit, we refer to Appendix C for a dedicated literature review section.
S HAPLEY EXPLANATION NETWORKS
Background We give a short introduction to SHAP explanations and their properties as originally introduced in Lundberg & Lee (2017).
Given a model v : R d (cid:55)→ R that is not inherently interpretable (e.g., neural nets), additive feature-attribution methods form a linear approximation of the function over simpliﬁed binary inputs, denoted z ∈ { 0 , 1 } d , indicating the “presence” and “absence” of each feature, respectively: i.e., a local linear approximation u ( z ) = a 0 + (cid:80) di
=1 a i z i .
While there are different ways to model “absence” and “presence”, in this work, we take a simpliﬁed viewpoint: “presence” means that we keep the original value whereas “absence” means we replace the original value with a reference value, which has been validated in Sundararajan & Najmi (2020) as Baseline
Shapley.
If we denote the reference vector for all features by r , then we can deﬁne a simple mapping function between z and x as Ψ x , r ( z ) =
z (cid:12) x
+ (1 − z ) (cid:12) r , where (cid:12) denotes element-wise product (eg, Ψ x , r ([0 , 1 , 0 , 1 , 1]) =
[ r 1 , x 2 , r 3 , x 4 , x 5 ] ).
A simple generalization is to group certain features together and consider including or removing all features in the group.
Lundberg & Lee (2017) propose three properties that additive feature attribution methods should intuitively satisfy.
The ﬁrst property called local accuracy states that the approximate model u at z = 1 should match the output of the model v at the corresponding x , i.e., v ( x ) =
u ( 1 ) =
(cid:80) di =
a i .
The second property called missingness formalizes the idea that features that are “missing” from the input x (or correspondingly the zeros in z ) should have zero attributed effect on the output of the approximation, i.e., z i =
⇒ a i = 0 .
Finally, the third property called consistency formalizes the idea that if one model always sees a larger effect when removing a feature, the attribution value should be larger (see (Lundberg & Lee, 2017) for full deﬁnitions of the three properties and their discussions).
Deﬁnition 1 (SHAP Values (Lundberg & Lee, 2017)) .
SHAP values are deﬁned as:
[Equation]
where (cid:107) z (cid:107) 1 is the (cid:96) 1 norm of z , given a set Z i =
z ∈ { 0 , 1 } d : z i =
2. S HAPLEY TRANSFORM AND S HAPLEY REPRESENTATION
For the sake of generality, we will deﬁne the Shapley transform in terms of tensors (or multidimensional arrays).
However, we ﬁrst make a distinction between tensor dimensions that need to be explained and other tensor dimensions.
For example, in image classiﬁcation, usually an explanation (or attribution) value is required for each pixel (i.e., the spatial dimensions) but all channels are grouped together (i.e., an attribution is provided for each pixel but not for each channel of each pixel).
We generalize this idea in the following deﬁnition of explainable and channel dimensions.
Deﬁnition 2 (Explainable Dimensions and Channel Dimensions) .
Given a tensor representation x
[Equation]
, the tensor dimensions D ≡ d
[Equation] d [Equation] · · · that require attribution will be called explainable dimensions and the other tensor dimensions C ≡ c 1 × c 2 × · · · will be called the channel dimensions .
As a simplest example, if the input is a vector x ∈ R d × 1 , then D = d and C = 1 , i.e., we have one (since C = 1 ) importance value assigned to each features.
For images in the space R h × w × c where h , w denote the height and width, respectively, the explainable dimensions correspond to the spatial dimensions (i.e., D = h × w ) and the channel dimensions correspond to the single channel dimension (i.e., C = c ).
While the previous examples discuss tensor dimensions of the input,
our
Shapley transforms can also operate on latent representations.
For example, our latent representations for image models could be in the space R w × h × c 1 × c 2 , where the explainable dimensions correspond to spatial dimensions (i.e., D = h × w ) but there are two channel dimensions (i.e., C = c
1 × c 2 ).
Given this distinction between explainable and channel dimensions, we can now deﬁne the Shapley transform and Shapley representation.
Deﬁnition 3 (Shapley Transform) .
Given an arbitrary function f ∈ F : R D × C (cid:55)→ R C (cid:48) where
C (cid:48) (cid:44) c (cid:48) 1 × c (cid:48) 2 × · · · , the Shapley transform Ω: R D × C × F
(cid:55)→ R D × C (cid:48) is deﬁned as:
[Equation]
where I D denotes the set of all possible indices for values captured by the explainable dimensions D (and similarly for I C (cid:48) ) and f j denotes the scalar function corresponding to the j -th output of f .
Deﬁnition 4 (Shapley Representation) .
Given a function f ∈ F as in Def. 3 and a tensor x ∈ R D × C , we simply deﬁne the Shapley representation to be: Z (cid:44) Ω( x , f ) ∈ R D × C (cid:48) .
Notice that in the Shapley transform, we always keep the explainable dimensions D unchanged.
However, the channel dimensions of the output representation space, i.e., C (cid:48) , is determined by the co-domain of the function f .
For example, if f is a scalar function (i.e., C (cid:48) = 1 ), then the attribution for the explainable dimensions is a scalar.
However, if f is a vector-valued function (i.e., C (cid:48) =
c 1 ), [Table] [Table] [Table]
[Table] [Table] [Table]
[Table] +2oB1NjrqcRJ7ueb5CuTm4KlknT/
PFd0CQI8017qwHeF4q9IaqtA5Jy31VnuM3EL+ct1FFhtf8/msTAZvIfHK3zObmuWWZT3hOQ9y/o0tWzRb2q+VKrWx93Cxt7+Q38TK9oJf0CnFt0Tbt0C66iDP0lb7Tj6U/hVLhdWE9M11cyDHP6MpTqP0Ft5o7Ng==</latexit>
[Figure]
then the attribution is a vector (corresponding to the vector output of f ). A multi-class classiﬁer is a simple example of an f that is a vector-valued function.
In summary, the Shapley transform maintains the explainable tensor dimensions, but each explainable element of D may be associated to a tensor of attribution values corresponding to each output of f .
We now present a simple but useful property of Shapley transforms (proof in Appendix A).
Without loss of generality, we can assume that the explainable dimensions and the channel dimensions have been factorized so that D = d 1 d 2 d 3 · · · and C = c 1 c 2 c 3 · · · in the following lemma.
Lemma 5 (Linear transformations of Shapley transforms are Shapley transforms) .
The linear transform, denoted by a matrix A ∈ R C (cid:48) × C , of a Shapley representation, Z (cid:44) Ω( x
, f ) ∈ R D × C , is itself a Shapley transform for modiﬁed functions ˜ f , i.e.,
[Equation]
Remark 6 (Alignment of channel dimensions and number of classes) .
By the local accuracy property and our construction, the output of the model is simply the summation of the Shapley values.
This is natural if the output of a model is a scalar (e.g., binary classiﬁcation).
To make it work with multi-class models, we can force the dimensionality of the co-domain, C (cid:48) , of a function f to be able to decompose as C (cid:48)
=
c (cid:48) 1 × c (cid:48) 2 × c (cid:48) 3 × · · · , where c (cid:48) 1 is the number of classes, and for the sake of simplicity, we will use c as the number of output classes henceforth.
We perform summation for the dimensions captured by c 2 × c 3 × · · · to obtain the logit for each class.
In this formulation, we can have attribution values for each class assigned to each element in R D .
Note that the summation operation described here, which we denote by sum [ − 1] : R n × C [Equation] (cid:55)→ R n × c with n as any positive integer, is a simple linear transformation so we can use it to create various simpliﬁed Shapley transforms via Lemma 5.
2. S HALLOW S HAP N ET
We now concretely instantiate our S HAP N ET s by presenting our Shallow S HAP N ET and its accompanying Theorem 8 which is an extension of Lemma 5 with complete proof in Appendix A.
Deﬁnition 7 (Shallow S HAP N ET ) .
A Shallow S HAP N ET G : R D (cid:55)→ R c is deﬁned as
[Equation]
where sum [ D ] denotes the summation over explainable dimensions D , which satisﬁes local accuracy, and g ( x , f ) (cid:44) sum [ − 1] ◦ Ω( x
, f ) ∈ R D × c is the Shallow S HAP N ET explanation.
Theorem 8.
Shallow S HAP N ET s compute the exact Shapley values, i.e., g i,j ( x ; f )
=
φ i ( G j , x ) .
Remark 9 (Sparsity in active sets curbs computation) .
The bottleneck of computing Shapley values is the exponential complexity in the number of input features that renders large-scale computing of Shapley values infeasible.
To alleviate such issue, we would like to promote sparsity in the computation.
Consider a function f depending on a subset of input features, (e.g. f j ( x 1 , x 2 , x 3 )
= 5 x 2 + x 3 only depends on x 2 and x 3 ), we will refer to such a subset active set , A ( f ) ( A ( f j ) =
{ x 2 , x 3 } and for the example above).
In principle, we would like to keep the cardinality of the active set low: |A
( f j ) | (cid:28) d for j ∈ I C (cid:48) , s.t.
we can limit the computational overhead, as features in the complementary set of
A ( f j ) will be trivially assigned Shapley values of 0’s.
To enforce the sparsity as discussed in Remark 9, we propose a simple but novel wrapper around an arbitrarily complex function f
j derived from f called a Shapley module F that explicitly computes the Shapley values locally but only for the active set of inputs.
One of the examples is in Fig.
Deﬁnition 10 (Shapley Module) .
Given an arbitrarily complex function f : R D × C (cid:55)→ R C (cid:48) parameterized by a neural network, from which we can derive f j : R D × C (cid:55)→ R where j ∈ I C (cid:48) , and a reference vector r ∈ R D × C , a Shapley module F : R D × C (cid:55)→ R D × 1 is deﬁned as:
F ( x | f j , r ) (cid:44) [ φ 1 ( f j , x ) , . .
. , φ d ( f j , x )] T , where φ i ( f j , x ) =
if x i / ∈ A ( f j ) .
We apply Shapley Modules inside both Shallow S HAP N ET s, and, as shown in subsection 2.3, Deep
S HAP N ET s to make our model scalable and efﬁcient.
Notice that this is similar in spirit with L-Shapley in Chen et al.
(2019), though it computes local Shapley values based on the Markov assumption on data while we build such locally computed Shapley values directly into the model.
2. D EEP S HAP N ET
While Shallow S HAP N ET is quite powerful, it does not enable inter-layer Shapley representations within a deep model.
To enable this, we construct deep S HAP N ET s by cascading the output of
Shallow S HAP N ET s from Def.
7 with different co-domain R C (cid:48) of different underlying functions in different stages except the last co-domain which has to align with the desired output space per
Remark 6.
Note that effectively the Shapley representation Z in Def.
4 from a Shallow S HAP N ET can be seen as input x to the next Shallow S HAP N ET , making it possible to cascade the representations.
Deﬁnition 11 (Deep S HAP N ET ) .
A Deep S HAP N ET H is a cascade of pre-summation Shallow
S HAP N ET s:
[Equation]
(5) where h ( x ) ∈ R D × c is the Deep S HAP N ET explanation , Ω ( l ) is the l -th Shapley transform with its own underlying function f ( l ) , and all the reference values are set to 0 except for the ﬁrst Shapley transform Ω (whose reference values will depend on the application).
To ground our Deep S HAP N ET s, we present the following theoretical properties about the explanations of Deep S HAP N ET s (proof in Appendix B) in Theorem 12 and later in Lemma 13.
Theorem 12.
The Deep S HAP N ET explanation h ( x ) deﬁned in Def.
11 provides an explanation that satisﬁes both local accuracy and missingness with respect to H .
Deep S HAP N ET with Disjoint Pairs and Butterﬂy Connections It is immediately obvious that the computational graph determined by the choice of active sets in Shapley Modules dictates how the features interact with each other in a in Shallow S HAP N ET s and hence a Deep S HAP N ET .
For our experiments, we focus on one particular construction of a Deep S HAP N ET based on disjoint pairs (active sets of Shapley modules do not overlap: ∩ cj =1 A ( f j ) = ∅ ) in each layer and a butterﬂy permutation across different layers to allow interactions between many different pairs of features— similar to the Fast Fourier Transform (FFT) butterﬂy construction.
This also means that the cardinality of the active set of all the Shapley modules is set to 2, making the overhead for computing the Shapley values roughly 4 × that of the underlying function.
An example of this construction can be seen on the right of Fig.
We emphasize that this permutation is one of the choices that enable fast feature interactions for constructing deep S HAP N ET s from Shallow S HAP N ET s. We do not claim it is necessarily the best but believe it is a reasonable choice if no other assumptions are made.
One could also construct S HAP N ET s based on prior belief (an example for images below).
It is possible to learn the pairs by creating redundancy and pruning as discussed in subsection 2.5 and Appendix D.
Deep S HAP N ET for Images
Here we describe one of the many possibilities to work on image dataset with S HAP N ET s. This is largely inspired by works including Desjardins et al.
; Dinh et al.
; Kingma & Dhariwal (2018).
Still, the Deep S HAP N ET here consists of different layers of Shallow S HAP N ET s. We begin by describing the operations in each of the consecutive Shallow
S HAP N ET s, and then cascade from one stage to the next.
Canonical convolution operation is composed of three consecutive operations: sliding-window (unfolding the image representation tensor to acquire small patches matching the the ﬁlter), matrix multiplication (between ﬁlters and the small patches from the images), and folding the resulting representation into the usual image representation tensor.
We merely 1) replace the matrix multiplication operation with Shapley modules (similar to that in Desjardins et al.
(2015)) and 2) put the corresponding output vector representation in space R c (cid:48) back in the location of the original pixel (and performed summation of the values overlapping at a pixel location just as canonical convolution operation would), since we have the same number of pixels.
To create a hierarchical representation of images, we use à-trous convolution (Yu & Koltun, 2016) with increasing dilation as the Shapley transform goes from one stage to the next, similar to Chen et al.
; Chen et al.
To reduce computation, we can choose a subset of the channel dimensions of each pixels to take as inputs, similar to Dinh et al.
; Kingma & Dhariwal (2018).
For the ﬁnal prediction, we do a global summation pooling, given that the ﬁnal representation conform to Remark 6.
See subsection H.3.
2. E XPLANATION REGULARIZATION DURING TRAINING
One of the important beneﬁts to S HAP N ET s is that we can regularize the explanations during training .
The main idea is to regularize the last layer explanation so that the model learns to attribute features in ways aligned with human priors–e.g. sparse or smooth.
This is quite different from smoothing the explanation post-hoc as in saliency map smoothing methods (Smilkov et al., 2017; Sundararajan et al., 2017; Yeh et al., 2019), but falls more into the lines of usage of interpretations of models as in Ross et al.
; Liu & Avci (2019); Erion et al. ; Rieger et al. Tseng et al. (2020); Noack et al.
For (cid:96) 1 regularization, the main idea is similar to sparse autoencoders in which we assume the latent representation is sparse.
This is related to the sparsity regularization in LIME (Ribeiro et al., 2016) but is fundamentally different because it actually changes the learned model rather than just the explanation.
Note that the explanation method stays the same. For (cid:96) ∞ regularization, the idea is to smooth the Shapley values so that none of the input features become too important individually, be useful for example in a security setting where the model should not be too sensitive to any one sensor because the sensors could be adversarial attacked.
Finally, with ad-hoc knowledge on the attribution, the user could specify different regularizations for different tasks.
2. C ONDITIONAL DYNAMIC PRUNING
Lemma 13 allows for pruning during inference time and learning during training time (Appendix D).
Lemma 13 (Missingness in Deep S HAP N ET s) .
In any of the Shapley representations, Z ( l ) i ∈ R C , from each Shallow S HAP N ET inside a Deep S HAP N ET , Z ( l ) , if the Shapley value vector corresponding to a certain input feature i ∈ I D is equal to the reference vector r i ∈ R C
(zeros for Shallow S HAP N ET s other than the ﬁrst) for the same feature , we have Z ( m ) i = 0 for any m > l .
Lemma 13 is a simple extension of Theorem 12: if a feature’s importance becomes 0 at any stage, the importance of the same feature will be 0 for all the stages that follow.
In particular, the computation of a Shapley Module could be avoided if its features in active sets are all set to zeros.
In other words, we prune the Shapley Modules whose inputs are zeros instead of outputs so the computation would not happen in the ﬁrst place.
In practice, we set a threshold close to zeros, and skip the computations involving the input features with importance values lower than the threshold.
Note that the importance values for the features are conditioned on the input instances instead of a static pruned model (Ancona et al., 2020), allowing more ﬂexibility.
Moreover, with the help of (cid:96) 1 regularization discussed in subsection 2.4 (which can applied on each Z ( l ) ), we can avoid even more computation.
An illustration can be found in Fig.
3 where we mask the pixels with small (cid:96) 1 norm after normalization, and pruning results are shown against overall performance of the model in Fig.
We draw similarity to prior works including Sun et al. Zeng et al. ; Han et al.
; Bacon (2015); Ioannou et al.
(2016); Bengio et al. ; Theodorakopoulos et al. ; Bolukbasi et al. [Figure]
E XPERIMENTS & V ISUALIZATIONS
We will (1) validate that our S HAP N ET models can be quite expressive despite the intrinsic explanation design, (2) demonstrate that our intrinsic S HAP N ET explanations perform comparably or better than posthoc explanations, and (3) highlight the novel capabilities.
More details in Appendix
Datasets First, we create a synthetic regression dataset by sampling the input v ∈ [ − 0 .
5]from the uniform distribution where the true (synthetic) regression model is f ( v ) =
(cid:81)
i v i
+ (cid:80) i v 2 i
+ 0 .
05 (cid:15) , where (cid:15) is sampled from a standard normal distribution.
Second, we choose ﬁve real-world datasets from Dua & Graff (2017):
Yeast ( d = 8 ) and Breast Cancer Wisconsin (Diagnostic) ( d = 30 ),
MNIST (LeCun & Cortes, 2010), FashionMNIST (Xiao et al., 2017), and Cifar-10 (Krizhevsky, 2009) datasets to validate S HAP N ET on higher-dimensional datasets.
S HAP N ET model performance We would ﬁrst like to validate that our Deep and Shallow S HAPN ET models can be comparable in performance to other models—i.e., that our Shapley module structure does not signiﬁcantly limit the representational power of our model.
Thus, we deﬁne DNN models that are roughly comparable in terms of computation or the number of parameters—since our networks require roughly four times as much computation as a vanilla DNN with comparable amount of parameters.
For our comparison DNNs, we set up a general feedforward network with residual connections (He et al., 2016).
Note that for the vision tasks, we compare with two other models with our Deep S HAP N ET s: a comparable convolutional-layer based model which shares the same amount of channels, layers, and hence parameters with only LeakyReLU activation (Xu et al., 2015) with no other tricks other than normalizing the input and random cropping after padding (same as our Deep S HAP N ET s), and the state-of-the-art on each dataset (Byerly et al., 2020; Tanveer et al., 2020;
Foret et al., 2021).
The performance of the models are shown in Table 1 and Tablein which the loss is shown for the synthetic dataset (lower is better) and classiﬁcation accuracy for the other datasets (higher is better).
While there is a very slight degradation in performance, our lower-dimensional models are comparable in performance even with the structural restriction.
While there is some gap between state-of-the-art for image data, our image-based deep S HAP N ET models can indeed do reasonably well even on these high-dimensional non-linear classiﬁcation problems, but we emphasize that they provide fundamentally new capabilities as explored later.
See Appendix H.
Explanation quality We now compare the intrinsic S HAP N ET explanations with other post-hoc explanation methods.
First, because our intrinsic S HAP N ET explanations are simultaneously produced with the prediction, the explanation time is merely the cost of a single forward pass (we provide a wall-clock time experiment in Appendix E).
For the lower- dimensional datasets, we validate our intrinsic explanations compared to other SHAP-based post-hoc explanations by computing the difference to the true SHAP values (which can be computed exactly or approximated well in low dimensions).
Our results show that Shallow S HAP N ET indeed gives the true SHAP values up to [Table]
[Table]
[Table]
[Table]
numerical precision as proved by Theorem
[Table]
and Deep S HAP N ET explanations are comparable to other post-hoc explanations (result in Table 3 and more discussion in Appendix F).
We will evaluate our high-dimensional explanations in the MNIST dataset based on 1) digit ﬂipping experiments introduced in Shrikumar et al.
(2017), where higher relative log-odds score is better, 2) removing the least- k salient features, where lower fraction of change in the output is better, 3) dropping the top- k salient features and measuring the remaining output activation of the original predicted class, where ideally, the top features would drastically reduce the output logits—showing that our explanations do highlight the important features.
We compare, with the same baselines for all values, Deep S HAP N ET explanations with DeepLIFT (rescale rule in Shrikumar et al. , DeepSHAP (Lundberg & Lee, 2017), Integrated Gradients (Sundararajan et al., 2017), Input × Gradients (Shrikumar et al., 2016) and gradients (saliency as in Baehrens et al. (2010); Simonyan et al.
From Fig.
4, Deep
S HAP N ET offers high-quality explanations on different benchmarks.
Layer-wise explanations With the layer-wise explanation structure
, we can probe into the Shapley representation at each stage of a Deep S HAP N ET as in Fig.
3 and perform dynamic pruning.
Hence we also provide a set of pruning experiment with retaining accuracy shown in Fig.
5, visualization shown in Fig. 3, and discussion in Appendix G, where we show how pruning the values in the Shapley representations of different layer changes model performance, and argue that a large portion of the computation can be avoided during inference time and the model can retain almost the same accuracy.
This showcases this ability to perform pruning dynamically conditioned input instances.
We note that though this has been done before, our models is, to the best of our knowledge, the only model capable of learning the importance of different computational components in tandem with the importance of each feature, performing conditional pruning during inference time, and explanation regularization.
[Figure] [Figure]
pixels pruned 0. Regular 20 40 60 80 100 % of pixels pruned 0.
Output ‘ ∞
Layer
Layer Layer
Layer Layer
Layer Layer
Figure
1) Pruning values from earlier layers have at least equal or stronger effects on the performance than from later layers due to Lemma 13; 2) (cid:96) 1 regularized model performs better especially when the regularization is applied on all all the layers instead of just the output; 3) Most of the values can be removed and retain the accuracy.
We prune the values in the order from the least salient (in magnitude of (cid:96) 1 norm) to the most pixel by pixel.
This is measured on the MNIST test set.
[Figure]
Explanation regularization during training We experiment with (cid:96) 1 and (cid:96) ∞ explanation regularizations during training to see if the regularizations signiﬁcantly affect the underlying model behavior.
For the lower-dimensional datasets, we compute the accuracy, the sparsity, and the coefﬁcient of variation (CV, standard deviation divided by mean) for our Deep S HAP N ET .
We see in Table 4 that our (cid:96) ∞ regularization spreads the feature importance more evenly over features (i.e., low CV), (cid:96)regularization increases the sparsity (i.e., higher sparsity), and both either improve or marginally degrade the accuracy.
For MNIST data, we visualize the explanations from different regularization models in Fig.
6, and we perform the top- k feature removal experiment with different regularizers in
Fig.
7, which shows that our regularizers produce some model robustness towards missing features.
Fig.
5 shows (cid:96) 1 regularized model achieves the best result under pruning across different layers, especially when applied on all layers, further increasing the amount of computation to be pruned. A CKNOWLEDGMENTS
R.W. and D.I. acknowledge support from Northrop Grumman Corporation through the Northrop
Grumman Cybersecurity Research Consortium (NGCRC) and the Army Research Lab through contract number W911NF-2020-221.
X.W. acknowledges support from NSF IIS #1955890.
In addition, the authors would like to thank anonymous reviewers of ICLR 2021, the conversations with whom were both productive and inspiring.
A P ROOF OF T HEOREM 8 & L EMMA
Proof of Lemma 5.
For each row of W (cid:44) AZ , we can show that it is the Shapley value for ˜ f
( k ) using the linearity of Shapley values (the original motivating axiom of Shapley values, see Shapley (1953, Axiom 4 & Axiom 5)):
[Equation]
Combining this result for all rows with Def. 3, we arrive at the result.
We deﬁne a feature selector function π ( w ) as follows:
Deﬁnition 14 (Feature selector function) .
π ( w ) : R d (cid:55)→ R | w | (for any w ∈ W , where W again indexes the set of all component Shapley modules in a given Shallow S HAP N ET as in Def. 7) is deﬁned to select the input features by the index w , which contains the indices of the supposed input features of F ( w ) :
[Equation]
Lemma 15.
The composite function F (cid:48) ( w ) for a given Shapley module F ( w ) , deﬁned as
[Equation]
8) has the following Shapley values for its input features
[Equation]
[Equation]
Proof.
This holds since the absence (set to reference values) of a feature does not make a difference to the ﬁnal output of F (cid:48)
( w ) .
Now we are ready to present proof for Theorem 8:
Proof.
We consider a simpler, and yet more general case: consider where each of the modules deﬁned in Def.
10, we ﬁrst transform any Shapley module F ( w ) in the given Shallow S HAP N ET G to F (cid:48) ( w ) .
Then, by Def. 7, the zeros in Lemma 15 do not make any difference whether to the output of G or
its associated β
i ’s, and yet β i ’s are nothing but summations of Shapley values of different F ( w ) ’s and by
Lemma 5, [Equation]
[Equation] which completes the proof.
B P ROOF OF T HEOREM
Proof.
The proof of local accuracy is trivial by construction since H ( x ) (cid:44) sum
[ D ] ( h ( x )) from
Def. 11.
The proof of missingness is straightforward via induction on the number of layers, where the claim is that if x i =
r i (i.e., the input for one feature equals the reference value), then ∀ (cid:96), h ( (cid:96) )
i ( x ) =(i.e., the pre-summation outputs of the Shapley modules in all layers corresponding to feature i will all be zero).
For the base case of the ﬁrst layer, we know that if x i
= r
i , then we know that Z (1) i
=by the construction of Shapley modules.
For the future layers, the inductive hypothesis is then that if previous layer representation Z ( (cid:96) − 1) i of a feature i is 0 , then the representation from the current layer (cid:96) for this feature remains the same, i.e., Z ( (cid:96) )
i = 0 .
Because the reference values for all layers except the ﬁrst layer are zero, then we can again apply the property of our Shapley modules and thus the inductive hypothesis will hold for (cid:96) > 1 .
Thus, the the claim holds for all layers (cid:96) , and h i ( x ) = sum
[ c ] Z ( (cid:96) )
i will also be zero—which proves missingness for Deep S HAP N ET s.
C D EDICATED L ITERATURE R EVIEW C. E XPLANATION M ETHODS
Our methodology sits in between post-hoc and intrinsic approaches by constructing an explainable model but using SHAP post-hoc explanation concepts and allowing more complex feature interactions.
Post-hoc feature attribution Post-hoc feature-attribution explanation methods gained traction with the introduction of LIME explanations (Ribeiro et al., 2016) and its variants (Ribeiro et al., 2018; Guidotti et al., 2018a;b), which formulate an explanation as a local linear approximation to the model near the target instance.
LIME attempts to highlight which features were important for the prediction.
Saliency map explanations for image predictions based on gradients also assign importance to each feature (or pixel)
(Simonyan et al., 2014; Shrikumar et al., 2017).
SHAP explanations attempt to unify many previous feature attribution methods under a common additive feature attribution framework (Lundberg & Lee, 2017; Lundberg et al., 2020), which is the main motivation for our work.
From a different route, gradient-based saliency map is another family of methods to highlight important components of each input (Ancona et al., 2018).
Input gradients (saliency as in Baehrens et al. (2010); Simonyan et al. (2014)) tries to identify the important features for a prediction value by taking the gradient from the output w.r.t the input. Input × Gradients (Shrikumar et al., 2016) uses the saliency map acquired by the input gradient method and performed element-wise multiplication with the input instance, which tried to sharpen the saliency map. Integrated Gradients (Sundararajan et al., 2017), on the other hand, sets up a reference values from which the gradient is integrated to the current input values, and integrate the input gradient taken along a chosen path from the reference values to the current input instance.
Intrinsically interpretable models Intrinsically interpretable models are an alternative to post-hoc explanation methods with a compromise on expressiveness of the models.
In particular, GAM (Lou et al., 2012) and its extension (Lou et al., 2013; Caruana et al., 2015; Wang et al., 2018) construct models that can be displayed as series of line graphs or heatmaps.
Because the entire model can be displayed and showed to users, it is possible for a user to directly edit the model based on their domain knowledge.
In addition, Alvarez Melis & Jaakkola (2018) proposed an architecture that are complex and explicit if trained with a regularizer tailored to that architecture.
Continuing that line of work, Plumb et al.
(2019) proposed E XP O, a regularizer to help improve the interpretability of black boxes.
E XP O was designed not for a speciﬁc structure but for all neural network models. C. A PPROXIMATING S HAPLEY VALUES To alleviate the computational issue, several methods have been proposed to approximate Shapley values via sampling (Strumbelj & Kononenko, 2010) and weighted regression (Lundberg & Lee, 2017, Kernel SHAP), a modiﬁed backpropagation step (Lundberg & Lee, 2017, Deep SHAP), utilization of the expectation of summations (Ancona et al., 2019, DASP), or making assumptions on underlying data structures (Chen et al., 2019, L-Shapley and C-Shapley). In terms of computational complexity, Sampling based methods (Strumbelj & Kononenko, 2010; Lundberg & Lee, 2017) are asymptotically exact in exponential complexity in the number of input features.
Deep Approximate Shapley Propagation (DASP) approximates in polynomial time, L-Shapley and C-Shapley do in linear with the Markov assumption, Deep SHAP approximates with an additional backward pass after the forward pass, and our method performs approximation with a constant complexity with a single forward pass, which additionally allows regularizing the explanations according to human priors and hence modifying the underlying model during training. C. R EGULARIZING ON THE EXPLANATIONS Our S HAP N ET s allow the users to modify the underlying function based on the explanations.
This falls in line with previous works (Ross et al., 2017; Liu & Avci, 2019; Erion et al., 2020; Rieger et al., 2020; Tseng et al., 2020), each of which adds an extra term in the loss function measuring the disagreement between the explanations and human priors.
Ross et al.
(2017) proposed to regularize the input gradients as explanations, which creates the need for second-order gradient.
Liu & Avci
(2019) uses Integrated Gradients (Sundararajan et al., 2017, IG), which is an extension to Shapley values (Sundararajan & Najmi, 2020).
Due to the use of IG, several samples (for one explanation) are additionally warranted.
Erion et al.
proposed Expected Gradient to rid of reference values in IG and suggested the usage of a single sample.
Tseng et al.
(2020) proposed to force the model to focus on low-frequency component of the input data by applying Fourier Transform on input gradients.
Rieger et al.
(2020) adds to this line of work with Contextual Decomposition (Murdoch et al., 2018; Singh et al., 2019) which avoids the need of computation of second-order gradients and extra samples, and in addition enable regularization on interactions.
Our work differs with these ﬁve in that we do not need a second order gradient and that we are focusing on the more theoretically-grounded Shapley values.
The explanation-related regularizations in Alvarez Melis & Jaakkola (2018); Plumb et al. Noack et al.
(2021) differ with ours in that the regularizers are used to promote interpretability and ours uses regularization to modify the underlying model to conform to prior belief.
(Noack et al. (2021) aims at connecting interpretability with We also draw close similarity to works that that attempt to smooth saliency map explanations (Smilkov et al., 2017; Sundararajan et al., 2017;
Yeh et al., 2019) which acts on explanation methods to induce certain properties in explanations but not the actual models.
C. C ONDITIONAL P RUNING
In contrast to prior work where people perform static structure pruning based on Shapley values (Ancona et al., 2020), we compute the importance values on the ﬂy during inference time.
Conditional Computation in Deep Neural Networks Conditional computation refers to a certain fashion where the neural network only activates a certain subset of all computations depending on the input data (Bengio et al., 2016).
This is desirable in low-power settings where the device can run only a fraction of the entire model to save memory.
Works investigating this computation style include
Sun et al.
Zeng et al.
; Han et al.
; Bacon (2015); Ioannou et al. (2016); Bengio et al. (2016); Theodorakopoulos et al.
; Bolukbasi et al. D L EARNED FEATURE INTERACTIONS With both (cid:96) 1 regularization and missingness in Deep S HAP N ET as discussed in subsection 2.5, we can discuss how it is possible to learn the speciﬁc choices for Shapley Modules in each Shallow
S HAP N ET in a Deep S HAP N ET (how features interact in a Deep S HAP N ET ).
Note that we have already discussed how we can perform instance-based pruning during inference.
To learn the interactions, we ﬁrst create redundancies in each Shallow S HAP N ET inside a Deep
S HAP N ET .
During training, we automatically gain the importance values for each features by construction, from which we will determine the modules that are worth computing by comparing input representation of the features in their active sets to 0 s. If the input representation is close enough to 0 s, we can simply ignore that module.
Of course, the underlying function can always change during training, and we argue that we can slowly strip away modules from the latter layers to the ﬁrst layers as the training progresses to allow for more expressiveness and more room to correct early mistakes by the training procedure.
E W ALL - CLOCK TIME FOR EXPLANATION EXPERIMENT
To validate the efﬁciency of computing an explanation, we compare the wall-clock time of our
S HAP N ET explanations to other approximation methods related to Shapley-values including Deep
SHAP and Kernel SHAP with different numbers of samples—40, 77, and default ( 2 × d + 2 11 ).
The wall-clock computation time in seconds can be seen in Table 5.
We note importantly that this comparison is a bit odd because other Shapley-value explanation methods are post-hoc explanations, whereas S HAP N ET is both a model and an intrinsic explanation method.
Thus, the most fair comparison in terms of time is to have the post-hoc explanation methods explain our S HAP N ET model as seen in the ﬁrst column of Table 5.
Nevertheless, we also show the wall-clock times for the post-hoc methods on our comparable DNNs in columns two and three of Table 5—note that [Table]
S HAP N ET cannot explain other DNNs hence the N/A’s.
From the ﬁrst column, we can see that our method is inherently faster than the others when applied to our model.
However, smaller model (i.e.,
DNN w. equivalent parameters) may come with faster explanations.
F L OWER - DIMENSIONAL COMPARISON BETWEEN S HAP N ET EXPLANATIONS AND POST - HOC EXPLANATIONS
For lower-dimensional datasets, we seek validation for the explanations h ( x ) provided by Deep
S HAP N ET s by comparing to the true Shapley values of our models (which can be computed exactly or aproximated well in low dimensions).
Our metric is the normalized (cid:96) 1 norm between the true
Shapley values denoted φ and the explanation denoted γ deﬁned as:
[Equation]
where the number of dimension d is used as an effort to make the distance Note that we measure the average difference from vector-output classiﬁers by means discussed in subsection F.1.
The experimental setup and results are presented in subsection F.2.
F. C OMPARING SHAP APPROXIMATION ERRORS WITH VECTOR OUTPUT MODELS
To compare the approximation errors for every output of the model, we use a weighted summation of the explanations for different outputs, where the weights are the models’ output after passing into the soft-max function.
Thus, we up weight the explanations that have a higher probability rather than taking a simple average.
Thus, in the extremes, if the predicted probability for a label is zero, then the difference in explanation is ignored, while if the predicted probability for a label is one, then we put all the weight on that single explanation.
Concretely, for a classiﬁcation model C : R d (cid:55)→ R n , where d is the number of input features and n is the number of classes, we have the output vector of such model with an input instance x ∈ R d as C ( x ) ∈ R n .
For each output scalar value [ C ( x )]
j , we compute the approximation of the SHAP values w.r.t.
that particular scalar for feature indexed i , denoted by γ ( j ) i , and hence its corresponding errors as measured in normalized (cid:96) 1 distance as discussed in Eqn.
11: (cid:96) ( j ) 1 .
The ﬁnal error measure that we compare between classiﬁers (cid:96) ∗ 1 is simply a weighted sum version of the normalized (cid:96) 1 :
[Equation]
where the softmax is taken on the non-normalized output of the classiﬁers.
Note that (cid:96) j 1 is also taken on the raw output.
F. S ETUP AND RESULTS
Because the explanation difference from SHAP is independent of the training procedure but solely depends on the methods, we compare the performance of the explanation methods on untrained models that have randomly initialized parameters (i.e., they have the model structure but the parameters are random)—this is a valid model it is just not a model trained on a dataset.
Additionally, we consider the metric after training the models on the synthetic, yeast, and breast cancer datasets.
Note that for even a modest number of dimensions, computing the exact Shapley values is infeasible so we [Table] use a huge number of samples ( 2 16 ) with the kernel SHAP method to approximate the true Shapley values as the ground truth for all but the Yeast dataset, which only requires 2 8 samples for exact computation.
Note that the extended Shapley value (ext.) requires 2 14 times of model evaluation for a single explanation which is already extremely expensive, and thus included merely for comparison.
Results of the average difference can be seen in Table 3.
From the results in Table [Table] we can see that indeed our Shallow S HAP N ET model produces explanations that are the exact Shapley values (error by ﬂoating point computation) as Theorem 8 states.
Additionally, we can see that even though our
Deep S HAP N ET model does not compute exact Shapley values, our explanations are actually similar to the true Shapley values.
We also provide in Table 6 the standard deviation of the approximation errors as in Table 3.
These are the same runs as those in Table 3.
G P RUNING EXPERIMENT
We also conduct a set of pruning (set to reference values) experiment.
This is, again, enabled by the layer-wise explainable property of Deep S HAP N ET .
Recent work has investigated the usage of
Shapley values for pruning (Ancona et al., 2020; Ghorbani & Zou, 2020), but our work means to showcase the ability even further.
Results are shown in Fig.
We perform the pruning experiment with all four MNIST-traiend models as discussed in subsection 2. and subsection 2.5 with no, output (cid:96) 1 , layer (cid:96) 1 , and (cid:96) ∞ regularizer, respectively.
The values are removed (set to reference) by the order of their magnitude (from smallest to largest).
We can see that most of the values might be dropped and the model performance stays unhinged.
Moreover, (cid:96) 1 seems to have the best robustness against pruning, which makes sense as it induces sparsity in explanation values.
Future works are to be done on the computational cost reduction under such pruning technique.
H M ORE EXPERIMENTAL DETAILS
For all the experiments described below, the optimizers are Adam (Kingma & Ba, 2014) with default parameters in PyTorch (Paszke et al., 2019).
H. T IMING EXPERIMENTS
For Table 5, the timing is for the untrained (randomly initialized) models with 16 features.
The numbers after Kernel SHAP indicates the number of samples allowed for Kernel SHAP’s sampling method.
Time is measured in seconds on CPU with 1000 independent trials per cell as Kernel SHAP supports solely CPU from the implementation of the authors of Lundberg & Lee (2017).
The model of the CPU used for testing is Intel(R)
Xeon(R)
Silver 4114 CPU @
2.20GHz.
We perform explanation on randomly initialized 16-feature Deep S HAP N ET model for 1000 rounds.
The 16-feature setup means we will have (16 / 2) × log 2 16 = 32 Shapley modules inside the entire model spanned out in 4 stages according to the Butterﬂy mechanism.
The inner function f for every module in this setting is set as a fully-connected neural network of 1 hidden layer with 100 hidden units.
For the two reference models, the model equivalent in computation time has 8 layers ofhidden units in each layer, while the model equivalent in parameters has 11 layers of 123 hidden units in each layer.
The output of the models are all scalars.
H. Y EAST AND B REAST C ANCER DATASETS EXPERIMENT SETUPS
This setup applies for the experiments involving both datasets, including those experiments for model performance and those that showcase the ability for explanation regularization.
The experiments on the two datasets were run for 10 rounds each with different random initialization and the end results were obtained by averaging.
For each round, we perform 5-fold cross validation on the training set, and arrive at 5 different models, giving us 50 different models to explain for each datasets.
For the two datasets, we do 9 training epochs with a batch size of 32.
For the preprocessing procedures, we ﬁrst randomly split the data into training and testing sets, with 75% and 25% of the data respectively.
Then we normalize the training set by setting the mean of each feature to 0 and the standard deviation to 1.
We use the training sets’ mean and standard deviation to normalize the testing set, as is standard practice.
The inner functions of the Shapley modules are fully connected multilayer neural networks.
The ﬁrst and second stages of two models on the two datasets are identical in structure (not in parameter values).
The ﬁrst stage modules have 2 inputs and 25 outputs, with a single hidden layer of size 50.
The modules in the second stage have 100 input units and 50 output units with two hidden layer of size 100.
The non-linearity is introduced by ReLU (Nair & Hinton, 2010) in the inner functions.
The rest of the speciﬁcs about the inner function are discussed below.
Note that the dimensionality of the output from the last stage differs with that of the input to the next stage, as is expected by Shapley module where the input has twice the dimensionality of that of the output.
Model for Yeast dataset The Yeast dataset has 8 input features and hence 4 Shapley modules at each of the log 2 8 = 3 stages.
Within each stage, the structure of the inner function is identical except for the parameters.
For the Yeast dataset, the third (last) stage comprises four-layer fully-connected model with inputs from 50 × 2 = 100 units to the number of classes, which is 10, with two hidden layers of 150 units.
Model for Breast Cancer Wisconsin (Diagnostic)
This dataset has 30 input features, which is not a exponent of 2, but we can still construct a Deep S HAP N ET by setting two extra variables to zero.
In fact, in doing so we can simplify the Shapley modules since we know which of the modules in the network always have one or both of its input being 0 at all times.
To construct the model, we realize that the number of stages is 5 and the number of modules at each stage is 16.
For the Breast
Cancer Wisconsin (Diagnostic) dataset, in the the third stage of the model, we use a model withinputs units and 75 output units with two hidden layers of 150 units.
The fourth stage has input units of 75 × 2 = 150 and output of 100 units with two hidden layers of size 200 . The ﬁfth (last) stage has a input dimension of 200 , two hidden layers of 250 units and output of size 10 .
H. S PECIFICS FOR VISION TASKS H.3. S TRUCTURE USED FOR VISION EXPERIMENTS
For all of the vision tasks, we use the same structure with the exception of input channel counts.
We use a simple Linear-LeakyReLU
(Xu et al., 2015) inside each Shapley Module, and the following are the output dimension of the linear layer: 1, 128, 128, 128, 128, 128, 128, 10, with 1 being the input channel of images in the case of MNIST & FashionMNIST or 3 with Cifar-10.
À-trous convolution À-trous convolution is used for learning hierarchical representations of images.
In our model, the dilations are set as follows: 1, 3, 5, 7, 9, 11, 13.
H.3. T RAINING RECIPE
Learning process As discussed before, all the models are trained with Adam in PyTorch with the default parameters.
The batch size is set to 64 for all the vision tasks.
No warm-up or any other scheduling techniques are applied.
The FashionMNIST & MNIST models in Table 1 are trained for 10 epochs while the MNIST models with which we investigate the new capabilities (layer-wise explanations & Shapley explanation regularizations) in Section 3 are trained with just 5 epochs.
The
Cifar-10 models are trained for 60 epochs.
Preprocessing For both MNIST dataset and FashionMNIST dataset, we normalize the training and testing data with mean of 0.1307 and standard deviation of 0.3081, this gives us non-zero background and we would therefore set the reference values to 0 ’s, allowing negative or positive contribution from the background.
Both FashionMNIST and MNIST share the same preprocessing pipeline: 1) for training dataset, we extend the resolution from 28 × 28 to 36 × 36 and then randomly crop to 32 × 32 , and then normalize. 2) for testing data, we simply pad the image to 32 × 32 and normalize them.
For Cifar-10 dataset, during training, we ﬁrst pad 4 pixels on four sides of the image and then randomly crop back to 32 × 32 , perform a horizontal ﬂip with probability 0.5, and normalize the data with means 0 . 4914 , 0 . 4822 , 0 . 4465 and standard deviaiton of 0 . 2023 , 0 . 1994 , 0 . 2010 for each channel, respectively.
For testing, we simply normalize the data with the same mean and standard deviation.
H. I LLUSTRATIONS OF EXPLANATION REGULARIZATION EXPERIMENTS
We present visual representations of our models’ explanation on MNIST (LeCun & Cortes, 2010) dataset in Fig.
8, Fig. 9, and Fig.
In all three ﬁgures, the gray images from left to right are explanations laid out for all 10 classes from 0 to 9, where the red pixels indicate positive contribution and blue pixels indicate negative contribution, with magnitudes presented in the color bar below.
From an intuitive visual interpretation, we argue that our explanation regularizations are indeed working in the sense that (cid:96) ∞ regularization is indeed smoothing the explanations (to put more weight on a large number of pixels) and that (cid:96) 1 regularization limits the number of pixels contributing to the ﬁnal classiﬁcation.
One can notice that, by comparing the color bars below the ﬁgures, the contribution magnitudes for (cid:96) ∞ and (cid:96) 1 regularized models are lower than those of the unregularized counterpart.
We have expected this to happen as both regularization techniques require to make one or more of the attribution values to be smaller.
However, we note that this change in scale of the attribution values should not impact the accuracy of the prediction as the softmax operation at the end will normalize across the outputs for all of the classes.
More interesting discussions are provided in the caption of Fig.
8, Fig. 9, and Fig.
and we strongly encourage readers to have a read.
H. M ORE RESULTS ON D IGIT FLIPPING
More results on digit ﬂipping are in Fig.
We also provide the t-scores and the degrees of freedom (instead of p-values because it is so statistically signiﬁcant that p-values are extremely close to 0’s.) of the paired t-tests to show that the explanation methods of ours provide boost in performance that is statistically signiﬁcant in Table 7.
[Figure]
[Figure] [Figure]
y
[Table]
[Figure] [Figure] [Figure] [Figure]