C AN W E F AITHFULLY R EPRESENT A BSENCE S TATES TO C OMPUTE S HAPLEY V ALUES ON A DNN?
Anonymous authors
Paper under double-blind review
A BSTRACT
Although many methods have been proposed to estimate attributions of input variables, there still exists a significant theoretical flaw in masking-based attribution methods, i.e. it is hard to examine whether the masking method faithfully represents the absence of input variables.
Specifically, for the masking-based attributions, setting an input variable to the baseline value is a typical way of representing the absence of the variable.
However, there are no studies investigating how to represent the absence of input variables and verify the faithfulness of baseline values.
Therefore, we revisit the feature representation of a DNN in terms of causality, and propose to use causal patterns to examine whether the masking method faithfully removes information encoded in input variables.
More crucially, it is proven that the causality can be explained as elementary rationale of the Shapley value.
Furthermore, we define the optimal baseline value from the perspective of causality, and we propose a method to learn the optimal baseline value.
Experimental results have demonstrated the effectiveness of our method.
I NTRODUCTION
Many attribution methods (Zhou et al., 2016; Selvaraju et al., 2017; Lundberg and Lee, 2017) have been proposed to estimatie the attribution of input variables to the model output, which represents an important direction in explainable AI.
In this direction, many methods estimated attributions of input variables based on masking some input variables.
For example, the Shapley value (Shapley, 1953) of an input variable can be considered as the average marginal utility of masking the variable in different contexts.
To this end, Dabkowski and Gal (2017); Ancona et al.
Covert et al.
(2020b) proposed different ways to compute baseline values to represent the masked state of input variables, as shown in Fig.
Theoretically, the trustworthiness of attribution methods highly depends on whether the current baseline value can really remove the information of the input variable without bring in new outof-distribution (OOD) features.
However, there is no criterion to evaluate the information removal of masking methods.
To this end, we need to first break the blind faith that seemingly reasonable baseline values can faithfully represent the absence of input variables, and seemingly OOD baseline values definitely cause abnormal features.
In fact, because different DNNs have different inference logic, seemingly OOD baseline values are not necessarily equivalent to generating OOD features.
Fortunately, Ren et al.
(2021a) have provided a new perspective to analyze feature representation of a DNN. In fact, it has been widely believed that the learning of DNN is a fitting problem, instead of explicitly formulating causality like how graphical models do.
However, Ren et al. (2021a) ; Deng et al.
(2022) have surprisingly discovered that when the DNN was sufficiently trained, the sparse and symbolic AND relationships between input variables emerge. Different AND relationships act like different causal patterns and construct a sparse causal graph to explain the DNN.
As Fig.
1 shows, the first layer of the causal graph represents input variables.
The second layer contains many causal patterns, each representing a specific AND interaction between input variables.
For example, the pattern of a bird head consists of the forehead , eyes , and beak , i.e.,
S = { forehead, eyes, beak } = { f, e, b } for short.
Only if input variables f , e , and b co-appear, the causal pattern S is triggered; otherwise not.
The third layer is the output of the causal graph.
Each triggered causal pattern S has a numerical contribution I S to the output, which is termed its causal effect .
[Figure]
Blurring [Figure] Therefore, we can consider causal patterns in the second layer as a set of interactive concepts memorized by the DNN, and the causal graph can use the triggering of causal patterns to explain the DNN.
It is because given each input sample with n variables, we can generate 2 n different masked samples.
It is proven that we can use a sparse causal graph with a relatively small number of causal patterns to accurately mimic the network output on all 2 n masked samples.
From this perspective, whether baseline values look reasonable is no longer the key factor to determine the trustworthiness of baseline values.
Instead, we check whether setting an input variable to its baseline value can remove all potential AND relationships related to this variable without creating new causal patterns. It is because setting incorrect baseline values let partial effects of the AND relationship be mistakenly explained as new causal patterns without being removed. • For example, let the AND pattern S = { f, e, b } contribute a score E to the output.
If the baseline value b f of the variable f just blur the f ( forehead ) , rather than fully remove its appearance, then partial effects of E is remained not explained by the causal effect I { f,e,b } , i.e. , masking the forehead cannot remove all score E . The remaining score E − I { f,e,b } will be explained as I { e,b } . Thus, masking forehead and deactivating the effect I { f,e,b } cannot remove all effects E of the AND relationship.
• If baseline values of { f, e, b } are set all black regions, then masking all the three regions may generate a new pattern of a black line, as a new causal pattern that influences the network output.
Therefore, we consider the optimal baseline values, which faithfully reflect the true inference logic, usually most reduce the overall strength of existing causal effects, without introducing new causal effects.
However, we find that most existing masking methods are not satisfactory from this perspective (see Section 2.1 and Table 1).
The masking method based on conditional distribution (Covert et al., 2020b; Frye et al., 2021) performs a bit better but still has problems.
In particular, we notice that Shapley values can be derived from effects of causal patterns in theory, i.e., the causal patterns are proven to be elementary effects of Shapley values.
Therefore, we propose a new method to learn optimal baseline values for Shapley values, which is verified to be effective in removing original causal effects of absent input variables while not introducing new causal effects.
Contributions of this paper can be summarized as follows.
(1) We propose a metric to examine whether the masking approach in attribution methods could faithfully represent the absence state of input variables.
Based on this metric, we find that most previous masking methods are not reliable.
(2) We define and develop an approach to estimating optimal baseline values for Shapley values, which ensures the trustworthiness of the attribution.
P ROBLEMS WITH THE REPRESENTATION OF THE MASKED STATES
The Shapley value (Shapley, 1953) was first introduced in game theory, which measures the contribution of each player in a game.
Actually, given an input x with n input variables, i.e., x = [ x 1 , . . .
, x n ] , we can consider a deep model as a game with n players N = { 1 , 2 , · · · , n } .
Each player i is an input variable x
i ( e.g. a input dimension, a pixel, or a word).
In this way, the problem of fairly estimating attributions of input variables in the DNN is equivalent to the problem of fairly assigning the total reward in the game to each player .
To this end, the Shapley value is considered as the unique approach that satisfies the linearity, dummy, symmetry, and efficiency axioms (please re- fer to Appendix B), and it fairly allocates attributions to input variables (Weber, 1988).
The Shapley value of the i -th input variable ϕ
i is defined as follows.
[Figure] [Figure] [Table] [Figure]
[Table] [Figure] [Figure] where v ( x S ) ∈
R denotes the model output when variables in S are present and variables in N \ S are absent.
Specifically, v ( x ∅ ) represents the model output when an empty image is given.
The Shapley value of the variable i is computed as the weighted marginal contribution of i when the variable i is present w.r.t.
the case when the variable i is absent, i.e. v ( x S ∪{ i } ) − v ( x S ) .
Originally, the Shapley value is defined in game theory, in which the presence and absence of a player can be defined without ambiguity.
However, when we explain a DNN, a core challenge is how to faithfully define the absence of an input variable.
Although we admit some alternative methods (Covert et al., 2020b; Frye et al., 2021) compute v ( x S ) based on the conditional distribution, the most classical way is to use baseline values (or called reference values ) b
= [ b 1 , b 2 , . . .
, b n ] to mask variables to represent their absence.
Specifically, we construct the following masked sample x S based on x , where only variables in S are present, and variables not in S are masked.
[Equation]
We aim to learn optimal baseline values b to faithfully represent absent states of input variables.
Representing a DNN using the causal graph.
Given a trained DNN v and an input sample x , it has been proven by Ren et al.
(2021a)
that the three-layer causal graph in Fig.
1 and Fig.
2 can faithfully explain the inference logic of the trained DNN.
The first layer contains n source nodes, and each node is an indicator variable A i ∈ { 0 , 1 } to present the presence of the variable
x
i ( A i = 1 ) and the absence of x i ( A i = 0 ).
Nodes in the second layer are indicator variables C S that represent the AND relationship between input variables in the causal pattern S .
For example, the head pattern of a bird consists of { forehead , eyes , beak } .
When the three parts co-appear, the head pattern S
= { forehead, eyes, beak } is triggered and C S
= 1 ; otherwise, C S = 0 .
In the third layer, the single sink node Y denotes the output of the causal graph.
Thus, the transition probability of the causal graph is given as follows.
[Equation]
where P ( C S =
| A 1 , A 2 , . . . ,
A n )
= 1 − P ( C S =
|
A 1 , A 2 , . . . ,
A n ) .
Ω denotes the set of all causal patterns.
1 ( · ) denotes the indicator function.
U S denotes the effects of the causal pattern S on the output Y .
According to Eq.
(3), the causal relationship between C S and the output Y can be rewritten as the following structural causal model (SCM) (Neuberg, 2003), i.e. , the output Y of the causal graph sums up causal effects of all the triggered causal patterns.
[Equation]
where I S ( x ) denotes the actual causal effect of the causal pattern S .
In implementations, the causal effect U S is computed as Harsanyi dividend (Harsanyi, 1982), i.e., U S =
(cid:80) S ′ ⊆ S ( − 1) | S |−| S ′
| · v ( x S ′ ) .
According to the definition of the Harsanyi dividend, U S measures the pure numerical contribution (causal effect) of the interaction between all variables in S to the DNN output.
But for each smaller subset L ⊊ S , the interaction between variables in L is measured by U L , rather than U S .
Therefore, each causal pattern represents an AND relationship between input variables, and can be considered as an interactive concept memorized by the DNN.
In the aforementioned example, if the forehead , eyes , and beak of the bird co-appear, then the head pattern S = { forehead , eyes , beak } is triggered and makes a causal effect I S
= U S on the output Y .
Otherwise, if any part is masked, the causal pattern S will not be triggered, i.e., C S = 0 , according to Eq. Then, the DNN’s inference score Y ( x ) will not receive the causal effect U S , and I S = C S =0 .
[-= 
-]Note that the causal pattern { eyes , beak } is not deactivated by the absence of forehead , because this pattern represents the AND relationship between eyes and beak , and it does not contain forehead . Because different input variables interact with each other to form AND relationships, when we mask an input variable i , it is supposed to remove not only independent effect of the variable i , but also all AND relationship related to the variable i .
To this end, Appendix E proves that masking the variable i will remove all causal effects of causal patterns containing this variable i . As both proved in Theorem 3 and discussed in the introduction, an incorrect baseline value b i will make partial effects of the AND relationship of the variable i be mistakenly explained as effects of redundant patterns without being removed, thereby complicating the explanation. Therefore, the optimal baseline value is supposed to generate the simplest causal graph explanation. Compared to the causal graph generated by sub-optimal baseline values, the simplest causal graph removes as much as existing causal effects without introducing additional causal effects.
Theorem 1 ( Faithfulness , proven by Ren et al. (2021a) and Appendix D.1) Let us consider a DNN v and a specific sample x with n input variables.
Given any of the 2 n arbitrarily masked input x S , the causal graph defined in Eq.
(4) can always well mimic the output of the DNN, i.e., ∀ S ⊆
N, Y ( x S )
= v ( x S ) , when we set the causal patterns in the causal graph are Ω=2 N = { S | S ⊆ N } .
Remark 1 ( Sparsity ) Empirically, the causal graph to explain the DNN is actually very sparse.
Most nodes S ∈ 2 N have almost zero causal effects U S
≈ 0 .
A subgraph of the causal graph with a few nodes (usually less than 100 nodes with salient causal effects in most cases) are already enough to approximate the DNN, i.e., ∀ S ⊆
N, Y ( x S )
≈ v ( x S ) .
Remark 1 has been verified on different DNNs learned for various tasks by experiments in both Appendix F.2 and (Ren et al., 2021a).
Theorem 1 and Remark 1 prove that a sparse causal graph can faithfully explain the DNN.
Theoretically, given an input x with n variables, we can construct 2 n different masked samples x S w.r.t.
all subsets S ⊆ N .
The causal graph can accurately mimic diverse outputs of the DNN on all 2 n masked samples.
Therefore, the causal graph is considered to faithfully explain the DNN’s inference.
2. P ROBLEMS WITH PREVIOUS MASKING METHODS
We consider that nodes with salient effects as interactive concepts memorized by the DNN, and use such salient concepts to evaluate the quality of masking methods.
Specifically, we examine whether baseline values remove most causal effects depending on x i in the causal graph, and whether baseline values generate new causal effects to the causal graph.
Theoretical support from three perspectives.
The evaluation of the masking methods based on salient causal effects is theoretically supported from the following three perspectives.
First, according to Theorem 1 and Remark 1, the diverse inference on all 2 n masked samples can be well mimicked by a few salient nodes (usually less than 100 nodes with large | I S | values) in the causal graph.
Such a phenomenon has been illustrated in Appendix F.2 and (Ren et al., 2021a).
Masking input variables variables will remove causal effects of the related causal nodes in the causal graph.
Theorem 2 (proven by Harsanyi (1982) and Appendix D.2)
We can directly derive Shapley values from the effects I S of causal patterns.
The Shapley value can be considered as uniformly allocating each causal pattern S ’s effect I S to all its variables, i.e. ϕ i = (cid:80) S ⊆ N \{ i } 1 | S | +1 I S ∪{ i } .
Second, Theorem 2 shows that Shapley values are a re-allocation of causal effects to input variables.
Reducing effects of salient patterns means removing elementary factors of Shapley values.
Besides, in order to verify that the reduction of causal patterns can really represent the absence of input variables, we have conducted experiments to find that salient patterns triggered by white noise inputs were much less than those triggered by normal images.
Please see Appendix F.3 for details.
Remark 2 (proof in Appendix D.3) Let us consider a function with a single causal pattern f
( x S )=
w S (cid:81) j ∈ S ( x j − δ j ) .
Accordingly, ground-truth baseline values of variables are obviously { δ j } , because setting any variable ∀ j ∈ S, x j = δ j will deactivate this pattern.
Given the correct baseline values b ∗ j = δ j , we can use a single causal pattern to regress f ( x S ) , i.e., I S = f
( x S ) , ∀ S ′
̸
= S, I S ′
= 0 .
Theorem 3 (proof in Appendix D.3) For the function f ( x S )= w S (cid:81) j ∈ S ( x j − δ j ) , if we use m ′ incorrect baseline values { b ′ j | b ′ j ̸ = δ j } to replace correct ones to compute causal effects, then the function will be explained to contain at most 2 m ′ causal patterns.
Theorem 4 (proof in Appendix D.3) If we use m ′ incorrect baseline values to compute causal effects in the function f ( x S ) = w S (cid:81) j ∈ S ( x j − δ j ) , a total of (cid:0) m ′ k
−| S | + m ′ (cid:1) causal patterns of the k -th order emerge, k ≥ | S |− m ′ .
A causal pattern of the k -th order means that this causal pattern represents the AND relationship between k variables. [Table]
Third, Remark 2, Theorems 3 and 4 provide a new perspective to understand how incorrect baseline values generate new causal patterns.
Remark 2 proves that correct baseline values faithfully explain the simplest causality in the model.
Theorems 3 and 4 show that incorrect baseline values will use an exponential number of redundant low-order patterns to explain a single high-order pattern.
For example, we are given the function f ( x )=
w ( x p − δ p )( x q − δ q ) s.t. x p =3 , x q =4 , δ p =2 , δ q =3 .
If we use ground-truth baseline values { δ p , δ q } , then the function is explained as simple as a single causal pattern Ω
= {{ p, q }} , which yields correct Shapley values ϕ p = ϕ q = 0 .
5 · w , according to Theorem 2.
Otherwise, if we use incorrect baseline values { b ′ p =1 , b ′ q =1 } , then this function will be explained as four causal patterns Ω= {∅ , { p } , { q } , { p, q }} , i.e., f ( x )=
U ∅ C ∅ + U { p } C { p } + U { q } C { q } + U { p,q } C { p,q } , where U ∅ = 2 w , U { p } = − 4 w , U { q } = − 3 w , and U { p,q } = 6 w are computed using incorrect baseline values.
Incorrect baseline values increase complicate causal patterns and lead to incorrect Shapley values ϕ p = − w, ϕ q =0 .
In fact, the existence of most newly introduced causal patterns is due to that the effects of a high-order causal pattern are not fully removed, besides OOD causal patterns (new OOD edges or shapes) caused by baseline values.
Evaluating previous baseline values.
Therefore, we can compare causal patterns in the masked sample with causal patterns in the original sample to evaluate the following baseline values.
(1) Mean baseline values.
As Fig.
1 shows, the baseline value of each input variable is set to the mean value of this variable over all samples (Dabkowski and Gal, 2017), i.e. b i =
E x [ x i ] .
However, empirically, this method actually introduces additional information to the input.
For example, mean values introduce massive grey dots to images and may form new edges as abnormal causal patterns, which has been verified by experiments in Table 1.
Experimental details are introduced later.
(2) Zero baseline values.
Baseline values of all input variables are set to zero (Ancona et al., 2019; Sundararajan et al., 2017), i.e. ∀ i ∈ N, b i = 0 .
As Fig.
1 shows, just like mean baseline values, zero baseline values also introduce additional information (black dots) to the input (verified in Table 1).
(3) Blurring input samples.
Fong and Vedaldi (2017) and Fong et al.
(2019) blur image pixels x i using a Gaussian kernel as its masked state.
Covert et al.
(2020a)
; Sturmfels et al.
(2020) mentioned that this approach only removed high-frequency signals, but failed to remove low-frequency signals.
(4) For each input variable, determining a different baseline value for each specific context S .
Instead of fixing baseline values as constants, some studies use varying baseline values to compute v ( x S ) given x , which are determined temporarily by the context S in x .
Some methods (Frye et al., 2021; Covert et al., 2020b) define v ( x S ) by modeling the conditional distribution of variable values in N \ S given the context S , i.e. v ( x S ) =
E p ( x ′ | x S )
[ model ( x S ⊔ x ′ N \ S )] .
The operation ⊔ means the concatenation of x ’s dimensions in S and x ′ ’s dimensions in N \ S .
By assuming the independence between input variables, the above conditional baseline values can be simplified to marginal baseline values (Lundberg and Lee, 2017), i.e. v ( x S )=
E p ( x ′ )
[ model ( x S ⊔ x ′ N \ S )] .
We conducted experiments to examine whether the above baseline values remove all causal patterns in the original input and whether baseline values introduce new causal patterns.
We used the metric
R = E x (cid:104) ( (cid:80) S ⊆
N | I ′ S ) |− (cid:80) S ⊆
N | I ( noise ) S | ) /
( (cid:80) S ⊆
N | I S | ) (cid:105) to evaluate the quality of masking.
We generated a set of samples based on x , where a random set of input variables were masked, and I ′ S denote the causal effect in such masked samples.
I S denote the causal effect in the original sample x , which was used for normalization.
I ( noise ) S denote the causal effect in a white noise input, and it represents unavoidable effects of huge amounts of noise patterns.
Thus, we considered the I ( noise ) S term as an inevitable anchor value and removed it from R for more convincing evaluation.
The masking method would have two kinds of effects on causal patterns.
(1) We hoped to remove all existing salient patterns in the original sample.
(2) We did not expect the masking method to introduce new salient patterns.
Interestingly, the removal of existing salient patterns decreased the R value, while the triggering of new patterns increased the R value.
Thus, the R metric reflected the both effects.
A small value of R indicated a good setting of baseline values.
We used 20 images in the MNIST dataset (LeCun et al., 1998) and 20 images in the CIFARdataset (Krizhevsky et al., 2009) to compute R , respectively.
We split each image into 8 × 8 grids.
We randomly masked some grids using the zero baseline, mean baseline, blur baseline, and the baseline based on conditional distribution, and computed the metric of R ( zero ) , R ( mean ) , R ( blur ) , and
R ( conditional ) , respectively.
Table 1 shows that the ratio R by using previous baseline values were all large.
Although the masking method based on conditional distribution (Frye et al., 2021) performed better than some other baseline values, our method exhibited the best performance.
It indicates that previous masking methods did not remove most existing patterns and/or trigger new patterns.
2. A BSENCE STATES AND OPTIMAL BASELINE VALUES
In the original scenario of game theory, the Shapley value was proposed without the need to define the absence of players.
When people explain a DNN, we consider the true absence state of variables as triggering the least causal patterns.
This is because the absence of input variables is supposed to remove all causal effects containing these masked variables.
To this end, Remark 2 and Theoremshow that correct baseline values usually generate the simplest causality-based explanation, i.e., using the least number of causal patterns to explain the DNN.
If an incorrect baseline value b i does not fully remove all effects of AND relationships of the variable i , then the remained effects will be mistakenly explained as a large number of other redundant patterns.
The above proof well fits Occam’s Razor, i.e. , the simplest causality with the minimum causal patterns may potentially represent the essence of the DNN’s inference logic.
This perspective also supports us to consider the baseline values that minimize the number of salient causal patterns ( i.e. , achieving the simplest causality) as the optimal baseline values.
Therefore, the learning of the baseline value b ∗ i of the i -th variable can be formulated to sparsify causal patterns in the deep model.
Particularly, such baseline values are supposed to remove existing causal effects without introducing many new effects.
[Equation]
where I S ( x | b ) denotes the causal effect computed on the sample x by setting baseline values to b .
E STIMATING BASELINE VALUES
Based on Theorem 3, we derive Eq.
(5) to learn optimal baseline values, but the computational cost of enumerating all causal patterns is exponential.
Thus, we explore an approximate solution to learning baseline values.
According to Theorem 4, incorrect baseline values usually mistakenly explain high-order causal patterns as an unnecessarily large number of low-order causal patterns, where the order m of the causal effect I S is defined as the cardinality of S , m = | S | .
Thus, we aim to learn baseline values that correctly reflect high-order causal patterns in the model without introducing redundant low-order patterns.
The objective of learning baseline values is roughly transformed to a loss function that penalizes effects of low-order causal patterns, in or- der to prevent learning incorrect baseline values that mistakenly represent the high-order pattern as an exponential number of low-order patterns.
[Equation]
An approximate-yet-efficient solution.
When each input sample contains a huge number of vari- ables, e.g. , an image sample, directly optimizing Eq.
(6) is NP-hard.
Fortunately, we find the multiorder Shapley value and the multi-order marginal benefit in the following equation have strong connections with multi-order causal patterns (proven in Appendix G), as follows.
[Equation]
where ϕ ( m ) i ( x | b ) and ∆ v i ( S | x , b ) denote the m -order Shapley value and the m -order marginal benefit computed using baseline values b , respectively, where the order m is given as m = | S | .
According to the above equation, high-order casual patterns I S are only contained by high-order
Shapley values ϕ ( m ) i and high-order marginal benefits ∆ v i .
Therefore, in order to penalize the effects of low-order causal patterns, we penalize the strength of low-order Shapley values and low-order marginal benefits, respectively, as an engineering solution.
We design to minimize the following
[Table] [Table]
two loss functions to penalize low-order causal patterns
[Table]
which boost the computational efficiency.
In experiments, these loss functions were optimized via SGD
[Table]
[Equation]
where λ ≥ m denotes the maximum order to be penalized.
We have conducted experiments to verify that baseline values b learned by loss functions in Eq.
(8) could effectively reduced causal effects of low-order causal patterns in Eq.
Please see Appendix F.4 for results.
Most importantly, we still used the metric R in Section 2 to check whether the learned baseline values removed original causal patterns in the input while not introducing new patterns.
The low value of R ours in Table 1 shows that baseline values learned by our method successfully removed existing salient causal patterns without introducing many new salient patterns.
E XPERIMENTS 4. V ERIFICATION OF CORRECTNESS OF BASELINE VALUES AND S HAPLEY VALUES
Correctness of baseline values on synthetic functions.
People usually cannot determine the ground truth of baseline values for real images, such as the MNIST dataset.
Therefore, we conducted experiments on synthetic functions with ground-truth baseline values, in order to verify the correctness of the learned baseline values.
We randomly generated 100 functions, whose causal patterns and ground truth of baseline values could be easily determined.
This dataset will be released after the paper acceptance.
The generated functions were composed of addition, subtraction, multiplication, exponentiation, and the sigmoid operations (see Table 3).
For example, for the function y = sigmoid (3 x 1 x 2 − 3 x 3 − 1 . 5) − x 4 x 5 + 0 .
25( x 6 + x 7 ) 2 , x i ∈{ 0 , 1 } , there were three causal patterns ( i.e. { x 1 , x 2 , x 3 } , { x 4 , x 5 } , { x 6 , x 7 } ), which were activated only if x i = 1 for i ∈ { 1 , 2 , 4 , 5 , 6 , 7 } and x 3 =0 .
In this case, the ground truth of baseline values was b ∗ i =0 for i ∈{ 1 , 2 , 4 , 5 , 6 , 7 } and b ∗ 3 =1 .
Please see Appendix F.5 for more discussions about the setting of ground-truth baseline values.
We used our method to learn baseline values on these functions and tested the accuracy.
Note that | b i − b ∗ i |∈ [0 , 1] and b ∗ i ∈{ 0 , 1 } .
If | b i − b ∗ i | < 0 .
5 , we considered the learned baseline value correct.
We set λ =0 .
5 n in both L Shapley and L marginal .
The results are reported in Table 4 and are discussed later.
Correctness of baseline values on functions in (Tsang et al., 2018).
Besides, we also evaluated the correctness of the learned baseline values using functions in Tsang et al.
Among all the 92 input variables in these functions, the ground truth of 61 variables could be determined (see Appendix F.5).
Thus, we used these annotated baseline values to test the accuracy.
Table 4 reports the accuracy of the learned baseline values on the above functions.
In most cases, the accuracy was above 90%, showing that our method could effectively learn correct baseline values.
A few functions in (Tsang et al., 2018) did not have salient causal patterns, which caused errors in the learning.
Besides, in experiments, we tested our method under three different intializations of baseline values ( i.e., 0, 0.5 and 1).
Table 4 show that baseline values learned with different initialization settings all converged to similar and high accuracy.
Correctness of the computed Shapley values.
Incorrect baseline values lead to incorrect Shapley values.
We verified the correctness of the computed Shapley values on the extended AdditionMultiplication dataset (Zhang et al., 2021).
We added the subtraction operation to avoid all baseline values being zero.
Theorem 2 considers the Shapley value as a uniform assignment of effects of each causal pattern to its compositional variables.
This enabled us to determine the ground-truth
Shapley value of variables without baseline values based on causal patterns.
For example, the function f ( x ) = 3 x 1 x 2 + 5 x 3 x 4 + x 5 s.t. x
= [1 , 1 , 1 , 1 , 1] contained three causal patterns, according to [Table] The function is model ( x ) = − 2 . 62 x 1 − 5 x 3 − 1 .
[-Accordingly, the ground-truth Shapley values were ˆ ϕ 
-]98 x 6 ( x 4 − 0 . 94)+1 .
[-= ˆ ϕ 2 = 3 / 2 , ˆ ϕ 
-]15( x 5 − 0 . 91) − 4 .
23 x 7 , and the input is x = {+[0 , 1 , 1 , 1 , 1 , 1 , 1] . [Table] the principle of the most simplified causality. Accordingly, the ground-truth Shapley values were ˆ ϕ 
+}
= ˆ ϕ 2 =3 / 2 , ˆ ϕ 3 = ˆ ϕ 4 =5 / 2 , and ˆ ϕ 5 =1 .
See Appendix F.6 for more details.
The estimated Shapley value ϕ i was considered correct if | ϕ i − ˆ ϕ i | ≤ 0 . 01 ; otherwise, incorrect.
Then, we computed the accuracy of the estimated Shapley values as the ratio of input variables with correct Shapley values.
We compared our methods with six previous choices of baseline values in Section 2, i.e. zero baseline values (Ancona et al., 2019), mean baseline values (Dabkowski and Gal, 2017), blurring images (Fong et al., 2019), and three implementations of varying baseline values in SHAP (Lundberg and Lee, 2017), SAGE (Covert et al., 2020b), and (Frye et al., 2021).
Discussion on why the learned baseline values generated correct Shapley values.
We computed
Shapley values of variables in the extended Addition-Multiplication dataset using different baseline values, and compared their accuracy in Table 5.
The result shows that our method exhibited the highest accuracy.
Table 6 shows an example of incorrect Shapley values computed by using other baseline values.
Our method generated correct Shapley values on this example.
For the variable x 6 , due to its negative coefficient − 1 . 98 , its contribution should be negative.
However, all other baseline values generated positive Shapley values for x 6 .
The term − 4 .
23 x 7 showed the significant effect of the variable x 7 on the output, but its Shapley value computed using baseline values in
SHAP was just − 0 . 010 , which was obviously incorrect.
Also, the absolute value of the Shapley value of x 1 computed using baseline values in SHAP was even larger than that of x 7 , demonstrating the unreliability of SHAP.
Please see Appendix F.7 for more examples.
4. R ESULTS AND EVALUATION ON REALISTIC DATASETS AND MODELS
Learning baseline values.
We used our method to learn baseline values for MLPs, LeNet (LeCun et al., 1998), and ResNet-20 (He et al., 2016) trained on the UCI South German Credit dataset (namely credit dataset ) (Dua and Graff, 2017), the UCI Census Income dataset (namely income dataset ) (Dua and Graff, 2017), the MNIST dataset (LeCun et al., 1998), and the CIFARdataset (Krizhevsky et al., 2009), respectively.
We learned baseline values by using either L Shapley or
L marginal as the loss function.
In the computation of L Shapley , we set v ( x S )
= log p ( y truth | x S ) 1 − p
( y truth | x S ) .
In the computation of L marginal , | ∆ v i ( S ) | was set to | ∆ v i ( S ) | =
∥ h ( x S ∪{ i } ) − h ( x S ) ∥ 1 , where h ( x S ) denotes the output feature of the penultimate layer given the masked input x S , in order to boost the efficiency of learning.
Please see Appendix F.8 for other potential settings of v ( x S ) .
We set λ = 0 .
2 n for the MNIST and the CIFAR-10 datasets, and set λ =0 .
5 n for the simpler data in two UCI datasets.
Given baseline values, we used the sampling-based approximation (Castro et al., 2009) to estimate Shapley values, and we have verified the stability of the estimated Shapley values in Appendix F.10.
We used two ways to initialize baseline values before learning, i.e. setting to zero or mean values over different samples, namely zero-init and mean-init , respectively.
Fig.
3 (left) and Fig.show that baseline values learned with different initialization settings all converged to similar baseline values, except for very few dimensions having multiple local-minimum solutions (discussed in
Appendix F.12), which proved the stability of our method.
Comparison of attributions computed using different baseline values.
Fig.
3 shows the learned baseline values and the computed Shapley values on the income dataset.
We found that attributions generated by zero/mean baseline values conflicted with the results of all other methods.
Our method obtained that the occupation had more influence than the marital status on the income, which was somewhat consistent with our life experience.
However, baseline values in SHAP and SAGE sometimes generated abnormal explanations.
In this top-right example, the attribute capital gain was zero, which was not supposed to support the prediction of “the person made over 50K a year.”
However, the SAGE’s baseline values generated a large positive Shapley value for capital gain . In the [Figure]
[Figure] [Table] [Table] bottom-right example, both SHAP and SAGE considered the marital status important for the prediction.
SHAP did not considered the occupation as an important variable.
Therefore, we considered these explanations not reliable.
Attribution maps and baseline values generated on the CIFARand the MNIST datasets are provided in Appendix F.11.
Compared to zero/mean/blurring baseline values, our baseline values were more likely to remove noisy variables on the background, which were far from the foreground in images.
Compared to SHAP, our method yielded more informative attributions.
Besides, our method generated smoother attributions than SAGE.
Quantitative evaluation of attributions for image classification.
In order to quantitatively evaluate Shapley values computed by different baseline values on the MNIST dataset, we constructed an And-Or decision tree following (Harradon et al., 2018), whose structure directly provided the ground-truth Shapley value for each input variable.
We used different attribution methods to explain the decision tree.
Please see Appendix F.13 for more about the And-Or tree.
Table 7 shows that our method generated more accurate Shapley values than other baseline values.
4. R EMOVING ADVERSARIAL PERTURBATIONS FROM THE INPUT
Let x denote the normal sample, and let x adv = x + δ denote the adversarial example generated by (Madry et al., 2018).
According to (Ren et al., 2021b), the adversarial example x adv mainly created out-of-distribution bivariate interactions with high-order contexts, which were actually related to the high-order interactions (causal patterns) in this paper.
Thus, in the scenario of this study, the adversarial utility was owing to out-of-distribution high-order interactions (causal patterns).
The removal of input variables was supposed to remove most high-order causal patterns.
Therefore, the baseline value can be considered as the recovery of the original sample.
In this way, we used the adversarial example x adv to initialize baseline values before learning, and used L marginal to learn baseline values.
If the learned baseline values b satisfy ∥ b − x ∥ 1 ≤∥ x adv − x ∥ 1 , we considered that our method successfully recovered the original sample to some extent.
We conducted experiments using LeNet, AlexNet (Krizhevsky et al., 2012), and ResNet-20 on the MNIST dataset ( ∥ δ ∥ ∞ ≤ 32 / 255 ) and the CIFAR-10 dataset ( ∥ δ ∥ ∞ ≤ 8 / 255 ).
Table 8 shows that our method recovered original samples from adversarial examples, which demonstrated the effectiveness of our method.
Please see
Appendix F.14 for more discussions.
C ONCLUSIONS
In this paper, we have defined the absence state of input variables in terms of causality.
Then, we have found that most existing masking methods cannot faithfully remove existing causal patterns without triggering new patterns.
In this way, we have formulated optimal baseline values for the computation of Shapley values as those that remove most causal patterns.
Then, we have proposed an approximate-yet-efficient method to learn optimal baseline values that represent the absence states of input variables.
Experimental results have demonstrated the effectiveness of our method.
E THIC S TATEMENT
This paper aims to examine the masking approach in previous explaining methods.
We find that previous settings of the masking approach cannot faithfully represent the absence of input variables, thereby hurt the trustworthiness of the obtained explanations.
Therefore, we propose a new method to learn optimal baseline values to represent the absence of input variables.
In this way, the trustworthiness of explanations of the DNN is further boosted.
There is no ethic issues with this paper.
R EPRODUCIBILITY S TATEMENT
We have provided proofs for all theoretical results in Appendix D and Appendix G. We have also provided experimental details in Section 4 and Appendix F. Furthermore, we will release the code when the paper is accepted
A R ELATED WORKS
No previous methods directly examined the faithfulness of the masking methods.
Instead, we made a survey in a larger scope of attribution methods and other explainable AI studies, and put them in the appendix.
Nevertheless, we will put this section back to the main paper if the paper is accepted.
In the scope of explainable AI, many methods (Simonyan et al., 2014; Yosinski et al., 2015; Mordvintsev et al., 2015; Dosovitskiy and Brox, 2016; Zhou et al., 2015) have been proposed to explain the DNN.
Among all methods, the estimation of attributions for each input variable represents a classical direction (Zhou et al., 2016; Selvaraju et al., 2017; Lundberg and Lee, 2017; Shrikumar et al., 2017).
In this paper, we mainly focus on attributions based on Shapley values.
Shapley values.
The Shapley value (Shapley, 1953) in game theory was widely considered as a fair distribution of the overall reward in a game to each player (Weber, 1988).
(Sen et al., 1981) and (Gr¨omping, 2007) used the Shapley value to attribute the correlation coefficient of a linear regression to input features.
(ˇStrumbelj et al., 2009; ˇStrumbelj and Kononenko, 2014) used the
Shapley value to attribute the prediction of a model to input features.
(Bork et al., 2004) used the
Shapley value to measure importances of protein interactions in large, complex biological interaction networks.
(Keinan et al., 2004) employed the Shapley value to measure causal effects in neurophysical models.
(Sundararajan et al., 2017) proposed Integrated Gradients based on the AumannShapley(Aumann and Shapley, 2015) cost-sharing technique.
Besides above local explanations, (Covert et al., 2020b) focused on the global interpretability.
In order to compute the Shapley value in deep models efficiently, (Lundberg and Lee, 2017) proposed various approximations for Shapley valus in DNNs.
(Lundberg et al., 2018) further computed the Shapley value on tree emsembles.
(Aas et al., 2021) generalized the approximation method in (Lundberg and Lee, 2017) to the case when features were related to each other.
(Ancona et al., 2019) further formulated a polynomial-time approximation of Shapley values for DNNs.
Baseline values.
In terms of baseline values of Shapley values, most studies (Covert et al., 2020a;
Merrick and Taly, 2020; Sundararajan and Najmi, 2020; Kumar et al., 2020) compared influences of baseline values on explanations, without providing any principles for setting baseline values.
Shrikumar et al.
(2017) proposed DeepLIFT to estimate attributions of input variables, and also mentioned the choice of baseline values.
Besides, Agarwal and Nguyen (2021) and Frye et al.
(2021) used generative models to alleviate the out-of-distribution problem caused by baseline values.
Unlike previous studies, we rethink and formulate baseline values from the perspective of gametheoretic causality.
We define the absent state of input variables, and propose a method to learn optimal baseline values based on the number of causal patterns.
Using causality to explain neural networks .
Causality and causal learning have been used in deep models.
Hoyer et al.
(2008) and Pearl (2009) studied the causal structure between some observed variables in the data.
Kocaoglu et al.
(2018) built a generative model based on causality.
The causality has also been used to explain the relationship between inputs and intermediate features/outputs of DNNs (Alvarez-Melis and Jaakkola, 2017;
Chattopadhyay et al., 2019; Harradon et al., 2018).
In comparison, in this study, the causal graph is constructed to represent the inference logic in the
DNN.
Interactions.
Many people defined interactions between input variables or weights to explain different models from different perspectives.
(Sorokina et al., 2008) proposed an approach to detect interactions of input variables in an additive model.
(Tsang et al., 2018) measured interactions of weights in a DNN.
(Murdoch et al., 2018; Singh et al., 2018; Jin et al., 2020) used the contextual decomposition (CD) technique to extract variable interactions.
(Cui et al., 2020) proposed a non-parametric probabilistic method to measure interactions using a Bayesian neural network.
In game theory, (Grabisch and Roubens, 1999; Lundberg et al., 2018) proposed and used the Shapley interaction index based on Shapley values.
(Janizek et al., 2021) extended the Integrated Gradients method (Sundararajan et al., 2017) to explain pairwise feature interactions in DNNs.
(Sundararajan et al., 2020) defined the Shapley-Taylor index to measure interactions over binary features.
In this paper, we use the multi-variate interaction of input variables based on the Harsanyi dividend (Harsanyi, 1982).
Our interaction metric has strong connections to (Grabisch and Roubens, 1999), but represents elementary causal patterns in a more detailed manner, and satisfies the efficiency axiom.
B A XIOMS OF THE S HAPLEY VALUE
The Shapley value satisfies the following four axioms (Weber, 1988).
(1) Linearity axiom : If two games can be merged into a new game u ( x S )
= v ( x S ) + w ( x S ) , then Shapley values in the two old games also can be merged, i.e. ∀ i ∈ N , ϕ i,u
= ϕ
i,v + ϕ i,w .
Dummy axiom and nullity axiom :
The dummy player i is defined as a player without any interactions with other players, i.e. satisfying ∀ S ⊆ N \ { i } , v ( x S ∪{ i } ) = v ( x S )
+ v ( x { i } ) .
Then, the dummy player’s Shapley value is computed as ϕ i = v ( x { i } ) .
The null player i is defined as a player that satisfies ∀ S ⊆ N \ { i } , v ( x S ∪{ i } )
= v ( x S ) .
Then, the null player’s Shapley value is ϕ i = 0 .
(3) Symmetry axiom : If ∀ S ⊆ N \ { i, j } , v ( x S ∪{ i } )
= v ( x S ∪{ j } ) , then ϕ i = ϕ
j .
(4) Efficiency axiom :
The overall reward of the game is equal to the sum of Shapley values of all players,
i
e. v ( x N ) − v ( x ∅ ) =
(cid:80) i ∈
N ϕ i .
C E XTENDED DISCUSSIONS ABOUT THE H ARSANYI DIVIDEND
This section provides extended discussions about the causal effect based on the Harsanyi dividend (Harsanyi, 1982), and provides proofs of axioms and properties that the Harsanyi dividend satisfies.
Given a trained model and the input sample x with n variables N = { 1 , 2 , · · · , n } , let v ( x S ) denote the model output when only variables in S are given.
Then, v ( x N ) − v ( x ∅ ) represents the overall benefit owing to all input variables in x , w.r.t.
the model output without given any variables.
In an input sample, different input variables interact with each other for inference, instead of working individually.
The Harsanyi dividend I S can be considered to measure the benefit to the model output from the interaction between variables in the subset S ⊆ N .
Furthermore, the Harsanyi dividend ensures that the overall benefit v ( x N ) − v ( x ∅ ) can be decomposed into the sum of I S of different subsets { S | S ⊆ N, S ̸ = ∅} .
[Equation]
For example, let us consider the S 1 = { head } denote the bird head pattern in Figure 2 of the main paper.
Patches inside S 1 collaborate with each other to form the head pattern, and to contribute an benefit I S 1 for the model output.
Then, the overall benefit of all patches in the image v ( x N ) − v ( x ∅ ) can be represented as the sum of various patterns, such as the head pattern S 1 with its benefit I S 1 , the tail pattern S 2 with the benefit I S 2 , the torso pattern S 3 with the benefit I S 3 , etc.
The Harsanyi dividend I S can be understood as measuring the additional benefit from the collaboration of input variables in S , in comparison with the benefit when they work individually or form smaller patterns.
Specifically, let v ( x S ) − v ( x ∅ ) denote the overall benefit from all variables in S , then we remove the marginal benefits owing to collaborations of all smaller subsets L of variables in S , i.e. { I L | L ⊊ S, L ̸ = ∅} .
[Equation]
Axioms and properties of the Harsanyi dividend .
The Harsanyi dividend is a trustworthy metric for the interaction utility, because it has been proven to satisfy efficiency, linearity, dummy, symmetry, anonymity, recursive and interaction distribution axioms.
The Harsanyi dividend can also serve as an elementary explanation for existing game-theoretic interaction metrics (Grabisch and Roubens, 1999; Sundararajan et al., 2017).
C. P ROOF OF AXIOMS OF THE H ARSANYI DIVIDEND
Here, we provide details and proofs for the seven axioms that the Harsanyi dividend satisfies.
These axioms are also proved in (Ren et al., 2021a).
In the following part, we use v ( S )
= v ( x S ) to denote the model output given only variables in S for simplicity.
Before proving the axioms and properties of the Harsanyi dividend, we first prove the following lemma.
Lemma 1 ∀ S ⊆ N , if S ̸ = ∅ , (cid:80) L ⊆ S ( − 1) | S |−| L | = 0 ; otherwise, (cid:80) L ⊆ S ( − 1) | S |−| L | = 1 .
• Proof :
[Equation]
Axiom 1 (Linearity)
If we merge outputs of two models, u ( S ) = w ( S ) + v ( S ) , then, ∀ S ⊆ N , the interaction I uS w.r
t. the new output u can be decomposed into I uS =
I wS
+ I vS .
• Proof :
[Equation]
Axiom 2 (Dummy)
The dummy variable i ∈ N satisfies ∀ S
[Equation]
N \{ i } , v ( S ∪{ i } )
= v
S )
+ v ( { i } )
It means that the variable i has no interactions with other variables, i
e. ∀ S ⊆
N \ { i } , S ̸
= ∅ , I S ∪{ i } = 0 .
• Proof :
[Equation]
(cid:88)
L ⊆ S ( − 1) | S | +1 −| L | v ( L ) + (cid:88)
L ⊆ S ( − 1) | S |−| L | [ v ( L ) + v ( { i } )]
= (cid:88)
L ⊆ S ( − 1) | S | +1 −| L | [ − v ( L ) + v ( L )]
+ (cid:88)
L ⊆ S ( − 1) | S |−| L | v ( { i } )
=  (cid:88)
L ⊆ S ( − 1) | S |−| L | 
 v ( { i } )
= 0 % according to Lemma 1 and S ̸
= ∅
Axiom 3 (Symmetry)
If input variables i, j ∈ N have same cooperations with other variables
∀
S
⊆
N \{ i, j } , v ( S ∪ { i } )
= v ( S ∪ { j } ) , then they have same interactions with other variables, ∀ S ⊆
N \ { i, j } , I S
∪{ i } = I S ∪{ j } .
• Proof :
I S ∪{ i } =
(cid:88)
L ⊆ S ∪{ i } ( − 1) | S | +1 −| L | v ( L ) =
(cid:88)
L ⊆ S ( − 1) | S | +1 −| L | v ( L ) + (cid:88)
L ⊆ S ( − 1) | S |−| L | v ( L ∪ { i } ) = (cid:88)
L ⊆ S ( − 1) | S | +1 −| L | v ( L ) + (cid:88)
L ⊆ S ( − 1) | S |−| L | v ( L ∪ { j } ) = (cid:88)
L ⊆ S ∪{ j } ( − 1) | S | +1 −| L | v ( L ) =
I S ∪{ j }
Axiom 4 (Efficiency)
The output of a model can be decomposed into interactions of different subsets of variables, v ( N ) = v ( ∅ ) + (cid:80) S
[Table]
N,S
[Table]
= ∅ I S .
• Proof : right = v ( ∅
[Table]
+
[Table]
according to LemmaAxiom 5 (Anonymity) For any permutations π on N , we have ∀ S ⊆ N, I vS
=
I πvπS , where πS ≜ { π ( i ) | i ∈ S } , and the new model πv is defined by ( πv )( πS )
= v ( S ) .
• Proof:
[Equation]
Axiom 6 (Recursive)
The interaction benefit can be computed recursively.
For i ∈ N and S ⊆ N \{ i } , the interaction benefit of the pattern S
∪ { i is equal to the interaction benefit of
S with the presence of i minus the the interaction benefit of S with the absence of i , i
e. I S ∪{ i } = I S
| i is always present − I S
(s.t. S ⊆
N \{ i } ).
I S | i is always present denotes the interaction benefit when the variable i is always present as a constant context,
i
e. I S
| i is always present =
(cid:80) L ⊆ S
[Equation]
| · v
[Equation]
L
[Equation]
i
[Equation]
• Proof: I
[Equation]
Axiom 7 (Interaction distribution)
This axiom characterizes how interactions are distributed for “interaction functions” (Sundararajan et al., 2020).
An interaction function v T parameterized by a context T satisfies ∀ S : T ⊆ S ⊆ N, v T ( S )
=
c , and v T ( S )
=
if T ⊊ S .
Then, we have I T
=
c , and ∀ S ̸
= T , I S
= 0 .
• Proof: If S ⊊ T , we have
[Equation]
= 0 .
If S = T , we have
I S
= I
T = (cid:88)
L ⊆ T ( − 1) | T |−| L | v ( L ) = v ( T ) + (cid:88)
L ⊊ T ( − 1)
| T |−| L | v ( L ) (cid:124)(cid:123)(cid:122)(cid:125) =
= c.
If S ⊊ T , we have
[Equation]
C. P ROOF OF CONNECTIONS BETWEEN THE H ARSANYI DIVIDEND AND OTHER GAME - THEORETIC INTERACTION METRICS
In this section, we prove the relationship between the Harsanyi dividend and other game-theoretic interaction metrics, including the Shapley interaction index (Grabisch and Roubens, 1999) and the
Shapley Taylor interaction index (Sundararajan et al., 2020).
Such relationship has been proven in (Ren et al., 2021a).
Lemma 2 (Connection to the marginal benefit) ∆ v T ( S ) =
(cid:80) L ⊆ T ( − 1)
| T |−| L | v ( L ∪ S ) denotes the marginal benefit (Grabisch and Roubens, 1999) of variables in T ⊆
N \ S given the environment
S .
∆ v T ( S ) can be decomposed as ∆ v T ( S ) =
(cid:80) S ′ ⊆
S I T ∪ S ′ .
• Proof:
[Equation]
(cid:125)
[Equation]
  
 % according to Lemma= (cid:88)
S
[Equation]
I S
[Equation]
Property 1 (Connection to the Shapley interaction index)
Given a subset of input variables T ⊆ N , I Shapley T
= (cid:80) S ⊆
N \ T | S | !
( | N |−| S |−| T | )!
( | N |−| T |
[Equation]
∆ v T ( S ) denotes the Shapley interaction index (Grabisch and Roubens, 1999) of T .
The Shapley interaction index can be represented as I Shapley T
= (cid:80)
S
[Equation]
N
[Equation]
• Proof:
[Equation]
Then, we leverage the following properties of combinatorial numbers and the Beta function to simplify the term
[Equation]
(i)
A property of combinitorial numbers.
m · (cid:0) nm (cid:1) =
n · (cid:0) n − 1 m − 1 (cid:1) .
(ii)
The definition of the Beta function.
For p, q > 0 , the Beta function is defined as B ( p, q ) =
(cid:82)0 x p − 1 (1 − x ) 1 − q dx .
(iii) Connections between combinitorial numbers and the Beta function.
◦
When p, q ∈ Z + , we have B ( p, q ) =
[Equation]
◦
For
m, n ∈ Z + and n > m , we have
(cid:0) nm (cid:1)
=
[Equation]
Hence
[Equation]
we leverage the properties of combinitorial numbers and the Beta function to simplify w L .
[Equation]
(cid:32)
[Equation]
N | − | L | −
[Equation]
T
[Equation]
k (cid:33) · B (cid:16)
[Equation]
N | − | L
[Equation]
−
[Equation]
T
[Equation] [Equation]
| L
[Equation]
k (cid:17) · · · 1 ⃝ + | N |−| L |−| T | (cid:88) k
= k · (cid:32)
[Equation]
N | − | L | −
[Equation]
k (cid:33) · B (cid:16)
[Equation]
N
[Equation]
−
[Equation]
− |
[Equation]
+
[Equation]
|
[Equation]
+
[Equation]
(cid:17) · · · 2 ⃝
Then, we solve 1 ⃝ and 2 ⃝ respectively.
For 1 ⃝ , we have 1 ⃝ =
(cid:90)
[Equation] [Equation]
k
[Equation]
=
[Equation] [Equation]
= (cid:90)
[Equation]
|
[Equation]
|
[Equation]
L
[Equation]
1 dx =
For 2 ⃝ , we have 2 ⃝ = | N
[Equation]
k
[Equation] [Equation]
N
[Equation]
L |
[Equation]
| T | ) (cid:32)
[Equation]
N
[Equation]
L
[Equation]
k
[Equation]
B (cid:16)
[Equation]
N
[Equation]
− | L | − | T | − k + 1 , | L | + k (cid:17) =( | N | − | L | − | T | ) | N |−| L |−| T |− 1 (cid:88) k ′ =
(cid:32) | N | − | L | − | T | −k ′ (cid:33) · B (cid:16) | N | − | L | − | T | − k ′ , | L | + k ′ + 1 (cid:17) =( | N | − | L | − | T | ) (cid:90)
| N |−| L |−| T |− 1 (cid:88) k ′ =
(cid:32) | N | − | L | − | T | −k ′ (cid:33) · x | N |−| L |−| T |− k ′ − 1 · (1 − x ) | L | + k ′ dx =( | N | − | L | − | T | ) (cid:90)  
| N |−| L |−| T |− 1 (cid:88)
k ′ =
(cid:32) | N | − | L | − | T | −k ′ (cid:33) · x | N |−| L |−| T |− k ′ − 1 · (1 − x ) k ′ 
 (cid:124) (cid:123)(cid:122) (cid:125) = · (1 − x ) | L | dx =( | N | − | L | − | T | ) (cid:90)0 (1 − x ) | L | dx =
| N | − | L | − | T | | L | +
Hence, we have
[Equation]
Therefore, we proved that I Shapley T
[Equation]
Property 2 (Connection to the Shapley Taylor interaction index)
Given a subset of input variables T ⊆ N , let I Shapley-Taylor T denote the Shapley Taylor interaction index (Sundararajan et al., 2020) of order k for T .
The Shapley taylor interaction index can be represented as the weighted sum of interaction utilities, i.e. I Shapley-Taylor T
[Equation]
if | T | = k ; and I Shapley-Taylor T
[Equation]
0 if | T | > k .
• Proof
: By the definition
of
the Shapley
Taylor interaction index,
[Equation]
When | T | < k , by the definition of the interaction benefit in Equation (4) in the main paper, we have
[Equation]
When | T | =
k , we have
[Equation]
| N |
[Equation]
| S
[Equation]
[Equation] [Equation]
=
k | N | | N |− k
[Equation]
m =
[Equation]
S
[Equation]
N \
[Equation]
| S | = m
[Equation]
(cid:0) | N |−
[Equation]
| S | (cid:1)
[Equation]
( S ) =
k | N | | N |− k (cid:88)
[Equation]
(cid:88)
S ⊆ N \ T | S | = m 1 (cid:0) | N |− 1 | S | (cid:1)  (cid:88)
L
[Equation]
S
I L ∪ T 
 =
k | N | (cid:88)
L ⊆ N \ T | N |
[Equation]
k (cid:88) m =
| L | 1 (cid:0) | N |− 1 | S | (cid:1) (cid:88)
S ⊆ N \ T | S | = m S ⊇ L
I L ∪ T = k
| N | (cid:88)
L ⊆ N \ T | N |− k (cid:88) m =
| L | 1 (cid:0) | N |− 1 | S | (cid:1) (cid:32) | N | − | L | − k m − | L | (cid:33)
I L ∪ T =
[Equation]
(cid:88)
L ⊆ N \ T
I L
[Equation]
T | N |
[Equation]
| L |−
[Equation]
(cid:88) m =
[Equation]
(cid:0)
[Equation]
N |− 1 | L |
[Equation]
m (cid:1) (cid:32) |
[Equation]
| L | − k m (cid:33) (cid:124) (cid:123)(cid:122) (cid:125) w L
Just like the proof of Property 1, we leverage the properties of combinitorial numbers and the Beta function to simplify w L .
[Equation]
Then
[Equation]
we solve 1 ⃝ and 2 ⃝ respectively.
For 1 ⃝
[Equation]
we have
[Equation]
For 2 ⃝ , we have
[Equation]
Hence, we have w L = 1 ⃝ + 2 ⃝ = 1 (cid:0) | L | + k − 1 k − 1 (cid:1) + | N | − | L | − k
( | L | + 1) (cid:0) | L | + k k − 1 (cid:1) =
| L | ! · ( k − 1)!
( | L | + k − 1)!
+ | N | − | L | − k
| L | +· ( | L | + 1)!
· ( k − 1)!
( | L | + k )!
=
| L | ! · ( k − 1)!
( | L | + k − 1)!
+ | N | − | L | − k
| L | + k · | L | ! · ( k − 1)!
( | L | + k − 1)!
= (cid:20) 1 + | N | − | L | − k | L |
+ k (cid:21) · | L | ! · ( k − 1)!
( | L | + k − 1)!
=
| N | | L | + k · | L | !
· ( k − 1)!
( | L | + k − 1)!
=
| N | k · | L | !
· k ! ( | L | + k )!
=
| N | k · 1 (cid:0) | L | + k k (cid:1)
Therefore, we proved that when | T | =
k , I Shapley-Taylor T =
k | N | (cid:80) L ⊆ N \ T w L · I L ∪ T = k | N
[Equation]
| N | k ·
[Equation]
( | L | + k
[Equation]
(cid:1)
[Equation]
L
[Equation]
T .
D P ROOFS OF THEOREMS
This section provides proofs of theorems in the main paper.
D. P ROOF OF T HEOREM
Theorem 1 ( Faithfulness ) Let us consider a DNN v and a specific sample x with n input variables.
Given any of the 2 n arbitrarily masked input x S , the causal graph defined in Eq.
(4) can always well mimic the output of the DNN, i.e., ∀ S ⊆
N, Y ( x S )
= v ( x S ) , when we set the causal patterns in the causal graph are Ω = 2 N = { S | S ⊆ N } .
Proof: According to the SCM in Eq.
(4), we have Y ( x S ) =
(cid:80) S ′ ∈
Ω U S ′
· C S ′ ( x S ) =
(cid:80) S
′ ⊆
S U S ′ .
Therefore, the faithfulness requirement can be equivalently re-written as ∀ S ⊆ N , v ( x S ) =
(cid:80)
S ′ ⊆
S U S ′ .
According to the definition of the Harsanyi dividend, we have ∀ S ⊆ N , (cid:88)
[Equation]
(cid:88)
L
[Equation]
S ′ ( − 1) | S
[Equation]
|−| L | v ( x L ) =
(cid:88)
L ⊆ S (cid:88)
S ′ ⊆ S : S ′ ⊇ L ( − 1)
| S ′ |−| L | v ( x L ) =
(cid:88)
L ⊆ S | S | (cid:88) s ′ =
| L | (cid:88)
S ′⊆ S : S ⊇ L | S ′| = s ′ ( − 1) s ′
−| L | v ( x L ) = (cid:88)
L ⊆ S v ( x L ) | S |−| L | (cid:88) m = (cid:32) | S | − | L | m (cid:33) ( − 1) m
= v ( x S )
D. P ROOF OF T HEOREM
Theorem 2 Harsanyi dividends can be considered as causal patterns of the Shapley value.
[Equation]
In this way, the effect of an causal pattern consisting of m variables can be fairly assigned to the m variables.
This connection has been proved in (Harsanyi, 1982).
• Proof
: right = (cid:88)
S ⊆ N \{ i } | S | + 1 I S ∪{ i } =
(cid:88)
S ⊆ N \{ i } | S | + (cid:88)
L ⊆ S ( − 1) | S | +1 −| L | v ( L ) + (cid:88)
L ⊆ S ( − 1) | S |−| L | v ( L ∪ { i } ) 
 = (cid:88)
S ⊆ N \{ i } | S | +(cid:88)
L ⊆ S ( − 1) | S |−| L | [ v ( L ∪ { i } ) − v ( L )]
= (cid:88)
L ⊆ N \{ i } (cid:88)
K ⊆ N \ L \{ i } ( − 1) | K | | K | + | L | + 1 [ v ( L ∪ { i } ) − v ( L )]
% Let
[Table]
=
S \ L = (cid:88)
L ⊆ N \{ i }   n − 1 −| L | (cid:88) k
= ( − 1) k k + | L | +(cid:32) n − 1 − | L | k (cid:33) 
[ v ( L ∪ { i } ) − v ( L )]
% Let k = | K | = (cid:88)
L ⊆ N \{ i } | L | !
( n − 1 − | L | )!
n !
[ v ( L ∪ { i } ) − v ( L )] % by the property of combinitorial number = [Table]
i = left
D. P ROOF OF R EMARK 2, T HEOREM 3, AND T HEOREM
Remark 2 Let us consider a function with a single causal pattern f
( x S ) =
w S (cid:81) j ∈ S ( x j − δ j ) .
Accordingly, ground-truth baseline values of variables are obviously { δ j } , because setting any variable ∀ j ∈ S, x j = δ j will deactivate this pattern.
Given the correct baseline values b ∗ j = δ j , we can use a single causal pattern to regress f ( x S ) , i.e., I S = f
( x S ) , ∀ S ′
̸
= S, I S ′
= 0 .
Theorem 3 For the function f ( x S ) = w S (cid:81) j ∈ S ( x j − δ j ) , if we use m ′ incorrect baseline values { b ′ j | b ′ j ̸ = δ j } to replace correct ones to compute causal effects, then the function will be explained to contain at most 2 m ′ causal patterns. Theorem 4 If we use m ′ incorrect baseline values to compute causal effects in the function f ( x S )=
w S (cid:81) j ∈ S ( x j − δ j ) , a total of (cid:0) m ′ k
−| S | + m ′ (cid:1) causal patterns of the k -th order emerge, k ≥ | S |− m ′ .
A causal pattern of the k -th order means that this causal pattern represents the AND relationship between k variables.
• Theoretical proof: Without loss of generality, let us consider an input sample x , with ∀ j ∈ S, x j ̸ = δ j .
Based on the ground-truth baseline value { δ j } , we have (1) v ( x S ) =
f ( x S ) =
w S (cid:81) j ∈ S ( x j − δ j ) ̸ = 0 , (2) ∀ S ′ ⊊ S, v ( x S ′ )
=
w S (cid:81) j ∈ S ′ ( x j − δ j ) (cid:81) k ∈ S \ S ′ ( δ k − δ k ) = 0 ,
Accordingly, we have I S
=
(cid:80) S ′ ⊆ S ( − 1) | S |−| S ′
| v ( x S ′ )
= v ( x S ) ̸
= 0 .
For S ′ ⊊ S , we have I S ′
= (cid:80) L ⊆ S ′ ( − 1)
| S ′ |−| L | v ( x L ) =
(cid:80) L ⊆ S ′
= 0 .
(3) ∀ S ′ ̸
= S , let S ′
=
L ∪ M , where L ⊆ S and M ∩ S = ∅ .
Then, we have [Table]
[Table]
[Table]
I
[Table]
′ = (cid:88) T ⊆ S ′ ( − 1) | S ′ |−| T | v ( x T ) = (cid:88)
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
Therefore, there is only one causal pattern with non-zero effect I S .
In comparison, if we use m ′ incorrect baseline values { δ ′ j } , where (cid:80) j ∈ S 1 δ ′ j ̸ = δ j = m ′ , then the function will be explained to contain at most 2 m ′ causal patterns.
For the simplicity of notations, let
S = { 1 , 2 , ..., m } , and δ ′
= δ 1 + ϵ 1 , ..., δ ′ m ′ =
δ m ′ + ϵ m ′ , where ϵ 1 , ..., ϵ m ′ ̸
= 0 .
Let T = { 1 , 2 , . . .
, m ′ } .
In this case, we have (1) v ( x S ) =
f ( x S ) ̸ = 0 (2) ∀ S ′ ⊊ S, | S ′ | < m − m ′ , v ( x S ′ )
= w S (cid:81) j ∈ S ′ ( x j − δ j ) (cid:81) l ∈ S \ S ′ ( δ ′ l − δ l ) .
Because | S | − | S ′ | > m ′ , there is at least one variable with ground-truth baseline value in S \ S ′ .
Therefore, v ( x S ′ ) = 0 .
Furthermore, I S ′
=
(cid:80) L ⊆ S ′ ( − 1) | S ′ |−| L | v ( x L ) =(3) ∀ S ′ ⊊ S, | S ′
| =
k ≥ m − m ′ , v ( x S ′ )
=
w S (cid:81) j ∈ S ′ ( x j − δ j ) (cid:81) l ∈ S \ S ′ ( δ ′ l − δ l ) .
If S \ T ⊆ S ′ , then
S \ S ′ ⊆
T and v ( x S ′ )
̸
= 0 .
Otherwise, v ( x S ′ )
= 0 .
Then,
[Equation]
=
[Equation]
x
[Equation]
If the above I S ′ = 0 , it indicates that S \ T ⊈ S ′ .
In this case, there is no subset L ⊆ S ′
s.t.
S \ T ⊆ L .
In other words, only if S \ T ⊆ S ′ , I S ′ ̸ = 0 .
In this way, a total of (cid:0) m [Equation] m
[Equation]
causal patterns of the k -th order emerge, where the order k of a causal pattern means that this causal pattern S ′ contains k = | S ′ | variables.
There are totally (cid:80) mk = | S |− m ′ (cid:0)
[Equation]
causal patterns in x .
For example, if the input x is given as follows,
[Equation]
ϵ i , i ∈
S \ T = { m ′ + 1 , . . .
, m }
where ϵ i ̸ = 0 are arbitrary non-zero scalars.
In this case, we have ∀ S ′ ⊆
T, I S ′ ∪{ m ′
+1 ,...,m } =
ϵ 1 ϵ 2 ...
ϵ m ̸ = 0 .
Besides, if { m ′ + 1 , ..., m } ⊈ S ′ , we have I S ′
= 0 .
In this way, there are totally 2 m ′ causal patterns in x . • Experimental verification: We further conducted experiments to show that the incorrect setting of baseline values makes a model/function consisting of high-order causal patterns be mistakenly explained as a mixture of low-order and high-order causal patterns.
To show this phenomenon, we compare causal patterns computed using ground-truth baseline values and incorrect baseline values in Table 9, and the results verify our conclusion.
We find that when models/functions contain complex collaborations between multiple variables ( i
e. high-order causal patterns), incorrect baseline values usually generate fewer high-order causal patterns and more low-order causal patterns than ground-truth baseline values.
In other words, the model/function is explained as massive low-order causal patterns.
In comparison, ground-truth baseline values lead to sparse and high-order salient patterns.
E P ROVING THAT MASKING INPUT VARIABLES REMOVES CAUSAL EFFECTS
In this section, we prove that for the causal pattern S ∋ i , if the input variable i is masked, then the causal effect w S = 0 . Proof: let S = S ′ ∪ { i } . If i ∈ S is masked, then ∀ L s.t. i / ∈
L, x L = x L ∪{ i } .
Therefore, v ( L ∪ { i } ) = v ( L ) .
According to the definition of Harsanyi dividend (Harsanyi, 1982), we have
[Equation]
− 1) | S ′ | +1 −| L | v ( L ) + (cid:88)
L ⊆ S ′ ( − 1) | S ′ |−| L | v ( L ∪ { i } ) =
(cid:88)
L ⊆ S ′ ( − 1) | S ′ | +1 −| L | v ( L ) + (cid:88)
L ⊆ S ′ ( − 1) | S ′ |−| L | v ( L ) =
(cid:88)
L ⊆ S ′ (cid:16) ( − 1) | S ′ | +1 −| L | + ( − 1)
| S ′ |−| L | (cid:17) v ( L ) = (cid:88)
L ⊆ S ′ ( − 1 + 1)( − 1)
| S ′ |−| L | v ( L ) =
F M ORE EXPERIMENTAL DETAILS AND RESULTS F. T HE VISUALIZATION OF CAUSAL PATTERNS
This section provides the visualization of causal patterns.
Figure 4 visualizes the distribution of textual pixels j that forming causal patterns with a certain pixel i
In Figure 4, the pixel value in the heatmap is computed as p ( j | i ) =
E S ∈ Ω [ 1 j ∈ S ] , where i ∈ N and Ω denotes the set of patterns S whose values | ∆ v i ( S ) | are relatively large among all S ⊆ N \{ i } .
∆ v i ( S ) is defined in
Equation (11) of the main paper.
[Figure]
Due to the exponential computational cost of enumerating all causal patterns to obtain the salient ones in Ω , we used the following greedy strategy for approximation.
We first randomly sample an input variable i and a context S ⊆ N \ { i } , and compute the corresponding value of | ∆ v i ( S ) | .
We sample multiple times and find the pair of ( i, S ) that yields the largest value of | ∆ v i ( S ) | .
Then, we add/remove some variables to/from the context S to obtain a larger | ∆ v i ( S ) | .
Alg.
1 shows the pseudo-code of this algorithm.
F. V ERIFICATION OF THE SPARSITY OF THE CAUSAL GRAPH
In this subsection, we conducted experiments to verify the sparsity of causal effects, which is introduced in Remark 1.
To this end, we computed causal effects U S of all 2 n causal patterns encoded by a DNN.
Specifically, we trained a three-layer MLP on the income dataset and computed causal effects in the model.
Figure 5 shows the distribution of absolute causal effects | U S | of causal patterns in the first five samples of each category of the income dataset.
These results show that most causal patterns had insignificant causal effects, U S ≈ 0 .
Only a few causal patterns had salient causal effects.
Moreover, we also conducted experiments to demonstrate the universality of this phenomenon.
We trained the five-layer MLP, CNN, LSTM, ResNet-32, and VGG-16 on the UCI census income dataset, the UCI TV news channel commercial detection dataset, the SST-2 dataset, and the MNIST dataset, respectively.
Figure 6 shows the absolute causal effects U S in the descending order.
These results show that various DNNs learned on different tasks could be explained by a sparse causal graph.
F. V ERIFICATION OF USING CAUSAL PATTERNS TO EXAMINE THE STATE OF INPUT VARIABLES
In this subsection, we conducted experiments to verify that causal patterns reflect the states of removing existing patterns.
Given causal effects I S in the normal input image and causal effects [Figure] do do
end else
S ( k ) max ∪ { j } ) | end end end if | d ( k ) max − d ( k ) tmp | <
ϵ then
Break end end Ω ← Ω ∪ { S ( k ) max } 31 end 32 return i max , Ω index of causal patterns s t r e n g t h o f c a u s a e ff e c t s [Figure] [Figure]
[Figure]
[Figure]
I
[Figure] S in the white noise input, we compared their distributions in Figure 7. Note that we assumed that the white noise input naturally contained information for classification than the normal input image.
We found that most causal effects in the white noise input were close to zero [Figure] and there were few salient causal patterns.
Besides, we computed the average strength of causal effects in the above two inputs.
In the normal input, the average strength of causal effects E S ⊆ N | I S | = 5 . 5285 , while in the white noise input, the average strength was much smaller, E S ⊆
N | I ( noise ) S | = 0 . 2321 .
These results indicated that salient causal patterns could reflect the information encoded in the input.
F. E FFECTS OF THE PROPOSED METHOD ON MULTI - ORDER S HAPLEY VALUES AND MULTI - ORDER MARGINAL BENEFITS
In this section, we conducted experiments to verify that baseline values b learned by the two proposed loss functions in Eq.
(8) could effectively reduced causal effects of low-order causal patterns in Eq.
Besides, we also conducted experiments to show the effect of the proposed loss in Section 3 on multi-order Shapley values ϕ ( m ) i and multi-order marginal benefits ∆ v i ( S ) .
To this end, we computed the metric E x
[( E | S | = m | I S | ) / ( v ( x N ) − v ( x ∅ ))] to measure the relative strength of causal patterns of a specific order m , in order to evaluate the effectiveness of baseline values. Fig. 8(a) shows that compared to zero baseline values, our method effectively reduced low-order causal patterns.
In addition, Fig. 9 and Fig. 8(b) verify that the loss L Shapley in Eq. (8) reduced the number of salient causal patterns in Ω , which means L Shapley avoided the exponential number of causal patterns caused by incorrect baseline values. In order to examine the effect of the proposed loss in Section 3 on multi-order Shapley values ϕ ( m ) i and multi-order marginal benefits ∆ v i ( S ) , we compared multi-order Shaplely values ϕ ( m ) i w.r.t.
the function proposed in (Tsang et al., 2018) between the following three settings, (1) using zero baseline values, (2) using the baseline values learned with L Shapley , and (3) using the baseline values learned with L marginal , respectively.
Similarly, we also compared marginal benefits ∆ i v ( S ) w.r.t.
the function in (Tsang et al., 2018) between the same three settings of baseline values.
Figure 10 shows the ratio of multi-order Shapley values and multi-order marginal benefits.
Both L shapley and L marginal effectively reduced low-order Shapley values and low-order marginal benefits.
F . D ISCUSSION ABOUT THE SETTING OF GROUND - TRUTH BASELINE VALUES .
This section discusses the ground truth of baseline values of synthetic functions in Section 4.1 of the main paper.
In order to verify the correctness of the learned baseline values, we conducted experiments on synthetic functions with ground-truth baseline values.
We randomly generatedfunctions whose causal patterns and ground truth of baseline values could be easily determined.
As
[Figure]
[Figure]
[Table] Table 10 shows, the generated functions were composed of addition, subtraction, multiplication, exponentiation [Table] and sigmoid operations
[Table]
The ground truth of baseline values in these functions was determined based on causal patterns between input variables.
In order to represent absence states of variables, baseline values should activate as few salient patterns as possible, where activation states of causal patterns were considered as the most infrequent state.
Thus, we first identified the activation states of causal patterns of variables, and the ground-truth of baseline values were set as values that inactivated causal patterns under different masks.
We took the following examples to discuss the setting of ground-truth baseline values (in the following examples, ∀ i ∈ N, x i ∈ { 0 , 1 } and b ∗ i ∈ { 0 , 1 } )
• f ( x ) =
x 1 x 2 x 3 + sigmoid ( x 4 + x 5 − 0 .
Let us just focus on the term of x 1 x 2 x 3 in f ( x ) .
The activation state of
this causal
pattern is x 1 x 2 x 3 = 1 when ∀ i ∈ { 1 , 2 , 3 } , x i = 1 .
In order to inactivate the causal pattern, we set ∀ i ∈ { 1 , 2 , 3 } , b ∗ i = 0 .
• f ( x ) = − x 1 x 2 x 3 + ( x 4 + x 5 ) 3 + · · · .
Let us just focus on the term of − x 1 x 2 x 3 in f ( x ) .
The activation state of this causal pattern is − x 1 x 2 x 3 = − 1 when ∀ i ∈ { 1 , 2 , 3 } , x i = 1 .
In order to inactivate the causal pattern, we set ∀ i ∈ { 1 , 2 , 3 } , b ∗ i = 0 .
• f ( x ) =
( x 1 + x 2 − x 3 ) 3 + · · · .
Let us just focus on the term of ( x 1 + x 2 − x 3 ) 3 in f ( x ) .
The activation state of this causal pattern is ( x 1 + x 2 − x 3 ) 3 = 8 when x 1 = x 2 = 1 , x 3 = 0 .
In order to inactivate the causal pattern under different masks, we set b ∗ 1 =
b ∗ 2 = 0 , b ∗ 3 = 1 .
• f ( x ) = sigmoid (3 x 1 x 2 − 3 x 3 − 1 . 5) + · · · .
Let us just focus on the term of sigmoid (3 x 1 x 2 − 3 x 3 − 1 . 5) in f ( x ) .
In this case, x 1 , x 2 , x 3 form a salient causal pattern because sigmoid (3 x 1 x 2 − 3 x 3 − 1 . 5) > 0 .
5 only if x 1 = x 2 = 1 and x 3 = 0 .
Thus, in order to inactivate causal patterns, ground-truth baseline values are set to b ∗ 1 =
b ∗ 2 = 0 , b ∗ 3 = 1 .
Ground-truth baseline values of functions in (Tsang et al., 2018).
This section provides more details about ground-truth baseline values of functions proposed in (Tsang et al., 2018). We evalu-
[Table] ated the correctness of the learned baseline values using functions proposed in (Tsang et al., 2018).
Among all the 92 input variables in these functions, the ground truth of 61 variables could be determined and are reported in Table 11.
Note that some variables cannot be 0 or 1 ( e.g. x 8 cannot be zero in the first function), and we set ∀ i ∈ N, x i ∈ { 0 . 001 , 0 . 999 } for variables in these functions instead.
Similarly, we set the ground truth of baseline values ∀ i ∈ N, b ∗ i ∈ { 0 . 001 , 0 . 999 } .
Some variables did not collaborate/interact with other variables ( e.g. x 4 in the first function), thereby having no causal patterns.
We did not assign ground-truth baseline values for these individual variables, and these variables are not used for evaluation.
Some variables formed more than one causal pattern with other variables, and had different ground-truth baseline values w.r.t.
different patterns.
In this case, the collaboration between input variables was complex and hard to analyze, so we did not consider such input variables with conflicting patterns for evaluation, either.
F. D ISCUSSION ABOUT THE SETTING OF GROUND - TRUTH S HAPLEY VALUES .
This section discusses the ground truth of Shapley values in the extended Addition-Multiplication dataset (Zhang et al., 2021), which is used in Section 4.1 of the main paper.
In order to verify the correctness of the Shapley values obtained by the optimal baseline values in this paper, we conducted experiments on the extended Addition-Multiplication dataset (Zhang et al., 2021) with ground-truth Shapley values.
The Addition-Multiplication dataset in (Zhang et al., 2021) contained functions that only consisted of addition and multiplication operations.
For example, f ( x ) = x 1 x 2 + x 3 x 4 where each input variable x i ∈ { 0 , 1 } was a binary variable.
Given x = [1 , 1 , 1 , 1] , the function contained two salient causal patterns, i.e. { x 1 , x 2 } and { x 3 , x 4 } , and their benefits to the output were
I { x 1 ,x 2 } = I { x 3 ,x 4 } = 1 , respectively.
According to (Harsanyi, 1982), the Shapley value is a uniform distribution of attributions.
Therefore, the effect of an causal pattern was supposed to be uniformly assigned to variables in the pattern.
Thus, the ground-truth Shapley values of variables were ˆ ϕ
= ˆ ϕ
= 1 / 2 ,
and ˆ ϕ 3 = ˆ ϕ 4 = 1 / 2 .
However, if the input x = [1 , 0 , 1 , 1] , then the pattern { x 1 , x 2 } was deactivated and I { x 1 ,x 2 } = 0 .
In this case, ˆ ϕ 1 =
ˆ ϕ 2 =
while ˆ ϕ 3 = ˆ ϕ 4 = 1 / 2 .
According to the analysis in Appendix F.5, ground-truth baseline values in the AdditionMultiplication dataset were all zero.
Then our method is equivalent to the zero baseline values.
Therefore, in order to avoid all ground-truth baseline values being zero, we added the subtraction operation.
We also added a coefficient before each term in the function to boost the diversity of functions.
For example, f ( x ) = 3 .
2 x 1 x 2 + 1 .
5 x 3 ( x 4 − 1) .
This function also contained two causal patterns, but the ground-truth baseline values of variables were different from the aforementioned function.
Here, b ∗
= b ∗
= b ∗ 3 = 0 and b ∗ 4 = 1 .
Given the input x = [1 , 1 , 1 , 0] , all patterns were activated and f ( x ) =
I { x 1 ,x 2 } + I { x 3 ,x 4 } = 3 .
+( − 1 . 5) = 1 . 7 .
Ground-truth Shapley values of input variables were ˆ ϕ
= ˆ ϕ 2 = 3 . 2 /
and
ˆ ϕ 3 = ˆ ϕ 4 = − 1 .
However, for the input x = [1 , 1 , 1 , 1] , the pattern { x 3 , x 4 } was deactivated, thereby ˆ ϕ 3 =
ˆ ϕ 4 = 0 .
Note that the above function can also be considered to contain three patterns ( f ( x ) = 3 .
2 x 1 x 2 + 1 .
5 x 3 x 4 − 1 .
5 x 3 ).
According to Occam’s Razor, we follow the principle of the most simplified interaction to recognize causal patterns in the function, i.e. using the least number of causal patterns.
Thus, we consider the above function f ( x ) = 3 .
2 x 1 x 2 + 1 .
5 x 3 ( x 4 − 1) containing two salient causal patterns.
Based on the extended Addition-Multiplication dataset, we randomly generated an input
[Table] [Table] [Table] following the Bernoulli distribution, i.e. p ( x i = 1) = 0 .
Therefore, for the mean baseline, baseline values of different input variables were all 0 .
For the baseline value based on the marginal distribution, which was used in SHAP (Lundberg and Lee, 2017), p ( x ′ i ) ∼ Bernoulli (0 . 7) .
Then, we compared the accuracy of the computed Shapley values of input variables based on zero baseline values, mean baseline values, baseline values in SHAP, and the optimal baseline values defined in this paper, respectively.
The result in Table 4 of the main paper shows that the optimal baseline values correctly generated the ground-truth attributions/Shapley values of input variables.
F. M ORE EXAMPLES OF INCORRECT S HAPLEY VALUES
This section provides more examples of incorrect Shapley values computed by using other baseline values, as discussed in Section 4.1 of the main paper.
Table 12 and Table 13 show two examples of incorrect Shapley values computed by using other baseline values.
For the example in Table 12, Shapley values of all input variables should be the same because the output is there production, but the baseline value used in SHAP generated different
Shapley values for these variables.
For the example in Table 13, Shapley value of the variable x 8 should be zero because − 4 .
52 x 8 = 0 .
However, mean baseline values and baseline values in
SHAP generated non-zero and large Shapley values for this variable.
These results demonstrate the unreliability of other baseline values.
F. O THER POTENTIAL SETTINGS OF v ( x S ) .
In the computation of Shapley values, people usually use different settings of v ( x S ) , although the settings of v ( x S ) do not affect the applicability of our method.
Our method of formulating baseline values is applicable to various settings of v ( x S ) .
Lundberg and Lee (2017) directly set v ( x S )
=
p ( y truth | x S ) .
Covert et al.
(2020b) used the cross-entropy loss as v ( x S ) .
In this paper, we use v ( x S )
=
log p ( y truth | x S ) 1 − p ( y truth | x S ) in L Shapley .
Besides, we use | ∆ v i ( S )
| =
∥ h ( x S ∪{ i } ) − h ( x S ) ∥in
L marginal on the MNIST dataset to boost the optimization efficiency, where h ( x S ) denotes the intermediate-layer feature.
It is because h ( x S ∪{ i } ) makes the optimization of L marginal receive gradients from all dimensions of the feature.
F. S TABILITY OF THE LEARNING OF BASELINE VALUES
In this section, we conducted experiments to verify the stability of the learning of baseline values.
We repeated experiments in Section 4.1, which verified the correctness of the learned baseline values, for three times with different random seeds.
Then, we computed the mean and standard deviation of the accuracy of the learned baseline values.
Table 14 shows that mean accuracy was close to 100% and the standard deviation was relatively small.
Therefore, the learning of baseline values was relatively stable. [Table]
[Figure]
F. S TABILITY OF THE ESTIMATED S HAPLEY VALUES
In this section, we provide more details and discussions about the instability of the estimated Shapley values. Given a specific setting of baseline values, we used the sampling-based approximation (Castro et al., 2009), which provided an unbiased estimation of Shapley values.
Besides, for the varying baseline values, we used the code released by (Lundberg and Lee, 2017) and (Covert et al., 2020b).
We conducted experiments to examine how to set the sampling number in (Castro et al., 2009) to ensure stable estimation of Shapely values.
Given a certain input sample, we fixed the sampling number and repeatedly computed the Shapley value multiple times.
Then, we measured the instability of the estimated Shapley value ϕ
i for a certain input variable i , which indicated that whether we could obtain similar Shapley values considering the randomness in each sampling process.
The instability was computed as E u,v [Figure] u ̸ = v | ϕ [Figure]
u [Figure] i − ϕ [Figure]
| [Figure] where ϕ ( u ) i denote the estimated Shapley value in the u -th time. Then, we computed the average instability value over Shapley values of all variables in 20 input samples.
Figure 11 shows that that when the sampling number was larger than 1000, the instability of the estimated Shapley values was usually lower than 0.1 for all baseline values.
This meant that the computed Shapley values were stable for all baseline values.
Therefore, we set the sampling number to 1000 in all other experiments in this paper.
F. E XPERIMENTAL RESULTS ON THE MNIST AND THE CIFAR-10 DATASETS
Experimental results on the MNIST datasets.
Experimental settings on the MNIST dataset have been introduced in Section 4.2 of the paper.
Figure 12 shows the learned baseline values on the
MNIST dataset by using L Shapley and L marginal in Section 3 of the paper, respectively.
Figure 13 shows the computed Shapley values using different baseline values on the MNIST dataset.
Compared to zero/mean/blurring baseline values, the learned baseline values by our method removed noisy variables on the background, which were far from digits in images.
Compared to SHAP, our method yielded more informative attributions.
Shapley values computed using baseline values in SAGE were dotted.
In comparison, our method generated smoother attributions.
[Figure] [Figure] [Figure] Experimental results on the CIFAR-10 datasets.
First, let us clarify the experiment settings on the CIFAR-10 dataset.
We split each image into 8 × 8 grids.
Each grid contained 4 × 4 pixels, which shared a specific RGB color as their baseline value.
Thus, we needed to learn a total of 8 × 8 × 3 color values as baseline values.
We learned baseline values by using the loss L Shapley .
Then, we computed
Shapley values for grids by using zero baseline values and computed Shapley values by using the learned baseline values for comparison.
Figure 14 shows that Shapley values computed with the learned baseline values mainly focused on objects in images.
In comparison, Shapley values based on zero baselines values mainly focused on the background.
F. E XPERIMENTAL RESULTS ON THE UCI S OUTH G ERMAN C REDIT DATASET .
This section provides experimental results on the UCI South German Credit dataset (Dua and Graff, 2017).
Based on the UCI datasets, we learned MLPs following settings in (Guidotti et al., 2018).
Figure 15 shows the learned baseline values by our method, and Figure 16 compares Shapley values computed using different baseline values.
Just like results on the UCI Census Income dataset, attributions (Shapley values) generated by our learned baseline values are similar to results of the varying baseline values in SHAP and SAGE.
However, the zero/mean baseline values usually generated conflicting results with all other methods.
Besides, we also noticed that baseline values learned with different initialization settings ( zero-init and mean-init ) all converged to similar baseline values, except for very few dimensions having multiple local-minimum solutions, which proved the stability of our method.
More specifically, an input variable might have different optimal baseline values in real applications from different perspectives.
For example, in the function of y =
( a − 0 . 5)( b − 3)( c +2)+( a +0 . 5)( d − 3)( e +2)+ f , the variable a had two equivalent baseline value ± 0 .
F. D ETAILS ABOUT THE QUANTITATIVE EVALUATION OF ATTRIBUTIONS FOR IMAGE CLASSIFICATION
This section introduces experimental details about the quantitative evaluation of attributions on the
MNIST dataset.
We constructed a decision tree (Song et al., 2013) for each category in the MNIST dataset.
Specifically, for each category (digit), we first computed the average image over all training [Figure] Using   (zero-init) Using   (mean-init)
Using   (zero-init)
Using   (mean-init) Baseline values learned by our methods [Figure] [Figure]
samples in this category.
Let ¯ x ( c ) ∈ R n denote the average image of the c -th category.
Then, we built a decision tree by considering each pixel as an internal node.
The splitting rule for the decision tree was designed as follows.
Given an input x in the category c , the splitting criterion at the pixel (node) x i was designed as (cid:16) (¯ x ( c ) i > 0 . 5)&( x i > 0 . 5) (cid:17) 2 .
If (¯ x ( c ) i > 0 . 5)&( x i > 0 . 5) =
True, then the pixel value x i was added to the output; otherwise, x i was ignored.
In this way, the output of the decision tree was f ( x ) =
(cid:80) i ∈
V x i , where V =
{ i ∈ N | (¯ x ( c ) i > 0 . 5)&( x i > 0 . 5) =
True } denoted the set of all pixels that satisfied the above equation.
For inference, the probability of x belonging to the category c was p ( c | x ) = sigmoid ( γ ( f ( x ) − β )) , where γ = 40 was a constant and β ∝ (cid:80) i ∈ N 1 ¯ x ( c ) i > 0 .
In this case, we defined v ( x N ) =
log p ( c | x ) 1 − p
( c | x ) .
Thus, the co-appearing of pixels in V formed a causal pattern to contribute for v ( x N ) .
In other words, because ∀ i ∈ N, x i ≥ 0 , the absence of any pixel in V might deactivate this pattern by leading to a small probability p ( c | x ) < 0 .
5 and a small v .
This pattern can also be understood as an AND node in the And-Or decision tree (Song et al., 2013).
In the above decision tree, the ground-truth Shapley values of input variables (pixels) were easy to determine.
The above decision tree ensured that the absence of any variable in V would deactivate the causal pattern.
Therefore, according to Theorem 2 in the paper, the output probability should be fairly assigned to pixels in V , i.e. , they shared the same Shapley values ˆ ϕ i
= v ( x N ) | V | .
For other pixels that were not contained in the output, their ground-truth Shapley values were zero.
We estimated Shapley values of input variables in the above decision tree by using zero baseline values, mean baseline values, baseline values in SHAP, and the learned baseline values by our method, respectively.
Let ϕ i denote the estimated Shapley value of the variable i .
If | ϕ i − ˆ ϕ i | ≤ 0 .
01 , we considered the estimated Shapley value ϕ
i correct; otherwise, incorrect.
In this way, we computed [Table] the accuracy of the estimated Shapley values, and Table 15 shows that out method achieved the highest accuracy.
F. D ISCUSSION ABOUT THE BASELINE VALUES LEARNED ON ADVERSARIAL EXAMPLES .
In Section 4.3 of the main paper, we show that the learned baseline values can recover original samples form adversarial examples.
This section provides more discussions about this experiment.
Let x denote the normal sample, and let x adv = x + δ denote the adversarial example generated by adversarial attacks (Madry et al., 2018).
A previous study (Ren et al., 2021b) defined the bivariate interaction with different contextual complexities, and found that adversarial attacks mainly created out-of-distribution bivariate interactions with large contexts.
In the scenario of this study, we can consider such sensitive interactions with large contexts related to high-order multi-variate causal patterns.
It is because the bivariate causal between ( i, j ) with large contexts ( i.e. under many contextual variables S ) actually considers the collaboration between i, j and contextual variables in
S .
Thus, it also partially reflects I S ∪{ i,j } , which is defined in this paper.
Therefore, from the perspective of multi-variate causal patterns, the adversarial utility can be considered as introducing out-of-distribution high-order causal patterns in the model.
To this end, setting input variables to baseline values is supposed to remove related causal patterns to represent absence states.
Particularly, if we set all input variables to their baseline values, many causal patterns will be eliminated.
Thus, if we initialize baseline values as the adversarial example, and optimize baseline values using our method, the learned baseline values are supposed to remove OOD high-order causal patterns in the adversarial example, and recover the original sample.
F. C OMPUTATIONAL COMPLEXITY OF THE PROPOSED LOSS FUNCTIONS
In this section, we will introduce both the theoretical complexity of the loss function and the real complexity of approximating the loss in real applications.
In terms of the theoretical complexity, the complexity of the loss functions L Shapley and L marginal in Eq.
(8) is Dn · (cid:80) λm
=0 (cid:0) nm (cid:1) , where D denotes the number of samples x in the dataset.
n is the number of input variables in the input.
(cid:80) λm
=0 (cid:0) nm (cid:1) denotes the computational complexity of all terms ϕ ( m ) i ( x | b ) of different orders m ≤ λ .
In terms of real implementation, as mentioned in Section 4.2, we used the sampling-based approximation (Castro et al., 2009) to compute the loss function.
We randomly sampled K input variables in each epoch, and we randomly sampled T contexts of each order m w.r.t.
each variable i to approximate ϕ
( m ) i ( x | b ) .
Thus, the real computational cost of L Shapley and L marginal was reduced to
DK · (cid:80) λm
=0 T , where we set T = 100 in implementation.
We conducted an experiment to show the time cost of learning the optimal baseline values.
We used the MLP, LeNet, and ResNet-20 trained on the income dataset, the credit dataset, the MNIST dataset, and the CIFAR-10 dataset.
For the MNIST and the CIFAR-10 datasets, we randomly sampled
K = 100 pixels as the variable set N to compute the Loss L Shapley and optimize their baseline values.¡/font¿
The following table reports the time cost of optimizing L Shapley for an epoch w.r.t.
DNNs trained on different datasets.
The time cost was measured by using PyTorch 1.10 on Ubuntu 20.04, with the Intel(R) Core(TM) i9-10900X CPU @ 3.70GHz and one NVIDIA Geforce RTX 3090 GPU.
[Table] G M ULTI - ORDER S HAPLEY VALUES AND MARGINAL BENEFITS In Section 3 of the main paper, we propose the Shapley values of different orders ϕ ( m ) i , and marginal benefits of different orders ∆ v i ( S ) .
Furthermore, we find that high-order causal effects are only contained by high-order Shapley values and marginal benefits.
Actually, the loss function on marginal benefits L marginal is more fine-grained than the loss function on the multi-order Shapley value L Shapley .
This section provides proofs for the above claims.
First, the Shapley value ϕ i can be decomposed into the sum of Shapley values of different orders ϕ ( m ) i and marginal benefits of different orders ∆ v i ( S ) , as follows.
[Equation]
[Equation]
where the Shapley value of m -order ϕ ( m ) i def =
E
[Equation]
S |
[Equation]
v ( x S
[Equation]
{ i } ) − v ( x S ) (cid:3) , and the marginal benefit ∆ v i ( S ) def = v ( x S ∪{ i } ) − v ( x S ) .
• Proof :
[Equation]
n − 1 − | S | !) n ! (cid:2) v ( x S ∪{ i } ) − v ( x S ) (cid:3) = n − 1 (cid:88) m = (cid:88)
S ⊆ N, | S | = m | S | ! ( n − 1 − | S | !) n ! (cid:2) v ( x S ∪{ i } ) − v ( x S ) (cid:3) = 1 n n − 1 (cid:88) m
= (cid:88) S ⊆ N, | S | = m | S | ! ( n − 1 − | S | !) ( n − 1)!
(cid:2) v ( x S ∪{ i } ) − v ( x S ) (cid:3) = 1 n n − 1 (cid:88) m = E S ⊆ N, | S | = m (cid:2) v ( x S ∪{ i } ) − v ( x S ) (cid:3) = 1 n n − 1 (cid:88) m
= ϕ ( m ) i = 1 n n − 1 (cid:88) m = E S ⊆ N \{ i } , | S | = m ∆ v i ( S )
Connection between multi-variate interactions and multi-order marginal benefits.
The morder marginal benefit can be decomposed as the sum of multi-variate interaction benefits, as follows.
Therefore, high-order causal effects I S are only contained in high-order marginal benefits.
∆ v i ( S ) = (cid:88) L ⊆ S I L ∪{ i } (13)
Proof : right = (cid:88)
L ⊆ S
I L ∪{ i } = (cid:88)
L ⊆ S   (cid:88)
K ⊆ L ( − 1)
| L | +1 −| K | v ( x K ) + (cid:88)
K ⊆ L ( − 1) | L |−| K | v ( x K ∪{ i } ) 
 = (cid:88)
L ⊆ S (cid:88)
K [Equation]
= (cid:88)
L ⊆ S (cid:88)
K ⊆ L ( − 1) | L |−| K | ∆ v i ( K ) = (cid:88)
K ⊆ S (cid:88)
P ⊆ S \ K ( − 1) | P | ∆ v i ( K ) % Let P =
L \ K = (cid:88)
K ⊆ S   | S |−| K | (cid:88)
p = (cid:32) | S | − | K | p (cid:33) ( − 1)
p 
 ∆ v i ( K ) % Let p = | P | = (cid:88)
K ⊆ S (cid:104) (1 + ( − 1) | S |−| K | ) (cid:105) ∆ v i ( K ) % Let p = | P | = (cid:88)
K ⊊ S 0 · ∆ v i ( K ) + (cid:88)
K = S   | S |−| K | (cid:88)
p = (cid:32) | S | − | K | p (cid:33) ( − 1)
p 

∆ v i ( K ) =
∆ v i ( S ) = left
Connection between multi-order interactions and multi-order Shapley values.
The m -order
Shapley value can also be decomposed as the sum of interaction benefits, as follows.
Therefore, high-order causal effects I S are only contained in high-order Shapley values.
[Equation]
Proof :
[Equation]
= m  (cid:88)
L ⊆ S
I L ∪{ i } 
