E XPLORING UNFAIRNESS IN I NTEGRATED G RADIENTS BASED ATTRIBUTION METHODS
Anonymous authors
Paper under double-blind review
A BSTRACT
Numerous methods have attempted to explain and interpret predictions made by machine learning models in terms of their inputs.
Known as “attribution methods” they notably include the Integrated Gradients method and its variants.
These are based upon the theory of Shapley Values, a rigorous method of fair allocation according to mathematical axioms.
Integrated Gradients has axioms derived from this heritage with the implication of a similar rigorous, intuitive notion of fairness.
We explore the difference between Integrated Gradients and more direct expressions of Shapley Values in deep learning and ﬁnd Integrated
Gradients’ guarantees of fairness weaker; in certain conditions it can give wholly unrepresentative results.
Integrated Gradients requires a choice of “baseline”, a hyperparameter that represents the ‘zero attribution’ case.
Research has shown that baseline choice critically affects attribution quality, and increasingly effective baselines have been developed.
Using purpose-designed scenarios we identify sources of inaccuracy both from speciﬁc baselines and inherent to the method itself, sensitive to input distribution and loss landscape.
Failure modes are identiﬁed for baselines including Zero, Mean, Additive Gaussian Noise, and the state of the art
Expected Gradients.
We develop a new method, Integrated Certainty Gradients, that we show avoids the failures in these challenging scenarios.
By augmenting the input space with “certainty” information, and training with random degradation of input features, the model learns to predict with varying amounts of incomplete information, supporting a zero-information case which becomes a natural baseline.
We identify the axiomatic origin of unfairness in Integrated Gradients, which has been overlooked in past research.
I NTRODUCTION
Attribution, the identiﬁcation of the input features most salient to a model prediction, is an increasingly important requirement for neural networks.
It is a core part of model interpretability, which is valuable as a research tool and design aid, but also increasingly as an output requirement in its own right for uses as diverse as medical imaging (Singh et al., 2020) to loan applications (Bhatt et al., 2020).
A “right to explanation” of machine decisions is even provided in the European Union’s General Data
Protection Regulation (Goodman & Flaxman, 2017).
Gradients of predictions with respect to model inputs can be calculated using backpropagation and have been used for feature attribution (“Vanilla Gradient”)
(Erhan et al., 2009; Simonyan et al., 2014;
Yosinski et al., 2015).
The gradients indicate which features are most sensitive to a perturbation, causing the largest change in prediction outcome.
Empirically, this method often does highlight areas relevant to prediction, but it may also miss important areas due to the nonlinearity of the model and saturation of the gradients (Sundararajan et al., 2016).
The method of Integrated Gradients (Sundararajan et al., 2016; 2017) overcomes these issues by generating attributions based on the theory of Shapley values (Sundararajan et al., 2017).
The method integrates the gradients as the input varies linearly between a baseline and the ﬁnal input of interest:
A xx
B xx
C xx
D xx
E xx
F xx
G xx
Failure modes: B-D are single static baselines so exhibit baseline blindness poor baselines
AGN fails the Local Minimum scenario (section
Burnt Snacks
[Equation]
(1) where F ( · ) denotes the prediction of the model, x is the input vector of interest, x (cid:48) is the baseline input vector, and i indexes to the feature of interest.
A numerical approximation is normally used to calculate the integral in practice.
The baseline represents a missing or neutral input, a concept required by the Shapley values theory, but shared with several other attribution methods (Covert et al., 2020).
In the remainder of this paper, we ﬁrst give an overview of baselines discussed in the literature (Section 2).
Then we discuss the theoretical cause of a failure mode of IG ( attribution transfer ) using a zero baseline example (Section 3).
In Section 4 we introduce a new method, Integrated
Certainty Gradients .
Finally we show in experiments that other baselines including the state of the art Expected Gradients do not prevent attribution transfer, while in the tested scenarios Integrated
Certainty Gradients does, and discuss additional failure cases of speciﬁc baselines (Section 5).
Note on notation It will be convenient to imply an equivalence between attribution methods which take a baseline, ϕ ( F, x (cid:48) , x , i ) , and those which do not ϕ
( F, x , i ) .
A baseline free method can be adapted to take a baseline by ϕ B ( F, x (cid:48) , x , i ) =
ϕ ( F, x , i ) − ϕ
( F, x (cid:48) , i ) .
C HOICE OF BASELINE
Baseline choice has a major effect on attribution outcome.
Effective choice of baselines in practice is a topic of ongoing research.
A recent review of four state-of-the-art models for tabular data did not ﬁnd a best performing choice (Haug et al., 2021).
As this paper noted, the concept of missingness within an arbitrary space is domain-speciﬁc.
Therefore applying attribution techniques to a new domain requires this concept to be determined.
Another review came to similar conclusions for image classiﬁcation attribution (Sturmfels et al., 2020).
Baselines can be generalized from single inputs to distributions by taking the expected value of
Integrated Gradients using each input in the distribution.
In practice X (cid:48) is often from an empirical (discrete uniform) distribution so the expectation becomes simply the mean over X (cid:48) .
A summary of common baselines used in Integrated Gradients and other attribution methods follows.
For a more comprehensive review, one can refer to Haug et al.
; Sturmfels et al.
Visualizations of integration paths for various baselines are shown in Figure 1.
Constant value baselines include the uniform black/zero baseline proposed with the original method (Sundararajan et al., 2016) and choices that attempt to be unbiased with respect to the input distribution, including the component-wise mean of the input distribution, the middle of the input range, and uniform noise.
If the input matches the baseline, the x i − x (cid:48)
i term in Equationdisappears, any such areas will not be attributed, even when they are essential to the model’s prediction (“baseline blindness”)
(Sturmfels et al., 2020).
When the baseline is constant this undesirable behavior cannot be avoided.
This behavior can be seen clearly in attributions of MNIST images with a zero baseline, for example in Adebayo et al.
Distribution baselines avoid baseline blindness by combining multiple attributions.
Uniform noise has been used in this way, although concerns have been identiﬁed about the risk of interaction between the baseline and high frequency features (Sundararajan et al., 2017; Sturmfels et al., 2020).
The recently developed Expected Gradients method (Erion et al., 2020) uses the input (typically training) distribution as the baseline.
It computes the attribution for a given input example efﬁciently by sampling baselines randomly from the training distribution, and interpolation weights from the the [0, 1] range (Monte Carlo integration), rather than using the many equally spaced interpolations of the more common Riemann Sum method.
Dynamic baselines change based on the attribution input.
The maximum distance baseline uses the farthest point in the input space from the input vector (Sturmfels et al., 2020).
This avoids baseline blindness but abandons the concept of neutrality.
Sturmfels et al.
(2020) cite Fong & Vedaldi (2017) as the inspiration for blurring as a possible method of feature ablation to produce a baseline.
Gaussian noise may be added to the input vector during attribution and when averaged over several iterations, the resulting attributions were found to be more detailed and less noisy (Smilkov et al., 2017).
Adding noise to input vectors during training was also found to give a similar and cumulative effect.
The combination of input vector with noise can be used by itself as a dynamic distribution baseline (Sturmfels et al., 2020) or as an augmentation technique in combination with another baseline.
The justiﬁcation of this method for use with attribution appears to be mainly empirical, and a question regarding the theoretical justiﬁcation has been raised (Sturmfels et al., 2020).
A TTRIBUTION FAIRNESS
Assessment of attribution methods is not currently straightforward because there is not yet a uniﬁed general procedure to do so.
Rather, there are a number of tests for various indicative characteristics (Hooker et al., 2019; Adebayo et al., 2018; Yang & Kim, 2019).
We focus our attention on fairness: whether the relative attributions of input components are reasonable.
This is the central concern of
Shapley Values, a celebrated method from game theory for allocating value, with a strongly justiﬁed mathematical foundation (Shapley, 1952), and the background to the Integrated Gradients methods (Sundararajan et al., 2017).
Assessing fairness is not trivial.
As a reference we consider two other attribution methods deﬁned using the original Shapley Values procedure: BShap (Sundararajan & Najmi, 2020) and SHAP (Lundberg & Lee, 2017).
Shapley Values is a procedure that, for a given set S , requires a value for every possible subset of elements: v : P ( S )
→ R , where P is the power set of S , and gives back values for the elements themselves (the “Shapley values”): φ sv ( v, e ) .
The Shapley values sum to the value of the complete set: v ( S ) =
(cid:80) e ∈ S φ sv ( v, e ) , and certain axioms, widely considered as naturally related to fairness, are maintained.
As such it gives a way of sharing the value of the complete set fairly among the elements, based on the values that are achieved by each subset.
Model inputs cannot typically represent missing components, so to allocate attribution to the components of an input vector using Shapley Values, presence/absence of a component must be replaced by another binary property.
In the case of BShap, ‘present’ and ‘absent’ components are represented with values taken from the equivalent components of the input vector x or a baseline x (cid:48) respectively.
The value function v is simply the model’s prediction F .
SHAP represents subsets using random vectors, and the value function v by the expectation of the model’s prediction E ◦ F .
Each vector has a distribution f
( X | X i
= x i ∀ i ) where X is the distribution of possible inputs, x is the input being attributed, and i are the ‘present’ components of the vector.
Further information about Shapley Values and these methods is given is Section A.1 of the supporting material.
To summarize the relationships:
[Equation]
v, e ) v sets elements e
S {} ϕ BShap ( F, x (cid:48) , x , i )
F vectors components i x x (cid:48)
ϕ SHAP ( F, x , i )
E ◦ F random vectors components i a = x a ∼ X
3. A TTRIBUTION TRANSFER
Integrated Gradients integrates partial derivatives along a straight path between a baseline and the input vector.
This can be roughly imagined as accumulating the contours ∂f ( x ) of the input-output landscape that fall across the integration path.
To explore fairness in IG we consider the scenario shown in Figure 2.
The scenario is to classify whether either of two components x 1 , x 2 fall within
[Figure]
A
B x [Figure] x (cid:48)
A xxx (cid:48)
Subﬁgure
A xxx (cid:48)
Subﬁgure
[Equation]
the range 0.25 - 0.5, and the shown model gives a continuous approximation of this (transitioning linearly from 0.05 either side of the boundary).
The ﬁgure shows IG attribution paths for two inputs
A and B. Attribution results for these inputs are shown in Table 1.
The results for path A could be considered ‘maximally misleading’ as discussed in Figure 2.
The other methods in Table 1 are consistent with this.
To understand why IG gives this result, we compare the scenario with a case where there is a ground truth attribution.
For additive models f ( x ) = (cid:80) i f
i ( x i ) each component’s contribution can be considered independently: ϕ
( f, x (cid:48) , x , i )
=
f i ( x i ) − f i ( x (cid:48) i ) .
Linear models are a subset.
In this regime BShap and IG give matching results, as does SHAP if the baselines x (cid:48) were chosen such that f i ( x (cid:48) i ) =
E ( f i ) and the input distribution is uniform.
The model in Figure 2 deviates from additivity where the target regions of each component overlap.
By passing through this area the integration path can enter the target region of x 2 without crossing any ‘horizontal’ contours ( ∂f ( x ) ∂ x 2 > 0 ), yielding an unintuitive result.
We name this phenomenon “attribution transfer”: the change in attribution caused by the integration path crossing a region not representative of the wider landscape.
This is a precise concept when the model is mostly additive.
For general models, as it becomes harder to say what is “representative” this becomes less precisely deﬁned, although it is still useful for discussing the effect of speciﬁc perturbations in the landscape, for example.
Path B attributes an input vector with both components above the target range.
IG explains the zero result as the balance of two competing contributions, unlike BShap and Shap which consider the components to contribute equally.
Due to sharp features of the input-output landscape near the integration path, the IG attribution will undergo an abrupt reversal as the input crosses the x 1 = xboundary, even though neither the input nor baseline are near a decision boundary, nor in the target range.
IG attribution results can be arbitrarily affected by small sharp regions that fall across the path.
Is it reasonable to be concerned these effects may occur in practice?
We feel it is given that gradients are known to ﬂuctuate sharply (Smilkov et al., 2017), in addition to the results we present.
3. A XIOMATIC FOUNDATIONS
Like the other methods, IG is based on Shapley theory and has rigorous axioms.
It might then be surprising that IG permits ‘maximally misleading’ results.
IG is based on a method in the Shapley
Values lineage known as Aumann-Shapley (Aumann & Shapley, 1974), and has similar but not identical axioms.
The names of the axioms of IG and BShap include those of the original Shapley
Values: Efﬁciency , Linearity , Symmetry , and Dummy (Sundararajan & Najmi, 2020).
However, IG takes a more liberal generalization of the Dummy axiom.
For clarity we will give separate names to each variant:
Deﬁnition 1 (Weak Dummy) .
An attribution method ϕ satisﬁes Weak Dummy iff
[Equation]
This simply means that if the model does not use an input feature, it has zero attribution.
Deﬁnition 2 (Strong Dummy (BShap)) .
Let a be a vector with each component a k drawn from the corresponding component of either of two given vectors: x k or x (cid:48)
k , with a i
= x
i for a given i .
Let b
= a except that b i = x (cid:48)
i .
Then an attribution method ϕ satisﬁes
Strong Dummy iff f
( a ) − f ( b ) = 0 ∀ a ⇒ ϕ
( f, x (cid:48) , x , i ) =
Strong Dummy is identical to Weak Dummy except that only component values from the target and baseline vector are considered, not the whole range.
The weaker conditions make the axiom stronger.
Both IG and BShap are formulated to satisfy Weak Dummy, but BShap additionally possesses
Strong Dummy.
This precludes it from exhibiting attribution transfer and from producing unintuitive attributions of the kind given by IG in Table 1.
The situation indicated in Figure 2 Subﬁgure 1 is prohibited; any positively attributed component must be responsible for an increase in output value relative to another location in the input space (proof in Section A.2 of the supporting material).
Attribution transfer is not possible because it requires sensitivity to the region between the baseline and target value.
Conversely, Weak Dummy guarantees that a component does not participate in attribution transfer only if it has no inﬂuence on the model at all.
We feel a clear distinction between the axioms in the (original) discrete case and the continuous case may be helpful to avoid any misconception about the strength of the guarantees provided, which are considered extremely robust for Shapley Values in their original context.
I NTEGRATED C ERTAINTY G RADIENTS
We have shown that certain features of the input-output landscape can lead to unintuitive IG attributions.
To contrast this behavior in our experiments we introduce a new IG based method, “Integrated
Certainty Gradients”, which does not traverse the original input space, but rather the space of conditional probabilities, in a manner related to SHAP.
Sundararajan & Najmi (2020) note that SHAP (“CES” in their work) is not immediately applicable on ﬁnite continuous-valued distributions (such as a dataset) due to their feature sparsity.
Our method resolves this by having the model learn a generalization of the dataset.
As a Shapley Values method, direct calculation of SHAP is intractable for high dimensional data, and our method is an Aumann-Shapley alternative to SHAP in the same way IG is for BShap.
Our method adds additional information, the certainty , c , to the model input, so the model bases its prediction on the input and the certainty together: ( x , c ) .
The dimension of c is the same as the number of input features in x , and it takes values between [0 , 1] that indicate the amount of information the corresponding feature provides.
The per-pixel, per-input certainty c is drawn from a random distribution throughout the training of a model, and the model must be trained to associate low certainty values with a concept of missingness suitable for the Shapley theory.
This is done using a training process that applies a damage function D to the input images, so the model receives damaged inputs : ( D ( x , c ) , c ) .
A diagram showing the damage process is given in Figure 3.
First, damage data is required, of the same dimension as the undamaged input x .
This can take many forms, but for now we will use random noise where each component is i.i.d.
uniformly in [0 , 1] .
Then a damage mask is generated by treating each component
c i as the probability of keeping the
[Figure]
Certainty
[Figure]
copy
Network
Ordinary training
ICG training
Ordinary attribution (e.g. IG)
ICG attribution
Interpolations
Network
Query network for input gradients
Network
[Figure]
Interpolations corresponding feature x i .
If x i is not kept it is replaced the corresponding value from the damage data .
A method is needed to generate c during training.
Our method works reasonably well if each component c i (and thus the keep probability of each input feature) is i.i.d.
uniformly in [0 , 1] .
Regular (undamaged) inputs to the model correspond to using c i = 1 ∀ i .
The features D ( x , c ) i of an input prepared in this way have varying chances, equal to their certainties c
i , to provide information about the original input x , and therefore the class ˆ y .
During training the model learns to ignore features unrelated to ˆ y , discounting features with lower certainty, and develops the required uncertainty semantics.
If the model is trained well, it will ignore features with zero certainty, because these give no information about the original input:
[Equation]
In this case, c = 0 provides a natural baseline to represent missing data, as the model will ignore the input values entirely.
We complete the Integrated Certainty Gradients attribution method by combining this baseline with
IG.
The integral of Equation 1 is taken from c = 0 to c = 1 , holding x constant at the input we wish to attribute.
Note that x is not damaged while calculating the attribution.
It is convenient to note when implementing that the distance term from Equation 1 disappears because c i − c (cid:48)
i
= 1 .
Deep networks are capable of detecting and ignoring noise, so the damage data described above may not guarantee learning a correct interpretation of certainties.
To avoid this, samples from the input dataset can be used as damage data , ensuring certainties must be used to distinguish original and damaged features.
During ICG the model will encounter interpolations with total mean certainty from 0 to 1.
Most inputs generated with certainty sampled I.I.D. from U(0, 1) will have mean certainty
A
B
C
D (logit)
E
F
G
H (logit)
[Figure]
close to 0.5.
To ensure the model is trained on a wider range of certainty inputs we developed a parameterized distribution based on the Continuous Bernoulli (Loaiza-Ganem & Cunningham, 2019) where the mean can be varied betwen 0 and 1.
Full information about the training methods used is given in Section A.3.2 of the supporting material.
E XPERIMENTS 5. B URNT S NACKS SCENARIO
We use a purpose designed scenario to demonstrate attribution transfer of methods including Expected
Gradients.
The scenario is notionally similar to an Optical Sorting task in which overcooked food items are identiﬁed (Pedreschi et al., 2016).
The model must determine whether ‘overcooked’ (dark) squares (25% - 50% lightness) are present (class 1), or not (class 0), amidst light squares and other shapes on a dark background.
The input-output landscape for an input with two squares with lightnesses x 1 , x 2 is similar to Figure 2.
Two characteristics are designed to mislead Expected
Gradients.
First, the average pixel value is very dark.
This will tend to darken the features of an attributed image when it is interpolated.
Second, the light squares are lighter versions of the target objects (dark squares), but are not relevant to the prediction outcome.
This combination will cause light squares to often present as dark squares in interpolations, despite having no correlation with them, confusing Expected Gradients.
It is easy to justify that light squares should not be attributed, because their presence does not inﬂuence the class (relative to black background).
A selection of results on this scenario are shown in Figure 4.
We trained a certainty aware model to 98.7% accuracy (for undamaged inputs).
We attribute probabilities, in line with a prominent guide (Google, 2021), and also logits.
Zero and mean baseline Although a zero baseline seems appropriate, both zero and mean baselines markedly show attribution transfer.
Example G for the zero baseline case is equivalent to path B of
Figure 2.
There are two light squares, one with negative and one positive attribution.
By checking the interpolation gradients, we saw the positive attributed square enter the target region (at 25% lightness) ﬁrst, picking up attribution.
The second square followed it but gathered no attribution due to saturation of the output.
On leaving (at 50% lightness) the situation was reversed.
Attribution transfer also occurs in class 1 scenarios where a light square is present, for example Zero Baseline example
C.
This situation is equivalent to path A of Figure 2.
As a result, the process concludes with only the light square, irrelevant to the prediction, attributed.
Expected Gradients Results for Expected Gradients are shown in the 5th column of Figure 4.
In no case does EG identify the dark squares when they are present when using probabilities (class 1, images A-C).
Instead, it transfers attribution to the light squares, most strongly the lightest.
In case
A it appears no square is light enough to attract attribution, and artifacts from the random sampling process are shown.
Since class 0 is an absence class the expected outcome is not as clear.
Images E and F show diffuse artifacts from the interpolation process and arguably this is a fair result.
However image G strongly attributes a speciﬁc square and this seems more clearly misleading, as the square has no signiﬁcance to the classiﬁcation outcome.
Integrated Certainty Gradients ICG does not suffer from the previously described problems.
Because the integration is across the uncertainty space, the relative location of features in the input space and the input distribution are not important.
For class 0, ICG attributions are very diffuse and largely uniform, which could be considered an appropriate outcome, arguably the most intuitive of any method.
For class 1, it always attributes a dark square (correct target feature).
The attributions are strong and precisely located.
However, only one square is attributed even when multiple are present.
By inspecting the interpolations and using variations of the image with each square removed, we found that the model’s conﬁdence in each square transitions abruptly during the certainty increase.
Because of this, the ﬁrst square to gain the model’s conﬁdence causes the prediction to saturate, preventing it accruing to other squares.
This is attribution transfer in the certainty space.
This shows that though irrelevant objects are no longer brought into in the attribution, there can still be transfer between independent relevant objects.
We leave further consideration of this aspect to future research.
Additive Gaussian Noise Additive Gaussian Noise correctly attributes the location of the dark squares in class 1 images.
The attributions are noisy and not as precisely localised to the squares as those of ICG, however, all squares are attributed.
In class 0 attribution focuses around the triangles.
This is unsurprising because the addition of noise could easily make them resemble squares.
[Figure]
function f for the Local Minimum scenario.
The total score is the sum of the scores of the covariates
S ( x ) =
(cid:80) i f
( x i ) .
Component values range from 0 to 1.
The score is − 0 .
75 for component values 0.
and below, and between 0.5 andfor values above 0.5.
Logits
For most methods logits show intermixed opposite attributions on both light and dark squares.
Dark squares have overall more positive attribution, but nonetheless this presentation is imperfect and potentially confusing.
ICG performs excellently in this scenario, no longer missing squares, attributing all dark squares clearly.
5. L OCAL M INIMUM SCENARIO
Additive Gaussian Noise uses baselines drawn mainly from a region local to the input vector.
We introduce a scenario designed to challenge local baselines.
Input vectors are classiﬁed according to a total score S ( x ) , which is the sum of componentwise scores f ( x i ) .
Inputs are in class 1 if S ( x ) > 0 , classotherwise.
Note that although S is an additive function, the classiﬁer as a whole is not.
The component-wise score function f is shown in Figure 5.
f has a positive local minimum.
This local minimum makes a positive contribution to class 1 even though it is less contributing than nearby values.
Attribution methods based on local baselines would be expected to give the local minimum an incorrect negative attribution.
We trained a certainty-aware network on this classiﬁcation scenario to 93%
[Table]
accuracy.
Attribution results for this network using an example input vector from class 0 are shown in Table 2.
Component values of 0.50546 were used for the BShap baseline, as this was determined to be close to the model’s prediction boundary.
BShap, ICG and EG performed well in the scenario, with ICG and BShap in excellent agreement.
Only IG and BShap accurately represent the uniform negative region.
All other methods performed badly.
As expected, AGN incorrectly gives the local minimum a negative attribution.
Zero IG shows strong signs of attribution transfer.
Zero IG and AGN both exhibit blindness in the negative score region due to sharing this negligible gradient region with the baselines.
The results for mean IG vary widely.
The shown result is a randomly chosen example.
The distribution mean lies on the discontinuity of the score function (0.5) so small variations around it make major differences to the result.
C ONCLUSION
We have described a situation in which Integrated Gradients produces extremely unintuitive results, and explained how this arises from areas where the prediction function is non-additive (“attribution transfer”).
This behavior may be unexpected because IG is founded in the theory of Shapley Values, an axiomatic framework for fair allocation.
We explain this seeming contradiction by identifying a difference in the formulation of an axiom derived from the original theory.
In experiments we have shown that attribution transfer occurs even for state of the art methods like
Expected Gradients in the presence of a highly skewed but plausible data distribution.
Avoiding detrimental interactions by restricting IG to a small area of the input domain, as Additive Gaussian
Noise does, does not prevent misleading results when the local prediction landscape does not reﬂect the wider input space.
We show these failures of IG may be avoidable, by developing a method with a new kind of baseline,
Integrated Certainty Gradients, that requires a special training process to learn the data distribution.
This method is the Aumann-Shapley alternative to SHAP (as IG is for BShap) and by not traversing the input space it avoids the failure cases we demonstrate in other methods.
A CKNOWLEDGMENTS
LEFT
IN ANONYMOUS VERSION
E THICS S TATEMENT
Attribution is of key importance in many scenarios with ethical implications, for example providing reasons for life affecting decisions such as loan assessments, or determining whether decisions are made based on legitimate or protected personal characteristics.
As such, the development of fair attribution methods is ethically important and this paper has been produced with appreciation of that.
The work itself undertaken to produce this paper did not raise any ethical concerns
R EPRODUCABILITY S TATEMENT
Model architectures and training hyperparameters are given in Appendix A.5.
Anonymized source code is available at https://github.com/account48294/iclr
A A PPENDIX A. A DDITIONAL THEORY AND CALCULATIONS A.1. S HAPLEY VALUES
Integrated Gradients is a method axiomatically founded in the theory of Shapley Values (Sundararajan et al., 2017).
Shapley Values is a way to share responsibility between contributing factors when their combined outcome (expressed numerically) need not be equal to the sum of the outcomes each factor would give on their own.
As such it is a way of sharing the beneﬁt (or disadvantage) of cooperation.
To do this it requires counterfactual knowledge of the outcomes that would be achieved by each subset of the factors alone.
Formally: given a set S , and a value function v : P ( S ) → R , with P ( S ) the power set of S , there is a Shapley value φ sv ( v, e ) for each element e ∈ S .
Its signiﬁcance is that it is the unique method that, given this information, allocates responsibility according to the following axioms, intuitively associated with fairness:
Efﬁciency Also called Completeness.
The Shapley values allocated to each of the factors sum to the value of the combined outcome.
v ( S ) =
(cid:80)
[Equation]
Symmetry Factors which have the same effect have the same Shapley value.
Formally: if a pair of factors e 1 , e 2 ∈ S have v ( S (cid:48) ∪ e 1 )
= v ( S (cid:48) ∪ e 2 ) for every S (cid:48)
⊂ S not containing e 1 or e 2 , then φ sv ( v, e 1 ) =
φ sv ( v, e 2 ) .
Null player Also called Dummy (Strong Dummy).
If v ( S (cid:48) )
= v ( S (cid:48) ∪{ e } ) for all S (cid:48)
then the Shapley value of e is 0.
Linearity The Shapley Values method is a linear mapping of the value function: φ sv ( v 1 , e ) + φ sv ( v 2 , e ) =
φ sv ( v 1 + v 2 , e ) and φ sv ( av, e ) =
aφ sv ( v, e ) .
To calculate the Shapley values we deﬁne the marginal contribution of a factor e to a set S (cid:48) ⊂ S , e / ∈ S (cid:48) as v ( S (cid:48) ∪ { e } ) − v ( S (cid:48) ) .
Then we consider every possible ordering of the factors in S .
The
Shapley value of e is the mean of the marginal contributions it makes to the subset of elements that precede it in each ordering.
A.1. BS HAP , SHAP, AND I NTEGRATED G RADIENTS
Shapley Values assigns attribution to factors based on the differences in outcomes when they are absent.
To represent missing components
BShap uses the components of a baseline vector.
The normal Shapley Values procedure is then followed, representing subsets of components by vectors.
These vectors take values from the target vector x for components in the subset, and from the baseline x (cid:48) for the other components.
SHAP does not use a baseline.
Instead it takes the expectation of the model output over the missing components, conditioned on the present components.
In this paper we focus only on the model and so assume the input distribution uniform when considering SHAP.
Therefore each subset in the Shapley Values calculation is represented by an integral across the possible values of the absent components, with the other components ﬁxed to their values in the target vector x .
IG does not follow the Shapley Values procedure but rather the method described in
Section 1, based on the Aumann-Shapley method for sharing attribution.
Figure 6 shows the regions of the input space used by each method.
A.1. C ALCULATIONS
The calculations for Table 1 follow.
These follow the methods explained in the earlier parts of
Section A.1.
The model outputs are F ( x A ) = 1 , F ( x B ) = 0 , and F ( x (cid:48) )
= 0 .
BShap With two features, there are two possible orderings for the Shapley calculations: x 1 , x 2 and x 2 , x 1 .
Starting from the baseline x (cid:48) and swapping in components from the input x in these orders gives:
[Equation]
For B the value at all of these locations is zero so the attribution is zero for both components.
Evaluating these for A:
[Equation]
Which give attributions for x 1 : 12 (1 − 1 + 0 − 0)
=
and x 2 : 12 (1 − 0 + 1 − 0) = 1 .
IG It is not necessary to perform an explicit calculation for IG.
For A the path does not encounter ‘vertical’ gradients so the ∂F (˙ x ) ∂ ˙ x i term in Equation 1 must be zero for i =2.
Hence the second component has attribution 0.
The attributions of all features must sum to the difference between the target input and baseline, giving the result (1 , 0) .
A similar argument can be developed for B by breaking the path into two parts and considering the traversal of the ‘horizontal’ and ‘vertical’ contours separately.
SHAP There are again two possible orderings, this time of conditional expectations.
Conditioned on both components E x ( F ( x ) | x = x ) is just F ( x ) .
Unconditioned E x ( F ( x ))
≈ 0 .
434 is the mean value over the whole input.
Conditioned on only one component, the expectation integral is 1 if it is along one of the ‘arms’ of the cross, and 0.25 if it is across one of the arms.
This gives attributions:
A: x 1 : 12 (1 − 1 + 0 .
25 − 0 . 474)
≈ − 0 .
[Equation]
−
[Equation]
+ 1 −
[Equation] [Equation]
[Equation] [Equation]
B: x 1 : 12 (0 − 0 .
25 + 0 .
25 − 0 .
≈ − 0 .
[Equation]
0 − 0 .
25 + 0 .
25 −
[Equation]
≈ −
[Equation]
A. P ROOF THAT S TRONG D UMMY IMPLIES C ONSISTENCY
We deﬁne an attribution method ϕ to be “Consistent” if given a model F , input x and feature i , an attribution ϕ ( F, x , i ) > 0 implies there exists an a with a i
= x
i such that F ( a ) >
F ( a + λ ˆ x i ) for some λ
ˆ x i , a vector zero except for component i .
That is to say, if a component is attributed positively, there must be at least one instance where changing that component of a vector to match the input vector results in an increase in value.
This section demonstrates that adding Strong Dummy to the axioms Completeness, Symmetry, and Linearity guarantees Consistency.
BShap follows all of the four axioms.
We will show it is the unique such method.
BShap takes an input x and baseline x (cid:48) .
It is evaluated using the Shapley Values procedure on all vectors a having a i =
x
(cid:48)
i or a i
= x i for each i .
Let A be a set of vectors so constructed.
For a given A the axioms
Completeness, Symmetry, Linearity, and Strong Dummy of an attribution method on the continuous space are equivalent to the Shapley Values axioms on the a space: Completeness requires the attributions sum to the total difference in output value between the baseline and target.
This transfers directly to the Shapley Values context by subtracting the baseline value from every F ( a ) .
Symmetry requires that the attribution method be invariant under permutations of the element/component labels; this is the same in both settings.
In the continuous space the model functions F are all the functions from input vectors to the possible attribution vectors.
The functions relevant to the Shapley Values perspective are all those from A , a subset of the input space, to the same attribution vectors as before.
Hence the functions on A are a linear subspace of the original function space, and so the linearity of an attribution method on the original space will apply to them also.
Last, the deﬁnition of Strong
Dummy was based on the Shapley Values Dummy axiom.
This demonstrates for any x (cid:48) and x , ie.
everywhere, the four axioms are equivalent to the Shapley
Values axioms.
Given that Shapley Values is known to be the unique solution, BShap must be unique also.
Shapley Values / BShap is consistent (if an element has positive value it must make at least one positive marginal contribution, a stronger guarantee) therefore Strong Dummy in combination with the other 3 axioms guarantees consistency.
This discussion highlights the difference between
Weak Dummy and the other axioms.
Most of the axioms were strengthened when generalizing to the continuous setting, whereas Weak Dummy was weakened.
A. C ERTAINTY T RAINING A.3. V ALIDATING TRAINING
For effective training, methods to assess correct interpretation of certainty are required.
We introduce two techniques for this.
Information absence To ensure fair attributions, the model should interpret all baselines in the same way, as formalized by equation 2.
This may be veriﬁed by varying x with c =
and conﬁrming predictions do not change.
Integrated Gradients gives zero attribution when components match the baseline (baseline blindness).
However, Integrated Certainty Gradients relies on the model itself also ignoring zero certainty pixels, so that model and attribution behavior are aligned.
This means that zero certainty pixels should not accrue attribution by any method, not just ICG.
This makes c = 0 in a sense a universal baseline.
This motivates a distinctive visual test.
If the model is well trained, it should be possible to suppress the attribution of an accurate method to a negligible amount by masking out that area with zero certainty during the attribution process.
This can be tested by using IG with baselines x (cid:48)
=
and x (cid:48)
= 1 : the result should show an obvious suppression of attribution in the uncertain region in both cases.
This test is visualized in Figure 7.
Neutrality Zero certainty baselines should provide no input information to the model, so its output should be unbiased.
It is therefore reasonable to expect the model prediction in the baseline space to match its expectation on the training set: F ( x , c = 0 )
≈ E ( F ( X train ))
∀ x .
The expectation of each class in the training distribution is the proportion of samples in that class.
So for example in one-hot representation, ˆ y i should be approximately the proportion of the training distribution occupied by class i .
Using these tests we assessed a number of methods for certainty damage.
The following methods improved training compared to using uniform distributions.
A.3. D AMAGE FUNCTIONS
The basic damage function described in section 4 can produce informative attributions but more effective damage functions have been identiﬁed.
Variable damage extent With uniform random certainty , the trained model often widely violates the neutrality condition.
Due to regression towards the mean, i.i.d.
uniform random certainty results in an image with mean certainty 1 n (cid:80)
i c i typically close to 0.5.
It is hypothesized that the the neutrality condition is not learned due to the lack of training inputs with mean certainty near zero.
We deﬁne damage extent as the mean uncertainty of an input: 1 − 1 n (cid:80)
i c i .
A damage function which results in damage extents distributed uniformly in [0, 1] will expose the model to inputs along more of the certainty interpolation space.
Sampling from the continuous Bernoulli distribution (Loaiza-Ganem & Cunningham, 2019) was chosen as a new certainty generating function because it produces samples in the range [0, 1] and has a parameter which allows the expectation to be varied in [0, 1] as well as other desirable properties.
Even though these are not certaintyaware attribution methods, they
[Figure]
The parameter does not vary the expectation uniformly but instead concentrates it towards the center of the range.
To counteract this we reparameterized the distribution.
Let e ( λ ) be the expectation of the distribution with parameter λ .
Then λ (cid:48) :
= e − 1 ( λ ) , where e − 1 is calculated numerically from the analytical form of e .
Using this new parameter gives a distribution parameterized by the mean.
It was found that varying damage extent using this distribution does in fact improve neutrality.
Adversarial damage Random noise is typically not strongly biased towards an input class.
Uniform random damage data therefore results in an input which is a combination of data from the original class and unbiased data.
As a result, the input is always positively biased towards the original class.
Furthermore, if the model can detect and exclude random noise, it will be able to determine the true class from the remaining data.
These considerations suggest the model might be able to ignore c and still achieve good predictions.
Motivated by this, adversarial damage was tested: using a random alternative sample from the test dataset as the damage data .
When the alternative and original input are from different classes, damaged features D ( x , c ) i provide evidence for a different class from the true one and with a large damage extent the majority of features can indicate the alternative class.
It is not possible for the model to make an accurate prediction in such circumstances without considering c , forcing the model to use it.
Adversarial damage was found to train certainty awareness faster and with more accuracy than random uniform damage.
Another consideration is that it avoids exposing the model to out of distribution feature values, although the implications of this were not investigated.
Binary uncertainty Continuous Bernoulli and uniform damage distributions have been described above.
Discrete Bernoulli damage was also tested, producing c i values in { 0 , 1 } .
It was found that this improves learning rate compared to uniform random damage.
The reason for this has not been determined, although it may be since c i can be only 0 or 1, the value components no longer require a probabilistic interpretation, which makes the scenario easier for the model to learn.
We found combining adversarial and uniform damage data , binary and uniform uncertainty , and using variable damage extents resulted in the most effective training in terms of speed and ﬁnal evaluation.
This training was used for all results which are discussed below.
A.3. L EARNING RATE AND ACCURACY
Certainty awareness increases the classiﬁcation task complexity.
To assess the impact of this we trained MNIST models with and without input damage.
The results are shown in Figure 8.
The red and blue curves show training with and without damage respectively.
Both are evaluated on undamaged inputs.
They show that damage training has not resulted in reduced accuracy on MNIST for this model, although training takes twice as many batches.
Variance in accuracy and loss is
[Figure]
also increased.
Given that on average half the components of the original image are lost during damage, this seems a modest penalty.
The green curve shows the damage trained model on damaged inputs.
This is included for completeness but not especially relevant to the comparison.
A. A DDITIONAL RESULTS A.4. I MAGENETTE
To test ICG on more realistic scenarios, we trained a ResNet50 (He et al., 2015) using artiﬁcial uncertainty training on the Imagenette dataset (fast.ai, 2021).
The only modiﬁcations made to the architecture were the addition of an extra channel to represent certainty, and an additional 1x1 ( channel) convolution layer at the input, with ReLu activation and without batch normalization.
The model was trained to 82.4% top-1 accuracy on the validation data.
Attributions using ICG were compared with those generated using EG, chosen because it is a contemporary baseline-free attribution method.
It is established that single-path attribution methods such as
ICG can produce clearer results when averaged over multiple slightly perturbed inputs (Smilkov et al., 2017).
For comparison with EG, a multi-path method, we averaged 5 ICG attributions over inputs with added low-intensity noise to make each ﬁnal result.
We tested including noise augmentation when generating EG interpolations but it did not signiﬁcantly affect the outcome.
Results can be seen in Figure 9.
In most cases ICG attributions have a clear association with the object of interest.
The attributions tend to be more tightly localized than those of EG.
Backgrounds can form part of a model’s classiﬁcation decision, so stronger localization is not a full guarantee of greater attribution faithfulness.
However, it is notable that in row 2 EG allocates signiﬁcant attribution in the top black “border” region.
It seems reasonable to assume this is one of the lowest information-density areas of the image, and not a representative region for attribution.
We have not seen a case where
ICG signiﬁcantly attributes the border regions.
Note that this is not baseline blindness: black/dark areas in images can be strongly attributed through ICG.
[Figure]
EG blurred notably
ICG strongly highlights background regions.
In all other cases ICG clearly indicates the object of interest,
6), where ICG focuses attribution primarily on the
We hypothesize the reason for the difference in behavior between EG and ICG.
We have noticed output artifacts which relate to objects in individual EG interpolation baseline images.
If EG were performed with enough interpolations to truly represent the input distribution it may be that the effects of interpolation baselines would exactly ‘average out’ in low attribution areas.
It may be that this cannot be achieved in larger images due to the high dimensional input space.
This would also explain the excellent performance of EG on the MNIST dataset (where the input distribution is rather densely represented).
It would be reasonable to imagine that ICG might overcome this problem by approximating the input space as part of the deep learning model, given that deep learning methods are a current state-of-the art technology for representing high dimensional data.
Whether or our hypothesis is correct, the Imagenette attributions give evidence to ICG as an effective baseline-free attribution method in practical scenarios.
[Figure]
A.4. MNIST IN S PACE
Figure 10 compares attribution results for several baselines on a random example from the MNIST training test discussed in section A.3.3.
The zero baseline attribution has strong baseline blindness: only the ‘digit itself’ is attributed, not the space around it.
It shares several intuitively reasonable features with the other results.
The crossbar at the top is positive, which is a characteristic feature of a “5”.
“7” is the only other numeral having one, and it terminates on the opposite side.
The bottom curve is also indicated, a feature shared only with “3”.
The slope in the middle is more similar a 6 or an 8 (attributions for classes 6 and 8 attributions support this), and hence attributed negatively (in blue).
Of the remaining results there is a notable distinction between distribution-aware methods (mean baseline and EG) and unaware (middle baseline and AGN).
For example in the unaware methods there is a negative attribution point where the crossbar terminates, indicating a preference for a longer crossbar, but not in the aware methods.
This point is outside the region where the distribution varies, as can be seen from the distribution mean image, so it makes sense the distribution aware methods would not indicate it.
ICG shares many characteristics with the distribution-unaware methods, but the most strongly attributed area, in the crossbar, matches to the aware methods.
It is plausible it acts as an intermediate between the distribution aware and unaware methods: learning a distribution of the data during certainty training, but incorporating the model’s generalization prior as it does so.
To explore this idea brieﬂy we introduce a modiﬁed version of MNIST.
We make the input images wider and introduce a translation-invariant model architecture.
We can train a model using regular
MNIST images in the left region, and the model is able to classify digits correctly anywhere in the extended image due to the architecture.
When classifying digits in the right region they will be out-of-training-distribution.
Results from this scenario are shown in Figure 11.
The Expected
Gradients result appears more sharply deﬁned because of baseline blindness (the right side of all training images is black).
A faint ‘shadow’ of the training distribution is visible in the EG result.
This does not represent a model behavior since the model is translation invariant.
EG is formulated as a fair baseline for a given distribution, but it does not follow the model’s generalization onto OOD inputs.
Because the distribution knowledge ICG uses is learned by the model, it is generalized by the model architecture and follows the model generalization onto new inputs.
Like distribution unaware methods there is no ‘shadow’, because knowledge of the ungeneralized distribution is not retained.
A. E XPERIMENTAL DETAILS
Model architectures are shown in Figures 14, 12, and 13.
[Figure]
The damage function D used during training is speciﬁed by a combination of damage data and certainty generator.
For all models damage data was taken from either U (0 , 1) or adversarially from another training input.
Within each input, c i values were taken i.i.d from either the Bernoulli or our reparameterization of the continuous Bernoulli distribution.
Both have a parameter equal to their expectation, and for each image this was sampled from U (0 , 1) .
The four combinations of damage data selection method and certainty generation method were each used in equal proportion.
Xavier initialization was used for network weights and network biases were initialized to zero.
Burnt snacks Training data was generated procedurally hence there is no epoch size or train-test split.
Adam optimization was used with learning rate 0.001 and batch size of 64.
Local minimum Training data was generated procedurally hence there is no epoch size or train-test split.
The network did not converge for several training attempts, and was reinitialized several times until an initial state which did converge was found.
The network was initially trained with Adam optimization and a learning rate of 0.001.
After initial training, reduction in the learning was found to improve accuracy, and training was continued with reduced learning rates down to a minimum of 10 − 6 .
A batch size of 128 was used for initial training and a batch size of 256 for the reduced learning rate stage.
MNIST The dataset was split into 60,000 training examples and 10,000 test examples across all classes.
Adam optimization was used for training with learning rate 0.001 and batch size of 64.