Making Sense of Dependence: Efﬁcient Black-box Explanations Using Dependence Measure
Abstract
This paper presents a new efﬁcient black-box attribution method based on HilbertSchmidt Independence Criterion (HSIC), a dependence measure based on Reproducing Kernel Hilbert Spaces (RKHS).
HSIC measures the dependence between regions of an input image and the output of a model based on kernelembeddings of distributions.
It thus provides explanations enriched by RKHSrepresentation capabilities.
HSIC can be estimated very efﬁciently, signiﬁcantlyreducing the computational cost compared to other black-box attribution methods.
Our experiments show that HSIC is up to 8 times faster than the previousbest black-box attribution methods while being as faithful.
Indeed, we improveor match the state-of-the-art of both black-box and white-box attribution methods for several ﬁdelity metrics on Imagenet with various recent model architectures.
Importantly, we show that these advances can be transposed to efﬁcientlyand faithfully explain object detection models such as YOLOv4.
Finally, weextend the traditional attribution methods by proposing a new kernel enablingan orthogonal decomposition of importance scores based on HSIC, allowingus to evaluate not only the importance of each image patch but also the importance of their pairwise interactions.
Our implementation is available at https://anonymous.4open.science/r/HSIC-Attribution-Method-C684 .
Introduction
Artiﬁcial Intelligence has established itself as the reference technique for tackling many real-worldautomation tasks.
Consequently, the diversity of its applications is growing and reaching ﬁelds whereits outputs can contribute to critical decision-making.
In such cases, it is essential to be able toprovide explanations for each link of the decision chain, including AI algorithms.
Over the pastdecade, many techniques have emerged to explain the predictions of these algorithms [48, 40, 45,17, 58, 36, 35, 30, 29, 42], marking the birth of a new ﬁeld called Explainable Artiﬁcial Intelligence(XAI).
The tools developed in this research ﬁeld, mostly designed to explain neural networks, havealready proven helpful. For instance, it has been used in model debugging, identiﬁcation of newdevelopment strategies for practitioners, and failure understanding.
Initial approaches are based on analyzing the internal state of neural networks during inference,often relying on input gradients or activation values of hidden layers [48, 45, 17, 28].
However,the gradient only reﬂects the model’s operation in an inﬁnitesimal neighborhood around an inputand can therefore be misleading [20].
Furthermore, their applicability is limited to the case wherethe ﬁnal user has access to the implementation of the model.
Therefore, such methods cannot beapplied in the most common use cases, e.g. when models are made available by third parties through
API calls or specialized hardware.
In order to address this issue, some black-box approaches have been recently proposed, relying on the perturbation of the input and the observation of its effect on the output [61, 40, 36, 13].
One challenge of such perturbation methods is assessing this effecttogether with into account complex interactions inherent to deep neural networks.
for these characteristics black- methods resort to complex Monte Carlo methods that
In [13, 32], the authors propose methods to reduce the required number of forward passes, but theobtained performance improvements still do not make them close to white-box methods.
In this work,similarly to [13], we cast perturbation studies as Global Sensitivity Analysis (GSA) [38].
However,we rely on a whole different approach based on dependence measure rather than analysis of variance.
We measure the dependence between patch-wise perturbations of an input image and the model’soutput by comparing the distribution of perturbed inputs and outputs embedded in a Reproducing
Kernel Hilbert Space (RKHS).
More speciﬁcally, we use Hilbert-Schmidt Independence Criterion(HSIC), a dependence measure based on the Hilbert-Schmidt norm of the empirical cross-covarianceoperator evaluated between the represented distribution.
HSIC leverages the rich theory of RKHS,thereby capturing more diverse information than variance-based indices such as Sobol.
In addition,it can be estimated more efﬁciently, even bridging the performance gap between black-box andwhite-box methods.
Our contributions are as follows: (1) we introduce a new efﬁcient black-box attribution methodrelying on HSIC; (2) we derive a new kernel that confers an orthogonal decomposition property,allowing us to go beyond usual attribution methods and evaluate interactions between patches of theimage; (3) we conduct experiments to assess the ﬁdelity of our method on ImageNet and show thatit improves or matches the state-of-the art for different metrics while bridging the computationalgap between black-box and white-box attribution methods; (4) we demonstrate its versatility and itspotential by successfully applying it to a less common test case: explanations of object detection; anda new test case: evaluation of pairwise interactions between patches of the input image.
Related work
Our work builds on prior efforts aiming to develop attribution methods in order to explain theprediction of a deep neural network by pointing to input variables that support the prediction –typically pixels or groups of pixels, i.e. patches in the image – which lead to importance maps.
Attribution methods for white-box models A large number of attribution methods have beendeveloped relying on the gradient of the decision studied.
The ﬁrst method was introduced in [4]and improved in [48, 61, 56] and consists of explaining the decisions of a convolution model byback-propagating the gradient from the output to the input, indicating which pixels affect the decisionscore the most.
However, this family of methods is limited because they focus on the inﬂuence ofindividual pixels in an inﬁnitesimal neighborhood around the input image in the image space.
Forinstance, it has been shown that gradients often vanish when the prediction score to be explained is near the maximum value [58].
Integrated Gradient [58] and SmoothGrad [50] partially address thisissue by accumulating gradients.
Another family of attribution methods relies on the neural network’sactivations.
Popular examples include CAM [62], which computes an attribution score based on aweighted sum of feature channel activities – right before the classiﬁcation layer.
GradCAM [45]extends CAM via the use of gradients, reweighting each feature channel to take into account theirimportance for the predicted class.
Nevertheless, the choice of the layer has a huge impact on thequality of the explanation.
In comparison, our proposed approach is model-agnostic and hence doesnot require access to internal computations.
Attribution methods for black-box models In this paper, we extend the problem by restrictingit to a black-box model: the analytical form and potential internal states of the model are unknown.
Several methods compute inﬂuence scores for each individual pixel or group of pixels.
The ﬁrst method, Occlusion [61], masks individual image regions – one at a time – with an occludingmask set to a baseline value and assigns the corresponding prediction scores to all pixels withinthe occluded region.
Then the explanation is given by these prediction scores and can be easilyinterpreted.
However, occlusion fails to account for the joint (higher-order) interactions betweenmultiple image regions.
For instance, occluding two image regions – one at a time – may onlydecrease the model’s prediction minimally (say a single eye or mouth component on a face), whileoccluding these two regions together may yield a substantial change in the model’s prediction if thesetwo regions interact non-linearly as is expected for a deep neural network.
Our work, together withrelated methods such as LIME [40], RISE [36] and more recently Sobol [13] addresses this problemby randomly perturbating the input image in multiple regions at a time.
Surprisingly, RISE [36] and Sobol Attribution [13] have recently shown that black-box attributionmethods can rival and even surpass the white-box methods commonly used without recourse tointernal states.
However, despite the efforts in [13] to limit their computational overhead, black-boxmethods remain far from white-box methods in terms of execution time.
In this work, we show that itis possible to match or even surpass the performances of current black-box methods while reachingcomputation times lower than some white-box methods by using dependence measure-based Global
Sensitivity Analysis (GSA).
Global sensitivity analysis using dependence measures Our attribution method builds on the GSA framework.
The approach was introduced in the 70s [9] and was popularized with variancebased sensitivity analysis and Sobol indices [51].
It consists of evaluating the sensitivity of a model’soutput of interest to some input design variables.
GSA is currently used in many ﬁelds, especiallyfor the study of physical phenomena [27, 38].
Recently, dependence measure-based sensitivityanalysis was introduced in [10] and was shown to circumvent some practical issues of variancebased sensitivity analysis.
In particular, by relying on the representation capabilities of RKHS, thedependence measure that we use in this work, HSIC [22], captures more diverse information thantraditional variance-based indices for far fewer model evaluations.
Explanations using Hilbert-Schmidt Independence Criterion
In this section, we describe sensitivity analysis-based attribution methods, deﬁne Hilbert-Schmidt
Independence Criterion (HSIC) [22] and explain how we can use it and adapt it to design a newblack-box attribution method whose efﬁciency competes with white-box methods.
We also explainthe theoretical advantages of HSIC that we can build on to go beyond traditional attribution methods.
3.1 Sensitivity analysis of perturbed black-box models
Let f : X = X 1 , ..., X n → Y be the model under study, x i ∈
X i the input variables and y =f
( x 1 , ..., x n ) ∈ Y the output value of the model f .
GSA studies the sensitivity of y to each input x iby considering them as iid (independent and identically distributed) random variables and assessingthe link between their distribution and that of the output after an initial input sampling.
Given an inputvector X = ( x 1 , ..., x n ) , a prediction y =
f ( X ) can thus be explained using sensitivity analysisby applying random perturbations x
= ( x 1 , ..., x n ) , x i ∼ P X i of the original X and analyzing theimportance of each x i for explaining the variations of y - which is considered a random variable,y ∼ P y .
For image data, the inputs x i are pixels.
However, pixel perturbations would only emphasize lowlevel explanations.
To obtain high level and meaningful explanations, we rather consider a randomperturbation mask M
=
( M 1 , ..., M d ) ∈ [0 , 1] d .We upsample this mask using a Nearest Neighbor interpolation method to obtain u ( M ) ∈ [0 , 1] n , a patch-perturbated vector that we apply on the inputimage X using a mask operator τ : X × [0 , 1] d → X .
More speciﬁcally, we use the inpaintingoperator deﬁned by τ ( X , M ) =
X (cid:12) u ( M )
+ (1 − u ( M )) µ , with (cid:12) the Hadamard product and µa baseline value (here, µ is a {+black image with all pixels’ value = 
+}
For eachimage patch, the mask values deﬁne a drift from the original image towards the baseline value µ .
Wethereby assess the effect of each image patch, represented by M i , on the output.
The perturbation methodology thus consists of (1) sampling p masks { M (1) , ..., M ( p ) } from
M ∼ P M (with P M = P M 1 × ...
× P M p ) , (2) applying them to the original input vector, leadingto p perturbed input vectors (e.g., partially masked images) { τ ( X , M (1) ) , ..., τ ( X , M ( p ) ) } (3)computing the predictions { y (1) , ..., y ( p ) } =
{ f ( τ ( X , M (1) )) , ..., f ( τ ( X , M ( p ) )) } and (4) statistically assessing the effect of each mask M i on y by estimating a sensitivity measure between each 137 P M i and P y from the previous sampling.
In this paper, we consider that the more independent M i isfrom y , the less important the corresponding image patch is to explain it [10].
In the following, wedescribe HSIC, a dependence measure, and how to use it in practice.
3.2 Hilbert-Schmidt Independence Criterion
Let x
and y be two random variables of probability distribution P x and P y deﬁned on X and Y .
HSIC measures the dependence between P x and P y based on their embedding in Reproducing Kernel
Hilbert Space (RKHS).
Let ϕ : X → F and ψ : Y → G two continuous feature mapping between X ,
Y , and two RKHS F , G , such that the inner product between the feature embeddings of x, x (cid:48) ∈ X in
F is given by the kernel k ( x, x (cid:48) )
= (cid:104) ϕ ( x ) , ϕ ( x (cid:48) ) (cid:105) (and l ( y, y (cid:48) )
= (cid:104) ψ ( y ) , ψ ( y (cid:48) ) (cid:105) for y, y (cid:48) ∈ Y ).
Thecross-covariance operator C xy : G → F between the random variables x and y is deﬁned in [19] andcan be written.
[Equation]
where µ x = E x [ ϕ ( x )] and µ y =
E y
[ ψ ( y )] are the mean embedding of x and y in F and G .
When
F and G are universal RKHS on the compact domains X and Y , then (cid:107) C xy (cid:107) HS
= 0 if and only ifx and y are independent, where (cid:107) · (cid:107) HS denotes the Hilbert Schmidt norm (see [23]).
In [22], theauthors deﬁne HSIC as (cid:107) C xy (cid:107) 2 HS , which can be written:
[Equation]
where x , x (cid:48) and y , y (cid:48) are pairwise iid.
HSIC can also be expressed in terms of Maximum Mean
Discrepancy (MMD), which is a distance between mean embeddings deﬁned in an RKHS [57].
Morespeciﬁcally, let the product RKHS P = F × G with kernel v (( x, y ) , ( x (cid:48) , y (cid:48) ))
=
k ( x, x (cid:48) )
l ( y, y (cid:48) ) .
Then, HSIC ( x , y ) =
γ 2 v ( P x P y , P x , y ) , where γ v is the MMD operator on P .
HSIC thus measuresthe distance between P xy and P y P x embedded in P [10].
Since x ⊥ y ⇒ P xy =
P y P x , the closerthese distributions are, in the sense of the MMD, the more independent they are.
Thus, given a set of inputs (e.g., random masks) { x 1 , ..., x p } and the associated outputs { y 1 , ..., y p } ,[22] shows that HSIC can be estimated by
[Equation]
where H, L, K ∈ R p × p and K ij = k
( x i , x j ) , L i,j = l ( y i , y j ) and H ij =
δ ( i = j ) − p − 1 .
Usingthis formula, H p x , y can be computed with a O ( p 2 ) complexity.
In the following, we compute H pM i , yi.e. the estimation of the HSIC between a patch M i and the output y , for each patch ( i ∈ { 1 , ..., d } ),see Algorithm 1. We denote H pi :
=
H pM i , y for clarity.
In the next section, we discuss the kernels kand l and show that we can obtain a valuable orthogonal decomposition property for HSIC-basedindices, allowing to assess interactions between input variables.
3.3 Orthogonalisation of HSIC to enable interactions evaluation
One question of interest in explainability is the measurement of the importance of a speciﬁc groupof variables.
Indeed, it is notorious that neural networks are highly non-linear, and as it has beendemonstrated in several works [16, 13], the effects of the groups of variables are far from beingadditive.
Concretely, some areas of the image may only be important in interaction with other areas,affecting the output only when both areas are perturbated at the same time - as we shall see in Section4.4 (for instance, for mustaches of a puma).
Classical GSA based on variance analysis enjoys an orthogonal decomposition property that isvaluable for that purpose.
Let { x 1 , ..., x n } ∈ X n be a set of n univariate random variables.
For anysubset A = { l 1 , ..., l | A | } ⊆ { 1 , ..., n } , we denote x A
=
( x l 1 , ..., x l | A | ) the vector of input variableswith indices in A .
When using Sobol indices based GSA, it is possible to measure the interactionsbetween variables thanks to an orthogonal (or ANOVA) decomposition property.
We describe thisproperty in Appendix A, and introduce an example to illustrate why this property is necessary inorder to properly assess the importance of interactions.
This property was lacking for dependencemeasure-based sensitivity analysis until the work of [11], which shows that when using HSIC, aspeciﬁc choice of kernel k can enable this decomposition.
For any choice of l and characteristic univariate kernel k , it is possible to obtain the decompositionproperty by deﬁning the input kernel
k
A such that
[Equation]
These conditions can be stringent, especially the right one, which implies to compute integrals(analytically or empirically), which can be non trivial for continuous input distributions p x andclassical kernel choices such as Radial Basis Function (RBF) of bandwidth σ , k ( x, x (cid:48) )
=
exp( (cid:107) x −x (cid:48) (cid:107) / 2 σ 2 ) .
This condition is alleviated when using discrete input variables of known densities, forwhich the integrals can be computed analytically.
In particular, in this work, we rely on Proposition 1:
Proposition 1. Let x i be a Bernoulli variable of parameter p = 12 , and δ ( x = x (cid:48) )
the Dirac kernel.
Let k 0 be deﬁned as in equation (3).
Then, the following kernel satisﬁes the decomposition property:
[Equation]
The proof is in Appendix A. As a practical consequence, if we sample binary masks from a Bernoullivariable of parameter p = 1 / 2 , i.e. M i ∼ B ( p ) for i ∈ { 1 , ..., d } , and use the kernel deﬁned inequation (4), we can assess not only the importance of each patch in the image but also the importanceof the interactions between patches.
It allows to go beyond classical attribution methods and revealareas of the image that are only important in interaction with other areas, i.e. that affect the outputonly when both areas are perturbated at the same time.
Concretely, for two image patches indexed byi and j , the interaction HSIC, H i × j , can be obtained with [11]:
[Equation]
We insist on the fact that if the decomposition property is not valid, subtracting H pM i , y and H pM j , yto H p
( M i ,M j ) , y does not ensure that we assess the importance of the interactions only.
Traces ofthe independent importance of M i and M j may remain in the obtained metric.
Some qualitativebeneﬁts of such a property are illustrated in Section 4.4.
This decomposition holds for any choice ofkernel l : Y → G .
Therefore, in the following, we use the RBF kernel, with the common practice ofchoosing the bandwidth as the median of the output [10, 22, 54].
3.4 Sample efﬁciency of HSIC estimator
Several types of metrics are classically used for sensitivity analysis.
The most famous one, Sobolindices [52] and its variants [18, 6] classically require p 2 model evaluations [25] to reach an estimationerror of O ( 1 √ p ) .
Recent design of experiments managed to reduce this requirement to p × ( d + 2)[43] (Theorem 1), but with the increase in complexity of state-of-the-art architectures and the highdimensionality of inputs (although mitigated by the use of d patches instead of all pixels), it canstill be cumbersome.
Despite this drawback, [13] uses Sobol indices to obtain explanations and stillachieves execution time improvement compared to other state-of-the-art attribution methods, whichare even less efﬁcient in terms of samples requirements.
HSIC is much less expensive to estimate than Sobol indices: for a same estimation error of O ( 1 √ p ) ,p forward passes are needed instead of p × ( d + 2) [22] (Theorem 1).
This allows using far fewersamples to obtain relevant explanations, thereby dramatically increasing the efﬁciency of the methodcompared to previous black-box approaches.
This huge advantage is empirically illustrated in Section4.2, where we demonstrate that our method deﬁnes a new standard in terms of efﬁciency for black-boxattribution methods.
It even bridges the efﬁciency gap between black-box and white-box approaches.
3.5 Implementation of the method
A summary of the whole attribution method is provided in Algorithm 1.
The computation of H piis
O ( p 2 ) but it is possible to vectorize it using any library optimized for tensor operations (e.g.tensorﬂow).
As a result, the computation time of H pi is negligible compared to that of the p forwardpasses.
Furthermore, we implemented a sampling based on Latin Hypercube Sampling [33], a QuasiMonte Carlo (QMC) method designed to efﬁciently ﬁll the input space in Monte Carlo integration.
Once the grid of H pi is obtained, we use a bilinear upsampling to be able to apply it to the image.
Experiments
This section showcases the beneﬁts of our approach compared to other attribution methods.
Thesebeneﬁts are threefold.
First, the computational cost of HSIC attribution method is signiﬁcantly lowerthan previous state-of-the-art methods, even bridging the performance gap between black-box andwhite-box methods.
Second, our method improves state-of-the-art for several ﬁdelity metrics, forblack-box as well as white-box methods.
Finally, the orthogonal decomposition property of HSICallows to go beyond usual attribution methods and assess interactions between image patches.
In Section 4.1, we compute explanations of the predictions in the ILSVRC-2012 [12] classiﬁcationtask (ImageNet), for four common architectures, namely MobileNet [44], ResNet50 [24], EfﬁcientNet[59] and VGG16 [49].
Then, we compare those explanations with these of other state-of-the-artblack-box and white-box attribution methods in terms of ﬁdelity and efﬁciency.
In Section 4.2, weinvestigate the convergence of our method by measuring its correlation with a high sample estimatorand comparing it with RISE [36] and Sobol [13].
In the remaining sections, we conduct additionalexperiments that show the versatility of our method.
In Section 4.3, we evaluate HSIC attributionmethod to explain object detection on COCO dataset [31] with YOlOv4 [39].
We conclude theexperiments with Section 4.4, where we showcase the use of the HSIC orthogonal decompositionproperty to assess interactions between image patches.
4.1 Fidelity of classiﬁcation explanations
In this section, we evaluate the ﬁdelity of the explanations with three ﬁdelity metrics.
The ﬁrst,
Deletion [36], assumes that the more faithful an explanation is, the quicker the prediction score shoulddrop when pixels that are considered important are shut down.
The second one, Insertion [36], insteadadds pixels on a baseline image, starting with pixels that are associated with the highest importancescores of the explanation.
Finally, µ Fidelity [5] creates random pixels subsets which are assigned abaseline value and measure the correlation between the drop in the score and the importance of theexplanation.
Those metrics are further described in Appendix F.
In Table 1, we report the results of several different attribution methods for explaining the classiﬁcationof MobileNet [44], ResNet50 [24], EfﬁcientNet [59] and VGG16 [49] on 1000 images sampled fromthe ImageNet validation dataset.
The models used for the experiments have been accessed fromtensorﬂow [1] with the keras API [8].
We introduce two variants of our method, H pi eff.
and H pi acc.
The words "eff" and "acc" stand for efﬁcient and accurate because we use p = 764 and p = 1536samples, respectively.
We use our method with a grid size of 7 × 7 ( d = 49 ).
To evaluate the differentmethods, we use the Xplique [15], a library dedicated to explainability.
For black-box and white-boxmethods, we bold the best result and underline the second.
When the differences between somemethods are not statistically signiﬁcant, we highlight both.
Note that for µ Fidelity, the estimationvariance is high (typically about 20% ), so we only use the bold notation and leave the Table in
Appendix B.
The exact error bars are also left in the Appendix to make the presentation lighter.
For the Deletion metric, our method obtains the best results among the tested black-box methods forall the architectures in both its efﬁciency and accurate variants.
Except for EfﬁcientNet, we even beat
[Table]
all tested white-box methods. RISE is still the best of black-box methods for the Insertion metric,but the accurate variant is systematically second.
Besides, our methods are among the best of bothblack-box and white-box methods.
These results, as such, are already satisfactory.
But it goes even further: we obtained these resultswith far fewer forward passes than other state-of-the-art black-box attribution methods.
With theefﬁcient variants, competitive results are obtained more than 8 times faster than RISE, the currentstandard of black-box attribution methods.
It improves on Sobol, a recent and promising attributionmethod that was already branded as more efﬁcient, by a factor 5 .
The time improvement factors forthe accurate variant of our method are still very appealing ( 5 and 3 ).
We even beat the execution timeof Integrated Gradients white-box method [58], a popular and successful white-box method.
Theefﬁciency of HSIC attribution method is investigated more thoroughly in the next section.
4.2 Estimator efﬁciency
The advantage of black-box attribution methods lies in providing explanations without access to themodel’s internal state or the gradients.
However, this advantage comes at a cost since many forwardpasses are needed to obtain meaningful explanations.
This cost is all the more constraining sincerecent architectures are increasingly heavy in terms of computational time.
Therefore, it is critical for such attribution methods to use as few forward passes as possible.
Results reported in Section 4.1 attest that our approach based on H pi shines in that regard, andwe refer to this section for more comments.
It motivates us to study the efﬁciency of ourmethod further.
To that end, we compute an "asymptotical" 1 explanation with 13 , 000 forwardpasses, for HSIC, Sobol and RISE attribution methods, on 100 images of ImageNet validation set.
We apply the three methods on EfﬁcientNetwith d = 7 × 7 masks and image patches,like in [36] and [13].
We then computeexplanations for an increasing number offorward passes and compare the obtainedexplanation with the baseline "asymptotical" explanation.
We use Spearman rank correlation [55] like theoretically and empirically argued in [21, 2, 60, 14].
Thisexperiment allows comparing the conver- gence speed of our method with RISE and
Sobol.
The curves are plotted in Figure 2and show that our method, HSIC converges much faster than RISE and Sobol.
4.3 Explanation of object detection
Explaining model’s predictions is more challenging for object detection than for image classiﬁcation.
Indeed, recent object detection models usually predict three pieces of information: localization(bounding box corners), objectness score (probability that a bounding box contains an object of anyclass), classiﬁcation information (probability of each different possible class).
Recently, it has beendemonstrated that it was possible to use attribution methods to explain object detection by constructinga score aggregating for the previous information.
In [37] the authors combine intersection over unionfor localization, cosine similarity for the class probability, and focus on high objectness areas.
As aresult, they can use RISE to explain the object detection using this score as the output of the model.
In this section, we test our method for explaining the object detections of YOLOv4 [39] on COCOdataset [31] compared to the approach presented in [37], D-RISE.
We also compare the explanationswith these of Kernel Shap [32], another black-box attribution method.
The explanations for 1 , 000validation images are evaluated with the Deletion, Insertion, and µ Fidelity metrics.
This experimentis time-consuming, so
we
[Table]
Even if our method is not the best for Deletion, it is for Insertion and µ Fidelity, and more importantly,it is 5 and 6 times faster than D-RISE and Kernel Shap.
Figure 3 displays visualizations of objectdetection explanations.
While the ﬁrst images show a standard detection explanation, the rightmostone is more interesting since it emphasizes an error of the object detector.
The model identiﬁes azebra as a cat, and our method manages to explain this error by emphasizing the cat at the bottomright corner of the image. Note that we did not obtain such an explanation with D-RISE, even with ahigh sample number and different grid sizes - visualizations can be found in Appendix C).
4.4 Finding spacial interactions in the model
Usual attribution methods provide explanations in the form of heat maps that assign each pixel (orpatch) an importance score.
However, the scope of such explanations is limited since the reason for aprediction may not be explained only by the single importance of independent patches.
In [13], theauthors use Sobol total indices that account for the importance of a patch in interaction with all otherpatches, but they cannot localize the interactions.
Thanks to its orthogonal decomposition property,our HSIC-based attribution method is able not only to assess the importance of each patch, but alsothe importance of interactions between speciﬁc patches by computing the HSIC of the joint patchesand subtracting the contribution of each patch taken independently 2 .
It is then possible to identifyregions of the image that affect the output only when both areas are perturbated jointly.
We illustrate this property in Figure 4.
For each image, we computed all the possible interactionsbetween patches and reported the most important ones.
Note that not all images exhibit signiﬁcantinteractions, so we performed a qualitative selection for pedagogical purposes.
On the upper row, weplot usual heatmaps that traduce the importance of each patch taken independently.
On the bottomrow, important interactions are represented in red.
We can see that the middle part of the snakebody is not the only important element in the picture, that one part of the mountain interacts withone of the skiers, so do the mustaches of the puma and two corners of the roof.
Note that for eachimage, the maximum values of H pi are 40 .
6 , 11 . 4 , 8 . 2 and 18 . 8 and the maximum values of H pi × j are19 . 6 , 6 . 6 , 9 . 2 and 6 . 3 respectively.
Thanks to the orthogonal decomposition property, these metricscan be compared and we can deduce that some interactions are as signiﬁcant as some importantindependent patches.
For the image with a puma, the interactions between the mustaches are evenmore important than the eye / forehead for identifying the animal ( H pi × j = 9 .
when i and j are theright and left mustaches and H p iff = 8 . 2 when i is the eye / forehead).
Conclusion
We have introduced a new attribution method based on a dependence measure, Hilbert-Schmidt
Independence Criterion, which leverages representation capabilities of Reproducing Kernel Hilbert
Spaces, thus being able to capture complex information.
This attribution method is black-box, soit is applicable even when the implementation of the neural network to explain is not available.
Nonetheless, it alleviates the computational burden of traditional black-box methods, improving onthe state-of-the-art of both black-box and white-box attribution methods while being closer to thelatter than the former in terms of efﬁciency.
In addition, we showed how the rich framework of RKHScould be used to assess and localize interactions between pairs of patches of the input image that arerelevant for explaining the output.
We hope that the introduced framework will open up researchavenues for attribution methods beyond traditional pixel-wise or patch-wise explanations.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] The limitations stem from the
complexity of computing HSIC mentioned in Sections 3.2 and 3.4.
We provided a
vectorized implementation to alleviate this limitation.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] ]
There is
no negative societal impact speciﬁc to this work that is not shared with common XAI
techniques.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] In Appendix A
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] URL is found
in page 1.(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[Yes] Error bars have been computed but left in the Appendix to make the presentation of Tables 1 and 2 lighter.
Nonetheless, they are taken into
account in the tables and in the comments to assess the statistical signiﬁcance of results
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]