CRAFT: Concept Recursive Activation FacTorization for Explainability
Anonymous Author(s)
AffiliationAddress
Abstract
Despite their considerable potential, concept-based explainability methods havereceived relatively little attention, and explaining what ’s driving models’ decisionsand where it’s located in the input is still an open problem.
To tackle this, we revisitunsupervised concept extraction techniques for explaining the decisions of deepneural networks and present CRAFT – a framework to generate concept-basedexplanations for understanding individual predictions and the model’s high-levellogic for whole classes.
CRAFT takes advantage of a novel method for recursivelydecomposing higher-level concepts into more elementary ones, combined with anovel approach for better estimating the importance of identified concepts with Sobol indices. Furthermore, we show how implicit differentiation can be used togenerate concept-wise attribution explanations for individual images.
We furtherdemonstrate through fidelity metrics that our proposed concept importance estimation technique is more faithful to the model than previous methods, and, throughhuman psychophysic experiments, we confirm that our recursive decompositioncan generate meaningful and accurate concepts.
Finally, we illustrate CRAFT’s potential to enable the understanding of predictionsof trained models on multiple use-cases by producing meaningful concept-basedexplanations.
Introduction
Interpreting the decisions of modern machine learning models such as neural networks remains amajor challenge.
The need for robust and reliable explainability methods has never been more urgentas machine learning is being applied to an ever increasing range of domains, including safety criticalones.
The application of the General Data Protection Regulation law (GDPR) [1] in the European Union has drawn the attention of the general public to the rights they should have on their data. This kickstarted a race for other needs, with more and more regulation agencies asking for the rightfor AI decisions to be explainable to users – e.g. European AI act [2], EASE concepts for designassurance [3].
In order to try to meet this need, an array of explainability methods have already been proposed.
Mostof these methods aim at explaining what inputs (or pixels in an image) are driving the model’s decision.
These so-called attribution methods yield heatmaps that indicate the importance of individual pixels.
Among the most notable ones is LIME [4], which was initially developed to try to locally – that is,at an instance level – understand models’ predictions to identify possible biases in vision models.
Multiple improvements have since been introduced – either by better harnessing the informationprovided by gradients to estimate the importance of individual pixels [5, 6, 7, 8, 9, 10, 11, 12], leveraging image perturbations to evaluate the sensitivity of a model’s output [13, 14] or, morerecently, via the use of formal methods to generate explanations [15].
However, all the aforementioned methods focus on one side of explainability – answering the questionof where – i.e., where in an image are the pixels that are critical to the decision located. They leavethe question of what – i.e., what visual features are actually driving decisions – entirely open.
Weargue that this limitation is one of the main reasons why these methods fail in some cases to helpusers, for instance, identify the source of a system’s bias or its failure cases as shown in [16].
Featurevisualization methods [17, 18] characterize the selectivity of individual neurons (or neural channels orarbitrary directions in the neural activation space) via the synthesis of input stimuli which maximizetheir responses and can partially answer this question.
Still in this vein, [19, 20, 21] proposed to usethe training dataset to identify the samples that contribute the most to the model’s decision.
Finally,closer to our work, a new line of research has recently been initiated [22] based on high-level concepts.
The goal of this branch is to find humanly interpretable concepts in the activation space of a layer in aneural network.
This approach can give positive results, but in its original formulation, it requiresprior knowledge on the relevant concepts, and more importantly, the labeling of a dataset for each ofthe concepts we want to extract.
Hence, several works have proposed to automate the concept searchbased only on the training dataset and without explicit human supervision.
The most prominentof these techniques, ACE [23], uses a combination of segmentation and clustering techniques, butrequires heuristics to remove outliers.
This method unlocks the possibility of large scale conceptextraction without additional labeling or human supervision.
Nevertheless, it suffers from severalproblems: each segment can only belong to one cluster, the choice of the layer from which to retrievethe concepts is not clear, and the amount of information lost during the outlier rejection phase canbe a cause of concern.
More recently, [24] proposes to leverage matrix decompositions on internalfeature maps to discover concepts.
It is important to note that current work does not offer a link between their global and local explanations, nor do they offer an answer to the question of which layer to choose to perform thedecomposition.
Building up on these conclusions, we revisit these concept extraction techniques byusing Non-Negative Matrix Factorization (NMF) and propose 3 different ingredients to answer thesequestions simultaneously, thereby introducing CRAFT, a new automatic concept extraction method.
We can summarize our main contributions as follows:
• A novel approach for the automated extraction of high-level concepts learned by deep neuralnetworks.
• A recursive procedure to automatically decompose concepts into sub-concepts, startingwith the last layer of the model and working our way inwards.
We validate the benefit ofthis recursivity – i
e. decomposing concepts into sub-concepts – with human psychophysic
experiments which show that (i) that the decomposition of a concept yields more coherent sub-concepts (ii) the groups of points formed by these sub-concepts are more refined andappear meaningful to humans (expert or non-expert).
• A novel technique to quantify the importance of individual concepts on a model’s predictionsusing Sobol indices coming from the field of Sensitivity Analysis.•
A novel Concept Attribution Map (CAM) method to backpropagate each of the conceptvalues independently into the pixel space by leveraging the implicit function theorem,allowing us to locate the concept in a given input image.
This effectively unlocks the abilityto apply all the white-box [5, 6, 7, 8, 9, 12, 25] and black-box [4, 26, 13, 14] explainability techniques in the literature to obtain concept-wise attribution maps.
• A demonstration of the approach combining local and global explanations to accurately explain predictions and understand complex failure cases.
Related Work
Explaining where The widespread use of black-box machine learning methods including deepconvolutional neural networks in myriads of computer vision tasks prompted a need to understandwhere in the input image the model looked to make predictions.
These explanatory heatmaps canbe generated through completely different approaches depending on whether access to gradients isprovided.
If it is indeed the case, there’s a plethora of different methods that harnesses intermediaryinformation inside the neural network to create these explanations [5, 8, 7, 28, 6, 29, 9, 12].
However,they have been found to induce confirmation bias [30] and to be vulnerable to adversarial attacks [31].
Somewhat differently, there are other methods [10, 11] that harness gradients to optimize masks to maximize the impact on the predictions, and thus determine the most important parts of theinput for the model.
However, if only the input and its corresponding output are available, othertechniques exist that enable the generation of attribution maps by locally estimating the importanceof each input pixel: LIME [4], RISE [13], and more recently, an attribution method based on Sobolindices [14, 32].
Crucially, they propose to input perturbed versions of the example one wishes toexplain and either construct a linear model to determine the importance of each region of the input,leverage Monte-Carlo methods to this end, or compute the Sobol indices [32] associated to them as ameasure of their influence on the model.
Concretely, we will be exploiting all this literature to locatethe important parts of images with respect to what we will call “high-level” concepts by generatingconcept-wise attribution maps.
Explaining «what» There have been studies [33, 34] that indicate that CNNs trained on the
ImageNet dataset [27] rely heavily on textures to classify, and largely disregard the shapes.
Forthis reason, some researchers suggest that attribution maps might not be enough to explain models’predictions [17], and that explainability methods revealing the role of the textures are a must.
Namely,in [18] and [17], explanations are generated as the inputs that would maximize the neural activation ofa given layer with respect to a given class.
However, these explanations may not be easily interpretableby humans.
Finally, other approaches suggest to modify the structure of the neural network, either byconstraining the convolutional layers to naturally provide visual explanations [35], or by forcing it togenerate prototypes for the classes [36], but our main focus are post-hoc methods that can be appliedto pre-trained neural networks and don’t need further training.
Concept discovery.
In [22], Kim et al.
proposed an alternative to explaining the what : they built adatabase with different concepts (such as “stripes”) to extract a concept vector in the latent space of agiven layer.
Then they proposed to estimate the importance of this concept vector using the directionalderivative of the model’s predictions with respect to this concept vector.
However, it is a supervisedapproach, and thus, only applicable when we have prior knowledge of the concepts in play.
Thenatural extension of this idea is automatic discovery of concepts in an unsupervised fashion, withoutthe need for prior knowledge or labelled concept datasets.
As such, in [23], a technique is proposed todiscover these “high-level” concepts: they perform segmentation at different resolutions on patchesof images, cluster them and select the most significant based on perception and Testing with Concept Activation Vectors (TCAV) [22] scores.
However, the quality of the result is highly dependent on thesegmentation scheme and on the layer used for perception scores.
Building up on this technique, [24]propose to generate a bank of concepts for each class by performing dimensionality reductions on theactivation maps flattened over the channel dimension.
Once the factorization done, the reconstructionof the activation of the image can then be interpreted as a combination of a set of concepts and acoefficient associated to these concepts.
Not all factorization-based methods are equal though.
Their large-scale human experiments show an interesting trend: Non-negative Matrix Factorization (NMF)is widely preferred over Principal Component Analysis (PCA) or ACE for generating meaningfulconcepts for humans.
Finally, [37] defines the notion of completeness of a concept bank and proposesa method to learn a complete set of concepts using Shapley values [26].
Overview of the method
In this section, we first describe our Concept Activations Factorization method by pointing outthe differences that set our technique apart from previous work.
We then proceed to introducethe three new ingredients that make up CRAFT: (1) a method to recursively decompose conceptsinto sub-concepts, (2) a new approach to better estimate the importance of extracted conceptsand (3) how we unlock any attribution method to create Concept Attribution Maps , using implicitdifferentiation [38, 39, 40].
Notation In this work, we consider a general supervised learning setting, where ( x 1 , ..., x n ) ∈
X nare n points and ( y 1 , ..., y n ) ∈ Y n their associated labels. Unless specified, all points are assumedto have the same labels.
We are given a (machine-learnt) black-box predictor f : X → Y , which atsome test input x predicts the output f ( x ) .
Without loss of generality, we assume that f is a neuralnetwork composed of k layers, and we denote f ( x ) =
h k ◦ h
k − 1 ◦ ... ◦ h 1 ( x ) with h l ( x ) ⊆
R pbeing the intermediate activations for the layer l and h l ( x ) i an activation for the same layer.
Further,we require non-negative activations: h l ( x ) i ≥ 0 : ∀ i ∈ { 1 , ..., p } , which amounts to choosing a layerwhose activation function σ l ( x ) ≥ 0 .
In particular, this assumption is verified by any architecture thatutilizes ReLU , but any non-negative activation function works.
Finally, we denote h l,k the functiongoing from the layer l to the output of the model f .
3.1 Concept Activations Factorization
As illustrated in Fig.3, we propose to use Non-negative matrix factorization activations to find a basisof concepts.
Inspired by ACE [23], we will use sub-regions of images to attempt to identify coherentconcepts.
Instead of using segmentation – which naturally introduces artifacts due to the inpaintingrequired by a baseline value –, we start by taking random crops of each image in our dataset (e.g, aset point of the same class) to form an auxilary dataset X ∈ R n × d such that X i =
τ ( x i ) with τ acrop function.
Given a layer l , we obtain the activations for the random crops A = h l ( X ) ∈ R n × p .
In the case where f is a convolutional neural network, a global average pooling is applied on theactivations.
We recall that all the elements of A are non-negative real numbers.
We are now ready to apply Non Negative Matrix Factorization (NMF) to decompose the positiveactivations A , into a product of non-negative, low rank matrices U ( A ) ∈ R n × r and W ∈ R p × r ,with:
[Equation]
Where || · || F denotes the Frobenius norm.
One of the appealing properties of NMF is the lowrank constraint r ≪ min ( n, p ) .
Simply put, NMF can be understood as the joint learning of W ,a dictionary of CAVs – “concept bank” in Figure 3 – that maps a R p basis onto R r , and U the coefficients of vectors A expressed in this new basis.
The minimization of the reconstruction error12 ∥
A − UW ∥ 2 F ensures that the new basis contains (mostly) relevant concepts.
Intuitively, thenon-negativity constraints U ≥ 0 , W ≥ 0 encourage (i) the sparsity of W (useful for creatingdisentangled concepts), (ii) the sparsity of U (convenient for selecting a minimal set of usefulconcepts) and (iii) the imputation of missing data [41], which corresponds to the sparsity patternof post-ReLU activations A .
We shall also note that each original activation A i coming from theinput x i can be approximated by its reconstruction h l ( τ ( x i ))
=
U i W T
=
(cid:80) rj
=1 U
i,j W Tj .
Thisapproach is attractive as each activation can be understood as a composition of concepts.
While other methods in the literature solve a similar problem (such as low rank factorization using
SVD or ICA), the NMF has stepped up as both fast, effective and is known to yield meaningfulconcepts to humans [42, 43, 24].
Finally, once the concept bank W is precomputed, we can associatethe concept coefficients U ( x ) to any new input x (e.g a full image) by solving the underlying Non-Negative Least Squares (NNLS) problem {+min U ≥ 0 
+}
∥
h l ( x ) − U ( x ) W T ∥ 2 F , and thereforehave its decomposition in the concept base.
In essence, the core of our method can be summarized as follows: using a set of images, we re-interprettheir embedding at a given layer l as a composition of concepts that can be easily understood byhumans.
In the next section we show how we can recursively apply concept activation factorizationson a layer l ′ <
l for an image containing a previously computed concept.
3.2 Ingredient 1: A Recursive Flavor
One of the most apparent issues in previous works [23, 24] is the choice of the layer at which theactivation maps are extracted.
Depending on this, certain concepts start getting amalgamated [44]into one, resulting in incoherent and indecipherable clusters, as illustrated in Fig 2.
We posit thatthis can be solved by iteratively applying our decomposition at different layer-depths, and for theconcepts that remain difficult to understand, look for their sub-concepts at earlier layers by isolatingthe images that contain them.
This allows us to build hierarchies of concepts for each class.
We offer a simple solution consisting of reapplying our method to a concept by performing a secondstep of Concept Activation Factorization on a set of points that contain the concept C in order torefine it and create sub-concepts (e.g., decompose C into {C 1 , C 2 , C 3 } ) see see Fig.2 for an illustrativeexample.
Note that we generalize current methods in the sense that taking points ( x 1 , ..., x n ) thatare clustered in the logits layer (belonging to the same class) and decomposing them in a previouslayer – as done in [23, 24] – is a valid recursive step.
For a more general case, let us assume thata set of points that contain a common concept is obtained using a first step of Concept Activation Factorization.
We then look for a set of points with a high coefficient for the concept of our choiceto perform the next factorization.
Formally, with a factorization for a layer l UW T and a conceptindex i , this set of points is defined as C = { τ ( x j ) : U ( A j ) i ≥ λ } In practice, we assume λ to beequal to the 90th percentile of the values of U i .
Given this new set of points, we can then re-apply the Concept Matrix Factorization method to a earlier layer l ′ – with l ′ <
l – to obtain the sub-concept’sdecomposition from the initial concept.
3.3 Ingredient 2: Sobol indices for enhanced concept importance estimation
A common concern with concept extraction methods is that what makes sense to humans is notnecessarily what is being used by the model to predict.
To avoid this kind of confirmation bias duringour concept analysis phase, we can estimate the global importance of the extracted concepts.
To doso, [22] proposed an estimator based on directional derivatives: the partial derivative of the modeloutput with respect to the concept vector.
While this measure is theoretically well founded, it relieson the same principle as gradient-based methods, and thus, suffers from the same pitfalls: neuralnetwork models have noisy gradients [5, 7].
Hence, the farther the chosen layer is from the output,the noisier the directional derivative score will be.
Since we essentially want to know which concept has the greatest effect on the output of the model,it is natural to consider the field of Sensitivity Analysis [45, 46, 32, 47].
In this section, we brieflyrecall the classical total Sobol indices and how to apply it to our problem.
The complete derivation ofthe Sobol-Hoeffding decomposition is presented in the appendix D.
Formally, we place ourselves at layer l and perform our Concept Activation Factorization, providing uswith U , W .
A natural way to estimate the importance of a concept
U
i is to measure the fluctuation ofthe model’s output h l,k ( UW T ) in response to meaningful perturbations of the concept
coefficient U i .
Concretely, with M =
( M 1 , ..., M r ) ∈ [0 , 1] r , here an i.i.d sequence of real-valued random variables,we introduce a concept fluctuation to reconstruct a perturbated activation ˜
A =
( U ⊙ M ) W T (e.g., the masks can be used to put a concept value to zero).
We can then propagate this perturbated activation to the model output Y =
h l,k ( ˜ A ) .
Thus, an important concept will have a large varianceon the model output while an unused concept will barely change it.
Finally, we can capture the importance that a concept might have as a main effect – along with itsinteractions with other concepts – on the model’s output by calculating the expected variance thatwould remain if all the indices of the masks except the M i were to be fixed.
This yields the generaldefinition of the Total Sobol indices.
Definition 3.1 ( Total Sobol indices ) .
The total Sobol index S T i , which measures the contributionof a concept U i as well as its interactions of any order with any other concepts to the model outputvariance, is given by:
[Equation]
In a practical way, this index can be calculated efficiently [48, 49, 50, 51, 52], more details on thesampling (Quasi-Monte Carlo) and the estimator used are left in appendix D.
3.4 Ingredient 3: Unlocking Concept Attribution Map
Attribution methods are useful for determining the regions deemed important by the model forthe decision, but they lack the information about what exactly triggered it.
We have seen that wecan already extract this information from the matrices U and W , but as it is, we cannot know towhich part of the image the model associates each concept, and thus, better comprehend the model’sdecisions.
In this section, we will show how we can unlock the set of attribution methods (forwardand backward mode) to find where a concept is located in the input image (see Fig.1).
Forwardattribution methods don’t rely on gradients and only use inference information, whereas backwardmethods require to back-propagate through the network’s layers.
By application of the chain rule,computing ∂ U ∂x requires access to ∂ U ∂ A .
To do so, it could be tempting to solve the linear system UW T =
A .
However, this problem isill-posed since W T is low rank.
A standard approach is to calculate the Moore-Penrose pseudoinverse ( W T ) + , which solves rank deficient systems by looking at the minimum norm solution [53]. In practice ( W T ) + is computed with the Singular Value Decomposition (SVD) of W T .
Unfortunately, SVD is also the solution to the unstructured minimization of 12 ∥ A − UW T ∥ 2 F by the
Eckart–Young–Mirsky theorem [54].
Hence, the non negativity constraints – i.e U ≥ 0 , W ≥ 0 – ofthe NMF are ignored, which prevents approaches based on solving U T W =
A T from succeeding.
Other issues stem from the fact that the U , W decomposition is generally not unique.
Our third contribution consists on tackling this problem to allow the use of attribution methods – i.e.
Concept Attribution Maps – by proposing a strategy to differentiate through the NMF layer.
Implicit differentiation of NMF layers The NMF problem 1 is NP-hard [55], and it is not convexwith respect to the input pair ( U , W ) .
However, fixing the value of one of the two factors and optimizing the other turns the NMF formulation into a pair of Non Negative Least Squares (NNLS)problems (see Equation 3), which are convex.
This ensures that alternating minimization (a standardapproach for NMF) of ( U , W ) factors will eventually reach a local (and global) minimum:
[Equation]
Each of the NNLS problem fulfills the KKT conditions[56, 57], which can be encoded in the so-called 255 optimality function F , see Equation 10 Appendix C.2.
The implicit function theorem [39] allows usto use implicit differentiation [38, 39, 58] to efficiently compute the Jacobians ∂ U ∂ A and ∂
W ∂
A withoutrequiring to back-propagate through each of the iterations of the NMF solver:
[Equation]
However, this requires the dual variables ¯ U and ¯ W , which are not computed by Scikit-learn’s
[59]popular implementation † .
Consequently, we leverage the work of [62] and we re-implement our ownsolver with Jaxopt [40] based on ADMM [63], a GPU friendly algorithm (see Appendix C.2).
We start by performing Concept Activations Factorization – i.e we precompute the concept bank Wby solving the NMF.
Concept Attribution Maps of a new input x are calculated by solving the NNLSproblem min U ≥ 0 12 ∥
h l ( x ) − UW T ∥ 2 F .
The implicit differentiation of NMF layer ∂ U ∂ A is integratedinto classical back-propagation to obtain ∂ U ∂x .
Most interestingly, this technical advance unlocks allwhite-box explainability methods [5, 6, 7, 8, 9, 12] to generate concept-wise attribution maps andtrace the part of the image that triggered the detection of the concept.
Additionally, it is even possibleto employ black-box methods [4, 13, 26, 14] since it only amounts to solving an NNLS problem.
Experimental evaluation
We used CRAFT to explain a ResNet50V2 trained on the ILSRVC2012 [27] data set (ImageNet).
Weselected a subset of 10 classes, each containing 1000 images (those recommended by ImageNette ‡ ).
In all of our experiments, r = 25 , like in [23] and the cropping function τ consists on randomlychoosing 10 square 64 × 64 patches for each image.
We start by qualitatively validating CRAFTby showing that: (1) the method yields more interpretable concepts than ACE (see Fig. 4), (2) thecombination of local and global explanations allows to explain complex failure cases otherwiseunexplainable with only the attribution methods (see Fig. 5).
Then, we validate independently the newingredients brought by the method by showing quantitatively that (3) recursivity allows us to refineconcepts, making them more meaningful to humans with the help of two psychophysics experiments, and (4) Sobol indices allow for a better estimation of concept importance.
Additional experiments,including a sanity check, a study of concept co-occurrence and an example of activation maximization(Deep dream) on the concept bank, as well as many more examples of local explanations for randomlypicked images from ILSVRC2012, are included in appendix B. We leave a discussion on the limitations of this method and on the broader impact in appendix A.
4.1 Example of CRAFT concepts
Figure 4 compares the examples of concepts found by CRAFT against those found by ACE [23] for 3classes of Imagenet.
For each class the concepts are ordered by importance (the highest being themost important).
ACE uses a clustering technique and TCAV to estimate importance, while CRAFTuses the method introduced in 3 and Sobol to estimate importance.
These examples illustrate oneof the weaknesses of ACE: the segmentation used can introduce biases through the baseline valueused [64, 10].
The concepts found by CRAFT seem distinct: (vault, cross, stained glass) for the Church class, (dumpster, truck door, two-wheeler) for the garbage truck, and (eyes, nose, fluffy ears)for the English Springer.
More examples can be found in the appendix.
4.2 Explaining complex failure cases
[Table]
One of the goals of explainability is toexplain the failure cases of the modelsstudied.
Figure 5 shows an exampleof an incorrect prediction: the modelin question – here still a ResNet50 –predicts ‘shovel’.
Moreover, the attribution method on the left – here RISE [13] – does not tell us muchexcept that the evidence for shovelseems to be located at the level of theground and the lower torso and legsof the astronaut.
With CRAFT, we can however study the concepts found by the model at theselocations.
There are two of them: the first concept in green, aims at the lunar ground and refers to therocks often seen next to shovels in the dataset.
The second concept in purple is aimed at the legs ofthe astronaut and refers to the legs of a person, often in a ski suit, which he takes for the astronaut’s.
4.3 Validation of Recursivity
To evaluate the meaningfulness of the extracted high-level concepts, we performed psychopshysicexperiments with human subjects, to whom we requested to answer a survey in two phases.
Further- more, we distinguished two different audiences: on the one hand, experts in machine learning, and onthe other hand, people with no particular knowledge in computer vision.
Both groups of participantswere volunteers and didn’t receive any monetary compensation.
Some examples of the developedinterface are available the appendix E.
Intruder detection experiment, we make users identify the intruder out of a series of five segmentsbelonging to a certain class, with the odd one being taken from a different concept but from the sameclass.
Now, we compare the results of this intruder detection with a concept (e.g., C 1 ) coming from alayer l and one of its sub-concepts (e.g., C 12 in Fig.2) extracted using our recursive method.
If theconcept (or sub-concept) is meaningful, then it should be easy for the users to find the intruder.
Table 1summarizes our results, showing that indeed both concepts and sub-concepts are meaningful, andthat recursivity can lead to a slightly higher understanding of the generated concepts (significant fornon-experts, not significant for experts) and might suggest a way to make concepts more interpretable.
Binary choice experiment, In order to test the meaningfulness of the sub-concept generatedwith recursivity, we showed participants a segment belonging to a subcluster and to the parentcluster (e.g., τ ( x ) ⊂ C 11 ⊂ C 1 ), and we asked which of the two clusters (i.e., C 11 or C 1 ) seemsthe more appropriate.
If our hypothesis is correct, then the concept refinement brought by recursion should help form more coherent clusters.
The results in Table 1 are satisfying, since inboth the expert and non-expert groups, the participants chose the sub-cluster by more than 74%.
We measure the significance of our results by fitting a binomial logistic regression to our data,and we find that both groups are more likely to choose the sub-concept cluster (at a p < 0 . 001 ).
100,000 images – shows that using Sobol indices yield to better estimates of important concepts.
4.4 Fidelity analysis
We propose to simultaneously verify thatthe concepts are faithful to the model andthat the concept importance estimator performs better than TCAV [22] by usingthe fidelity metrics introduced in [23, 24].
These metrics are similar to the one usedfor attribution methods, which consist onstudying the change of the logit score whenremoving/adding pixels considered important.
Nevertheless, we do not make thesemodifications in the pixel space but in theconcept space: once U , W are computed,we reconstruct the matrix A ≈ UW T using only the most important concept (or removing the most important concept for deletion), and studythe score in output of the model.
As can be seen from Fig. 6, ranking the extracted concepts using Sobol’s importance score results in much steeper curves than when they are sorted by their TCAVscores.
We confirm these results with other matrix factorization techniques (PCA, ICA, RCA) in the Appendix F.
Conclusion
In this paper, we introduced a method for automatically extracting human-scrutable concepts from
Deep Neural Network: CRAFT.
Our method allows to explain a pre-trained model both in a per-classand per-instance basis by highlighting both what the model saw when predicting the class labeland where it is located, which, as we have shown, exhibits complementary benefits.
The approachrelies on three novel ingredients: 1) exploiting the recursive nature of the feature extraction chains in
CNNs to find decompositions where each concept is clearly understandable; 2) measuring conceptimportance through Sobol indices to more accurately identify which concepts influence a model’sdecision for a given class; and 3) harnessing implicit differentiation to backpropagate through NMFblocks, thus enabling the use of any attribution method to create concept-wise local explanations thatwe call Concept Attribution Maps . Human experiments confirmed the validity of the approach andthat concepts identified by CRAFT are more meaningful.
We hope that this work will guide furtherefforts in the search for concept-based explainability methods and that further connections betweenlocal and global explanations will be made.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] In the appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] As a URL
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[N/A]
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[Yes] For the experiments comparing TCAV scores to our
concept importance score based on Sobol indices
(d) Did you include the total amount of compute and the type of resources used (e g., type
of GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No] The ILSVRC2012 dataset contain personally
identifiable information [65]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[Yes] Screenshot of the experiments are in the appendix
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[Yes]