A TTENTION - BASED I NTERPRETABILITY WITH C ONCEPT T RANSFORMERS
A BSTRACT
Attention is a mechanism that has been instrumental in driving remarkable performance gains of deep neural network models in a host of visual, NLP and multimodal tasks.
One additional notable aspect of attention is that it conveniently exposes the “reasoning” behind each particular output generated by the model.
Speciﬁcally, attention scores over input regions or intermediate features have been interpreted as a measure of the contribution of the attended element to the model inference.
While the debate in regard to the interpretability of attention is still not settled, researchers have pointed out the existence of architectures and scenarios that afford a meaningful interpretation of the attention mechanism.
Here we propose the generalization of attention from low-level input features to high-level concepts as a mechanism to ensure the interpretability of attention scores within a given application domain.
In particular, we design the ConceptTransformer, a deep learning module that exposes explanations of the output of a model in which it is embedded in terms of attention over user-deﬁned high-level concepts.
Such explanations are plausible (i.e. convincing to the human user) and faithful (i.e. truly reﬂective of the reasoning process of the model).
Plausibility of such explanations is obtained by construction by training the attention heads to conform with known relations between inputs, concepts and outputs dictated by domain knowledge.
Faithfulness is achieved by design by enforcing a linear relation between the transformer value vectors that represent the concepts and their contribution to the classiﬁcation log-probabilities.
We validate our ConceptTransformer module on established explainability benchmarks and show how it can be used to infuse domain knowledge into classiﬁers to improve accuracy, and conversely to extract concept-based explanations of classiﬁcation outputs.
Code to reproduce our results is available at: https://github.com/ibm/concept_transformer .
I NTRODUCTION
The spectacular gains in accuracy of recent large-scale machine learning models like deep neural networks have generally come at the cost of a loss of transparency into their functioning.
This “black box” aspect severely limits their applicability in safety-critical domains, such as medical diagnostics, healthcare, public infrastructure safety, visual inspection for civil engineering, to name just a few, where it is essential for decisions to be corroborated by robust domain-relevant knowledge.
In recent years, approaches focusing on explaining black box models have emerged, mostly with the goal of providing post-hoc explanations in terms of a set of relevant features used by the underlying model to make predictions (Ribeiro et al., 2016; Selvaraju et al., 2017; Lundberg & Lee, 2017; Smilkov et al., 2017).
While widely used, such post-hoc explainability methods have been criticized for operating on low-level features such as pixel values, or sensory signals that are combined in unintelligible ways and do not correspond to high-level concepts that humans easily understand (Kim et al., 2018;
Alvarez-Melis & Jaakkola, 2018; Kindermans et al., 2019; Su et al., 2019).
To overcome these limitations and sidestep the potential perils resulting from a misuse of posthoc explainability of black box models, some researchers have been vocally advocating for the use of inherently interpretable models (Rudin, 2019) that in particular would generate decisions based on human-understandable categories (i.e., concepts) grounded in domain expertise rather than raw features (Barbiero et al., 2021;
Ghorbani et al., 2019; Kim et al., 2018;
Yeh et al., 2020; Koh et al., 2020; Goyal et al., 2019; Kazhdan et al., 2020; Chen et al., 2020; Alvarez-Melis & Jaakkola, 2018;
Li et al., 2018; Chen et al., 2019).
For example, to identify a bird species, a model should focus on morphologically meaningful concepts, such as the shape, size and colors of beak, feathers or wings, rather than focusing on raw pixels, and combine them in ways that a domain expert (in this case an ornithologist) would reckon as intelligible to produce a classiﬁcation.
In addition, using high-level concepts emulates a human’s thinking process (i.e., structured into familiar concepts) and provides insights into the model’s reasoning in a human-understandable way.
The chasm between post-hoc explainability vs. inherently interpretable models closely reﬂects a related ongoing discussion in the NLP community on the interpretation of attention mechanisms (Bahdanau et al., 2014), and in particular on the interpretability of attention weights over input tokens, with researchers on one end of the debate claiming that attention provides interpretability, while others claim that “Attention is not explanation” (Jain & Wallace, 2019).
While the debate over what degree of interpretability that can be ascribed to attention weights is still not settled (Wiegreffe & Pinter, 2019), it is arguable that in many situations attention is not a “fail-safe indicator” (Serrano & Smith, 2019), particularly when decisions rely on the interaction of multiple interacting tokens as is typically the case in deep architectures.
Conversely then, a way to guarantee the interpretability of attention weights would be to make sure that they are not being processed by downstream operations that renders their relation to the decision outputs uninterpretable.
This is indeed something that had been proposed in the past, in particular in architectures that preserve the interpretability of “relevance scores” (akin to attention weights) by acting on them only through a restricted class of intelligible “aggregation functions” such as additive models (Alvarez-Melis & Jaakkola, 2018), which are a common functional elements in interpretable and white-box models (Caruana et al., 2015).
In this paper, we propose the ConceptTransformer (CT) , a transformer-based module (Vaswani et al., 2017) for classiﬁcation tasks, that can be used to enhance an arbitrary deep learning classiﬁer with domain knowledge in the form of plausible cross-attention weights between input features and highlevel interpretable concepts.
The CT can be used as a drop-in replacement for the classiﬁer head of any deep learning architecture.
The resulting model can then be trained end-to-end without any additional overhead on the training pipeline, except a modiﬁcation of the loss function that enforces plausibility of the explanation.
Importantly, the CT was speciﬁcally conceived to provide explanations that guarantee faithfulness by design and plausibility by construction .
Faithfulness is deﬁned as the degree to which the explanation reﬂects the decision and aims therefore to ensure that the explanations are indeed explaining the model’s operation (Lakkaraju et al., 2019; Guidotti et al., 2018).
In our model this is achieved by enforcing a linear relation between the transformer value vectors that represent the concepts and their contribution to the classiﬁcation log-probabilities.
Plausibility refers to how convincing the interpretation is to humans (Guidotti et al., 2018; Carvalho et al., 2019).
In the CT architecture, plausibility is achieved by construction by supervising the attention heads of the cross-attention mechanism to conform with inputs-concepts-outputs relations derived by domain knowledge.
We validate our approach on three image benchmark datasets, MNIST Even/Odd (Barbiero et al., 2021), CUB-200-2011 (Welinder et al., 2010), and aPY (Farhadi et al., 2009).
On these datasets we will examine how the faithfulness and plausibility of the CT explanations are practically translated into domain-relevant explanations behind particular output decisions or diagnostic insights about ensuing wrong classiﬁcations.
We will also quantify the beneﬁt of domain-expert knowledge in terms of statistical efﬁciency by showing that providing domain-relevant explanations to our CT model tends to improve the performance of the downstream classiﬁcation, in particular in low-data regime.
This for instance translates in an 8-9% improvement in accuracy on the bird classiﬁcation
CUB-200-2011 dataset when CT is trained in conjunction with part location annotations.
We note in addition that one of the strengths of our CT model is its versatility that allows it to be effortlessly applied to other data modalities as well by combining it with deep learning classiﬁers which are then rendered interpretable with no appreciable overhead or change in their training pipeline.
This is in stark contrast to other inherently interpretable models that are often speciﬁcally designed for the domain at hand, and require adhoc multi-stage training procedures.
CT on the other hand, is differentiable and can be ﬂexibly included in any end-to-end training pipeline that uses backpropagation, as we will showcase by combining it with a host of different deep learn- ing backbones ranging from convolutional architectures like Residual Networks (He et al., 2016) to more modern Vision Transformer (Dosovitskiy et al., 2020) and hybrid Compact Convolutional
Transformer models (Hassani et al., 2021).
R ELATED W ORK
In recent years, there have been signiﬁcant advancements towards designing interpretable models that quantify the importance of individual features with respect to the prediction output.
One general approach is post-hoc analysis, in which one interprets a trained model by ﬁtting explanations to the classiﬁcation outputs (Alvarez-Melis & Jaakkola, 2018;
Ribeiro et al., 2016; Lundberg & Lee, 2017).
In particular for CNNs, popular techniques are activation maximization (van den Oord et al., 2016;
Nguyen et al., 2016; Yosinski et al., 2015) and saliency visualization (Selvaraju et al., 2017; Smilkov et al., 2017; Sundararajan et al., 2017).
However, these post-hoc methods do not actually explain how the underlying model reached a particular classiﬁcation outcome.
In contrast, attention-based interpretable techniques aim to expose which parts of a given input a network focuses on, and therefore deems to be most relevant, when making a decision.
Examples of attention models are Zhang et al.
; Zhou et al.
Zheng et al.
Fu et al.
(2017); Girshick (2015); Girshick et al.
; Huang et al.
The problem with these models is that they focus on low-level individual features when providing an explanation.
Such features are often not intuitive for humans, are typically noisy and non-robust, or can be misleading when interpreted afterwards (Kim et al., 2018).
One of the recent advancements in the ﬁeld of interpretability was to design methods that explain predictions with high-level human understandable concepts (Ghorbani et al., 2019;
Kim et al., 2018;
Yeh et al., 2020;
Koh et al., 2020; Goyal et al., 2019; Kazhdan et al., 2020; Chen et al., 2020;
Barbiero et al., 2021; Li et al., 2018; Chen et al., 2019) – either by identifying common activation patterns in the last nodes of the neural network corresponding to human understandable categories or constraining the network to learn such concepts.
For instance, TCAV (Kim et al., 2018) proposes to deﬁne concepts from user-annotated examples in which concepts appear.
Others propose prototypesbased explanation models (Li et al., 2018; Chen et al., 2019), but they typically require specialized convolutional architectures to ensure feature extraction.
In particular, ProtoPNet (Li et al., 2018) uses previously learned prototypes to focus attention on various parts of an image.
This architectural design implies that object-level (global) concepts cannot be easily incorporated, and since prototypes are not learned together with the attention model, explanations based on these prototypes may lack faithfulness.
SENN (Alvarez-Melis & Jaakkola, 2018) proposes a network that transforms inputs into interpretable basic features, generates concept relevance scores and then aggregates concepts with relevance scores to explain predictions.
While it is out-of-the-box interpretable, it lacks concept localization.
Barbiero et al.
(2021) proposed a differentiable approach that allows the extraction of logic explanations from neural network models using First-Order Logic.
The approach relies on an entropy-based criterion to automatically identify the most relevant concepts that have contributed to a particular classiﬁcation output.
In our approach, high-level concepts are deﬁned with a set of related dimensions, and can be partspeciﬁc or global.
Such concepts are typically readily available in many domains and can be used to enhance the performance of the learning task while offering explainability at no additional cost for the network.
Obtained explanations are plausible and guaranteed to be faithful, since concepts participate in the model computation.
Finally, CT also allows in some cases to discover the presence concepts that were not annotated.
A PPROACH
The ConceptTransformer module.
The ConceptTransformer (CT) is a transformer-based module designed to be used as classiﬁer head in a deep learning architecture that generates classiﬁcation outputs using cross-attention (Vaswani et al., 2017) between input features and a set of embeddings representing domain-relevant concepts.
Fig.
1 shows the case where the inputs to the CT are embeddings of P visual patches of an input image that are linearly projected and concatenated into the query matrix Q ∈ R P ×
d m of a query-key-value cross-attention mechanism whose corresponding key matrix K ∈ R C ×
d m is the linearly projected concatenation of the embeddings representing the C concepts.
In addition, the concepts are linearly projected with a value projection matrix and concatenated to result in the value matrix V ∈ R C × d m .
Cross-attention then outputs an attention weight
[Equation]
with p = 1 , . . .
, P, c = 1 , . . .
, C, between each patch-concept pair, which are combined into an attention map matrix A =
[ α pc ] ∈
R P × C .
The ﬁnal output of the CT is the product obtained by multiplying the attention map A , the value matrix V and an output matrix O ∈ R d m × n c that projects onto the (unnormalized) n c logits over the output classes, and averaging over patches:
[Equation]
Notice that here for simplicity we described a single-head attention model, but in our experiments we will be using a multi-head version (Vaswani et al., 2017).
Equation 1 says that, given an input x to the network, the conditional probability of output class i is Pr
[Equation]
with β c with components ( β c ) i =
[ V O ] [Equation] [Equation] and γ c ( x ) are positive relevance scores that depend on x through the averaged attention weights: γ c ( x ) = 1 P (cid:80)
Pp =1 α pc .
The output of the CT is therefore essentially a simple multinomial logistic regression model over positive variables γ c ( x ) that measures the contribution of each concept.
Notice that this result follows from the linear relation between the value vectors and the classiﬁcation logits, which itself comes from the design choices of computing outputs from the value matrix V through the linear projection V O , and aggregating patch contributions by averaging.
[Figure]
outputs ...
C r o ss
A tt e n t i o n
A v e r a g e
Q
Q
Q conceptembeddings
ConceptTransformer
K V ...
Q ...
1 21 21 21 2... 1 2patchembeddings
Faithful concept-based explanations by design.
The CT was conceived as a drop-in replacement for the classiﬁer head of an arbitrary deep learning classiﬁer to provide concept-based explanations of the outputs that are guaranteed to be faithful by design .
We formalize this statement as follows:
Proposition 1 Each concept relevance score γ c ( x ) in Equation 2 is a faithful explanation of the output.
More speciﬁcally, the probability of choosing the preferred output i c = arg max i ( β c ) i of concept c (assuming it’s unique) is guaranteed to decrease if γ c ( x ) is externally set to zero.
Moreover, the correlation between γ c ( x ) and Pr ( i c | x ) is strictly positive.
Proof Proof of Proposition 1 is provided in Appendix A.
Note that the last statement in the Proposition above is a corollary of the ﬁrst one, and it speciﬁcally shows that CT is guaranteed to satisfy the technical deﬁnitions of faithfulness given for instance by
Alvarez-Melis & Jaakkola (2018).
Training and plausibility by construction.
As mentioned, the CT is a differentiable transformerbased module that can be embedded in a deep learning architecture trained end-to-end with backpropagation.
In addition, the fact that it exposes attention weights over concept tokens that can be user deﬁned gives us the freedom to shape these attention weights according domain-expert knowledge relevant for the problem under consideration.
This can be done by explicitly guiding the attention heads to attend to concepts in the input that are a priori known to be informative to correctly classify the input.
In practice this can be achieved by supervising the attention weights at training as for instance proposed by Deshpande & Narasimhan (2020) as a self-supervised technique for bidirectional language models.
In particular, given a desired distribution of attention H provided by domain knowledge (e.g., we know which patches in the input image contain which concepts that are relevant to classify the input) we can force the CT attention weights A by adding an “explanation cost” term to the loss function that is proportional to L expl =
||
A − H || 2 F , where || · || F is the
Frobenius norm.
The ﬁnal loss used to train the architecture then becomes L
=
L cls + λ L expl , where L cls denotes the original classiﬁcation loss, L expl the additional explanation loss, and the constant λ ≥ 0 controls the relative contribution of the explanation loss to the total loss.
Notice that setting λ = 0 essentially amounts to just minimizing the classiﬁcation loss and disregarding the prior domain knowledge as imparted into CT by guiding the attention heads.
R ESULTS
We have evaluated the proposed approach on three datasets: MNIST Even/Odd (mni), CUB-2002011 (Welinder et al., 2010) and aPY (Attribute Pascal and Yahoo)
(Farhadi et al., 2009).
Each dataset illustrates a slightly different use case and different visual backbone in combination with the
CT.
This variety is meant to showcase the ﬂexibility and versatility of the CT module.
With MNIST
Even/Odd we consider a case where the correspondence between concepts and output classes is many-to-one and deterministic.
In addition, concepts are global as opposed to being spatially localized.
That means that we will not use a patch-based representation of the input, and instead the concepts refer to the whole input, which is essentially equivalent to having only one patch.
For this task, CT is combined with a small Compact Convolutional Transformer architecture (Hassani et al., 2021).
In the case of CUB-200-2011 we consider a case where the relation between concepts and outputs is instead many-to-many and non-deterministic, and there exists a mixture of global and spatially localized concepts.
In order to handle both types of concepts we instantiate two CTs, one for spatial concepts and one for global concepts and we then average the logits provided by both in order to preserve interpretability (as explained in the previous section).
For this dataset, we use a Vision
Transformer as a backbone.
As inputs to the CT handling the spatial concepts we use the embeddings of the tokenized image patches, while as input to the CT in charge of the global concepts we use the embedding of the CLS token.
Finally, with the aPY (Attribute Pascal and Yahoo) dataset we consider another situation where concepts are exclusively global but this time are many-to-many.
The architecture of the visual backbone used on this dataset is a larger version of the Compact Convolutional Transformer architecture used on MNIST Even/Odd, but with a larger tokenizer consisting of a ResNet50 model pre-trained on
ImageNet.
4. E VALUATION ON MNIST E VEN /O DD
The objective of this evaluation is simply to illustrate how our architecture works.
We perform a simple binary classiﬁcation task, based on the MNIST
Even/Odd dataset, aiming at classifying 28x28 pixel images of hand-written digits ranging from 0 to 9 as either ‘even’ or ‘odd’.
In this case, we exploit the fact that we know the identity of each digit and use that as an explanation for the binary classiﬁcation prediction.
In other words, for instance a ‘7’ should be classiﬁed as ‘odd’, and a plausible explanation to support this prediction is that it is ‘odd’ because it is a ‘7’ .
Figure 2 shows the accuracy on the test set (left) and explanation loss during validation (right), relative to the number of samples used at training, which varies from 100 to 7000.
With the largest number of training samples, the CT achieves 99% accuracy, irrespective of whether concepts are leveraged ( λ = 2 . 0 ) or not ( λ = 0 . 0 ).
However, when the model is trained with signiﬁcantly fewer samples (e.g., 100 - 500), using concepts provides a performance boost, in the range of 5%.
Accordingly, we observe a more graceful decay of the explanation loss when λ = 2 . 0 .
[Figure]
Figure 3 (left) shows a test sample, a ‘7’ whose correct binary label is ‘odd’, which is itself a prediction that should be supported by the correct ground-truth explanation ‘7’ (i.e., “ the sample is ‘odd’, because it is a ‘7’ ”).
CT in this case outputs the correct prediction, and looking at the concept attention weights we can see that indeed this prediction is supported by the correct concept.
[Figure]
Figure 3 (right) on the other hand shows the example of a test sample which is being misclassiﬁed by CT: the sample (an ‘8’) is supposed to be classiﬁed as an even digit, but is instead classiﬁed as an odd number.
If we look at the concept attention weights for this sample we see that the reason for the incorrect prediction can be traced back to the fact that the CT strongly associated the sample to the ‘9’ concept and hence predicted the sample to be ‘odd’ because it “thought” that it was a ‘9’.
Notice that also the correct concept ‘8’ is being attended to by the architecture, but the wrong concept ‘9’ received in this case a higher attention score.
Interestingly, visually inspecting the sample it is not far from being a ’9’.
4. E VALUATION ON CUB-200-
The CUB-200-2011 (Welinder et al., 2010) dataset contains 11788 images of birds, classiﬁed inspecies.
Each image is annotated with a given number of concepts (e.g., shape of the beak, color of the back, etc.)
explaining the visual characteristics of the bird in the image.
It uses 312 concepts, but their distribution varies across images.
We therefore pre-process the dataset and retain concepts that are sufﬁciently representative in class-level annotations.
Speciﬁcally, for a given class, we retain only concepts that are present in at least 45% of the images of that class and subsequently present in at least 9 classes.
This led us to retain 108 concepts.
Table 1 compares our CT against other deep models (top models in bold), classiﬁed based on their training procedure in Multi-Stage (i.e., complex training) and End-to-end (i.e., training with backpropagation).
When concepts are leveraged to train the CT, we notice a signiﬁcant boost in accuracy of 8.2% (i.e., CT [w/o] vs. CT).
Therefore, not only does the CT provide explainability by design, but it also boosts the overall performance of the classiﬁcation task.
In future, we plan to examine ways of modeling and taking advantage of the relation between extracted concepts.
This has been shown to further boost performance by Barbiero et al.
(2021) by extracting First Order Logic clauses that relate concepts.
The test accuracy of our model is on par with the baseline, non-interpretable model (B-CNN) as well as with ProtoPNet, PN-CNN, SPDA-CNN, MA-CNN and RA-CNN, even though some of these techniques require signiﬁcantly more complex training than the CT.
In terms of the type of
[Table]
explanations provided, at the coarsest level, CAM highlights the entire object as the reason behind a classiﬁcation decision.
At a ﬁner level, most models offer part-level attention, however they differ in how attention is generated.
For example, Part R-CNN, SPDA-CNN, PS-CNN, PN-CNN use additional part-localization models previously trained with part annotations.
RA-CNN uses an additional neural network to decide where to focus attention, whereas MA-CNN uses convolutional feature maps to direct attention to various parts of the image.
ProtoPNet generates attention based on previously learned prototypes, namely it focuses on a speciﬁc region of the input image because the region is similar to a learned prototypical example.
This leads to two main drawbacks.
First, objectlevel concepts cannot be naturally incorporated, since the approach is based speciﬁcally on part-level concepts.
Second, ﬁnding learned prototypes for other models requires post-hoc analysis, speciﬁcally inspecting which region mostly activates a model’s convolutional ﬁlter.
Since such prototypes are not computed together with the model, resulting explanations cannot be always guaranteed to be faithful to the classiﬁer’s decisions.
In contrast, the CT makes use of both part-level ( spatial ) and object-level ( global ) concepts to generate explanations, as shown in Fig.
We highlight concepts with the highest attention scores and hence most relevant for the classiﬁer for four correctly predicted bird species.
In addition, the model guarantees faithful explanations by learning concepts together with the classiﬁcation model.
Specifically, this is achieved by enforcing an additive relation between the transformer value vectors that represent the concepts and their contribution to the classiﬁcation log-probabilities.
We show that explanations mimic the reasoning of the classiﬁer in Fig.
5 for one misdiagnosed bird species (left image).
The CT predicts the species to be Summer Tanager, when in fact the bird is a Scarlet Tanager.
If we look at the concept attention scores for this sample, we notice that there are two reasons for the incorrect prediction.
First, relevant parts of the bird such as upper parts color :: black and has primary color :: black are not fully visible in the left most image.
Both concepts are extremely important to ensure a correct classiﬁcation as Scarlet Tanager, as seen in the middle image.
Second, attention is not attended to the has wing color :: black concept (missing), but instead to the has shape :: perching − like concept, which is typical for the Summer Tanager species.
4. E VALUATION ON A PY (A TTRIBUTE P ASCAL AND Y AHOO )
The aPY (Attribute Pascal and Yahoo) dataset introduced by Farhadi et al.
(2009) consists of two subsets.
The aPascal dataset was created for the Visual Object Classes Challenge 2008 by Everingham et al.
and consists of 6340 training samples and 6355 test samples.
aYahoo is a dataset developed by Farhadi et al.
(2009) using Yahoo image search and comprises 2644 samples, which are typically used for testing only.
Each sample image in either dataset contains one or more objects which are enclosed by a bounding box and labeled with a single class.
In total there are 32 classes describing everyday objects (in the form of nouns) such as animals, vehicle types, or common items found in apartments.
In addition, for most of the objects there is a set of associated attributes, denoting present object parts or characteristics, e.g., “tail”, “hair”, “door”, or “screen”.
The number of attributes is 64, with an average of around 20 present attributes per object while the minimum number of attributes is 2 and the maximum is 43.
In contrast to the CUB dataset, there is no notion of object parts and associated attributes, i.e. the attributes are referring to the entire object (“global attributes”).
Furthermore, the combined dataset of aPascal and aYahoo contains 989 objects without any attribute associated.
We pruned these cases for our experiments so that 14350 objects remain.
A cropped image was created for each bounding box in the samples of the dataset, yielding one image per object.
At training time, the following augmentations were applied to the individual object samples: resizing to a standardized format ( H × W = 320 × 320 pixels), random horizontal ﬂipping with probability p = 0 .
5 , random rotations in the range of ± 15 ° based on an uniform probability distribution and normalization.
1 For validation and testing, only resizing and normalization were applied.
We evaluated the aPascal as well as the combined aPY dataset with varying regularization parameter values applied to the explanation loss.
Table 2 shows the negative log likelihood loss for the classiﬁcation, the mean squared error (MSE) loss for the concepts (explanation loss) and accuracy ﬁgures for the aPascal and the aPY datasets.
The accuracy numbers are in a narrow range, implying that the concept regularization does not seem to signiﬁcantly affect the classiﬁcation accuracy.
On the other hand, the decrease of the concept-related MSE loss for increasing λ values by approximately one order of magnitude suggests, that the labeled concepts are indeed learned by the model.
The experiments were conducted with six different data-seeds and it was conﬁrmed that the results are generalizable.
[Table]
Evaluations on the combined aPY dataset appear to be focused on speciﬁc applications like zero-shot learning and classifying attributes in the literature
[Figure]
which renders this dataset difﬁcult to baseline.
Still
[Figure]
Wang & Ji (2013) report 63.02% overall class prediction accuracy on the aPascal dataset, while (Mahajan et al., 2011) reached an accuracy of 67 .
33% on the same dataset
[Figure]
Figure 6 shows samples of correctly and incorrectly classiﬁed objects and the activated concepts (i.e., above a given threshold) for the aPY dataset.
Notably, our model is able to discover meaningful and obviously correct concepts that were not annotated in the ground truth, e.g., the “wheel”, “door” and “shiny” attributes that pertain to the ﬁrst image with the car, or the “snout” in the image with the dog.
The incorrectly classiﬁed objects were associated with concepts that seem, to a certain extent, understandable.
For example, the monkey (predicted as dog) or the bottle (predicted as person).
In the latter case, we ﬁnd that the classiﬁcation is consistent with the concepts that were discovered, i.e., if “head”, “torso”, “arm”, etc. are detected, it is coherent that the corresponding classiﬁcation is “person”.
2 It also shows that the concepts allow a person to better understand how the model arrived at the misclassiﬁcation as well as more easily to identify it as such.
We believe that the CT approach can be extended for applications in concept discovery and zero-shot learning.
C ONCLUSIONS
In this work, we generalize the notion of attention from low-level features to high-level concepts to ensure an interpretability of attention scores that agrees with a human’s reasoning for classiﬁcation tasks.
We propose the ConceptTransformer, a novel deep learning module that accommodates this form of interpretability and show on three benchmark image datasets that the explanations obtained are plausible and faithful.
Our architecture achieves performance that matches the state-of-the-art techniques, while being more versatile, signiﬁcantly less complex and easier to train.
R EPRODUCIBILITY
We provide code for the model implementation in the Appendix.
Upon acceptance of the paper, we plan to release the full code for the experiments that we reported in a public repository.
The datasets used for evaluation of our approach are public benchmarks, therefore already available in the community.
A A PPENDIX : A DDITIONAL DETAILS P ROOF OF P ROPOSITION
Let us ﬁx a speciﬁc c and let us denote Pr
( i c | x ) in Equation 2 after we externally set γ c ( x ) > 0 to zero with Pr( i c | x ) | γ c =0 .
We want to calculate the difference:
[Equation]
(cid:48) (cid:17) − exp (cid:16)(cid:80) c (cid:48) (cid:54) = c ( β c (cid:48) ) i c γ c (cid:48) (cid:17) (cid:80) i exp (cid:16)(cid:80) c (cid:48)
(cid:54) =
c ( β c (cid:48) ) i γ c (cid:48) (cid:17) = 0 , where the inequality comes from the assumption that the preferred output i c = arg max i ( β c ) i of concept c is unique, i.e. exp(( β c )
i c ) > exp(( β c ) i ) , ∀ i (cid:54)
=
i c .
This proves that setting c to zero causes a non-zero drop in Pr( i c | x ) .
The same type of inequality more in general proves that increasing/decreasing 0 < γ c ( x ) <increases/decreases Pr( i c | x ) , which proves the second part of Proposition 1 that γ c ( x ) is strictly positively correlated with Pr( i c | x ) .
B A PPENDIX : A DDITIONAL FIGURES
Artiﬁcially changing the color of a Cardinal from red to blue, causes an attention shifts from the concept has forhead color::red to has forhead color: blue (among others).
Correspondingly, the bird that was classiﬁed as a Cardigan is then being classiﬁed as Indigo Bunting .
[Figure]
C A PPENDIX : C ONCEPT T RANSFORMER PYTORCH CODE
[Table]
[Table] [Table] [Table]
Processes spatial and
[Table]
concepts in parallel andaggregates the log-probabilities
[Table]
the end
[Table]
[Table]
[Table]
[Table]
,n [Table] [Table] n
[Table]
[Table]
[Table]
[Table]
self
[Table]
[Table]
(torch.
[Table]
concepts
[Table]
dim), requires
[Table]
)nn
[Table]
[Table]
concepts
[Table]
/ math. [Table] [Table] if n_concepts > 0:self.concept_tranformer = CrossAttention(dim=embedding_dim,n_outputs=num_classes,num_heads=num_heads,attention_dropout=
attention_dropout,projection_dropout= projection_dropout,)
[Table]
for non-spatialif n_concepts > 0: self.token_attention_pool = nn.Linear(embedding_dim, 1) # [Table] Conceptsself
[Table]
_conceptsself.
[Table]
concepts
[Table]
.Parameter(torch.
[Table]
[Table]
[Table]
[Table]
nn.init.trunc_normal_(self.spatial_concepts, std=1.0 / math.sqrt(embedding_dim))if n_spatial_concepts > 0:self.spatial_concept_tranformer = CrossAttention(dim=embedding_dim,n_outputs=num_classes,num_heads=num_heads,attention_dropout=
attention_dropout,projection_dropout=
projection_dropout,) def forward(self, x):concept_attn, spatial_concept_attn
=
None, None out = 0if self.n_concepts > 0:token_attn = F.softmax(self.token_attention_pool(x), dim=1).transpose(-1, -2)x_pooled =
torch.matmul(token_attn, x) if self.n_concepts > 0: # Non-spatial streamout_n, concept_attn = self.concept_tranformer(x_pooled, self.concepts)concept_attn = concept_attn.mean(1)
# average over headsout = out + out_n.squeeze(1)
# squeeze token dimension if self.n_spatial_concepts > 0:
# Spatial streamout_s, spatial_concept_attn = self.spatial_concept_tranformer(x, self.spatial_concepts)spatial_concept_attn = spatial_concept_attn.mean(1) # average over headsout = out + out_s.mean(1)
# pool tokens sequence return out, concept_attn, spatial_concept_attn