Bounded logit attention: Learning to explain image classiﬁers
Abstract
Explainable artiﬁcial intelligence attempts to elucidate the workings of systems too complex to be directly accessible to human cognition through suitable sideinformation referred to as “explanations”.
We present a trainable explanation module for convolutional image classiﬁers we call bounded logit attention (BLA).
The BLA module learns to select a subset of the convolutional feature map for each input instance, which then serves as an explanation for the classiﬁer’s prediction.
BLA overcomes several limitations of the instancewise feature selection method “learning to explain” (L2X)
(Chen et al., 2018): 1) BLA scales to real-world sized image classiﬁcation problems, and 2) BLA offers a canonical way to learn explanations of variable size.
Due to its modularity BLA lends itself to transfer learning setups and can also be employed in a post-hoc fashion.
Beyond explainability, BLA may serve as a general purpose method for differentiable approximation of subset selection.
In a user study we ﬁnd that BLA explanations are preferred over explanations generated by L2X and the popular (Grad-)CAM method.
Introduction
A commonly quoted rule of thumb is that as machine learning systems increase in size and sophistication, it becomes increasingly hard to understand how these systems arrive at their predictions.
That is, an increase of performance is paid for by a lack of interpretability.
Uninterpretable, “blackbox” systems are undesirable for a multitude of reasons, both from a usability and engineering perspective.
In this work we deal with the explanation of predictions made by image classiﬁcation models based on deep convolutional neural networks.
Much work in this area is focused on generating explanations post-hoc .
This means that after building and training a model, an additional mechanism is put in place to produce explanations for the model’s behavior.
An explanation could be any additional humaninterpretable information suitable to improve the understanding of the workings of (some aspects of) a model.
In the ﬁeld of image classiﬁcation such information could be visual, e.g. highlighting or masking certain parts of an input image.
Considerably less popular than post-hoc methods are trainable explanation mechanisms incorporated directly into a model.
A drawback of this approach is that it cannot a priori be ruled out that such mechanisms could negatively affect the model’s performance.
However, building explainability mechanisms directly into a model as part of its architecture is appealing insofar as once trained such explanations provide by construction true, accurate insight into the workings of (individual aspects of) the model.
This is in contrast to post-hoc explanations, where one always has to question how faithful these explanations are to actual calculations performed by the model (Hooker et al., 2019).
Contributions.
We propose bounded logit attention (BLA), a trainable, modular explanation mechanism to be incorporated into convolutional image classiﬁcation networks.
The BLA module learns to select a subset of the convolutional feature map for each input instance, which then serves as an explanation for the classiﬁer’s prediction.
Relative to previous work on learned explanations by Chen et al.
our key contributions are: 1) BLA scales to real-world sized convolutional neural networks.
2) BLA offers a canonical way to produce explanations of variable size.
One of the merits of our method is that due to its modularity it can be used with typical transfer learning setups in computer vision, in particular pretrained feature extractors.
Similarly, our method may even be employed as a purely post-hoc method.
Beyond explainability, BLA can be used as a general purpose method for differentiable approximation of subset selection.
In our experiments we obtain favorable results according to both quantitative metrics and human evaluation.
Related work
There is no consensus on the deﬁnitions of notions such as interpretability, explainability , etc.
and we use them loosely throughout this work.
Roughly by interpretable we mean “accessible to human understanding” and and by explanation we mean “any piece of information that facilitates interpretability”.
Lipton (2018) investigates notions related to interpretability in a principled way.
In his taxonomy, the opposite of “blackbox-ness” is transparency .
In this sense, a built-in explanation mechanism like ours makes a model (partially) transparent.
Explanations in computer vision.
Linear models are easy to interpret, as long as they use a relatively small number of (interpretable) features.
As a consequence, many approaches to interpretability involve the construction of a linear surrogate model of some sort.
End-to-end differentiable models can be linearized around an input.
In the ﬁeld of computer vision this approach is known as saliency maps (Simonyan et al., 2013) which present gradient information graphically.
Since the gradients of large image classiﬁcation networks are noisy, averaging gradients over some neighborhood has been proposed by Sundararajan et al.
Smilkov et al.
Another idea is decomposing input images into coarser, more interpretable features, such as superpixels (cohesive segments of similar pixels).
Then, a neighborhood of this input is deﬁned, consisting of all images with any of the interpretable features either “present” or “absent” (corresponding to occluding the “absent” superpixels with black color).
LIME (Ribeiro et al., 2016) builds a local approximation of the model for this neighborhood.
Similarly, one may use the Shapley value (Shapley, 1953) of interpretable features when constructing explanations.
It turns out that Shapley values also ﬁt into the LIME framework (Charnes et al., 1988) as pointed out by Lundberg and Lee (2017) (there called SHAP).
Other perturbation-based methods, e.g. introduced by Fong and Vedaldi (2017);
Qi et al.
(2019), estimate the importance of input features by (partially) occluding the input and measuring the effect on the model output without employing a additional learning component.
Zhou et al.
(2016) propose class activation mappings (CAM) which assigns saliency to convolutional feature maps based on the coefﬁcients of a ﬁnal dense linear layer.
Selvaraju et al.
(2017) generalize this approach to Grad-CAM where in the case that more than one layer is used to process the convolutional features they use a gradient-based linear approximation of this calculation.
Examplebased methods (Koh and Liang, 2017; Chen et al., 2019; Hase et al., 2019) provide explanations that relate predictions to the training data.
User studies evaluating some of the explanation methods described in this section were conducted by Hase and Bansal (2020); Jeyakumar et al.
Learning to explain.
Our work is inspired by the learning to explain (L2X) method by Chen et al.
(2018), proposing “instancewise feature selection as a methodology for model interpretation”.
Given a d -dimensional input x ﬁrst a binary mask δ ∈ 2 d is computed in an explanation network (e.g. a multilayer perceptron).
Then, the masked input x (cid:12) δ is used as the input to a second network solving the task.
In order to make this setup end-to-end differentiable the computation of the discrete mask δ has to be approximated in a continuous fashion.
To this end, the explanation network learns a distribution of inclusion probabilities of each input feature.
Then, k features are drawn independently from this distribution, to approximate (coarsely) the sampling of a mask of size k
(i.e. a mask with exactly k of the d entries of δ equal to 1).
The discrete sampling itself is approximated by the Gumbel-softmax trick (Jang et al., 2016; Maddison et al., 2016).
At test time no sampling is performed and instead the k features with highest inclusion probabilities are used.
As a result, L2X is constrained to produce explanations of ﬁxed size k , with k a hyperparameter.
In this work, we propose an alternative way to compute masks that permits masks of different sizes for different inputs.
This is crucial for vision tasks as the sizes of regions of interest in an image vary between images.
For image classiﬁcation Chen et al.
(2018) compute the mask δ through a convolutional network.
As a consequence of pooling operations in the network, δ is of lower resolution than the input x and hence must be upsampled (with repetition) to ˜ δ before computing the masked input x (cid:12) ˜ δ .
The masked input image is then classiﬁed by a second convolutional network.
We ﬁnd that this approach scales poorly to real-life sized image classiﬁcation problems (Section 4.1).
Furthermore, Jethani et al.
(2021) identify that L2X can lead to explanations with limited ﬁdelity in the case a prediction is encoded in only one image patch.
They propose REAL-X, a method similar to L2X, except that the ﬁrst convolutional network is forced to learn meaningful masks that are not subject to the aforementioned encoding.
These approaches are computationally expensive, requiring two forward passes through convolution stacks.
Our proposed architecture is designed to alleviate these issues.
For further related work and the embedding of our approach into related approaches see Section 3.3.
Bounded logit attention
Image classiﬁcation.
We consider a standard supervised image classiﬁcation task for some dataset consisting of pairs ( x, y ) , with x an image annotated by some class-label y .
A model h for this tasks predicts labels
ˆ y =
h
( x ) .
We assume the following standard architecture for h .
Given an input image x a convolutional feature extractor F computes a list of feature vectors F ( x ) =
(cid:126)f = (cid:104) f
i : i < n (cid:105) , with n the size of the feature map.
To avoid clutter, we use a single index i < n for these features instead of the usual two indices indicating a feature’s position in the two-dimensional feature map, but of course each feature still corresponds to a region in x .
These features are then globally average-pooled to a feature vector v =
[Equation]
n (cid:88)
[Equation]
Finally, a classiﬁer L (e.g. a logistic regression head, or perhaps a multilayer perceptron) computes ˆ y =
L ( v ) .
Explanation module.
The features f i learned by the model h may be considered an abstract representation of properties of the corresponding regions of the input image x .
In this work, we are interested in selecting features that are “important” to the model h when predicting the label ˆ y = h ( x ) .
To this end, we propose to employ an explanation module E that outputs a subset of feature indices E ( (cid:126)f ) =
(cid:126)δ = (cid:104) δ
i : i < n (cid:105) ∈ 2 n .
A model h , as described above, may then be augmented by E by replacing the average-pooling operation by the reweighed pooling operation
[Equation]
Then (cid:126)δ = E ( (cid:126)f ) acts as an explanation for the prediction ˆ y
=
L ( v ) .
Indeed (cid:126)δ is easily humaninterpretable as each entry can be understood as a binary ﬂag encoding the use of the corresponding feature in the prediction, and the size of (cid:126)δ is typically small enough, e.g. n = 49 = 7 2 .
Our goal is to build the explanation module E in a way such that the model remains end-to-end differentiable, i.e. such that E may be learned.
3.1 Parameterizing the explanation module
Variational approximation.
We identify (cid:126)δ = E ( (cid:126)f ) with the probability distribution p ( i | (cid:126)f )
=
δ i (cid:107) (cid:126)δ (cid:107) 1 .
During training we approximate p ( ·| (cid:126)f ) by some q ( ·| (cid:126)f ) =
Q ( (cid:126)f ) where Q is an element of some variational family
[Equation]
q
[Equation] [Equation]
[Equation]
} parameterized by φ .
Hence, we need to ﬁnd a suitable variational family, which in deep learning terms means ﬁnding the appropriate neural architecture implementing Q .
We require Q to compute a probability distribution q approximating p .
Activation function.
The key trick we propose is using the simple, yet uncommon activation function
[Equation]
[Figure]
features (cid:126)f 1 × 1 -convolution β activation logits (cid:126)(cid:96) softmax activation q ( ·| (cid:126)f ) × (cid:80) classiﬁer L prediction ˆ y explanation module Q in the approximation of the logits of such discrete distributions.
The β -activation function is of course closely related to the popular ReLU function.
What is uncommon is applying it directly before the softmax function.
Concretely, we build Q as follows (Figure 1).
Given a feature map (cid:126)f , we apply a 1 × 1 -convolution with a single ﬁlter, mapping each f i to a scalar value g i =
u (cid:62) f
i , with u the weights of the convolution.
Then, we compute the logits (cid:126)(cid:96) =
(cid:104) (cid:96) i : i < n (cid:105) where (cid:96) i =
β ( g i ) .
Finally, we apply the softmax function σ to obtain (cid:126)q = σ ( θ · (cid:126)(cid:96) ) , with the temperature θ being a hyperparameter.
During training we plug Q into the original model h by replacing the pooling operation with
[Equation]
Discretization.
After training we obtain the explanation module E from its variational approximation Q by deﬁning δ i = 1 iff (cid:96) i = 0 .
The merit of the β -function now becomes clear: it allows for constructing explanations of variable size (the size of an explanation being (cid:107) (cid:126)δ (cid:107) 1 ) by giving us a canonical way of discretization.
Importantly, it forces q to be uniform on indices for which (cid:96) i = 0 , making the behavior of Q and its discretization E match more closely.
Thresholding.
Why may we hope that Q might learn to approximate reasonable explanations?
Remember that we are interested in ﬁnding “important” features.
The explanation module Q computes weights q
( i | (cid:126)f ) for each feature f i to use in the pooling operation in formula (1).
Thus during training Q should learn to assign more weight to features that are discriminative for the task, and to mostly ignore features that are not.
To encourage this behavior and put additional learning pressure on Q we propose to threshold weights in a modiﬁed pooling operation
[Equation]
where 1 [ q ( i | (cid:126)f ) > γ ] is 1 if q ( i | (cid:126)f ) > γ and 0 otherwise.
We propose γ = 1 n as the default threshold.
3.2 Application-speciﬁc aspects
Bounded logit attention (BLA).
One way to think about the explanation module described above is to consider it an attention mechanism , in the sense that Q computes soft attention, approximating hard attention in E .
With the β -activation function bounding the logits of q ( ·| (cid:126)f ) from above being the key property of this attention mechanism, we refer to our architecture as bounded logit attention (BLA).
Independently of explainability, BLA can serve as a general purpose method to produce differentiable approximations for the selection of subsets.
Transfer learning.
Note that the modularity of our method lends itself to transfer learning setups.
Having trained a model
h = F ◦ L , we may for example hold both F , L ﬁxed and only train an explanation module Q as a post-hoc addition to h .
We investigate this approach in Section 4.2 under the name BLA-PH.
In this setup we could even make predictions using the original model h (without
Q plugged in) and use the outputs of Q as a purely post-hoc explanation.
Section 4.3 provides justiﬁcation for this approach (that has the advantage that the outputs of Q are far cheaper to compute
V i s .
H a r d
S o f t
I npu t than the LIME scores
Finally, our method can readily be used with a pretrained feature extractor F while training both L Q
Visualization.
Like the (Grad-)CAM method (Zhou et al., 2016; Selvaraju et al., 2017), BLA produces explanations at the level of the convolutional feature map.
In Section 1 we mentioned that a conceptual advantage of the learned explanations produced by E is that they faithfully reﬂect the workings of (some aspect of) the model by construction.
However, we still need to make a choice how to present the outputs (cid:126)δ of E .
Following the convention of the (Grad-)CAM literature, we visualize (cid:126)δ using a colormap, which is upscaled and imposed on the input image (Figure 2).
The experiment in
Section 4.3 provides justiﬁcation for our choice.
Note that other choices are possible of course, e.g. to spread out each δ i over the corresponding receptive ﬁeld.
Both (Grad-)CAM and BLA achieve interpretability by restricting their semantics to the most easily interpretable aspect of convolutional features which is their location.
3.3 Alternative explanation modules
In Section 3.1 we proposed BLA as a possible architecture of the explanation module Q .
There exist several concepts in the literature one might consider for alternative explanation modules.
Fixed size explanations: L2X-F.
When employing the transfer learning approach discussed above, keeping the feature extractor F ﬁxed, and identifying each input x with its feature map (cid:126)f = F ( x ) , our approach resembles computing L2X (see Section 2) explanations for the classiﬁcation head L .
The key differences are: L2X a priori ﬁxes the size of the generated explanation to some k < n , while in our approach use of the β -function allows to generate explanations of a variable size.
Like BLA, L2X computes a probability distribution over the input points (which here are the convolutional feature vectors), but unlike BLA the training of L2X has a non-deterministic element in the approximate sampling from this distribution.
We refer to L2X at the feature level as L2X-F and use it as a baseline to compare our approach to.
Another way to think about the BLA explanation module is that q ( ·| f i ) computes an objectness score for the feature f i .
However, the prediction of the objectness of f i is not made independently of the objectness of f j , j (cid:54) =
i .
As a result of the softmax function there is global competition between these scores, in the sense that increasing q ( ·| f i ) necessarily means decreasing (cid:80) j (cid:54) =
i q ( ·| f j ) .
Thus the softmax function provides a weak form of global context in the explanation module.
This raises the question, which amount of context is most suitable.
More context: attention with global concept vector.
Jetley et al.
(2018) propose “an end-toend differentiable attention module for convolutional neural network architectures built for image classiﬁcation”.
Like in our approach, the authors compute probability distributions to weight features (and they do this for three different levels of convolutional features, not just the outputs of the feature extractor as we do).
The key differences to our approach is that a global concept vector g , computed as a learned linear transformation of the concatenation of all feature vectors is used in the computation of the logits of the probability distribution.
To this end, g is added to each feature vector f i , resulting in logits (cid:96) i = u (cid:62) ( g + f i ) .
We attempted incorporating a global concept vector in the BLA module in this way (while keeping the β -activation for the logits before the softmax).
Interestingly, the module learned to compute extremely negative logits (of magnitude ca.
− 10 3 ), essentially circumventing the β -activation function (Figure S19 in the supplemental material).
As a result, only a single feature is chosen in the discrete attention module E (since there is now a unique maximum of the logits (cid:96) i ).
We believe this is due to the concept vector enabling the explanation module to “cheat” in the sense that it can be very certain which feature vector it wants to choose.
Since the initial results were not promising, we did not pursue this approach further.
Less context: pointwise attention.
Park et al.
and Woo et al.
(2018) propose an attention module for convolutional neural networks they call CBAM.
They do not use the softmax function when computing their attention maps, but instead use pointwise sigmoid activations, i.e. they use no global context at all.
We found that for our purposes the lack of competition between features did not seem to put sufﬁcient learning pressure on an explanation module to choose between features, resulting in poorly focused attention maps.
Again, we did not pursue this approach further.
Experiments 4.1 Comparison to L2X
In this section we substantiate the claims that 1) BLA improves accuracy over L2X on small datasets and 2) unlike BLA, L2X scales poorly to larger datasets, hence a new method is indeed needed.
The MNIST dataset (LeCun et al., 2010) contains 28 × 28 grayscale images of handwritten digits.
We subsample all images of 3 and 8, resulting in 11982 training and 1984 validation images.
Chen et al.
(2018) report validation accuracy of 0.958 (using hard explanations, there called post-hoc accuracy) on this dataset for their L2X method, with ﬁxed explanation size k = 4 .
For their baseline model without learned explanations they report a validation accuracy of 0.997.
We imitate their CNN architecture, building a model consisting of three 5 × 5 convolutions with ReLU activations with 8, 16, and 16 ﬁlters, respectively.
The ﬁrst two convolutions are followed by 2 × 2 maximum pooling, and the third convolution is followed by a dense linear layer with a sigmoid output unit.
We augment this
CNN by a BLA explanation module as described in Section 3.1, and train the model end-to-end forepochs, using the Adam optimizer (Kingma and Ba, 2014) with a learning rate of 10 − 3 to minimize the cross-entropy loss.
We use temperature θ = 0 . 1 and thresholding with γ = 49 − 1 .
Over 20 runs we obtain a mean baseline accuracy of 0 .
994 ± 1.53e-3, and a mean accuracy of 0 .
993 ± 3.86e-4 for the interpretable model, with no statistically signiﬁcant difference (p=0.26 Mann-Whitney-U-test).
The cats vs. dogs ( CvsD ) dataset (Elson et al., 2007) consists of 23,262 color images, divided equally into two classes “cats” and “dogs”.
We resize all images to 224 × 224 .
We use 18,609 images for training and 4653 images for validation.
We adapt the setup of Chen et al.
(2018) to the CvsD dataset.
The explainer network uses an EfﬁcientNet-B0 (Tan and Le, 2019) convolution stack (pretrained on ImageNet and frozen in the subsequent experiments), producing explanations of size k = 8 ( selected 32 × 32 patches of the 224 × 224 input image), while the classiﬁcation network uses the same convolution stack, followed by a dense layer with sigmoid activation.
We train this setup for 3 epochs using the Adam optimizer (Kingma and Ba, 2014) with a learning rate of 10 − 3 .
L2X obtains a validation accuracy of 0 .
96 using soft explanations, but a validation accuracy of 0 .
5 when using the desired hard explanations (i.e. the random baseline).
According to our visual assessment, the produced explanations indeed failed to capture meaningful information.
We take the failure of this experiment as evidence that L2X is difﬁcult to scale to deep CNNs (while also being less efﬁcient than the BLA method).
4.2 Understanding the BLA explanation module
In this section we show that BLA improves accuracy and loss over the L2X-F baseline, implying that the improvements of BLA over L2X are not solely due to the switch to a single pass convolution architecture.
We proceed to show that thresholding slightly increases accuracy further, while increasing loss.
Finally, we show that employing BLA-T post-hoc does not decrease accuracy on average.
Datasets Additionally, we evaluate our method on the Stanford Dogs ( StanDogs ) dataset (Khosla et al., 2011; Deng et al., 2009), consisting of 20,580 color images of 120 dog breeds.
We use 12, images for training and 8,580 for validation.
Finally, the Caltech-UCSD Birds-200-2011 ( CUB-200 ) dataset (Welinder et al., 2010), consists of 11,788 color images of 200 bird species.
We use 5, images for training and 5,794 for validation.
For each dataset images are resized to 224 × 224 .
[Figure]
[Table]
[Figure]
We use EfﬁcientNet-B0 (Tan and Le, 2019) as our classiﬁcation model h
= F ◦ L , i.e. F is the
EfﬁcientNet-B0 convolution stack and L consists of a dropout layer (Srivastava et al., 2014), dropping units with probability 0 . 2 , followed by a dense linear layer with softmax activations.
The parameters of F are pretrained on ImageNet and held ﬁxed in our experiments.
Throughout this section we use the Adam optimizer (Kingma and Ba, 2014) with a learning rate of 10 − 3 , a batch size of 32, minimizing the cross-entropy loss.
Non-pretrained parameters are set using the Glorot uniform initalizer (Glorot and Bengio, 2010).
Baseline (BL).
We begin by training classiﬁcation heads L for each dataset (training for 2 epochs for
CvsD and 5 epochs for StanDogs, CUB-200), to obtain trained classiﬁcation models
h = F ◦L .
These models will serve as the uninterpretable baseline.
Note that our goal is not to reach state-of-the-art performance, but to investigate how the addition of explainability modules affects the performance, relative to the baseline.
We conduct every experiment in this section 20 times for each dataset, e.g. we begin by training 20 baseline models for each dataset.
To compare the results we use the Wilcoxon signed-rank test, setting the level of signiﬁcance to 0.05.
When talking about measurements (e.g. accuracy or loss) we always refer to the mean over all 20 runs.
When comparing mean measurements, their differences are always implied to be statistically signiﬁcant, unless stated otherwise.
In a next step, we augment our baseline models
h = F ◦ L with explanation modules Q .
We train the resulting models for 2 epochs for dataset CvsD and 5 epochs for StanDogs, CUB-200.
The parameters of F and L serve as a starting point for training, with Q randomly initialized as described above.
The convolution stack F continues to be held ﬁxed, while L , Q are trained, except in the last experiment where we hold the head L ﬁxed as well.
Validation accuracies and losses are always reported for the hard explanations generated by E (which are approximated during training by soft explanations in Q ).
The numerical results of the following experiments are listed in Table 1 and presented graphically in
Figure 3.
Examples of explanations for this experiment are shown in Figure 2 and further examples from experiments in this section are found in the supplemental material (Figure S7-S18).
L2X-F. First, we try adding L2X-F modules to obtain an interpretable baseline as discussed in
Section 3.3.
Accuracy and loss for all three datasets are considerably worse than the corresponding measurements for the baseline, i.e. adding interpretability through L2X-F modules comes at a considerable cost in model performance.
BLA.
Next, we investigate our BLA modules, with a temperature θ = 0 .
1 and without thresholding.
The accuracies improve greatly compared to the L2X-F interpretable baseline, while not quite reaching the uninterpretable baseline.
The losses also improve compared to the L2X-F baseline, but do not come quite as close to the baseline losses.
B L A - T
C A M
E q u a l 0.
B L A - T
L 2 X - F
E q u a l 0.
B L A - T
R a n d
E q u a l 0.
Dataset
LIME/CAM
LIME/BLA-T
CvsD 0.
StanDogs 0.
CUB0.
BLA-T.
Next, we investigate the BLA modules, again with temperature θ = 0 .
1 using thresholding with γ = 0 .
98 / 49 = 0 .
Compared to BLA (the same method without thresholding), the accuracies improved slightly.
For dataset CvsD the improvement is not statistically signiﬁcant, for StanDogs the improvement is signiﬁcant with the accuracy still signiﬁcantly worse than the uninterpretable baseline, and for CUB-200 the accuracy is no longer signiﬁcantly different to uninterpretable baseline.
Curiously, the losses increased a fair bit compared to BLA, but remained lower than the L2X-F losses.
BLA-PH.
Finally, we try the same setup as BLA-T, but with frozen heads L , i.e. only the explanation module Q being updated during training.
Except for the loss for dataset A, all of the measurements are not statistically signiﬁcantly different from the corresponding measurements for BLA-T.
This means models with L frozen perform on average as well as those where L is trained together with Q .
However, the variances of these measurements increased drastically.
We ran our experiments on NVIDIA GeForce GTX 1080 GPUs.
While each of the models for any of the three datasets can be trained in a matter of minutes, due the large number of runs the experiments described above take up about 11 hours of GPU time.
Our implementation is available at: https://github.com/fhstp/bla
ImageNet.
We demonstrate that our method scales to the ImageNet 2010 dataset by Russakovsky et al.
(2015), consisting of medium resolution color images of 1000 categories.
We train on 1,261, images and validate on 50,000, resizing to 224 × 224 .
In the fashion described above, we train one baseline model, and models augmented with BLA and BLA-T modules, for 2 epochs each.
We obtain an uninterpretable baseline accuracy of 0.469.
For BLA we obtain 0.458 and for BLA-T 0.525, improving upon the baseline accuracy.
An additional 8 hours of GPU time were used.
4.3 On faithfulness
While learned explanation mechanisms are faithful to the model by construction, getting back to points raised in Section 3.2 we may wonder: a) How faithful are BLA explanations to the uninterpretable baseline model, i.e. is it appropriate to use BLA in a post-hoc setup and b) is our choice of visualization faithful?
Inspired by an occlusion experiment in Selvaraju et al.
(2017), we use the LIME method Ribeiro et al.
(2016), which was designed for local faithfulness, as a baseline.
For the uninterpretable baseline model, we compute LIME scores for patches of size 32 × 32 of the 224 × 224 input images from the validation set, resulting in explanations of size 7 × 7 , by randomly sampling 1000 occlusion patterns and ﬁtting a least squares model.
We compute the Spearman rank correlation between these LIME explanations and the 7 × 7 soft explanations produced by Q (using
BLA-T).
Additionally we compute rank correlation between LIME and (Grad-)CAM (Table 2).
We observe that both BLA-T and (Grad-)CAM consistently achieve moderate rank correlation, except on dataset CUB-200 where the rank correlation for (Grad-)CAM is low.
We conclude that our method’s faithfulness is comparable to that of (Grad-)CAM on datasets CvsD, StanDogs and considerable more faithful on dataset CUB-200.
4.4 User study
In a user study we investigate the following questions: 1) How do users rate BLA-T explanations compared to explanations generated by the popular (Grad-)CAM method (Zhou et al., 2016; Selvaraju et al., 2017) which also produces explanations at the level of the convolutional feature map and 2) compared to L2X-F ﬁxed size explanations? 3) Can users tell the difference between BLA-T and random explanations from the same distribution (a BLA-T explanation from another, random image
Both systems decide for the class *briard* in this example.
Below, the two different systems show you on which parts of the image they base their decision.
Which system behaves more reasonably?”
in the dataset imposed on the original image)?
The latter question is intended as a sanity check to verify if our method actually produces meaningful output, as it is easy to fool oneself by wishful thinking when relying on one’s own visual assessment (Adebayo et al., 2018).
The participants in this study were 62 unpaid volunteers from diverse educational backgrounds (for further details refer to Figures S3-S6 in the supplemental material).
In each question, participants were presented with a random image from one of the three datasets and two explanations, corresponding to one of the three questions above.
The position of the explanations was randomized.
Participants were asked which of the two explanations, if any, they consider more reasonable.
For each question type and dataset, participants answered four questions (Figure 5).
We obtain the following results (Figure 4): 1) When compared to (Grad-)CAM, BLA-T was chosen 71% of times and (Grad-)CAM 14% of times.
We consider such clear user preference for our method over the well established (Grad-)CAM method to be strong evidence that BLA will be able to serve as a useful new tool in the ﬁeld of explainable artiﬁcial intelligence.
When compared to L2X-F,
BLA-T was chosen 53% of times and L2X-F 22% of times.
The difference between these methods is less decisive than for the previous question, as evidenced by the 24% “equally reasonable” responses, however overall there is still a clear preference for BLA-T.
Thus, our method is an improvement over “learning to explain” Chen et al.
(2018) on feature level not only conceptually by being able to produce explanations of variable size, but also according to user preference.
Our sanity check is passed easily, with 79% choosing the true BLA-T explanations and only 6% the random explanation.
A more detailed evaluation is found in the supplemental material (Figure S2).
Conclusion
We presented BLA as a method to learn explanations in deep image classiﬁcation networks, selecting “important” elements of the convolutional feature map for each instance.
The key idea of using the activation function β ( x ) =
min( x, 0) to bound the logits of the attention map allowed us to canonically produce explanations of variable size.
Beyond explainability, BLA can also serve a general purpose method to produce differentiable approximations for the selection of subsets.
The main concern with trainable, built-in explainability methods is that they affect model performance.
In Section 4.1 we showed that unlike L2X method of Chen et al.
our method does not affect accuracy on a subsample of the MNIST dataset.
Our experiment attempting to scale L2X to the cats vs. dogs dataset failed, demonstrating the need for a more scalable approach such as the one presented in this work.
In Section 4.2 we showed that L2X-F, i.e. using the L2X method on feature level, also comes at a considerable cost of decreased accuracy.
However, using BLA modules the performance came close to that of the uninterpretable baseline models.
The accuracies slightly improved further when using thresholding in the BLA module, which for Caltech birds datatset even resulted in accuracy statistically indistinguishable from the baseline accuracy.
However, thresholding came at the price of increased loss.
A pure transfer learning approach, only training Q while holding the head L ﬁxed, resulted in no change in performance on average, but increased variance.
Our quantitative experimental results are thus summarized as follows: built-in explainability for an image classiﬁcation model using the BLA attention module comes at a very moderate cost of accuracy, while (unlike post-hoc methods) providing by construction faithful insight into the workings of the classiﬁer.
Finally, conducting a user study, we found strong evidence that participants prefer our method over (Grad)-CAM and L2X on feature level.