Towards Better Visual Explanations for Deep Image Classiﬁers
Abstract
Understanding and explaining the decisions of neural networks is of great importance, for safe deployment as well as for legal reasons.
In this paper, we consider visual explanations for deep image classiﬁers that are both informative and understandable by humans.
Motivated by the recent FullGrad method, we ﬁnd that bringing information from multiple layers is very effective in producing explanations.
Based on this observation, we propose a new method, DeepMaps, that combines information from hidden activities.
We show that our method outranks alternative explanations with respect to metrics established in the literature, which are based on pixel perturbations.
While these evaluations are based on changes in the class scores, we propose to directly consider the change in the network’s decisions.
Noting that perturbation-based metrics can fail to distinguish random explanations from sensible ones, we propose to measure the quality of a given explanation by comparing it to explanations for randomly selected other images.
We demonstrate through experiments that DeepMaps outperforms existing methods according to the resulting evaluation metrics as well.
Introduction
Neural networks are well-established tools in a wide range of applications, such as image recognition, natural language understanding, recommendation systems, to name a few.
Their deployment, however, is in some cases met with reservation and comes with its own challenges (e g. legal or safety-related), given how little guarantees are available about their generalisation properties, or about their robustness to changes in the input data.
Therefore, there is an increasing demand for explanations (Goodman & Flaxman, 2017), especially urgent where machine learning algorithms support human decisions, e.g., in medical diagnosis, law, banking, among other domains.
In this paper, we add to the explainable AI effort by focusing on explaining decisions of deep image classiﬁers.
More precisely, we consider trained networks rather than the problem of image classiﬁcation itself (cf.
Rieger & Hansen, 2019; Kindermans et al., 2017), and we study instance-based explanations (Baehrens et al., 2010), i.e. explanations for a given image.
Supplying such visual explanations together with network decisions might increase trust of the end user, or, contrarily, help to spot deﬁciencies in the network’s decisions.
Thus, our work is relevant for deciding which networks are safe to deploy (cf.
Adebayo et al., 2020).
Gradients
GradCAM
DeepMaps
What we call a visual explanation takes the form of an image that matches the shape (array of pixels) of the given input image.
Such explanations can be easily visualised and even overlaid over the input image, boosting interpretability.
Speciﬁcally, we focus on saliency maps , which are visual explanations that highlight pixels that are most relevant for the network’s decision, such as shown in Fig.
As one can see, different saliency maps range from scattered points to big blobs through shapes which highlight the most relevant content in the image.
The ﬁrst attempts to produce visual explanations were based on input gradients (see, e.g. Fig.
1, Gradients), showing the local sensitivity of the class score, which is rather hard to interpret (Baehrens et al., 2010; Simonyan et al., 2014).
To address this issue, Selvaraju et al.
(2016) proposed to analyse the information obtained from the last convolutional layer rather than the input.
Their method yields explanations of lower resolution, which, while desirable, also tends to remove the ﬁne details of the object shape (Fig. 1, GradCAM).
We advocate for explanations that are of sufﬁciently low resolution to be localised around the object of interest, and at the same time sufﬁciently high resolution to keep the distinctive details of its shape.
Recent work by Srinivas & Fleuret (2019) goes in this direction, and argues that methods for extracting visual explanations should take bias parameters into account (see also Kindermans et al., 2017).
Applying their method to the original VGG-16 network (Simonyan & Zisserman, 2014) results in a vast improvement over the input-based saliency maps (Fig. 1, FullGrad).
However, an analysis of the behaviour of VGG-16 shows that the bias parameters contribute very little to the class scores and have negligible effect in the ﬁnal decisions.
Thus, the success of the F ULL G RAD method of Srinivas & Fleuret (2019) must be for some other reason.
We hypothesise that it is rather due to aggregating explanations over different depths, and propose a new method, D EEP M APS , that produces saliency maps based on hidden activities instead of biases.
Apart from producing better explanations, on which many studies have focused, developing evaluations for network explanations remains challenging.
We often rely on visual comparison to the ground truth images as a way to rank methods.
While such an evaluation is easy to understand, it is biased towards human perception and results in the introduction of many heuristics for smoothing or sharpening the saliency maps.
It is therefore hard to tell which part of the resulting explanations is related to the network behaviour and which part is related to the method heuristics, and so developing quantitative measures that are unambiguous as well as principled is of high interest.
Popular quantitative measures are built on image perturbation as a means for evaluating saliency maps.
In the most common pixel-removal approach, image regions are masked while the change in the class scores is monitored (Samek et al., 2016).
The masks are chosen according to the given saliency map, either selecting the most or least important pixels.
However, existing approaches have multiple drawbacks: they care about class scores instead of the actual classiﬁcation performance, and they are not always able to distinguish sensible explanations from obscure ones; for example, high frequency random perturbations are likely to confuse state-of-the-art neural networks (Samek et al., 2016; Fong & Vedaldi, 2017; Dabkowski & Gal, 2017), and hence can achieve high scores as explanations when considering the removal of the most important pixels.
To overcome the above difﬁculties, we propose new evaluation approaches.
First, rather than monitoring the score change while removing pixels, we monitor the change in decisions.
In addition, we propose a new evaluation method that is constructed to distinguish meaningful explanations from random ones.
This new evaluation method measures the quality of a given explanation by comparing it to a set of other explanations obtained by the same explanation method for random images from the dataset of interest.
We analyse the performance of D EEP M APS through extensive experiments and compare it to F ULLG RAD and other existing methods (see also Appendix E).
Arguably, it produces explanations that are visually more convincing, as they particularly highlight the ﬁne details of the object shape that are important to the predicted class (see, e.g. Fig.
1, DeepMaps).
We also demonstrate that in VGG16, D EEP M APS outperforms previous explanations with respect to evaluation metrics commonly used in the literature, and
the new ones proposed in this paper.
We also analyse D EEP M APS for
ResNet-50 (He et al., 2016), showing a competitive performance to previous methods.
To summarize, our contributions come in two angles: (i)
We challenge the use of biases for visual explanations and propose a new method that aggregates information from hidden activations instead.
(ii)
We rethink the evaluation of visual explanations and propose new metrics, and a novel evaluation paradigm, which have intuitively desirable properties.
The rest of the paper is organised as follows.
Previous work is presented in § 2.
The analysis leading to the development of D EEP M APS and the description of the method are presented in § 3.
Existing and new quality metrics are described in § 4, together with an in-depth analysis of the performance of the different methods according to the different metrics.
Notation and background
We consider feed-forward (convolutional) ReLU networks for image classiﬁcation, as sketched in
Fig.
2 (top).
A ReLU network composed of L layers takes an input image x ∈ R m × n × Φ of width m , height n , and composed of Φ colour channels, and progressively maps this image into subsequent hidden layers, which gives the “images” x l for l ∈ { 1 , . . .
, L − 1 } deﬁned by x l = (cid:2) W l x l − 1 + b l (cid:3) + , where x 0 = x is the original input, x l ∈ R m l × n l × Φ l for suitable dimensions, W l is a weight matrix of appropriate size, b l is a tensor with the same size as x l (although, in practice, bias values are often shared across space), and [ · ] + is the positive part function (applied entrywise to a tensor).
The network computes class scores f c for all possible classes c ∈ { 1 , . .
, C } by f c =
w Lc x L −
+ b Lc .
The predicted label for x is given by the class corresponding to the highest score.
Throughout the paper, each entry of a vector corresponding to a neuron in a layer (such as x l ) is indexed by its spatial position ( i, j ) and its channel φ .
2.1 From gradient-based to attribution-based explanations
The ﬁrst attempts to produce explanations were based on the study of the gradients, coinciding with the weights for the case of linear models.
Gradient visualisations provide insight into how the classiﬁer works locally, by quantifying how each class weighs every pixel in a small neighborhood of the input image.
This is especially true for ReLU networks, as the class scores are piecewise linear functions of the input x .
The earliest methods for producing visual explanations based on this concept of gradient backpropagation are due to (Baehrens et al., 2010; Simonyan et al., 2014).
Such methods simply compute the gradients ∂f c ( x ) ∂x ijφ (note that they exist in ReLU networks for almost all images, apart from a set of Lebesgue measure zero), and produce a saliency map s of the same size as the input image x , by taking, for example, s ij ( x )
= (cid:80) φ (cid:12)(cid:12) (cid:12) ∂f c ( x ) ∂x ijφ (cid:12)(cid:12) (cid:12) .
We refer to this method as G RADIENTS .
In a similar vein, a large family of attribution methods were developed, devising rules for decomposing the class score and back-propagating the decomposition to the input space (Bach et al., 2015; Selvaraju et al., 2016; Sundararajan et al., 2017; Smilkov et al., 2017; Shrikumar et al., 2017; Lundberg &
Lee, 2017).
Of those, particularly relevant to this work are gradient × input attributions (Srinivas &
Fleuret, 2019), deﬁned as
[Equation]
While these explanations are easy to compute, are intuitively related to the network behaviour, and result in saliency maps showing signal around the objects of interest; they are typically scattered and noisy-looking (similarly to the panel Gradients in Fig. 1).
In view of this, from a human-centric perspective, such explanations are hard to interpret.
[Figure]
f c =23.
One of the ﬁrst and most popular methods to produce low-resolution explanations is GRAD C AM (Selvaraju et al., 2016), which uses the gradients with respect to the activations in the last convolutional layer.
While the saliency maps produced by this method (e.g. Fig.
1, GradCAM) are more centered around the object of interest, they tend to render a low resolution signal, losing the shape of the object.
The concepts humans tend to look for lie in the spectrum of explanations in-between these two extremes.
More recently, Srinivas & Fleuret (2019) proposed the F ULL G RAD method that incorporates information from the biases.
Their method is derived by looking at complete decompositions of the class scores, i.e. attributions that sum up to the score.
For a network with zero-biases, due to the piecewise linearity of the output, this is obtained from gradient × input attributions:
[Equation]
In general, a complete decomposition of a class score requires taking bias parameters into account.
Deﬁning bias attributions as
[Equation]
Srinivas & Fleuret (2019) noted that f c ( x ) = (cid:80) i,j a cij ( x ) + (cid:80) l,φ a b,l cφ .
Mapping the biases back to a single image of the original size ( m × n ), they proposed the saliency map
[Equation]
where a c = ( a cij ) i,j is the matrix of input attributions (Eq.
Srinivas & Fleuret (2019) recommended using Ψ
[Equation]
bilinearUpscale(normalize
[Equation]
(5) where bilinearUpscale is a typical image resizing operation mapping its input image to size m × n , and | .
| and normalize are applied entrywise with normalize( a ij ) =
( a ij − min( a )) /
(max( a ) − min( a )) for a matrix a .
The resulting explanation is shown in Fig.
1 (FullGrad).
There is a clear improvement over saliency maps based on input attributions, which look just like the Gradients, Fig. 1.
In the next section, we look into the sources of this improvement, and based on our ﬁndings we propose a new explanation method.
DeepMaps: rethinking attributions
We illustrate the decomposition of the class score for an example image of a gazelle in Fig.
2 for a
VGG-16 network.
Note that our network was trained without layer normalisations, so there are no other parameters we need to take into account beyond the biases b l .
For each layer l , we sum up bias attributions over all channels, a b,lcij = (cid:80) φ a b,lcijφ .
We show the bias attributions a b,lcij in the bottom row of Fig. 2, with their sum A b,lc = (cid:80) i,j a b,lcij shown above the corresponding images.
From left to right, attributions decrease in size, reﬂecting the size of the corresponding convolutional layers (sketched on top), but are shown upscaled to the original size, similarly to the operation in Ψ above.
In the middle row of Fig.
2, we plot input attributions (left), as well as attributions of hidden activations in every layer, deﬁned as
[Equation]
with a h, 0 cij = a cij .
Similarly to the biases, above the attributions of each layer we note their sum
[Equation]
[Equation]
Similarly to (2), in a bias-free network we have f c ( x ) =
A h,lc ( x ) , that is, the class scores can be decomposed using the hidden-layer attributions.
For such a network, the numbers A h,lc over each a h,l panel would be identical and equal to the network output (top right).
From Fig.
2, we can make two important observations, which are generally true for other images, as well: (i) The noisy-looking input attribution has a signiﬁcantly higher value than the bias attributions, and the sum of attributions for each hidden layer is approximately the same, showing that the biases over these layers have little effect.
The most important bias attribution comes from the last convolutional layer and does not make sense from a visual perspective.
(ii) All the hidden-layer attributions focus on or around the gazelle, while the bias attributions seem less focused in the last layers.
In general, the attributions from the hidden layers seem to be of superior quality.
To summarise, for the studied VGG-16, the information conveyed by the biases seems to have a low impact on the class scores and the ﬁnal decision.
Despite this little impact on the score, F ULL G RAD explanations seem visually convincing (see Fig. 1).
Therefore, it is natural to ask: What is the key operation that makes the strength of F ULL G RAD ?
Our hypothesis is that, rather than relying on biases, the crucial contribution of F ULL G RAD is the aggregation of information from all the hidden layers of the network.
Hidden activations in deeper layers depend on correlations in larger input patches and hence shed some light on the non-linear structure of the network.
Furthermore, attributions derived from the hidden activations seem to be visually more convincing than those derived from the biases (as in Fig. 2).
Based on these observations, we propose D EEP M APS , a new method to generate explanations.
DeepMaps: combining attributions from multiple layers: Inspired by the superior quality of a h,l over a b,l attributions and the advantages of taking explanations from multiple layers (i.e. of multiple resolution), D EEP M APS aggregates hidden-layer attributions as
[Equation]
[Equation]
where the transformation Ψ is deﬁned in Eq.
Transforming attributions into saliency maps is mostly governed by heuristics.
Other than the transformation Ψ , there are also different choices for the level at which it is applied.
Note that we apply Ψ( · ) to a h,l , rather than to every feature map as in F ULL G RAD (cf.
Eq.
Thus, our approach emphasises attributions per layer, rather than per feature.
Commonly, the number of features grows for spatially smaller layers (Fig. 2 top), which means F ULL G RAD effectively overemphasises lowresolution visual explanations.
Note that in Fig. 2, hidden-layer attributions seem to have a similar scale ( A h,lc ( x ) ) to the input attribution.
D EEP M APS then corresponds, for this example, to giving explanations with equal weight on every resolution, which in turn leads to an explanation dominated by the important resolutions.
In Appendix D, we compare single-layer explanations generated by these two approaches.
Evaluating explanations
In this section, we analyse the explanations produced by D EEP M APS and compare them to previous methods.
We ﬁrst extensively analyse the original VGG-16 architecture Simonyan & Zisserman (2014), for which we observed a limited contribution of the biases.
We then extend our analysis to
ResNet-50, a network in which biases have a more important role (Appendix A).
In the sequel, all the visualizations and quantitative evaluations use images from the validation set of ImageNet.
All the metrics are evaluated over 5000 images taken from classes 0..99 (50 images per class).
Whenever we plot conﬁdence intervals, they refer to one standard deviation of the class-wide mean ( n = 50 ).
[Figure] [Figure]
We start with visually inspecting some outputs of D EEP M APS as well as some of the previous explanation methods for VGG-16.
Fig.
3 shows saliency maps obtained by computing the input gradients,
GRAD C AM (Selvaraju et al., 2016), F ULL G RAD (Srinivas & Fleuret, 2019) and D EEP M APS .
Those panels conﬁrm the tendency of the gradients to yield very scattered explanations and of GRAD C AM to produce blobby saliency maps.
F ULL G RAD and D EEP M APS are both a compromise between the noisy local explanations and the rough delineation of the object of interest.
They both highlight accurately the edges and the ﬁne details of the shapes.
However, if we look, for example, at the bird example in the fourth row, it appears that F ULL G RAD shows an almost uniform signal all over the bird, while D EEP M APS emphasizes more the beak and the eyes and at a lower level the rest of the animal.
Considering the number of birds categories in ImageNet (59), it is expected that the network focuses on ﬁner details than the body to distinguish between the different species, while the body is important to distinguish birds from other super-categories.
In addition, in the second row, F ULLG RAD seems to pick up some of the background with the ﬁshes, while D EEP M APS focuses more precisely on the latter, with again a higher emphasis on the eyes.
In general, D EEP M APS seems to produce saliency maps that are easy to analyse and that highlight the distinctive details of the object of interest.
These observations, however, are subjective and need to be supported by quantitative evaluations, which we provide below.
4.1 Score-based pixel removal metrics
In the literature, explanations are usually evaluated using image perturbations, that is, measuring how sensitive the network is to modifying the input image according to the ranking provided by a given saliency map (Samek et al., 2016; Srinivas & Fleuret, 2019; Tomsett et al., 2020).
To introduce a mild perturbation (and avoid introducing adversarial effects), selected pixels are usually masked out with their values (over all input channels) replaced with an average value computed over the dataset, in effect “removing” the pixel from the input image.
Pixel-removal strategies can be divided into two families (Samek et al., 2016): (i) strategies where the most salient pixels are removed,
[Figure]
designated as Most Relevant First (MoRF); (ii) strategies where the least salient pixels are removed.
designated as Least Relevant First (LeRF).
The effect of pixel-removal is most often studied through the change of the class scores.
To this end, we ﬁrst consider two such metrics for evaluation purposes.
The area over the perturbation curve ( AOPC ), deﬁned by Samek et al.
(2016), evaluates the average change in the score of the predicted class as the most relevant pixels are removed progressively (Appendix B).
If an explanation succeeds at detecting important regions of the input, the decrease in the class score is expected to be high.
AOPC is then expected to be higher for better explanations.
The LeRF absolute fractional output change ( AFOC LeRF ) is deﬁned as the change in the score of the originally predicted class relative to the original score when the pixels are removed in a LeRF fashion.
This metric is used by Srinivas & Fleuret (2019), who advocate for the LeRF approach as it lessens the effect of the artifacts introduced by pixel removal, which are particularly troublesome when removing random pixels from the salient parts of the image.
The decrease in the class score is expected to be lower for the more successful methods.
Therefore, for AFOC LeRF , lower values are better.
Note that other variants of these metrics are possible, for example, a LeRF version of AOPC or a MoRF variant of AFOC .
In Fig.
4, we compare D EEP M APS to random pixel removal, G RADIENTS , GRAD C AM and F ULLG RAD with respect to average AOPC (top) and AFOC LeRF (bottom).
The results demonstrate that our method outperforms competitors in detecting both the most important pixels on which the network is relying for its prediction ( AOPC ), and the least important pixels that can be ignored with minimal effect on the predicted class score ( AFOC LeRF ).
4.2 Rethinking the evaluation of saliency maps
While the previous metrics focus on the class scores, they ignore the main purpose of our networks, i.e. the classiﬁcation itself.
In this section, we introduce two new pixel removal evaluations that are based on the decision shift, rather than the change in the class scores.
One way to measure the importance of the selected pixels is to determine what proportion of the pixels can be removed before the predicted class changes.
4 When removing the most relevant pixels ﬁrst, an explanation is deemed better if the number of pixels n MoRF to be removed is less, while in case of removing the least relevant pixels ﬁrst, we want the number of pixels n LeRF which can be removed without changing the decision high.
A smaller n MoRF value means a more concise explanation.
However, as the MoRF perturbation procedure targets the most sensitive region of the input, it is expected to be noisy (albeit informative).
This can be complemented by looking at the n LeRF value, which shows how good an explanation is in selecting the irrelevant parts of an image.
In Fig.
5, we show images when a ﬁxed number of pixels are removed, as introduced for LeRF procedures by Samek et al.
(Fig. 5, top row) and the one we propose (bottom row).
This ﬁgure shows that continuing with the perturbations until the prediction changes, results in more sensible masks.
They cover a much larger part of the input, which are not essential for the network’s classiﬁcation.
While the decision-based pixel-removal metrics provide more direct measures than the score-based ones, they are not robust to the effects of unnatural image perturbation, which are known to affect state-of-the-art image classiﬁers.
We therefore propose a new evaluation procedure that directly addresses this issue.
The idea is to compare the quality of an explanation to the average quality of
[Figure]
explanations obtained with the same method for other images.
When the explanations are speciﬁc, we expect that the explanation ( s ) of the underlying image ( x ) will have a much higher quality than the explanation coming from another image.
For random or generic explanations (e.g. selecting the middle region of an image), the quality of explanation coming from any image should be similar.
Therefore, for any pixel removal metric σ (where σ ( x , e ) measures the quality of explanation e on image x ), independently of its category (LeRF or MoRF), we deﬁne
[Equation]
(9) where s ( · ) is some saliency method and the expectation is taken over images ξ selected uniformly at random.
In our experiments, the expectation is replaced by 1 K (cid:80) Kk =1 σ ( x , s ( ξ k )) for K = 5 images ( ξ 1 , . . .
, ξ K ) selected uniformly at random from our dataset for every image considered.
Note that all pixel-removal metrics are naturally sensitive to the size of the region of interest, which varies widely between images.
Therefore, averaging yields conclusions biased towards smaller or larger objects (depending on the evaluation method).
This leads to considering pairwise tests , i.e. to compare the effectiveness of the methods per image.
For example, one can look at the win -statistics
(Chattopadhyay et al., 2017), i.e. the proportion of images when one saliency method outperforms another.
Naturally, the disadvantage of such a comparison is that it does not take into account how much better the different explanations are for individual images.
4.3 Decision-based evaluation
In Fig.
6, we compare random pixel removal, G RADIENTS , gradient × input attributions (Eq.
GRAD C AM , F ULL G RAD , and D EEP M APS with respect to decision-based pixel-removal metrics (note that the direction of the y -axes is chosen so that higher means better).
The top panels (in the MoRF and the LeRF section, resp.) show the average impact of pixel-removal procedures.
When removing pixels randomly, about 20% of the image needs to be removed for the decision to change.
This number drops to about 10% for D EEP M APS when removing the most relevant pixels ﬁrst.
Interestingly, n MoRF for GRAD C AM and F ULL G RAD are similar to those reported for G RADIENTS and input attributions, and are somewhat worse than for D EEP M APS (about 1214%).
Considering ∆ n MoRF , the comparison to alternative explanations, changes the picture (Fig. 6, bottom left).
First note that the random method gets score zero.
This is because all random saliency maps are (on average) equally good in changing network decisions.
G RADIENTS , on the other hand, yield explanations that are clearly image-speciﬁc: On average, 10% image pixels less need to be removed to change the network decision.
For D EEP M APS , the difference in the number of pixels removed by alternative explanations ( ξ k ) and by the true one is almost 20%.
The metric also shows the advantages of F ULL G RAD and GRAD C AM versus the simpler input-gradient-based methods.
A complementary picture is revealed when targeting the least relevant pixels (Fig. 6, right).
According to n LeRF , there is a clear beneﬁt from using smooth explanations.
Similarly to the ﬁndings of
Srinivas & Fleuret (2019), F ULL G RAD seems even more accurate than GRAD C AM in delineating the irrelevant regions.
With D EEP M APS , a whooping 60% of the image can be removed (on average) without changing the network’s decision.
In this case, the ∆ n LeRF metric ﬁnds D EEP M APS ,
F ULL G RAD , and GRAD C AM less advantageous compared to the input-based measures than n LeRF .
This is because if one can remove over half of the image without changing the network’s decision,
RandomGradients Gradient x input
GradCAMFullGradDeepMaps
AFOC MoRF n MoRF AFOC MoRF n MoRF
R a n d o m G r a d i e n t s
G r a d i e n t x i n p u t
G r a d C A M F u ll G r a d D e e p M a p s
RandomGradients Gradient x input
GradCAMFullGradDeepMaps
AFOC LeRF
R a n d o m G r a d i e n t s
G r a d i e n t x i n p u t
G r a d C A M F u ll G r a d D e e p M a p s n LeRF
R a n d o m G r a d i e n t s
G r a d i e n t x i n p u t
G r a d C A M F u ll G r a d D e e p M a p s
AFOC LeRF
R a n d o m G r a d i e n t s
G r a d i e n t x i n p u t
G r a d C A M F u ll G r a d D e e p M a p s n LeRF       w i
n50 % it is more likely that such a vast region will overlap with an explanation (involving a similarly large region) for some other image.
This metric is then less informative than its MoRF counterpart.
Finally, in Fig. 7, we provide a statistical summary for the results shown in Fig.
6 as well as AFOC .
The results are presented in the form of pairwise comparisons, and also include the ∆ -metric (which is applicable to any base metrics).
Each panel of Fig.
7 shows the frequencies (after subtracting 50%) at which the methods displayed on the left of the heat map yielded values better than the methods displayed at the bottom.
The name of the metric used for the comparison is indicated on the top of each heat map.
All metrics, except for ∆ AFOC LeRF , rank D EEP M APS the best among the tested methods.
We noted our reservation towards ∆ n LeRF already.
For ∆ AFOC LeRF , the comparison with alternative explanations makes even less sense, as the perturbation removes a mere 10% of the irrelevant pixels.
In lots of images, there could be a lot of such regions that are equally valid.
This metric can therefore be misleading.
In Appendix C we provide further analysis about the behavior of the evaluation metrics, demonstrating the statistical signiﬁcance of our ﬁndings above, as well as looking into the consistency of the induced rankings of the different saliency methods.
4.4 Explaining ResNet
So far, we extensively analysed the VGG-16 architecture, showing interesting properties and a clear advantage for D EEP M APS .
However, D EEP M APS yields interesting explanations for other architectures as well.
In Appendix F, we show detailed experiments conducted to analyse ResNet-50 following a protocol similar to our previous VGG-16 experiments.
A slight advantage for D EEP M APS is noticeable in sample images (see Appendix F), with again a better focus on details.
Regarding quantitative metrics, GRAD C AM , F ULL G RAD , and D EEP M APS all seem to have comparable quality with respect to the LeRF methods, while D EEP M APS seems slightly better for the MoRF metrics.
Furthermore, it is notable that D EEP M APS has a comparable quality to F ULL G RAD while using information from a signiﬁcantly lower number of layers.
In our experiments, D EEP M APS computes attributions from the outputs of the 17 residual blocks of the model, while F ULL G RAD requires the biases of all the 53 layers (including layer normalisation parameters).
To conclude, this experiment conﬁrms that the hidden-layer attributions are a source of information that is worth considering to explain the decisions of ResNets as well, showing a more efﬁcient use of information with comparable (or better) results than previous methods.
Conclusion
We considered visual explanations of deep image classiﬁers.
We demonstrated that hidden activities in the different layers are a better source of information than biases recommended by earlier work, and proposed a new method, D EEP M APS , to generate saliency maps from hidden activities.
We also introduced new measures to evaluate saliency maps, which take into account the decisions of the classiﬁer rather than the class scores, and are also able to identify very generic or random explanations.
We demonstrated experimentally that D EEP M APS clearly outperforms competing explanation methods on VGG-16, and has advantages in explaining the decisions of ResNet-50.
Acknowledgements
We would like to thank Razvan Pascanu and the anonymous reviewers for their insightful comments, which helped improve the presentation of the manuscript.
Omar Rivasplata gratefully acknowledges funding from the U.K. Engineering and Physical Sciences Research Council (EPSRC) under grant number EP/W522636/1.
This work was done while Omar was a research scientist intern at DeepMind.
A The role of bias parameters in ImageNet classiﬁers
We surveyed networks trained on ImageNet and available via the Lucid package (Silberman &
Guadarrama, 2016) to estimate how bias parameters contribute to prediction accuracy.
In Fig. 8, the networks are arranged by the number of bias parameters.
Best performing networks do not require the highest number of biases (e.g. InceptionV2 slim ).
Removing biases destroys the performance of most classiﬁers considered (Fig. 8, bottom), except for those that do not use batch normalization: AlexNet, CaffeNet, VGG16/
VGG19, InceptionV (GoogleNet) and Inception V1 caffe .
In fact, without biases, only CaffeNet suffers a drop in performance larger than 1%.
[Table]
I n c e p t i o n
V 3 _ s li m
N a s n e t M o b il e _ s li m
M o b il e n e t V 2 __ s li m
R e s n e t V_ 50 _ s li m
I n c e p t i o n
V 4 _ s li m
R e s n e t V_ 50 _ s li m
R e s n e t V__ s li m R e s n e t V__ s li m
I n c e p t i o n
R e s n e t V_ s li m R e s n e t V__ s li m
P n a s n e t L a r g e _ s li m N a s n e t L a r g e _ s li m
R e s n e t V__ s li m
[Figure]
B AOPC/AFOC metrics
Samek et al.
(2016) propose to measure the area over the perturbation curve : for an input image x , they deﬁne
[Equation]
where c ( x ) =
arg max c (cid:48)
f c (cid:48)
( x ) is the predicted class for image x , and x ( k )+ is obtained by replacing the k pixels having the highest values in the saliency map with the mean RGB values over the training set (MoRF), and P is the maximum number of pixels removed this way.
The AOPC is then obtained as the average value over all images in the dataset (Samek et al., 2016).
In our experiments,
P is set to remove 10% of the pixels and we use 5000 test images spanning classes 0–99.
[Table]
GradCAMFullGradDeepMaps
[Table]
R a n d o m G r a d i e n t s
G r a d i e n t x i n p u t
G r a d C A M F u ll G r a d D e e p M a p s
[Table]
GradCAMFullGradDeepMaps
AFOC
[Table]
R a n d o m G r a d i e n t s
G r a d i e n t x i n p u t
G r a d
[Table]
A M
[Table] [Table]
gradientGradCAMFullGradDeepMaps
[Table]
R a n d
[Table]
m G r a d i e n t s
I n p u t x g r a d i e n t G r a d C A M F u ll G r a d
D e e p M a p
[Table]
gradientGradCAMFullGradDeepMaps
AFOC
[Table]
R a n
[Table]
m
[Table]
I n p
[Table]
M
[Table]
M
[Table]
R a
[Table]
m
[Table]
n p
[Table]
x g r a d i e n t G r a d C A M F u ll G r a d
D e e p M a p s
AFOC LeRF
R a n d o m G r a d i e n t s
I n p u t x g r a d i e n t G r a d C A M F u ll G r a d
D e e p M a p s n LeRF      l o g 10 ( p )
If an explanation succeeds at detecting important regions of the input, the decrease in the class score is expected to be high.
AOPC( x ) is then expected to be higher for better explanations.
Srinivas & Fleuret (2019) consider the absolute fractional output change for input x deﬁned as
[Equation]
(11) where x ( k ) − denotes the image after replacing its k least-relevant pixels with the average pixel over the training set (LeRF).
The other quantities are deﬁned as for AOPC .
Srinivas & Fleuret (2019) remove up to 10% of the image.
Note that both metrics can have other variants, for example a MoRF version of AFOC or a LeRF version of AOPC .
When both variants are used, they are distinguished by appropriate subscripts.
All the statistical evaluations (Fig. 7, Fig. 9 and Fig. 16) are based on AFOC measured for both the most and the least salient perturbations (removing 10% of the image).
C Further empirical analysis of the evaluation metrics
In this section we provide further analysis of the behavior of the metrics, including statistical significance of our ﬁndings as well as the consistency of the saliency metrics.
C.1 Pairwise tests for VGG-
In Fig.
9, we present p -values from the Wilcoxon signed-rank test ( scipy .
stats .
wilcoxon , SciPy.
org ), performed on data shown in Fig.
7, which is repeated in Fig.
9 for convenience.
Colours are chosen such that white corresponds to the null hypothesis (metric values derived from two saliency methods are statistically indistinguishable).
The darkest colour corresponds to p < 10 − 30 ; note, however, that many of the tests yield values as low as 10 − 200 , or even zero.
All metrics except for ∆ AFOC LeRF depict D EEP M APS as the best method for extracting visual explanations from VGG-16 at a highly statistically signiﬁcant level ( p < 10 − 30 ).
See below for our hypothesis why ∆ AFOC LeRF is of a limited use.
AFOC LeRF , on the other hand, appears much more informative, distinguishing between every single explanation method with high signiﬁcance (the least signiﬁcant p -values are on the order of 10 − 10 ).
While AFOC LeRF seems to be much better metric than ∆ AFOC LeRF , the opposite is true for the MoRF metrics based on network’s decisions: While n MoRF overestimates the quality of gradient × input attributions (see below), the ∆ n MoRF metric ranks saliency methods in the same, intuitive way as AFOC LeRF .
Higher signiﬁcance (lower p -values) are achieved with ∆ n MoRF than with n MoRF and with ∆ AFOC MoRF than with AFOC MoRF in most pairwise comparisons (Fig.bottom).
In summary, ∆ comparisons reﬁne the message carried by n MoRF , and they provide a complementary information to n LeRF .
For VGG-16, they have a small effect on AFOC MoRF and corrupt the message carried by the AFOC LeRF metric, as described below.
On gradient × input attributions Both the ∆ AFOC LeRF and n MoRF metrics indicate a supreme quality of gradient × input attributions.
This might be an unfortunate side effect of our choice of image perturbations, which replace pixels with their expected value — grey.
As a result, “removing” grey pixels amounts to no change.
Now, gradient × input saliency maps attribute the lowest scores to grey pixels (see Fig. 3).
This is because grey corresponds to zero value across all colour channels, curtailing the gradient × input score.
Thus, gradient × input saliency maps will get an automatic boost in all LeRF metrics.
In a symmetrical manner, we can consider the most extreme pixel values, corresponding to black and white.
Removing these amounts to the strongest possible change to the input.
On the other hand, black and white are favoured by gradient × input scores.
Thus, gradient × input saliency maps would induce stronger perturbations in the MoRF procedures, leading to a boost of the corresponding metrics.
We believe this automatic, input-induced mechanism underlies the apparent beneﬁt of gradient × input attributions method according to ∆ AFOC LeRF and n MoRF metrics.
We would advise against using these metrics.
C.2 Consistency of evaluations
Tomsett et al.
(2020) suggested that good metrics to evaluate explanation methods should be consistent in the sense that they should rank different explanation methods similarly on individual images.
Following their work, in this section we analyze the consistency of the evaluation metrics considered in the paper in ranking the different explanation methods, as well as the similarity of the different metrics (as baseline, we only present results for AFOC and not for AOPC , since the former is consistently slightly better and otherwise they behave similarly).
We consider two measures from
Tomsett et al.
(2020), taken from the psychometric literature (Peter, 1979): inter-rater reliability and internal consistency reliability.
Inter-rater reliability is measured by Krippendorff’s α coefﬁcient, which quantiﬁes the consistency of ranking over multiple images: a value of 1 (the maximum value) means that all explanation methods are ranked in the same way on every image and 0 that the rankings on different images are independent.
We used the Krippendorff-alpha package (Grill, 2017), and applied it to the ranks assigned by a given metric to the six saliency methods considered (Random, G RADIENTS , gradient × input,
GRAD C AM , F ULL G RAD , D EEP M APS ).
The average α coefﬁcients, computed over 500 randomly selected images (out of 5000) and repeated 50 times, are shown in Fig.
10 for VGG-16.
One can observe that our n LeRF and ∆ n MoRF saliency metrics are more consistent than the traditional metrics.
Moreover, our modiﬁcation ∆ σ (cf.
Eq.
(9)) to compare to the alternative explanations clearly improves the consistency for any MoRF metric σ .
On the other hand, this modiﬁcation decreases the
[Figure]
consistency for LeRF metrics.
A potential reason for this is that our images are often centered, thus the unimportant regions of the images may overlap quite a bit, hence alternative LeRF explanations can work reasonably well (see the paragraph ”On gradient × input attributions” in Appendix C.1 for another reason behind ∆ σ being less effective in the LeRF setup).
[Figure]
The internal consistency reliability, computed by Spearman’s rank correlation ρ , measures pairwise similarity between the saliency metrics.
Fig.
11 presents pairwise correlations between all evaluation metrics (we use the negative of the n MoRF and
AFOC LeRF metrics since for these, larger is better), for two saliency methods, Random and D EEP M APS (computed over our dataset of 5000 images).
Results on the Random saliency map conﬁrm that applying ∆ σ instead of the original evaluation method σ reduces the similarity in evaluating a meaningless (random) saliency map: while n MoRF , n LeRF , and AFOC MoRF , AFOC LeRF are highly correlated, 5 their ∆ -transformed versions are not.
On the other hand, the ∆ -transformation has much smaller impact on sensible visual explanations, such as D EEP M APS , for the evaluation metrics n LeRF and AFOC MoRF ( ρ = 0 .
68 and ρ = 0 .
76 , respectively).
Thus, ∆ n LeRF and ∆ AFOC MoRF measure similar qualities of explanations as intended with the original metrics.
These conclusions are similar to those we drew by comparing win statistics (Fig. 9).
Taken the consistency results into account, if one wants to use multiple metrics to evaluate saliency methods as recommended by Tomsett et al.
(2020), we suggest to use the metrics ∆ AFOC MoRF (or ∆ n MoRF ) and n LeRF , as they are the most consistent while measuring different aspects of the visual explanations, with ρ (∆ AFOC MoRF , n LeRF ) = 0 .
26 ( ρ (∆ n MoRF , n LeRF ) = 0 .
While all the results presented in this section were shown for VGG-16, we get very similar results for ResNet-50 (see Appendix F).
[Table]
[Figure]
FullGrad
DeepMaps
[Figure]
FullGrad
DeepMaps
We showed that D EEP M APS is competitive as a method for producing visual explanations.
Here, we take a step back to evaluate what makes the difference.
In Fig.
12, we use one of the most informative metrics we introduced, ∆ n MoRF , to evaluate the quality of visual explanations produced by each layer of VGG-16.
In the top panel of Fig.
12, our evaluation relates to per-layer attributions a h,l , a b,l (such as the ones shown in Fig. 2).
Our metrics conﬁrm the visual assessment: for every layer, hidden activities yield a better explanation than the corresponding biases.
The quality of the last convolutional layer (the problematic one, with the highest impact on the ﬁnal score, layer 13) is zero.
The bottom panel in Fig.
12 relates to the alternative way of aggregating explanations: while
D EEP M APS ﬁrst aggregates attributions by channels (summing over φ in Eq.
and then normalises with Ψ , F ULL G RAD normalises every feature attribution (such as a b,lφ ) and then sums these for all φ .
In the bottom panel the
Here, both methods aggregate over all layers in the network.
Given the variable quality of layer attributions, it is possible that aggregating a subset of layers would do a better job.
This requires the development of a proper procedure and is left for future work.
Using matrix factorisation, as done by Gur et al.
(2021), is another idea of combining layer- and channel-speciﬁc explanations, which is likely to beneﬁt approaches such as F ULL G RAD (based on input and bias attributions) or D EEP M APS (hidden layer attributions).
E S MOOTH G RAD
S MOOTH G RAD , by Smilkov et al.
(2017), is another approach to increase human readability of saliency maps.
The trick is to aggregate saliency maps for a bunch of samples from a local neighbourhood of a given image.
Average over samples appears less noisy than any sample and is often easier to decipher by eye, but is it more faithfully depicting the network’s sensitivity?
We combined S MOOTH G RAD with the methods studied in our paper.
While all saliency maps appear “smoothed” by this method, the metrics improve only for G RADIENTS , and the methods based on deeper activities remain better, with D EEP M APS without smoothing remaining the best, as shown in
Fig.
Thus, according to our metrics, S MOOTH G RAD does not produce more faithful saliency maps.
On the other hand, we could apply the smoothing idea of S MOOTH G RAD to the metrics as well (by performing perturbations on local neighbours of a given image).
This would prefer explanations which are more robust to input perturbations.
Analysing such metrics is left for future work.
[Figure]
F ResNET
Details of the ResNet-50 architecture are given by He et al.
In short, the network consists of 1 + 16 × 3 blocks of convolutions.
After the initial convolution (which reduces the output size from 224 × 224 to 112 × 112), the convolutions come in blocks of three, each triplet accompanied by a residual connection and followed by a ReLU nonlinearity (see Fig.
3 of He et al., 2016).
In our implementation of D EEP M APS , we extract explanations from ReLU layers that follow the initial convolution, and each of the triplets, i.e. every time the parallel routes of computation come together again.
Altogether, these amount to 17 hidden-activity attributions ( a h,l ).
In the implementation of F ULL G RAD , we had to extract all parameters due to the batch normalisation (see below).
These amounted to 53 blocks, and included parallel computational routes.
The only regular bias parameters used in this network are assigned to the output layer, and as such, they do not enter the computation of the F ULL G RAD saliency maps.
Batch normalisation Batch normalisation (Ioffe & Szegedy, 2015) results in the following additive component:
[Equation]
where β and γ are trainable parameters and x k is the activity of a given neuron (or a convolutional kernel).
During training, activity statistics are collected online, however, at test time, these act as parameters.
Saliency maps In Fig.
14, we show example saliency maps for ResNet-50.
Similarly to VGG-16, the explanations extracted by D EEP M APS contain ﬁner details than those extracted with F ULL G RAD and GRAD C AM .
All three are more interpretable than the G RADIENTS explanations.
A chequered pattern appears in F ULL G RAD saliency maps, it is much less prominent in D EEP M APS and absent from GRAD C AM , indicating that it is not due to the top convolutional layer.
Evaluation In Fig.
15, we present results for ResNet-50 using the same metrics as for VGG-16 in
Fig.
On average, random perturbations have the same effect in both networks — around 23% of the image needs to be removed for the classiﬁcation to change.
However, when guided by any other saliency map, it takes more pixels to change the decisions of ResNet-50 than of VGG-16, both for the MoRF and LeRF perturbations.
[Figure] [Figure]
There is a striking similarity between the shape of n MoRF for the two networks, with identical ranking of the explanation methods according to this metric.
Here again, gradient × input attributions appear competitive to the other, more interpretable methods, indicating that this metric is not faithfully reﬂecting our explainability prior (see Appendix C.1).
When comparing ∆ n MoRF in Fig.
6 and Fig.
15, there is a clear difference for GRAD C AM , which in
ResNet-50 appears to be as effective as F ULL G RAD and D EEP M APS .
A similar picture is derived from the n LeRF metric: both GRAD C AM and F ULL G RAD allow to remove more pixels in LeRF fashion than in the case of VGG-16.
Pairwise comparisons (Fig. 16) conﬁrm these observations: F ULL G RAD offers more concise explanations (highest n LeRF count, albeit not signiﬁcant, with p > 0 .
1 when compared to GRADC AM ); LeRF depiction is more image-speciﬁc in D EEP M APS (highest ∆ n LeRF count, p < 0 .
002 , win=52.7% compared to GRAD C AM ).
GRAD C AM appears most accurate in image-speciﬁc delineation of the most important regions (lowest ∆ n MoRF count), albeit the difference to the other methods is small ( p < 0 . 04 , win=51.1% compared to D EEP M APS ).
D EEP M APS is the best according to most of the MoRF metrics, although for AFOC MoRF and n MoRF , they compete with gradient × input attributions (the problem we deliberated on in Appendix C.1).
According to ∆ n MoRF and LeRF metrics, they are very similar to F ULL G RAD and GRAD C AM .
In summary, for ResNet-50, there is no clear choice for the explanation method.
Depending on whether we care about the change in classiﬁcation conﬁdence (as approximated by AFOC ) or classiﬁcation itself (measured by the number of pixels), we might choose D EEP M APS , F ULL G RAD or
GRAD C AM .
However note that in D EEP M APS , we have the ﬂexibility of choosing which maps to aggregate.
For example, limiting the aggregation to the top layers is likely to help against the chequered pattern evident in Fig.
14 for F ULL G RAD and D EEP M APS , but not for GRAD C AM , which focuses on the top layer
[Figure]
[Figure]
pace of upgrades to TensorFlow.
We veriﬁed that all the input and bias attributions sum up to the class score, f c ( x )
=
(cid:80) i,j a cij ( x ) + (cid:80)