Where do Models go Wrong? Parameter-Space Saliency Maps for Explainability
Anonymous Author(s)
AffiliationAddress email
Abstract
Conventional saliency maps highlight input features to which neural networkpredictions are highly sensitive.
We take a different approach to saliency, inwhich we identify and analyze the network parameters, rather than inputs, whichare responsible for erroneous decisions.
We first verify that identified salientparameters are indeed responsible for misclassification by showing that turningthese parameters off improves predictions on the associated samples, more thanturning off the same number of random or least salient parameters.
We furthervalidate the link between salient parameters and network misclassification errorsby observing that fine-tuning a small number of the most salient parameters on asingle sample results in error correction on other samples which were misclassifiedfor similar reasons – nearest neighbors in the saliency space.
After validating ourparameter-space saliency maps, we demonstrate that samples which cause similarparameters to malfunction are semantically similar.
Further, we introduce aninput-space saliency counterpart which reveals how image features cause specificnetwork components to malfunction.
Introduction
With the widespread deployment of deep neural networks in high-stakes applications such as medicalimaging [20], credit score assessment [49], and facial recognition [10], practitioners need to understand why their models make the decisions they do.
In fact, “right to explanation” legislation in the
European Union and the United States dictates that relevant public and private organizations must beable to justify the decisions their algorithms make [46, 14].
Diagnosing the causes of system failuresis particularly crucial for understanding the flaws and limitations of models we intend to employ.
Conventional saliency methods focus on highlighting sensitive pixels [37] or image regions thatmaximize specific activations [13].
However, such maps may not be useful in diagnosing undesirablemodel behaviors as they do not necessarily identify areas that specifically cause bad performancesince the most sensitive pixels may not be the ones responsible for triggering misclassification.
We develop an alternative approach to saliency which highlights network parameters that influencedecisions rather than input features.
These parameter saliency maps yield a number of useful analyses:
• We verify that identified salient parameters are indeed responsible for misclassification by
showing that turning these parameters off improves predictions on the associated samples
more than turning off the same number of random or least salient parameters.
• Nearest neighbors in parameter saliency space share common semantic information.
Thatis, samples which are misclassified for similar reasons and cause similar parameters to
malfunction are semantically similar.
[Figure]
• We further validate the link between salient parameters and network misclassification errors
by observing that correcting a small number of the most salient parameters by fine-tuning
them on a single sample results in error correction on other samples which were misclassified
for similar reasons.
• By first identifying the network parameters responsible for an erroneous classification,we can then visualize the image regions that interact with those parameters and trigger
the identified misbehavior obtaining interpretable insights into model mistakes and neuralnetwork’s reliance on spurious correlations.
1.1 Related Work
Neural network interpretability and parameter importance.
A major line of work in neuralnetwork interpretability focuses on convolutional neural networks.
Works visualizing, interpreting,and analysing feature maps [52, 51, 27, 25] provide insight into the role of individual convolutionalfilters.
These methods, together with other approaches for filter explainability [7, 56, 57] find thatindividual convolutional filters often are responsible for specific tasks such as edge, shape, andtexture detection.
Other ideas in neural network interpretability include identifying recurring patternsin model behavior [8].
The ideas of measuring neural network parameter importance has been studied in multiple contexts.
Notions of neuron [34], feature [26, 35, 41, 30] and parameter [40, 47] importance have been usedfor AI explainability, manipulating model behavior [6], model debugging [24, 53] and pruning[1, 23, 42, 32, 50] for improving model performance.
Input space saliency maps.
A considerable amount of literature focuses on identifying inputfeatures that are important for neural network decisions.
These methods include using deconvolutionapproaches [52] and data gradient information [37].
Several works build on these ideas and proposeimprovements such as Integrated Gradients [44], SmoothGrad [38], and Guided Backpropagation[39] which result in sharper and more localized saliency maps.
Other approaches focus on the useof class activation maps [55] with improvements incorporating gradient information [34] and morenovel approaches to weighting the activation maps [48].
In addition, various saliency methods arebased on manipulating the input image [15, 52, 30, 3].Another line of work is aimed at evaluating theeffectiveness of saliency maps and developing validation techniques [2, 4, 5, 54, 33].
Our work combines the ideas of saliency maps and parameter importance and evaluates saliencydirectly on model parameters by aggregating their absolute gradients on a filter level.
We leveragethe resulting parameter saliency profiles as an explainability tool and develop an input-space saliencycounterpart which highlights image features that cause specific filters to malfunction to study theinteraction between the image features and the erroneous filters.
Method
It is known that different network filters are responsible for identifying different image properties andobjects [52, 51, 27, 25].
This motivates the idea that mistakes made on wrongly classified images can
[Figure]
be understood by investigating the network parameters, rather than only the pixels, that played a rolein making a decision. We develop parameter-space saliency methods geared towards identifying andanalyzing neural network parameters that are responsible for making erroneous decisions.
Central toour method is the use of gradient information of the loss function as a measure of parameter sensitivity.
2.1 Parameter Saliency Profile
Let x be a sample in the validation set D with label y , and suppose a trained classifier has parametersθ that minimize a loss function L .
We define the parameter-wise saliency profile of x as a vectors ( x, y ) with entries s ( x, y ) i :=
|∇ θ i L θ ( x, y ) | , the magnitudes of the gradient of the loss with respectto each model parameter.
Because the gradients on training data for a model trained to convergenceare near zero, it is important to specify that D be a validation, or holdout, set.
Intuitively, a largergradient norm at the point ( x, y ) indicates a greater inefficiency in the network’s classification ofsample x , and thus each entry of s ( x, y ) measures the suboptimality of individual parameters.
Aggregation of parameter saliency.
Convolutional filters are known to specialize in tasks suchas edge, shape, and texture detection [51, 7, 27].
We therefore choose to aggregate saliency onthe filter-wise basis by averaging the gradient magnitudes of parameters corresponding to eachconvolutional filter.
This allows us to isolate filters to which the loss is most sensitive ( i.e. thosewhich, when corrected, lead to the greatest reduction in loss).
Formally, for each convolutional filter F k in the network, consider its respective index set α k ,which gives the indices of parameters corresponding to the filter F k (the number of parameterscorresponding to F k is then given by | α k | ).
The filter-wise saliency profile of a sample x with label yis defined to be a vector s ( x, y ) with entries
[Equation]
resulting from averaging the parameter-wise saliency profile of the sample x with label y on the filterlevel.
Standardizing parameter saliency.
Figure 1 exhibits the ResNet-50 [18] filter-wise saliency profileaveraged over the ImageNet [9] validation set, where filters within each layer are sorted from highestto lowest saliency.
One clear observation is the difference in the scale of gradient magnitudes –shallower filters are more salient than deeper filters.
This phenomenon might occur for a number ofreasons.
First, early filters encode low-level features, such as edges and textures, which are activeacross a wide spectrum of images.
Second, typical networks have fewer filters in shallow layers thanin deep layers, making each individual filter more influential at shallower layers.
Third, the effects ofearly filters cascade and accumulate as they pass through a network.
To isolate filters that uniquely cause erroneous behavior on particular samples, we find filters that areabnormally salient for a sample, x , but not for others.
That is, we further standardize the saliencyprofile of x with respect to all filter-wise saliency profiles of D .
Formally, let µ be the average filter-wise saliency profile across all x ∈ D , and let σ be an equallength vector with the corresponding standard deviation for each entry.
We use these statistics toproduce the standardized filter-wise saliency profile as follows:
[Equation]
The resulting tensor ˆ s ( x, y ) is of length equal to the number of convolutional filters in the network,and we henceforth call it the saliency profile for sample x .
By standardizing saliency profiles, wecreate a saliency map that activates when the importance of a filter is unusually strong relative to othersamples in the dataset.
This prevents the saliency map from highlighting filters that are uniformlyimportant for all images, and instead focuses saliency on filters that are uniquely important and servean image-dependent role.
In the rest of the paper, unless explicitly noted otherwise, we use ˆ s ( x, y )and refer to it as parameter saliency .
Incorrectly classified samples are more salient.
Empirically, we observe the saliency profiles ofincorrectly classified samples exhibit, on average, greater values than those of correctly classifiedexamples.
This bolsters the intuition that salient filters are precisely those malfunctioning — ifthe classification is correct, there should be few malfunctioning filters or none at all.
Moreover,we see deeper parts of the network appear to be most salient for the incorrectly classified sampleswhile earlier layers are often the most salient for correctly classified samples.
An example ofthese behaviors for ResNet-50 is shown in Figure 2 which presents standardized filter-wise saliencyprofiles averaged over the correctly and incorrectly classified examples from the ImageNet validationset.
Additionally, we note the improved relative scale of the standardized saliency profile acrossdifferent layers compared to the absolute gradient magnitudes in Figure 1.
Saliency profiles forother architectures could be found in Appendix A. Henceforth, we will focus specifically on saliencyprofiles of misclassified samples in order to explore how neural networks make mistakes.
2.2 Input-Space Saliency for Visualizing How Filters Malfunction
[Figure]
The parameter saliency profile allows us to identify filters that are most responsible for mistakes anderroneous network behavior.
In this section, we develop an input-space counterpart to our parametersaliency method to understand which features of the image affect the saliency of particular filters.
Geiping et al.
[17] show that the gradient information of a network is invertible, providing a linkbetween input space and parameter saliency space.
This work, along with existing input-spacesaliency map tools [37, 39, 38, 55, 34], inspires our method.
Given a parameter saliency profile ˆ s = ˆ s ( x, y ) for an image x with label y , our goal is to highlightthe input features that drive large filter saliency values.
That is, we would like to identify imagepixels altering which can make filters more salient.
To this end, we first select some set F of themost salient filters that we would like to explore.
Then, we create a boosted saliency profile s ′ F byincreasing the entries of ˆ s corresponding to the chosen filters F :
[Equation]
k > 1 (we used k = 100 in our experiments).
Now, we can find pixels that are important for makingthe chosen filters F more salient and, equivalently, making the filter saliency profile
ˆ s ( x, y ) close tothe boosted saliency profile s ′ F by taking the following gradients:
[Equation]
where D C ( · , · ) is cosine distance.
The resulting input saliency map M F contains input features (pixels) that affect the saliency of thechosen filters F the most.
Experiments
In this section, we aim to validate the meaningfulness of our parameter saliency method.
First,we verify that salient parameters are indeed responsible for misclassification by showing on the [Figure] dataset level that turning them off improves predictions on the associated samples more than off the same number of random or least salient parameters.
This experiment is similar in spirit toremoving salient features to validate input-space saliency methods [5, 33].
We then find that sampleswhich cause similar filters to malfunction are semantically similar.
We also show on the datasetlevel that fine-tuning a small number of the most salient parameters on a single sample results inerror correction on other samples which were misclassified for similar reasons and cause similarparameters to malfunction.
We then use our input-space saliency technique in conjunction with itsparameter-space counterpart as an explainability tool to explore how neural networks make mistakesand how salient filters interact with visual input features.
We evaluate our saliency method in the context of image classification on CIFAR-10 [21] and
ImageNet [9].
Images we use for visualization, unless otherwise specified, are sampled from
ImageNet validation set.
Throughout the experiments, we use a pre-trained ResNet-18 classifier[18] on CIFAR-10 and a pre-trained ResNet-50 on ImageNet.
Both models are trained in a standardfashion on the corresponding dataset 12 .
3.1 Turning Off Salient Filters
We begin validating our parameter-space saliency maps by observing the effect of turning off thesalient filters.
In order to remove the influence of a particular salient filter, we prune it – zero outthe filter weights and also the biases of the associated batch normalization layers.
This procedureguarantees that the corresponding input feature map to the next convolutional layer is always zero.
We gradually increase the number of pruned most salient filters and track three metrics: the changein the incorrect class confidence, the change in the true class confidence, and the percentage ofthe samples that flip their label to the correct class.
In every case, we compare pruning the mostsalient filters against pruning the same number of random filters and the least salient filters.
Theseexperiments are performed on the dataset level: we average the trends across all misclassified imagesin the ImageNet validation set.
As shown in Figure 3, pruning the most salient filters is significantly more effective for decreasingthe incorrect class confidence than random or least salient filters.
Specifically, gradually pruning thetop 100 salient filters achieves up to 30% drop in the incorrect class confidence score while pruningrandom filters yields only about 7% decrease.
We also note that pruning the least salient filters doesnot produce any effect on the incorrect class confidence.
We repeat the same experiment with the true class confidence and observe that the highest true classconfidence gain occurs when we prune around 20 most salient filters.
Pruning enough salient filterseventually leads to a gradual decrease in the true class confidence.
We note that this behavior isexpected since we are destroying, not correcting, the inference power of all of the most sensitivefilters for an image, some of which may be essential for inference.
Finally, pruning random filters [Figure] provides a much slower increase in the true confidence class while the least salient filters again donot produce a significant effect.
In addition, we count the number of images that were corrected as a result of pruning and find thatpruning around 30 most salient filters results in the best correct classification rate of 12%.
Similarto the true class confidence, the trend decreases beyond this point.
Pruning random filters increasesthe percentage of corrected samples at a much slower rate and does not perform better than the mostsalient filters when pruning up to 100 filters.
Notably, pruning the least salient filters manages tocorrect a nontrivial number of samples but still much smaller than pruning random filters.
These experiments suggest that the identified salient parameters are indeed responsible for misclassification since turning these parameters off improves predictions on the associated samples more thanturning off the same number of random or least salient parameters.
3.2 Nearest Neighbors in Parameter Saliency space
We validate the semantic meaning of our saliency profiles by clustering images based on the cosinesimilarity of their profiles.
In this section, we present visual depictions of a nearest neighbor searchamong all images in the ImageNet validation set.
We also conduct this analysis on CIFAR-10 images,and this can be found in Appendix A.
We find that the nearest neighbors of misclassified images in saliency space are mostly other misclassified images from the same pair of predicted and true classes but possibly in reverse order.
Forexample, in Figure 4, the reference image in (a) is a great Pyrenees misclassified as kuvasz, and the4 images with the most similar profiles exhibit either the same misclassification or the reverse (i.e.,kuvasz misclassified as great Pyrenees).
Intuitively, the common salient parameters across theseneighbors are those which are important for discriminating between the two classes in question butare not well-tuned for this purpose.
Note that we find the concept of “being similar” in parameter saliency space to be different from theone in image space.
The nearest neighbors we find are often not similar in a pixel-wise sense, butrather they are similar in their reason for causing misclassification.
For example, images in Figure 4(b) are beagles mistaken by a network for basset hounds and vice versa. We find that these pictures [Figure] d s
[Figure]
l
[Figure] s [Figure] [Figure]
r
[Figure]
g
[Figure]
are either taken from a high angle or do not include the dog’s legs, making the leg length, a major distinction between the two breeds, indistinguishable from the picture.
We include more images along with their nearest neighbors in Appendix A
In addition, we compute nearest neighbors when only considering filters in a specific range of layersin order to visualize the types of misbehavior triggered by network components (filters) at variousnetwork depths.
We search for similar images using parameter saliency in the shallow and deep layersof a VGG-19 network [36], which we divide into the shallow and deep parts that respectively occurup to and after layer relu4_1 .
The top row of figure Figure 5 shows neighbors found using shallowparameters, which share basic image attributes such as color histogram, while images in the bottomrow share more abstract similarities.
3.3 Correcting Mistakes by Fine-Tuning Salient Filters
To validate that salient filters are more responsible for the erroneous behavior of neural networks,we show that updating a limited number of salient filters on a single misclassified image can correctthe mistakes made by a neural network on other images that were misclassified because of similarlymalfunctioning filters – on nearest neighbors of that image in the parameter saliency space.
In thisexperiment we fine-tune a pretrained ResNet-50 for one step on a single image for which the networkmakes a wrong prediction and track the model’s performance on the image’s nearest neighbors foundthrough the process introduced in Section 3.2.
While filter saliency profiles are standardized withrespect to the ImageNet validation set, we find the nearest neighbors from an independent test set– ImageNet-v2, collected by Recht et al.
[31] separately from the original ImageNet data.
This way,the label information of the nearest neighbors is not used in fine tuning in any way.
We restrict the number of tunable filters to be no more than 1.0% of the total number of filters in thenetwork, and we update the chosen filters by taking one step of gradient descent with a normalizedstep.
We compare fine-tuning the most salient filters with two other choices of tunable filters: theleast salient filters and random filters.
We use misclassified images from the ImageNet validation set,making up to over 10,000 samples.
We evaluate the effect of fine-tuning on each of these imagesindependently and find the corresponding nearest neighbors from a holdout ImageNet-v2 test setwhere we limit the search scope exclusively to misclassified images too.
In Figure 6, we compare the average performance of our three choices of tunable filters under threemetrics.
Panel (a) presents a the percentage of samples that are corrected after fine-tuning as a sanitycheck and we see that updating 150 most salient filters ( ∼ 0.6% of total filters) is enough to achievethe same percentage of corrections as updating the entire network.
The second and third metrics evaluate the effect on the nearest neighbors of the training sample(Figure 6 (b),(c)).
Note that for a given training sample, its nearest neighbors are not involved in ourone-step single sample fine-tuning process.
By tracking model predictions and true class confidencescores among the 10 nearest neighbors of each sample, we find that fine-tuning salient filters issignificantly more effective than other choices.
Results in Figure 6, (b) and (c), also imply that thenearest neighbors found using our method are the images that are wrong for similar reasons and that [Figure] [Figure]
F il
[Figure]
e r s a li e n c y c h a n g e ( % ) [Figure] salient [Figure]
across
[Figure]
they can be corrected altogether by only updating the salient filters on a single image.
We note thatwe do not propose a new pruning or fine-tuning method.
Rather, we use these experiments to verifythat the salient filters are indeed responsible for misclassification.
3.4 Interpretable Network Failures: Visualizing Input Features Which Trigger FilterMalfunctions
Case study.
To present an example of how our approach could be used as an explainability tool, webegin this section with a case study of an image misclassified by ResNet-50 as “killer whale” (Figure7(a)).
The correct label of the image is “great white shark”.
Our goal is to study the interaction betweenthe most salient filters and input features.
We first identify filters most responsible for misclassificationby computing the filter saliency profile and visualize parts of the image that drive the high saliencyvalues for those filters using the input-space saliency counterpart (as introduced in Section 2.2).
Panel (b) of Figure 7 presents our image-space visualization for the ten most salient filters – the pixelswhich trigger misbehavior in these filters are highlighted.
For example, we see that the seal and boatare both triggers.
One natural hypothesis is that the seal looks like a killer whale to the network and isthe source of the classification error.
We check this by masking out the seal (Figure 7 (d)) .
However,although the probability of “killer whale” goes down and the probability of the correct class increases,the network still misclassifies the image as “killer whale”.
Now, if we mask out exactly the most salient areas of the image according to our visualization(see Figure 7 (b), (e)), the network manages to flip the label of the image and classify it correctly.
If we extend our mask to the less pronounced, but still salient, areas of the image as in Figure 7(f), we observe that the correct class confidence increases even more while the probability of theincorrect “killer whale” label further decreases.
Additionally, we find that masking out the non-salientparts of the image results in even worse misclassification confidence than that of the original image(see Appendix A).
In order to further investigate the effect of the salient region, we pasted it fromthis image onto other great white shark images (see Appendix A) and observed that this drives the [Figure] probability of “killer whale” up for 39 out of 40 examples of great white sharks from the ImageNetvalidation set with an average increase of 3 .
Our experiments suggest that secondary objects in the image are associated with the misclassification.
However, we see that the erroneous behavior of the model does not just stem from classifying anon-target object in the image.
It is possible that the model correlates the combination of sea creatures(e.g.
a seal) and man-made structures (e.g. a boat) with the “killer whale” label.
We note that imagesof killer whales in ImageNet often have man-made structures which look similar to the boat (see
Appendix A for examples).
Finally, at each step of our masking experiments, we recompute the saliency values of the originallychosen 10 filters (i.e. the filters that caused erroneous behavior on the reference image).
From Figure7 (c), we observe that as we mask out the input features according to our input-saliency, the saliencyvalues of those filters decrease gradually and reach an 80% drop, confirming that highlighted regionsindeed drive the high saliency of the chosen filters.
Dataset-level masking experiment.
We performed the experiment of masking the non-foregroundimage areas on the dataset level and observed similar trends to the case study.
Specifically, we traineda ResNet-50 on half of the ImageNet train set, and then used the incorrectly classified images from theother half of the train set with target object bounding boxes for the experiment (we used annotationsfrom the ImageNet website 3 ).
We computed our input saliency for every image and masked out themost salient regions of those images while preserving the target object.
We observed that as a resultof masking, the softmax output corresponding to the correct class increases (p < 0.05) on average anddecreases for the incorrect class (p < 0.05), while masking the same number of randomly selectedpixels (similarly outside of the target object bounding box) does not produce a statistically significantchange in either correct or incorrect class confidence scores (p = 0.854 and p = 0.695, respectively).
Figure 8 showcases input space visualizations with different illustrative examples of neural networkmistakes.
For a thorough discussion of mistake categories we find using our method, we refer to
Appendix A.9.
Discussion
Numerous applications demand that practitioners be able to understand the decisions their modelsmake, especially when those decisions are incorrect.
Existing methods for explainability focuson locating the input regions to which the network’s output is sensitive or on associating networkcomponents with specific roles.
In contrast, we develop a framework for finding the exact filterswhich are responsible for faulty predictions and studying the interactions between these filters andimages.
This direction yields an interpretable understanding of model behaviors.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Appendix C
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeAppendix C
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We include the
code in the supplementary material as well as detailed instructions on how to use it.
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Appendix B
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix B
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] See footnotes
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We include the code in the supplementary material as well as detailed instructions on
how to use it.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
We do not work with human subjects or with user data
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
We do not work with human subjects or with
user data
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]