W HERE DO M ODELS GO W RONG ? P ARAMETER -S PACE S ALIENCY M APS FOR E XPLAINABILITY
Anonymous authors
Paper under double-blind review
A BSTRACT
Conventional saliency maps highlight input features to which neural network predictions are highly sensitive.
We take a different approach to saliency, in which we identify and analyze the network parameters, rather than inputs, which are responsible for erroneous decisions.
We ﬁrst verify that identiﬁed salient parameters are indeed responsible for misclassiﬁcation by showing that turning these parameters off improves predictions on the associated samples, more than pruning the same number of random or least salient parameters.
We further validate the link between salient parameters and network misclassiﬁcation errors by observing that ﬁne-tuning a small number of the most salient parameters on a single sample results in error correction on other samples which were misclassiﬁed for similar reasons – nearest neighbors in the saliency space.
After validating our parameter-space saliency maps, we demonstrate that samples which cause similar parameters to malfunction are semantically similar.
Further, we introduce an input-space saliency counterpart which reveals how image features cause speciﬁc network components to malfunction.
I NTRODUCTION
With the widespread deployment of deep neural networks in high-stakes applications such as medical imaging (Kang et al., 2017), credit score assessment (West, 2000), and facial recognition (Deng et al., 2019), practitioners need to understand why their models make the decisions they do.
In fact, “right to explanation” legislation in the European Union and the United States dictates that relevant public and private organizations must be able to justify the decisions their algorithms make (United
States Congress Senate Committee on Banking and Housing and Urban Affairs, 1976; European
Commission, 2018).
Diagnosing the causes of system failures is particularly crucial for understanding the ﬂaws and limitations of models we intend to employ.
Conventional saliency methods focus on highlighting sensitive pixels (Simonyan et al., 2014) or image regions that maximize speciﬁc activations (Erhan et al., 2009).
However, such maps may not be useful in diagnosing undesirable model behaviors as they do not necessarily identify areas that
speciﬁcally
cause bad performance since the most sensitive pixels may not be the ones responsible for triggering misclassiﬁcation.
We develop an alternative approach to saliency which highlights network parameters that inﬂuence decisions rather than input features.
These parameter saliency maps yield a number of useful analyses:
• Nearest neighbors in parameter saliency space share common semantic information.
That is, samples which are misclassiﬁed for similar reasons and cause similar parameters to malfunction are semantically similar.
• By ﬁrst identifying the network parameters responsible for an erroneous classiﬁcation, we can then visualize the image regions that interact with those parameters and trigger the identiﬁed misbehavior.
• We verify that identiﬁed salient parameters are indeed responsible for misclassiﬁcation by showing that turning these parameters off improves predictions on the associated samples, more than pruning the same number of random or least salient parameters.
• We further validate the link between salient parameters and network misclassiﬁcation errors by observing that ﬁne-tuning a small number of the most salient parameters on a single
sample results in error correction on other samples which were misclassiﬁed for similar reasons.
After carefully delineating our methodology and experimentally validating the meaningfulness of our parameter saliency maps, we showcase the practical utility of this paradigm as an explainability tool with a case study in which we are able to uncover a neural network’s reliance on a spurious correlation which causes interpretable failures.
1. R ELATED WORK
Neural network interpretability and parameter importance.
A major line of work in neural network interpretability focuses on convolutional neural networks.
Works visualizing, interpreting, and analysing feature maps (Zeiler & Fergus, 2014;
Yosinski et al., 2015; Olah et al., 2017; Mahendran & Vedaldi, 2015) provide insight into the role of individual convolutional ﬁlters.
These methods, together with other approaches for ﬁlter explainability (Bau et al., 2017; Zhou et al., 2018; 2019)
ﬁnd that individual convolutional ﬁlters often are responsible for speciﬁc tasks such as edge, shape, and texture detection.
The idea of measuring neural network parameter importance has been studied in multiple contexts.
Notions of neuron and parameter importance have been used for AI explainability (Srinivas &
Fleuret, 2019; Selvaraju et al., 2017; Morcos et al., 2018; Shrikumar et al., 2017; Shrikumar et al.), manipulating model behavior (Bau et al., 2018), and parameter pruning (Abbasi-Asl & Yu, 2017; Liu & Wu, 2019).
Input space saliency maps.
A considerable amount of literature focuses on identifying input features that are important for neural network decisions.
These methods include using deconvolution approaches (Zeiler & Fergus, 2014) and data gradient information (Simonyan et al., 2014).
Several works build on these ideas and propose improvements such as Integrated Gradients (Sundararajan et al., 2017), SmoothGrad (Smilkov et al., 2017), and Guided Backpropagation (Springenberg et al., 2015) which result in sharper and more localized saliency maps.
Other approaches focus on the use of class activation maps (Zhou et al., 2016) with improvements incorporating gradient information (Selvaraju et al., 2017) and more novel approaches to weighting the activation maps (Wang et al., 2020).
In addition, various saliency methods are based on manipulating the input image (Fong &
Vedaldi, 2017; Zeiler & Fergus, 2014).
Another line of work is aimed at evaluating the effectiveness of saliency maps (Adebayo et al., 2018; Alqaraawi et al., 2020).
Although extensive work studies how different regions of images affect a network’s predictions, limited work (Srinivas & Fleuret, 2019) aims to distinguish important network parameters .
Our work combines the ideas of saliency maps and parameter importance and evaluates saliency directly on model parameters by aggregating their absolute gradients on a ﬁlter level.
We leverage the resulting parameter saliency proﬁles as an explainability tool and develop an input-space saliency counterpart which highlights image features that cause speciﬁc ﬁlters to malfunction to study the interaction between the image features and the erroneous ﬁlters.
M ETHOD
It is known that different network ﬁlters are responsible for identifying different image properties and objects (Zeiler & Fergus, 2014;
Yosinski et al., 2015; Olah et al., 2017; Mahendran & Vedaldi, 2015).
This motivates the idea that mistakes made on wrongly classiﬁed images can be understood by investigating the network parameters, rather than only the pixels, that played a role in making a decision.
We develop parameter-space saliency methods geared towards identifying and analyzing neural network parameters that are responsible for making erroneous decisions.
Central to our method is the use of gradient information of the loss function as a measure of parameter sensitivity and optimality of the network at a given point in image space.
2. P ARAMETER SALIENCY PROFILE
Let x be a sample in the validation set D with label y , and suppose a trained classiﬁer has parameters θ that minimize a loss function L .
We deﬁne the parameter-wise saliency proﬁle of x as a vector
[Figure]
s ( x, y ) with entries s ( x, y ) i :=
|∇ θ i L θ ( x, y ) | , the magnitudes of the gradient of the loss with respect to each model parameter.
Because the gradients on training data for a model trained to convergence are near zero, it is important to specify that D be a validation, or holdout, set.
Intuitively, a larger gradient norm at the point ( x, y ) indicates a greater inefﬁciency in the network’s classiﬁcation of sample x , and thus each entry of s ( x, y ) measures the suboptimality of individual parameters.
Aggregation of parameter saliency.
Convolutional ﬁlters are known to specialize in tasks such as edge, shape, and texture detection (Yosinski et al., 2015;
Bau et al., 2017; Olah et al., 2017).
We therefore choose to aggregate saliency on the ﬁlter-wise basis by averaging the gradient magnitudes of parameters corresponding to each convolutional ﬁlter.
This allows us to isolate ﬁlters to which the loss is most sensitive ( i.e. those which, when corrected, lead to the greatest reduction in loss).
Formally, for each convolutional ﬁlter F k in the network, consider its respective index set α k , which gives the indices of parameters corresponding to the ﬁlter F k .
The ﬁlter-wise saliency proﬁle of x is deﬁned to be a vector s ( x, y ) with entries s ( x, y ) k :
[Equation]
the parameter-wise saliency proﬁle aggregated by averaging on the ﬁlter level.
Standardizing parameter saliency.
Figure 1 exhibits the ResNet-
(He et al., 2016) ﬁlter-wise saliency proﬁle averaged over the ImageNet (Deng et al., 2009) validation set, where ﬁlters within each layer are sorted from highest to lowest saliency.
One clear observation is the difference in the scale of gradient magnitudes – shallower ﬁlters are more salient than deeper ﬁlters.
This phenomenon might occur for a number of reasons.
First, early ﬁlters encode low-level features, such as edges and textures, which are active across a wide spectrum of images.
Second, typical networks have fewer ﬁlters in shallow layers than in deep layers, making each individual ﬁlter more inﬂuential at shallower layers.
Third, the effects of early ﬁlters cascade and accumulate as they pass through a network.
To isolate ﬁlters that uniquely cause erroneous behavior on particular samples, we ﬁnd ﬁlters that are abnormally salient for a sample, x , but not for others.
That is, we further standardize the saliency proﬁle of x with respect to all ﬁlter-wise saliency proﬁles of D .
Formally, let µ be the average ﬁlter-wise saliency proﬁle across all x ∈ D , and let σ be an equallength vector with the corresponding standard deviation for each entry.
We use these statistics to produce the standardized ﬁlter-wise saliency proﬁle as follows:
[Equation]
The resulting tensor ˆ s ( x, y ) is of length equal to the number of convolutional ﬁlters in the network, and we henceforth call it the saliency proﬁle for sample x .
By standardizing saliency proﬁles, we create a saliency map that activates when the importance of a ﬁlter is unusually strong relative to other samples in the dataset.
This prevents the saliency map from highlighting ﬁlters that are uniformly important for all images, and instead focuses saliency on ﬁlters that are uniquely important and serve
[Figure]
an image-dependent role.
In the rest of the paper, unless explicitly noted otherwise, we use ˆ s ( x, y ) and refer to it as parameter saliency .
Incorrectly classiﬁed samples are more salient.
Empirically, we observe the saliency proﬁles of incorrectly classiﬁed samples exhibit, on average, greater values than those of correctly classiﬁed examples.
This bolsters the intuition that salient ﬁlters are precisely those malfunctioning — if the classiﬁcation is correct, there should be few malfunctioning ﬁlters or none at all.
Moreover, we see deeper parts of the network appear to be most salient for the incorrectly classiﬁed samples while earlier layers are often the most salient for correctly classiﬁed samples.
An example of these behaviors for ResNet-50 is shown in Figure 2 which presents standardized ﬁlter-wise saliency proﬁles averaged over the correctly and incorrectly classiﬁed examples from the ImageNet validation set.
Additionally, we note the improved relative scale of the standardized saliency proﬁle across different layers compared to the absolute gradient magnitudes in Figure 1.
Saliency proﬁles for other architectures could be found in Appendix A. Henceforth, we will focus speciﬁcally on saliency proﬁles of misclassiﬁed samples in order to explore how neural networks make mistakes.
2. I NPUT - SPACE SALIENCY FOR VISUALIZING HOW FILTERS MALFUNCTION
The parameter saliency proﬁle allows us to identify ﬁlters that are most responsible for mistakes and erroneous network behavior.
In this section, we develop an input-space counterpart to our parameter saliency method to understand which features of the image affect the saliency of particular ﬁlters.
Geiping et al.
(2020) show that the gradient information of a network is invertible, providing a link between input space and parameter saliency space.
This work, along with existing input-space saliency map tools (Simonyan et al., 2014; Springenberg et al., 2015; Smilkov et al., 2017; Zhou et al., 2016; Selvaraju et al., 2017), inspires our method.
Given a parameter saliency proﬁle ˆ s = ˆ s ( x, y ) for an image x with label y , our goal is to highlight the input features that drive large ﬁlter saliency values.
That is, we would like to identify image pixels altering which can make ﬁlters more salient.
To this end, we ﬁrst select some set F of the most salient ﬁlters that we would like to explore.
Then, we create a boosted saliency proﬁle s (cid:48) F by increasing the entries of ˆ s corresponding to the chosen ﬁlters F (e.g., multiplying by a large constant).
Now, we can ﬁnd pixels that are important for making the chosen ﬁlters F more salient and, equivalently, making the ﬁlter saliency proﬁle
ˆ s ( x, y ) close to the boosted saliency proﬁle s (cid:48) F by taking the following gradients:
[Equation]
where D C ( · , · ) is cosine distance.
[Figure]
The resulting input saliency map M F contains input features (pixels) that affect the saliency of the chosen ﬁlters F the most
E XPERIMENTS
In this section, we aim to validate the meaningfulness of our parameter saliency method.
First, we show on the dataset level that turning salient parameters off improves predictions on the associated samples thus verifying that the salient parameters are indeed responsible for misclassiﬁcation.
We then ﬁnd that samples which cause similar ﬁlters to malfunction are semantically similar.
We also show on the dataset level that ﬁne-tuning a small number of the most salient parameters on a single sample results in error correction on other samples which were misclassiﬁed for similar reasons.
We then use our input-space saliency technique in conjunction with its parameter-space counterpart as an explainability tool to explore how neural networks make mistakes and how salient ﬁlters interact with visual input features.
We evaluate our saliency method in the context of image classiﬁcation on CIFAR-10 (Krizhevsky, 2009) and ImageNet (Deng et al., 2009).
Images we use for visualization, unless otherwise speciﬁed, are sampled from ImageNet validation set.
Throughout the experiments, we use a pre-trained ResNet18 classiﬁer (He et al., 2016) on CIFAR-10 and a pre-trained ResNet-50 on ImageNet.
Both models are trained in a standard fashion on the corresponding dataset 12 .
3. P RUNING SALIENT FILTERS
We begin validating the meaningfulness of our parameter-space saliency maps by turning off the malfunctioning ﬁlters which cause misclassiﬁcation of the associated image.
We prune away the most salient convolutional ﬁlters (i.e., ﬁlters identiﬁed by our method as malfunctioning).
In order to remove the inﬂuence of a particular salient ﬁlter, we zero out the ﬁlter weights and also the biases of the associated batch normalization layers.
This procedure guarantees that the corresponding input feature map to the next convolutional layer is always zero.
Remarkably, we ﬁnd that this simple procedure improves the network’s behavior on the associated samples.
In particular, we gradually increase the number of pruned most salient ﬁlters and track three metrics: the change in the incorrect class conﬁdence, the change in the true class conﬁdence, and the percentage of the samples that ﬂip their label to the correct class.
In every case, we compare pruning the most salient ﬁlters against pruning the same number of random ﬁlters and the least salient ﬁlters.
These experiments are performed on the dataset level: we average the trends across all misclassiﬁed images in the ImageNet validation set.
As shown in Figure 16, pruning the most salient ﬁlters is signiﬁcantly more effective for decreasing the incorrect class conﬁdence than random or least salient ﬁlters.
Speciﬁcally, gradually pruning the top 100 salient ﬁlters achieves up to 30% drop in the incorrect class conﬁdence score while pruning random ﬁlters yields only about 7% decrease.
We also note that pruning the least salient ﬁlters does not produce any effect on the incorrect class conﬁdence.
We repeat the same experiment with the true class conﬁdence and observe that the highest true conﬁdence gain occurs when we prune around 20 most salient ﬁlters.
Pruning enough salient ﬁlters eventually leads to a gradual decrease in the true class conﬁdence.
We note that this behavior is expected since we are destroying, not correcting, the inference power of all of the most sensitive ﬁlters for an image, some of which may be essential for inference.
Finally, pruning random ﬁlters provides a much slower increase in the true conﬁdence class while the least salient ﬁlters again do not produce a signiﬁcant effect.
In addition, we count the number of images that were corrected as a result of pruning and ﬁnd that pruning around 30 most salient ﬁlters results in the best correct classiﬁcation rate of 12%.
Similar to the true class conﬁdence, the trend decreases beyond this point.
Pruning the random ﬁlters increases the percentage of corrected samples at a much slower rate and does not perform better than the most salient ﬁlters when pruning up to 100 ﬁlters.
Notably, pruning the least salient ﬁlters manages to correct a nontrivial number of samples but still much smaller than pruning random ﬁlters.
Given the trends in panels (b) and (c) of Figure 16, and given that pruning is a coarse tool for ﬁxing misbehavior, we explore the natural idea of correcting the most salient ﬁlters instead of removing them altogether in our ﬁne-tuning experiments in Section 3.3.
3. N EAREST NEIGHBORS IN PARAMETER SALIENCY SPACE
We validate the semantic meaning of our saliency proﬁles by clustering images based on the cosine similarity of their proﬁles.
In this section, we present visual depictions of a nearest neighbor search among all images in the ImageNet validation set.
We also conduct this analysis on CIFAR-10 images, and this can be found in Appendix A.
[Figure]
We ﬁnd that the nearest neighbors of misclassiﬁed images in saliency space are mostly other misclassiﬁed images from the same pair of predicted and true classes but possibly in reverse order.
For example, in Figure 4, the reference image in (a) is a great Pyrenees misclassiﬁed as kuvasz, and the 4 images with the most similar proﬁles exhibit either the same misclassiﬁcation or the reverse (i.e., kuvasz misclassiﬁed as great Pyrenees).
Intuitively, the common salient parameters across these neighbors are those which are important for discriminating between the two classes in question but are not well-tuned for this purpose.
Note that we ﬁnd the concept of “being similar” in parameter saliency space to be different from the one in image space.
The nearest neighbors we ﬁnd are often not similar in a pixel-wise sense, but rather they are similar in their reason for causing misclassiﬁcation.
For example, images in Figure(b) are beagles mistaken by a network for basset hounds and vice versa.
We ﬁnd that these pictures are either taken from a high angle or do not include the dog’s legs, making the leg length, a major distinction between the two breeds, indistinguishable from the picture.
We include more example images along with their nearest neighbors in Appendix A.
In addition, we compute nearest neighbors when only considering ﬁlters in a speciﬁc range of layers in order to visualize the types of misbehavior triggered by network components (ﬁlters) at various network depths.
We search for similar images using parameter saliency in the shallow and deep layers of a VGG-19 network (Simonyan & Zisserman, 2015), which we divide into the shallow and deep parts that respectively occur up to and after layer relu4 1 .
The top row of ﬁgure Figureshows neighbors found using shallow parameters, which share basic image attributes such as color histogram, while images in the bottom row share more abstract similarities.
3. C ORRECTING MISTAKES BY FINE - TUNING SALIENT FILTERS
To validate whether salient ﬁlters are more responsible for the erroneous behavior of neural networks, we show that updating salient ﬁlters alone is sufﬁcient for correcting the mistakes made by a neural network.
In this experiment, for a pre-trained image classiﬁcation network, we ﬁne-tune it for one step on a single image for which the network makes the wrong prediction.
We restrict the number of tunable ﬁlters to be no more than 1.0% of the total number of ﬁlters in a network, and we update the chosen ﬁlters by taking one step of gradient descent with a ﬁxed step size.
To validate the effectiveness of optimizing salient ﬁlters, we compare it with two other choices of tunable ﬁlters: the least salient ﬁlters and random ﬁlters.
For a more general evaluation, we use images from the
ImageNet validation set that are misclassiﬁed by a ResNet-50, making up to over 10,000 samples.
We evaluate the effect of ﬁne-tuning on each of these images independently. [Figure] A [Figure]
e
[Figure]
c
[Figure]
rr ec t e d n e
i ghbo
[Figure] i [Figure] In Figure 6, we compare the average performance of our three choices of tunable ﬁlters under three evaluation metrics.
For a given sample image and a set of tunable ﬁlters, an update step is considered to be effective if the updated network corrects its mistake on the sample image.
In addition, it is more useful if the updated network also corrects its mistake on other images that resemble the sample image but are not seen during ﬁne-tuning.
First, by inspecting the percentage of samples that are corrected after ﬁne-tuning (Figure 6 (a)), we ﬁnd that updating 150 salient ﬁlters ( ∼ 0.6% of total ﬁlters) can achieve the same result as updating the entire network.
The second and third metrics evaluate the effect on the nearest neighbors of the
[Figure] [Figure]
il
[Figure]
r s
[Figure]
li e
[Figure]
c
[Figure]
c
[Figure]
training sample (Figure 6 (b),(c)).
We ﬁnd nearest neighbors for each training sample through the process introduced in Section 3.2, where we limit the search scope exclusively to other misclassiﬁed images.
Note that for a given training sample, its nearest neighbors are not involved in our one-step single sample ﬁne-tuning process.
By tracking model predictions and true class conﬁdence scores among the 10 nearest neighbors of each sample, we ﬁnd that ﬁne-tuning salient ﬁlters is signiﬁcantly more effective than other choices.
Results in Figure 6, (b) and (c), also imply that the nearest neighbors found using our method are the images that are wrong for similar reasons and that they can be corrected altogether by only updating the salient ﬁlters on a single image.
We note that we do not propose a new pruning or ﬁne-tuning method.
Rather, we use these experiments to verify that the salient ﬁlters are indeed responsible for misclassiﬁcation.
3. I NPUT FEATURES THAT CAUSE FILTERS TO MALFUNCTION : A CASE STUDY
We consider a case study of an image misclassiﬁed by ResNet-50 as “killer whale” (Figure 7(a)).
The correct label of the image is “great white shark”.
Our goal is to study the interaction between the most salient ﬁlters and input features.
We ﬁrst identify ﬁlters most responsible for misclassiﬁcation by computing the ﬁlter saliency proﬁle and visualize parts of the image that drive the high saliency values for those ﬁlters using the input-space saliency counterpart (Section 2.2).
Panel (b) of Figure 7 presents our image-space visualization, which depicts the causes of misbehavior for the ten most salient ﬁlters – the pixels that trigger misbehavior in these ﬁlters are highlighted.
For example, we see that the seal and boat are both triggers.
One natural hypothesis is that the seal looks like a killer whale to the network and is the source of the classiﬁcation error.
We test this hypothesis by masking out the seal (Figure 7 (d)) .
However, although the probability of “killer whale” goes down and the probability of the correct class increases, the network still misclassiﬁes the image as “killer whale”.
Now, if we mask out exactly the most salient areas of the image according to our visualization (see Figure 7 (b), (e)), the network manages to ﬂip the label of the image and classify it correctly.
If we extend our mask to the less pronounced, but still salient, areas of the image as in Figure
[Figure]
(f), we observe that the correct class conﬁdence increases even more while the probability of the incorrect “killer whale” label further decreases.
Additionally, we ﬁnd that masking out the non-salient parts of the image results in even worse misclassiﬁcation conﬁdence than that of the original image (see Appendix A).
In order to further investigate the effect of the salient region, we pasted it from this image onto other great white shark images (see Appendix A) and observed that this drives the probability of “killer whale” up for 39 out of 40 examples of great white sharks from the ImageNet validation set with an average increase of 3 .
Our experiments suggest that secondary objects in the image are associated with the misclassiﬁcation.
However, we see that the erroneous behavior of the model does not just stem from classifying a non-target object in the image.
It is possible that the model correlates the combination of sea creatures (e.g. a seal) and man-made structures (e.g. a boat) with the “killer whale” label.
We note that images of killer whales in ImageNet often have man-made structures which look similar to the boat (see
Appendix A for examples of other “killer whale” images).
Finally, at each step of our masking experiments, we recompute the saliency values of the originally chosen 10 ﬁlters (i.e. the ﬁlters that caused erroneous behavior on the reference image).
From Figure 7 (c), we observe that as we mask out the input features according to our input-saliency, the saliency values of those ﬁlters decrease gradually and reach an 80% drop, conﬁrming that highlighted regions indeed drive the high saliency of the chosen ﬁlters.
More visualizations of input space saliency showcasing different illustrative examples of neural network mistakes can be found in Figure 8.
For a thorough discussion of mistake categories we identify using our saliency method, we refer to Appendix A.
D ISCUSSION
Numerous applications demand that practitioners be able to understand the decisions their models make, especially when those decisions are incorrect.
Existing methods for explainability focus on locating the input regions to which the network’s output is sensitive or on associating network components with speciﬁc roles.
In contrast, we develop a framework for ﬁnding the exact ﬁlters which are responsible for faulty predictions and studying the interactions between these ﬁlters and images.
This direction yields both an interpretable and intuitive understanding of model behaviors.
E THICS STATEMENT
Although our formulation of parameter saliency is not restricted to image datasets and CNNs, we only conduct experiments in these settings.
In contrast, real-world data and models come in many forms.
Explainability methods which shed light in some settings may fail to do so in others.
Moreover, we emphasize that some erroneous model behaviors are simply difﬁcult to understand through existing methods, and the capabilities of parameter saliency are limited.
In many applications, it is imperative that practitioners understand why their models behave as they do and that they are able to diagnose problems when they arise.
We hope that our work helps to enable solutions to real-world problems.
However, we caution against a false sense of security.
Our visual interpretations of model behavior should be viewed as approximations since neural networks are incredibly complex.
R EPRODUCIBILITY STATEMENT
We include our implementation of our parameter saliency method as well as the input-space saliency counterpart as a supplementary material.
All the datasets used in the experiments are publicly available.
The implementation details are available in Appendix B
A A DDITIONAL E XPERIMENTS A. P ARAMETER SALIENCY PROFILES FOR OTHER NETWORK ARCHITECTURES
In this section, we present average saliency proﬁles for several popular network architectures other than ResNet-50 (He et al., 2016).
Analogously to Figure 1, Figure 9 presents average gradient magnitudes for VGG-19 (Simonyan & Zisserman, 2015), Inception v3 (Szegedy et al., 2016)
[Figure]
and
DenseNet (Huang et al., 2017).
Similarly to Figure 2, we also present in Figure 10 standardized ﬁlter-wise saliency proﬁles for those architectures averaged across correctly and incorrectly classiﬁed
ImageNet (Deng et al., 2009) samples.
A. M ORE EXAMPLES OF NEAREST NEIGHBORS
We present more examples of nearest neighbors in our parameter saliency space.
Figure 12 are nearest neighbors in CIFAR-10 (Krizhevsky, 2009) dataset, where reference images are chosen from samples misclassiﬁed by our classiﬁer.
Figure 13 are examples from ImageNet, where images are captioned with the true label of the reference images.
In addition, in comparison with the nearest neighbor in our parameter saliency space, we also conduct the nearest neighbor search in the feature representation space. We take the feature representation from the con5 3 layer of a ResNet-50, and run the nearest neighbor search using the same reference images and candidate pool as in Figure 4. Results are shown in Figure 11. We ﬁnd that nearest neighbors in the feature space bare more resemblance in image structures, but it fails to identify samples that share the same mistakes.
In fact, most of the nearest neighbors in Figure 11 are correctly classiﬁed samples.
[Figure]
[Figure]
[Figure]
Furthermore, we present preliminary results on applying our method to language models.
We use a BERT (Devlin et al., 2018) model, pre-trained on the task of predicting the word at a masked position.
We consider an independent dataset for evaluation in our experiments, which consists of short sentences with masked words, provided by LAMA (F. Petroni & Riedel, 2019;
Petroni et al., 2020), an open-source language model analysis framework.
Similar to the ﬁlter-wise aggregation in convolutional networks, we adopt column-wise aggregation for obtaining our saliency proﬁles in the transformer-based architecture.
We conduct the nearest neighbor search by comparing sentences from the dataset in saliency space and analyzing the top-5 nearest neighbors.
We present four examples below, where the numbering indicates n -th nearest neighbor sentence, and the italic word is the ground truth.
Each example is accompanied with a description of similarities between the neighbors:
Reference: “Cany Ash and Robert Sakula are both Architects.” incorrectly predicted as: actors
“David
Castlles
-Quintana and
Vicente Royuela
are economists .”
incorrectly predicted as: actors 2.
“Raghuram Rajan is an economist .”
incorrectly predicted as: actor 3.
“Richard G. Wilkinson and Kate Pickett are British .” incorrectly predicted as: actors 4.
“Nathan Alterman was a poet .”
correctly predicted: poet 5.
“Zbigniew Badowski is an architect .” incorrectly predicted as: author
Note that all 5 neighboring sentences here share the common structure of being declarations of profession for speciﬁcally named people.
Interestingly, the ﬁrst three closest sentences to the reference incorrectly predict the profession to be an actor as well.
Moreover, the ground truth of the reference and its closest neighbor is economist/s.
Reference: “D’Olier Street is in Dublin.” incorrectly predicted as: Paris
“A group who call themselves Huguenots lives in Australia .” incorrectly predicted as: France 2.
“Huguenots and Walloons settled in Canterbury .” incorrectly predicted as: France 3.
“In the Treaty of Lisbon 2007 Ireland refused to consent to changes.” incorrectly predicted as: it 4.
“Samuel Marsden Collegiate School is located in Wellington .”
incorrectly predicted as: Melbourne 5.
“Konstantin Mereschkowski has Russian nationality.”
correctly predicted:
Russian
Again, we see all ﬁve nearest neighbor sentences are semantically similar to the reference, this time relating to national afﬁliations and geography.
In the ﬁrst two closest sentences, the model incorrectly ﬁlls in a location with France, which is similar to the reference which incorrectly predicts Paris.
Reference: “The Super Bowl sponsor
was
the Gap clothing
company.
” incorrectly predicted as Nike
“During Super Bowl 50 the Nintendo gaming company debuted their ad for the ﬁrst time.”
incorrectly predicted as: video 2.
“Experimental measurements on a model steam engine was made by Watt .” incorrectly predicted as: Siemens 3.
“ABC’s programming strategy was criticized in May 1961 by Life magazine.” incorrectly predicted as: Time 4.
“In 2009, Doctor Who started to be shown on Canadian cable station Space .” incorrectly predicted as: CBC 5.
“To emphasize the 50th anniversary of the Super Bowl the gold color was used.” incorrectly predicted as: blue
All but one of the nearest neighbors in this example relate to some kind of TV programming, and two also mention the Super Bowl.
Much like the reference sentence, which incorrectly predicts the name of a corporation, the ﬁrst four neighbors have a ground truth or incorrect prediction that is also a corporation.
Reference sample: “Tetzel’s collections of money to free souls from purgatory was objected by
Luther .” incorrectly predicted as: some
“Newcastle was granted a new charter in 1589 by Elizabeth .”
incorrectly predicted as: Parliament 2.
“The suggestion that imperialism was the “highest” form of capitalism is from Lenin .”
incorrectly predicted as: Aristotle 3.
“Fritschel said the man’s sleep was disturbed by dreams .” incorrectly predicted as: lightning 4.
“The concept that falling objects fell at the same speed regardless of weight was introduced by Galileo .” incorrectly predicted as: NASA 5.
“One of the earliest examples of Civil Disobedience was brought forward by the Egyptians .” incorrectly predicted as: government
This is an example where there is a weak relation between the incorrect classiﬁcations (three of ﬁve involve some kind of government or government organization) of the nearest neighbors, but the sentences are still highly semantically similar.
All of the neighbors except the third are declarations of historical actions performed by a speciﬁc person or group of people.
[Figure]
[Figure]
A. C ORRECTING MISTAKES ON OTHER DATASETS
We have shown in Section 3.3 that updating a few salient parameters of an ImageNet pre-trained network is enough for effectively correcting mistakes on the ImageNet validation set.
Moreover, in this section, we show that this effect can be extended to other independent datasets.
We use the
ImageNet-v2 test set, consisting of 10,000 images collected by Recht et al.
(2019), independent from the original ImageNet data.
In Figure 14, we observe the similar pattern as in Figure 6.
With test samples independent of the training data, salient parameters still demonstrate the most strength in correcting mistakes and their nearest neighbors.
The advantage is more prominent when inspecting model’s predictions of unseen nearest neighbors.
In this section, we conduct the ﬁne-tuning experiment introduced in Section 3.3 on a VGG-19 network trained on ImageNet, which has a total of 5504 ﬁlters.
The learning rate for training our VGG network [Figure] is 1/10 of that for the ResNet, so we decrease the ﬁne-tuning step size by 10 in this experiment
Figure 15 shows the effect of updating salient ﬁlters of a VGG-19.
Note that we use the same range for the number of tunable ﬁlters in this experiment; 300 ﬁlters correspond to 5.5% of total ﬁlters in a
VGG-19, while it is 1.2% for a ResNet-50. [Figure] r
[Figure]
t a g e o f c o rr ec t e
[Figure]
A. R ANDOM PERTURBATION OF SALIENT FILTERS
As an alternative to the pruning approach described in Section 3.1, random perturbations could be used to show that the most salient ﬁlters are indeed responsible for misclassiﬁcation.
We perturbed the ﬁlters using small Gaussian noise N (0 , 0 . 001) .
We observe similar trends to our pruning experiments in Section 3.1.
A. C ONNECTION TO ADVERSARIAL ATTACKS IN PARAMETER SPACE
Adversarial attacks in parameter space have been used for optimizers which ﬁnd ﬂat loss minima (Foret et al., 2020; Kwon et al., 2021; Du et al., 2021) and for improving model robustness through parameter-corruption-resistant training (Sun et al., 2020).
One could instead apply adversarial attacks to construct parameter saliency proﬁles – choose a constraint space and perturb parameters in order to minimize loss, subject to the constraint, using the perturbation to parameters as a saliency proﬁle (perhaps standardizing afterwards).
We notice several advantages and disadvantages of this alternative.
On the one hand, the “adversarial” approach requires a choice of constraint space and may require more compute (our method results in the exact
[Figure]
same saliency proﬁle as a single-step adversary).
On the other hand, the choice of constraint space and optimizer in the adversarial approach yields more ﬂexibility.
In this section
[Figure]
we compare our method with the adversarial-attack-based method.
More speciﬁcally, for a given sample ( x, y ) , we perturb parameters either to maximize or minimize the loss subject to an L2-norm constraint on the parameter change:
[Equation]
Then, given the adversarially perturbed parameters θ ∗ , we deﬁne the adversarial-attack-based parameter saliency proﬁle of the sample ( x, y ) as the absolute difference from the initial parameters θ 0 : s ( x, y )
=
| θ ∗ − θ 0 | .
We compare the resulting adversarial saliency proﬁles with our original method on a random sample of 100 images from the ImageNet validation set.
We observe that for a reasonably small constraint ( (cid:15) =
10 − 4 ), the resulting adversarial saliency proﬁles are similar to our original parameter saliency proﬁles (as one would expect for smooth loss) with the average cosine similarity between the saliency proﬁles generated by each method for the same images reaching 0.99 (the average is taken over the random sample of 100 images).
We also see that both methods agree on the top-k
(we tried k=100) most salient ﬁlters: on average, 95% of ﬁlters identiﬁed by our original method as top-k salient ﬁlters were also identiﬁed as top-k salient ﬁlters by the adversarial parameter saliency.
The differences were gradually more distinct with larger constraints, however, we note that the smaller epsilons are of greater interest since they reﬂect the intuition of perturbing only the most important parameters.
We also tried L1-regularized adversarial attacks.
We can similarly enforce sparsity in our original method by simply adding the L1 regularizer to our loss before computing the gradient:
[Equation]
This modiﬁcation can be seen as one step of the L1-regularized adversarial attack, and we experimentally checked that it produces very similar results with the cosine similarity between the resulting saliency proﬁles of 0.97 on average (for sufﬁciently large α = 0 . 99 ).
A. A DDITIONAL CASE STUDY FIGURES
Masking non-salient parts of the image.
As noted in section 3.4 and presented in Figure 17, masking the non-salient parts of the image results in even worse misclassiﬁcation conﬁdence with the incorrect class conﬁdence increasing compared to the reference image.
Pasting the salient region from the reference image onto other “great white shark” images.
As mentioned in section 3.4, in order to further investigate the effect of the salient region, we pasted it (i.e., the seal and the boat) from the original image onto other images with “great white shark” ground truth label that were correctly classiﬁed by ResNet-50 (see Figure 18 for examples).
We observed that this increased the probability of “killer whale” for 39 out of 40 examples of great white sharks from the ImageNet validation set with an average increase of 3 . [Figure]
Examples of images with “killer whale” label.
As we discussed in section 3.4, the model might have learned to correlate a combination of sea creatures (e g a seal) and man-made structures (e g. a boat) with the “killer whale” label.
Images of killer whales in ImageNet often have man-made structures which look similar to the boat, we provide examples of that in Figure 19. [Figure]
A. E XPLORING NEURAL NETWORK MISTAKES
In Section 3.4, we apply our parameter-space saliency method as an explainability tool using it alongside our input-space technique to study how image features cause speciﬁc ﬁlters to malfunction.
In our case study, the salient pixels that confuse the network are focused less on the target object than on other image features, and masking the salient regions which are not on the target object improves the network behavior.
Such examples expose the network’s reliance on spurious correlations and constitute an interesting type of model mistake.
The masking approach can be adopted to explore network mistakes and ﬁnd other interesting cases (e.g., cases where the salient pixels are not concentrated on the target object).
Investigating examples where masking the most salient pixels improves performance may provide insights into the model’s misbehavior as well as expose dataset noise and biases.
We select misclassiﬁed samples where masking the top 5% of salient pixels leads to an increase of at least 25% in conﬁdence corresponding to the correct class.
We showcase representative examples of different types of mistakes that we observe in those samples in Figure 8.
Many of the neural network misclassiﬁcations stem from classifying a non-target object in images with multiple objects (see Figure 8 (a), (b)).
However, other mistakes are triggered by background features (Figure 8 (d)), dataset biases (as our case study experiments in Figure 7 suggest), and label noise (Figure 8 (e)).
Interestingly, in some of the selected cases the salient regions still focus on the target object features (see Figure 8 (c)), and masking them improves the model’s behavior.
Masking salient target object features that confuse the network seems to be particularly beneﬁcial for images of animals; for example, masking dog ears helps the network identify the correct breed (see Figure 20(a)) or masking spot patterns helps distinguish different types of rays (see Figure 20(b)).
While masking the top 5 % of salient pixels results in correct classiﬁcation for all samples in Figureand in Figure 20(a), (b), this is, of course, not always the case.
Sometimes, pixels which cause large ﬁlter saliency values are focused densely on the target object, and masking them results in decreased conﬁdence corresponding to the correct class.
Selecting such samples can be used to ﬁnd situations where the network is genuinely confused by the target object rather than background features (Figure 20 (c)) or samples with multiple objects present and with salient pixels more focused on the target object (Figure 20 (d)).
A. C OMPARISON TO G RAD CAM
Existing input saliency maps used with the predicted label can highlight features which are related to misclassiﬁcation.
However, they are not speciﬁcally geared towards that goal.
Our input saliency technique highlights image features that cause speciﬁc ﬁlters to malfunction and those features, while they might in some cases coincide with the features that explain a high class conﬁdence score corresponding to the predicted label, may not be the same.
In this section, we will compare our input-space saliency technique which highlights pixels that drive high parameter saliency values of speciﬁc ﬁlters (i.e., pixels that confuse the network) to the visual explanations produced by GradCAM with the predicted label 3 (Selvaraju et al., 2017) – one of the most popular and high quality input-saliency methods.
Figures 21 and 22 present panels of images comparing our method to GradCAM explanations computed with the predicted label.
From the perspective of highlighting pixels responsible for neural network mistakes and for driving high ﬁlter saliency values, we note the following:
• GradCAM highlights the object that corresponds to the incorrect label, and the entire target object is highlighted in the images where only the target object is present (see Figure(c)-(e), Figure 22 (a)-(c)).
However, when our method focuses on the target object, it highlights speciﬁc features of that object.
Those are the features that confuse the network, and masking them can correct the misclassiﬁcation.
• In cases where the network classiﬁes the non-target object in the image (see Figure(a)-(b), Figure 22 (d)), both methods highlight the non-target objects.
However, GradCAM is more localized to the non-target object.
This is expected since GradCAM produces visual explanations for the predicted label (and has been shown to produce highly localized saliency maps (Selvaraju et al., 2017)) while our method highlights all pixels that drive the ﬁlter saliency, and these pixels may be located on the target object as well.
• In cases where the misclassiﬁcation does not stem from confusion by the target object or classifying the non-target object (see Figure 21 (d)-(e), Figure 22 (e)), our method highlights background features and/or a combination of non-target object features, while GradCAM still highlights the target object.
For example, in Figure 22 (e), our method highlights the boat and the sky much more than GradCAM, and our case study masking experiments in section 3.4 show that those regions indeed confuse the network.
• In addition, we emphasize that our input saliency technique is speciﬁc to the chosen ﬁlter set F and is introduced to study the interaction between the image features and the malfunctioning ﬁlters.
In contrast, GradCAM is not able to relate image-space mistakes to an arbitrary set of model parameters or ﬁlters chosen by the user.
To summarize, GradCAM (as well as many other input-space saliency methods) was designed to be highly localized to the object correponding to the label of interest, while our method highlights sparse ﬁne-grained features of images which we believe is a desirable property for our speciﬁc application.
Therefore, we opt for using input-gradient information similar to the original Vanilla
Gradient (Simonyan et al., 2014) method.
However, instead of class conﬁdence scores, we use a different loss – cosine distance to the boosted parameter-saliency proﬁle (as described in Section 2.2) which allows us to explore how image features cause speciﬁc ﬁlters to malfunction.
A. I NPUT - SALIENCY SANITY CHECK
To assure that our input-space saliency technique is model dependent, we performed the model randomization test from (Adebayo et al., 2018).
We can see that the input saliency map is model dependent.
We note that the data randomization test is not applicable in our case because our inputspace saliency map is based on the parameter-saliency proﬁle and parameter-saliency is designed to investigate a pretrained model with particular weights. [Figure]
B I MPLEMENTATION DETAILS B. H YPER - PARAMETER SETTING FOR FINE - TUNING SALIENT FILTERS
When tuning a small number of random or least salient ﬁlters, we re-normalize the gradient magnitude of these parameters to be the same as the salient ﬁlters for a fair comparison; otherwise the gradients for these parameters are always smaller than the salient ones by the deﬁnition of our saliency proﬁle, and updating them would make less change to a model than updating the salient ones.
In addition to re-normalizing the gradients, we also multiply them with a step size, similar to the concept of learning rate in stochastic gradient descent.
For ResNet-50, we set the step size to be 0.001, which equals to the learning rate of the last epoch when training the ResNet-50 on ImageNet from scratch.
For VGG-19, we also set the ﬁne-tuning step size to be the learning rate from the last training epoch, which is 0.0001. We also note that the batch normalization layers were set to the test mode for our ﬁne-tuning experiments.
B. I NPUT SALIENCY VISUALIZATION
The number of top salient ﬁlters to boost was chosen to be 10 in all input-space saliency experiments.
The ﬁlters were boosted by multiplying by 100.
For visualization, absolute input gradients were thresholded at 90-th percentile and Gaussian Blur with (3 , 3) kernel was applied.
B. H ARDWARE
The experiments were run on Nvidia GeForce RTX 2080Ti GPUs with 11Gb GPU memory on a machine with 4 cpu cores and 64Gb RAM.
The input-space and parameter-saliency proﬁles take seconds to compute for a single sample. [Figure] [Figure]