R ESTRICTING THE F LOW : I NFORMATION B OTTLENECKS FOR A TTRIBUTION
Anonymous authors
Paper under double-blind review
A BSTRACT
Attribution methods provide insights into the decision-making of machine learning models like artiﬁcial neural networks.
For a given input sample, they assign a relevance score to each individual input variable, such as the pixels of an image.
In this work we adapt the information bottleneck concept for attribution.
By adding noise to intermediate feature maps we restrict the ﬂow of information and can quantify (in bits) how much information image regions provide.
We compare our method against ten baselines using three different metrics on VGG-16 and
ResNet-50, and ﬁnd that our methods outperform all baselines in ﬁve out of six settings.
The method’s information-theoretic foundation provides an absolute frame of reference for attribution values (bits) and a guarantee that regions scored close to zero are not necessary for the network’s decision.
I NTRODUCTION
B i t s / P i x
e l
Deep neural networks have become state of the art in many realworld applications.
However, their increasing complexity makes it difﬁcult to explain the model’s output.
For some applications such as in medical decision making or autonomous driving, model interpretability is an important requirement with legal implications.
Attribution methods (Selvaraju et al., 2017; Zeiler & Fergus, 2014;
Smilkov et al., 2017) aim to explain the model behaviour by assigning a relevance score to each individual input variable.
When applied to images, the relevance scores can be visualised as heatmaps ov¨a er the input pixel space, thus highlighting salient areas relevant for the network’s decision.
Attribution maps can be hard to interpret.
If an attribution heatmap highlights subjectively irrelevant areas, this might correctly reﬂect the network’s unexpected way of processing the data, or the heatmap might simply be inaccurate.
Given an image of a railway locomotive, the attribution map might highlight the train tracks instead of the train itself.
Current attribution methods cannot guarantee that low relevance scores imply the network is ignoring the locomotive for prediction.
Yet, the low relevance values of the locomotive do not guarantee that the network will ignore the locomotive for prediction.
We propose a novel attribution method that estimates the amount of information an image region provides for the network’s decision.
We use a variational approximation to upper-bound this estimate and therefore can guarantee that areas with zero bits of information are not used.
Figure 1 shows an exemplary heatmap of our method.
Up to 3 bits per pixel are available for regions corresponding to the monkeys’ faces, whereas the tree is scored with close to zero bits per pixel.
We can thus guarantee that the tree is not necessary for predicting the correct class, a guarantee, to the best of our knowledge, no other method can provide.
To estimate the amount of information, we adapt the information bottleneck concept (Tishby et al., 2000; Alemi et al., 2017).
The bottleneck is inserted into an existing neural network and restricts the information ﬂow by adding noise to the activation maps.
Unimportant activations are replaced almost entirely by noise, removing all information for subsequent network layers.
We developed two approaches to learn the parameters of the bottleneck – either using a single sample (Per-Sample
Bottleneck), or the entire dataset (Readout Bottleneck).
We evaluate against ten different baselines.
First, we calculated the Sensitivity-n metric proposed by Ancona et al.
Secondly, we quantiﬁed how well the object of interest was localized using bounding boxes and extend the degradation task proposed by Ancona et al.
In all these metrics our method outperforms the baselines consistently.
Additionaly, we test the impact of cascading layer-wise weight randomizations on the attribution heatmaps (Adebayo et al., 2018).
For reproducibility, we share our source code ∗ .
Our method provides a theoretic upper-bound on the used information while demonstrating strong empirical performance.
This contributes to improving model explainability and to increasing trust in attribution results.
To summarize our contributions:
• We adapt the information bottleneck concept for attribution to estimate the information used by the network.
Information theory provides a guarantee that areas scored irrelevant are indeed not necessary for the network’s prediction.
• We propose two ways – Per-Sample and Readout Bottleneck – to learn the parameters of the information bottleneck.
• We contribute a novel evaluation method for attribution based on bounding boxes and we also extend the metric proposed by Ancona et al.
(2017) to provide a single scalar value and improve the metric’s comparability between different network architectures.
R ELATED W ORK
Attribution is an active research topic.
Gradient Maps (Baehrens et al., 2010) and Saliency Maps (Simonyan & Zisserman, 2014) are based on calculating the gradient of the target output neuron w.r.t.
to the input features.
Integrated Gradient (Sundararajan et al., 2017) and SmoothGrad (Smilkov et al., 2017) improve over gradient-based attribution maps by averaging the gradient of multiple inputs, either over brightness level interpolations of the input image or in a local neighborhood.
Other methods, such as Layer-wise Relevance Propagation (LRP) (Bach et al., 2015), Deep Taylor
Decomposition (DTD) (Montavon et al., 2017), Guided Backpropagation (GuidedBP) (Springenberg et al., 2014) or DeepLIFT (Shrikumar et al., 2017) modify the propagation rule.
PatternAttribution (Kindermans et al., 2018) builds upon DTD by estimating the signal’s direction for the backward propagation.
Perturbation-based methods are not based on backpropagation and treat the model as a black-box.
Occlusion (Zeiler & Fergus, 2014) measures the importance as the drop in classiﬁcation accuracy after replacing individual image patches with zeros.
Similarly, blurring image patches has been used to quantify the importance of high-frequency features (Greydanus et al., 2018) .
Grad-Cam (Selvaraju et al., 2017) take the activations of the ﬁnal convolutional layer to compute relevance scores.
They also propose to combine thier method with GuidedBP called GuidedGradCAM .
Ribeiro et al.
(2016) use image superpixels to explain deep neural networks.
To the best of our knowledge, we are the ﬁrst to estimate the amount of used information for attribution purposes.
Although many attribution methods exist, no common evaluation benchmark is established.
Thus, determining the state of the art is difﬁcult.
The performance of attribution methods is highly dependent on the used model and dataset.
Often only a pure visual comparison is performed (Smilkov et al., 2017; Springenberg et al., 2014;
Montavon et al., 2017; Sundararajan et al., 2017; Bach et al., 2015).
The most commonly used benchmark is to degrade input images according to the attribution heatmap and measure the impact on the model output.
(Kindermans et al., 2018; Samek et al., 2016; Ancona et al., 2017).
The Sensitivity-n score (Ancona et al., 2018) is obtained by randomly masking the input image and than measuring the correlation between removed attribution mass and model performance.
For the ROAR score (Hooker et al., 2018), the network is trained from scratch on the degraded images.
It is computationally expensive but avoids out-of-domain inputs – an inherent problem of masking.
Adebayo et al.
(2018) investigates the effect randomized parameters and shufﬂed labels have on the attribution output.
Adding noise to a signal reduces the amount of information (Shannon, 1948).
It is therefore a popular way to regularize neural networks (Srivastava et al., 2014; Kingma et al., 2015; Gal et al., 2017).
For regularization, the noise is applied independently from the input and therefore no attribution maps can be obtained.
In Variational Autoencoders (VAEs) (Kingma & Welling, 2013), noise restricts the information capacity of the latent representation.
In our work, we construct a similar information bottleneck that can be inserted into an existing network.
Deep convolutional neural networks have been augmented with information bottlenecks before to improve the generalization and robustness against adversarial examples (Achille & Soatto, 2018;
Alemi et al., 2017).
I NFORMATION B OTTLENECK FOR A TTRIBUTION
Instead of a backpropagation approach, we will quantify the ﬂow of information through the network in the forward pass.
Given a pre-trained model, we inject noise into a feature map, which restrains the ﬂow of information through it.
We optimize the intensity of the noise to minimize the information ﬂow, while simultaneously maximizing the original model objective, the classiﬁcation score.
The parameters of the original model are not changed.
3. I NFORMATION B OTTLENECK FOR A TTRIBUTION
Generally, the information bottleneck concept (Tishby et al., 2000) describes a limitation of available information.
Usually, the labels Y are predicted using all information from the input X .
The information bottleneck limits the information to predict Y by introducing a new random variable Z .
The information the new variable Z shares with the labels Y is maximized while minimizing the information Z and X share:
[Equation]
(1) where I [ X, Z ] denotes the mutual information and β controls the trade-off between predicting the labels well and using little information of X .
A common way to reduce the amount of information is to add noise (Alemi et al., 2017; Kingma & Welling, 2013).
For attribution, we inject an information bottleneck into a pretrained network.
The bottleneck is inserted into an early layer to ensure that the information in the network is still local, e.g. for the
ResNet the bottleneck is added after conv3 *.
Let R denote the intermediate representations at this speciﬁc depth of the network, i.e. R = f l ( X ) where f l is the l -th layer output.
We want to reduce information in R by adding noise.
Yet, as the neural network is already trained, we have to preserve the magnitude and variance of the input to the following layers.
Therefore, we also damp the signal
R when increasing the noise, effectively replacing the signal partly with noise.
In the extreme case, when no signal is transmitted, we replace R completely with noise of the same mean and variance as
R .
For this purpose, we estimate the mean µ R and variance σ 2 R of each feature of R empirically.
As information bottleneck, we then apply a linear interpolation between signal and noise:
[Equation]
where (cid:15) ∼ N ( µ R , σ 2 R ) and λ ( X ) controls the damping of the signal and addition of the noise.
The value of λ is a tensor with the same dimensions as R and λ i ∈ [0 , 1] .
Given λ i ( X ) = 1 at the feature map location i , the bottleneck transmits all information as Z i
= R i .
Whereas if λ i = 0 , then Z i = (cid:15) and thus all information of R i is lost and replaced with noise.
It could be tempting to think that Z from equation 2 has the same mean and variance as R .
This is not the case in general as λ ( X ) and R both depend on X (an analysis of this is shown in the appendix E).
In our method, we consider an area relevant if it contains useful information for classiﬁcation.
We therefore need to estimate how much information Z still contains about R .
This quantity is the mutual information I [ R, Z ] that can be written as:
[Equation]
where P ( Z | R ) and P ( Z ) denote the respective probability distributions.
We have no analytic expression for P ( Z ) since it would be necessary to integrate over the feature map p ( z ) =
[Equation]
– an intractable integral.
It is a common problem that the mutual information can not be computed exactly but is rather approximated (Poole et al., 2019; Suzuki et al., 2008).
We resort to a variational approximation Q ( Z ) =
N ( µ R , σ R ) which assumes that all dimensions of Z are normally distributed and independent, a reasonable assumption, as activations after linear or convolutional layers tend to have a Gaussian distribution (Klambauer et al., 2017).
The independence assumption will generally
[Figure]
not hold.
However, this will only lead to an overestimation of the mutual information as shown below.
Substituting Q ( Z ) into the previous equation 3, we obtain:
[Equation]
The derivation is shown in appendix D and follows the one of Alemi et al.
The ﬁrst term contains the KL-divergence between two normal distributions and can therefore be evaluated easily.
We will use the ﬁrst KL-divergence to approximate the mutual information.
The information loss function L I is therefore:
[Equation]
We know that L I overestimates the mutual information, i.e. L I ≥
I
[ R, Z ] as the second KLdivergence term D KL [ Q ( Z ) || P ( Z )] has to be positive.
If L I is zero for an area, we can guarantee that information from this area is not necessary for the network’s prediction.
Information from this area might still be used when no noise is added.
We aim to only keep the information needed for correct classiﬁcation.
Thus, the mutual information should be minimal while the classiﬁcation score should remain high.
Let L CE be the cross-entropy of the classiﬁcation.
Then, we obtain the following optimization problem:
[Equation]
where the parameter β controls the relative importance of both objectives.
A small β will result in more bits of information ﬂowing, whereas a higher β will result in more information getting discarded.
We propose two ways of ﬁnding the parameters λ – the Per-Sample Bottleneck and the Readout
Bottleneck.
For the Per-Sample Bottleneck, we optimize λ for each image individually, whereas in the readout bottleneck, we train a distinct neural network to predict λ .
3. P ER -S AMPLE B OTTLENECK
For the Per-Sample Bottleneck , we use the bottleneck formulation described above and optimize L for individual samples – not for the complete dataset at once.
Given a sample x , λ is ﬁtted to the sample to reﬂect important and unimportant regions in the feature space.
A diagram of the Per-Sample
Bottleneck is shown in Figure 2.
Parameterization: The bottleneck parameters λ have to be in [0 , 1] .
To simplify optimization, we parametrize λ
= sigmoid ( α ) .
This allows α to be chosen freely as α ∈ R d without having to explicitly clamp λ to [0 , 1] during optimization.
Initialization:
For training neural networks, the initialization of parameters matters and has a strong impact on the training output.
In the beginning, we want that all information is retained.
Therefore for all dimensions i , we initialize α i = 5 and thus λ i ≈ 0 .
993 ⇒ Z ≈ R .
At ﬁrst, The bottleneck has no practical impact on the model performance.
It then deviates from this starting point to suppress unimportant regions.
Optimization: We use a ﬁxed number of 10 iterations using the Adam optimizer (Kingma & Ba, 2014) with learning rate 1 to train the mask α .
To stabilize the training, we copy the single sample
[Figure]
ou t pu t
R ea dou t
N e t w o r k
i npu
t ou t pu t Bottleneck r e s i ze ……
l a y e r s i npu t ……
l a y e r s …… l a y e r s ……
l a y e r s
Readout Features ………
…… l a y e r s b l u r λ times and apply different noise to each.
In total, we execute the model 100 times to create a heatmap, comparable to other methods such as SmoothGrad.
After the optimization, the model usually predicts the target with probability close to 1, indicating that only negative evidence was removed.
Measure of information: For a measure of importance of each feature in Z , we can simply evaluate
D KL ( P ( Z | R ) || Q ( Z )) per dimension: It measures where in the network the information ﬂows and is less suppressed after training.
To obtain a two-dimensional heatmap m , we sum over the channel axis: m
[ h,w ] = (cid:80) ci =0 D KL ( P ( Z
[ i,h,w ] | R [ i,h,w ] )
|| Q ( Z [ i,h,w ] )) .
As convolutional neural networks preserve the locality in their channel maps, we use bilinear interpolation to match the map to the spatial dimensions of the input.
The bottleneck is inserted into an early layer to ensure that the information in the network is still local, e.g. for the ResNet the bottleneck is added after layer conv3 *.
Choosing a later layer with lower spatial resolution would also increase the blurriness of the attribution maps due to the required interpolation.
The effect of different depth choices for the bottleneck is visualized in ﬁgure 4.
Enforcing local smoothness: Pooling operations and convolutional layers with stride greater thanare ignoring parts of the input.
This causes the Per-Sample Bottleneck to overﬁt to a grid structure as shown in Appendix B. To obtain a robust and smooth attribution map, we convolve the sigmoid output with a ﬁxed Gaussian kernel with standard deviation σ s .
Smoothing the mask during training is not equivalent to smoothing the resulting attribution map, as during training also the gradient is averaged locally.
Combining everything, the parametrization for the Per-Sample Bottleneck is:
[Equation]
3. R EADOUT B OTTLENECK
In this section, we train a second neural network to predict the mask α .
In contrast to the Per-Sample
Bottleneck, this model is trained on the entire training set.
In Figure 3, the Readout Bottleneck is depicted.
The readout concept was ﬁrst introduced by K¨ummerer et al.
(2014) for gaze prediction.
Similarly, we collect feature map values from different depths in a ﬁrst forward pass without adding any noise.
As the spatial resolution of the feature maps differ, we interpolate them bilinearly to match the spatial dimensions of the bottleneck layer.
The readout network then predicts the information mask based on the collected feature maps.
In a second forward pass, we insert the bottleneck layer into the network and restrict the information ﬂow.
Except from this formulation as a function of the readout values, the Readout Bottleneck is identical to the mechanism of the Per-Sample Bottleneck.
The measure of information works in the same way as for the Per-Sample Bottleneck and we also use the same smoothing.
Given a new sample, we can obtain a heatmap simply by collecting the feature maps and executing the readout network.
[Table]
[Figure] [Figure]
varying layer depth and β on the Per-Sample Bottleneck for the ResNet-50.
The colorbars measure the bits per input pixel.
The resulting output probability of the correct class p = p ( y | x, β ) is decreasing for higher β .
The readout network consists of three 1x1 convolutional layers.
ReLU activations are applied between convolutional layers and a ﬁnal sigmoid activation yields λ ∈ [0 , 1] .
As the input includes the upscaled feature maps, the ﬁeld-of-view is large although the network itself only has 1x1 kernels.
E VALUATION 4. E XPERIMENTAL S ETUP
As neural network architectures, we selected the ResNet-50 (He et al., 2016) and the VGGSimonyan & Zisserman (2014), using pretrained weights from the torchvision package (Paszke et al., 2017; Marcel & Rodriguez, 2010).
These two models cover a range of concepts: Different dimensionality reduction methods (by stride, max-pooling, average-pooling), residual connections, batch normalization, dropout, low depth (16-weight-layer VGG) and high depth (50-weight-layer
ResNet).
This variety makes the evaluation less likely to overﬁt on a speciﬁc model type.
They are commonly used in literature concerning attribution methods.
For PatternAttribution on the VGG-16, we obtained weights for the signal estimators from Kindermans et al.
As naive baselines, we selected random attribution, Occlusion with patch sizes 8x8 and 14x14, and
Gradient Maps.
SmoothGrad and Integrated Gradients cover methods that accumulate gradients.
With
PatternAttribution, GuidedBP, and LRP, we include three methods with a modiﬁed backpropagation rule.
As PatternAttribution and LRP do not support skip connections, we report no results for them on the ResNet-50.
We also include Grad-CAM and its combination with GuidedBP, GuidedGrad-CAM.
For the compared methods, we use the hyperparameters suggested by the original authors with one exception.
For LRP (Bach et al., 2015), the original hyperparameters produced unsatisfactory results.
With (cid:15) = 5 and β = − 1 , we found LRP to yield best visual results.
For our methods, the hyperparameters are obtained using grid search with the degradation metric as objective (Appendix
[Figure]
F).
The readout network is trained over the training set of the ILSVRC12 dataset ( et al., 2015) for E = 20 epochs, i.e. the same dataset the original models were trained on.
The optimization objective of the bottleneck is L CE + β L I as given in equation 6.
Generally, the information loss L I is larger than the classiﬁer loss by several orders of magnitude as it sums over height h , width w and channels c .
We therefore use k = hwc as a reference point to select β .
In ﬁgure 4, we display the effect of different orders of β and the effect of layer depth for ResNet- (see Appendix C for VGG-16).
A uniform uninformative heatmap is obtained for β ≥ 1000 /k
– all information is discarded, resulting in an output probability of 0 for the correct class.
The heatmap for β = 0 .
1 /k is similar to β = 1 /k
but generally more information is passed, i.e. less noises is added.
In Table 1, a comparison of pre- to post-training accuracy and the estimated mutual information is shown for both Bottleneck types.
The Per-Sample Bottleneck can learn to suppress negative evidence for each sample and the accuracy is close to 1 for β ≤ 10 /k .
For β = 100 /k , also positive evidence is discarded and the accuracy decreases to 0 .
The Readout Bottleneck learns to suppress negative evidence for small β ≤ 0 .
1 /k and slightly improves the ﬁnal accuracy.
For the Per-Sample
Bottleneck, we investigate β = 1 /k, 10 /k, 100 /k .
The Readout network is trained with the best performing value β = 10 /k .
4. Q UALITATIVE A SSESSMENT
In Figure 5, the heatmaps of all evaluated samples are shown (for more samples see Appendix A).
Subjectively, both the Per-Sample and Readout Bottleneck identify areas relevant for the classiﬁcation well.
While Guided Backpropagation and PatternAttribution tend to highlight edges, the Per-Sample
Bottleneck focuses on image regions.
For the Readout Bottleneck, the attribution is concentrated a little more on the object edges.
Compared to Grad-CAM, both our methods are more speciﬁc, i.e. less pixels are scored high.
4. S ANITY C HECK : R ANDOMIZATION OF M ODEL P ARAMETERS
Adebayo et al.
(2018) investigates the effect of parameter randomization on attribution masks.
A sound attribution method should depend on the entire network’s parameter set.
Starting from the last layer, an increasing proportion of the network parameters is re-initialized until all parameters are random.
We excluded PatternAttribution as the randomization would require to re-estimate the signal directions.
The difference between the original heatmap and the heatmap obtained from the randomized model is quantiﬁed using SSIM (Wang et al., 2004).
In the Appendix G.3, we visualise the effect the randomization has on the heatmaps and provide implementation details.
Figureshows that our methods are less sensitive to randomization than SmoothGrad and Int.
Gradients but more dependent on the model’s parameters than GuidedBP and LRP.
Importantly, randomizing the ﬁnal linear layer drops the mean SSIM by around 0.4 for our methods.
Higher β values result in a higher dependency to the models’ parameters.
The values for the Readout Bottleneck are of limited expressiveness as we did not re-train it after randomization.
[Figure]
i
[Figure]
C h
[Figure]
c kSS I M
[Figure]
i
[Figure]
4. S ENSITIVITY - N
Ancona et al.
(2018) proposed Sensitivity-n as an evaluation metric for attribution methods.
Sensitivity-n masks the network’s input randomly and then measures how strongly the amount of attribution in the mask correlates with the drop in classiﬁer score (see Appendix G.1 for implementation details).
In Figure 6, the Sensitivity-n scores are shown for ResNet-50 and VGG-16.
When masking inputs pixel-wise as done in Ancona et al.
(2018), the Sensitivity-n score across methods are not well discriminable, all scores range within the lower 10% of the scale.
We therefore ran the metric with a tile size of 8x8 pixels.
This resulted in an increase of the Sensitivity-n score to 30% of the scale.
Although not shown in the ﬁgure due to the zooming, Occlusion-8x8 performs perfectly for small n as its relevance scores correspond directly to the drop in logits per 8x8 tile.
We ﬁnd that the Per-Sample Bottlenecks β
= 10 /k perform best for both models above n = 2 · 10 3 pixels, i.e. when more then 2% of all pixels are masked.
4. B OUNDING B OX
To quantify how well attribution methods identify and localize the object of interest, we rely on bounding boxes available for the ImageNet dataset.
Bounding boxes may contain irrelevant areas especially for non-rectangular objects.
We restrict our evaluation to images with bounding boxes covering less than 33% of the input image.
In total, we run the bounding box evaluation on 11, images from the ImageNet validation set.
If the bounding box contains n pixels, we measure how many of the n -th highest scored pixels are contained in the bounding box.
By dividing by n , we obtain a ratio between 0 and 1.
The results are shown in Table 2.
The Per-Sample Bottleneck outperforms all baselines on both VGG-16 and
ResNet-50 each by a margin of 0.152.
We discuss the differences between attribution methods in
Appendix G.2.
4. I MAGE D EGRADATION
As a further quantitative evaluation, we rely on the degradation task as used by Ancona et al.
Kindermans et al.
; Hooker et al.
; Samek et al.
Given an attribution heatmap, the input is split in tiles which are ranked by the sum of attribution values within each corresponding tile of the attribution.
At each iteration, the highest ranked tile is replaced with a constant value, the modiﬁed input is fed through the network, and the resulting drop in target class score is measured.
A steep descent of the accuracy curve indicates a meaningful attribution map.
[Table]
Per-Sample β = 100 /k
[Table]
Readout
[Table]
= 10 /k
[Table]
When implemented in the described way, the most relevant tiles are removed ﬁrst (MoRF).
However,
Ancona et al.
(2017) argue that using only the MoRF curve for evaluation is not sufﬁcient.
For the
MoRF score, it is beneﬁcial to ﬁnd tiles that disrupt the output of the neural network as early as possible in the sequence of tiles.
Neural networks have been shown to be sensitive to subtle changes in the input (Szegedy et al., 2013).
Hence, the tiles do not necessarily have to contain meaningful information to disrupt the network.
Therefore, Ancona et al.
(2017) proposes to invert the degradation direction, removing tiles ranked as least relevant by the attribution method ﬁrst (LeRF).
The LeRF task favors methods that identify areas sufﬁcient for classiﬁcation.
Ancona et al.
(2017) measures the drop in target class logits, which may vary in scale for different models and does not account for scores of other classes.
To improve the comparability between different models, we measure the drop in ﬁnal target class probability.
Furthermore, we scale the model output probability p ( y | x ) such that it becomes independent from the model performance.
The scaled probability s ( x ) is then:
[Equation]
100 x - degragation of x [%]
[Figure]
[Figure]
The area between both curves is the ﬁnal degradation score.
where t 1 is the model’s average top-1 probability on the original samples and b is the mean model output on the fully degradad images.
Both averages are taken over the validation set.
A score of 1 corresponds to the original model performance.
The score s ( x ) still depends on the model, as every model behaves differently on degraded images.
Our extension of Ancona et al.
(2017) is a re-scaling and thus does not change the ordering of the attribution methods.
Both LeRF and MoRF degradation yield curves as visualized in
Figure 7, measuring different qualities of the attribution method.
In order to obtain a scalar measure of attribution performance and to combine both metrics, we propose to calculate the integral between the MoRF and LeRF curves.
This scalar can then be interpreted as the average difference of the drop in model accuracy when replacing most important and least important tiles ﬁrst, respectively.
The results for all implemented attribution methods on the degradation task are given in Table 2.
We evaluated both models on the full validation set using 8x8 and 14x14 tiles.
In Appendix H, we show the mean LeRF and MoRF curves for 14x14 tiles.
The Per-Sample Bottleneck outperforms all other methods in the degradation benchmark except for GuidedGrad-CAM on ResNet-50 where it scores comparably (score difference of 0 . 004 ).
The Readout Bottleneck generally achieves a lower degradation scores but still perform competitively.
C ONCLUSION
We propose two novel attribution methods that return an upper bound on the amount of information each input region provides for the network’s decision.
Our models’ core functionality is a bottleneck layer used to inject noise into a given feature layer and a mechanism to learn the parametrized amount of noise per feature.
The Per-Sample Bottleneck is optimized per single data point,
whereas the
Readout Bottleneck is trained on the entire dataset.
Our method does not constrain the internal network structure.
In contrast to several backpropagationbased methods, it supports any activation function and network architecture.
To evaluate our method we proposed a novel variant of the degradation task to quantify model performance deterioration when removing both relevant and irrelevant image tiles ﬁrst .
Our Per-Sample Bottleneck and Readout
Bottle both show competitive results on all metrics used, outperforming the state of the art with signiﬁcant margin for some of the tasks.
Generally, we would advise to use the Per-Sample Bottleneck over the Readout Bottleneck.
It performs better and is more ﬂexible as it only requires to estimate the mean and variance of the feature map.
The Readout Bottleneck has the advantage of producing attribution maps with a single forward pass once trained.
Images with multiple object instances provide the network with redundant class information.
The Per-Sample Bottleneck may therefore discard some of the class evidence.
Even for single object instances, the heatmaps of the Per-Sample Bottleneck may vary slightly due to the randomness of the optimization process.
The method’s information-theoretic foundation provides a guarantee that the network does not require regions of zero-valued attribution for correct classiﬁcation.
To our knowledge, our attribution methods is the only one to provide scores with units (bits).
This absolute frame of reference allows a quantitative comparisons between models, inputs and input regions.
We hope this contributes to a deeper understanding of neural networks and creates trust to use modern models in sensitive areas of application.
A CKNOWLEDGEMENT
Hidden for double-blind review.
A V ISUAL C OMPARISON OF A TTRIBUTION M ETHODS
A
R ES N ETInput
Grad-CAM
GuidedBP.
OcclusionPer-Sample
Readout
[Table]
r
S
[Table]
rr e l m onk e y
C r oqu e t
[Table]
a ll
L it ...
l u e h e r on
W i nd s o r ti e
[Table]
c o r n
T a r a n t u l a
A.
VGGInput
Grad-CAM
GuidedBP.
PatternAttr.
Per-Sample
Readout
B l ac k b ea r
S qu i rr e l m onk e y
C r oqu e t b a ll
L it ...
l u e h e r on
W i nd s o r ti e
A c o r n
T a r a n t u l a
B G RID A RTIFACTS WHEN NOT USING S MOOTHING
[Figure]
C E FFECTS OF DIFFERENT β AND LAYER DEPTH FOR THE VGG-
c onv    β = 0 .
1 /k p = 0 .0 .2 .
.β
= 1 /k p = 0 .
β = 10 /k p = 0 .
β
= 100 /k p = 0 .0 .0 .0 .β
= 1000 /k
p = 0 .
[Figure] [Figure]
conv1   conv2  conv40 .0 .
.conv50 .0 .conv
D D ERIVATION OF THE A PPROXIMATION OF M UTUAL I NFORMATION
For the mutual information I [ R, Z ] , we have:
[Equation]
E M EAN AND VARIANCE OF Z
The λ ( X ) linearly interpolate between the feature map R and the noise (cid:15) ∼ N ( µ R , σ 2 R ) , where µ R and σ R are the estimated mean and standard derivation of R .
Both R = f l ( X ) and λ ( X ) depend on the input random variable X .
[Equation]
For the mean of Z , we have:
E
[ Z ] =
E
[Equation]
( X
[Equation]
R ] + E
[(1 −
[Equation]
( X )
[Equation]
(cid:15) ] (cid:46) substituting in deﬁnition of Z = E
[ λ ( X ) R ] + E [1 − λ
( X )]
E [ (cid:15) ] (cid:46) independence of λ and (cid:15) =
cov ( λ ( X ) , R ) + E [ λ ( X )]
E [ R ] + E [1 − λ ( X )]
E [ (cid:15) ] (cid:46) cov ( A, B ) =
E [ AB ] − E
[ A ] E [ B ] ≈ cov ( λ ( X ) , R )
+ E [ (cid:15) ]
(cid:46) E [ (cid:15) ]
= µ R ≈ E
[ R ]
As λ ( X ) and R are multiplied together, they form a complex product distribution.
If they do not correlate, E [ Z ] ≈ E [ (cid:15) ]
≈ E
[ R ] .
A similar problem araises for the variance:
Var
[ Z ] =
E [ Z 2 ] − E
[ Z ]= E [( λ ( X ) R + (1 − λ ( X ) (cid:15) ) 2 ] − ( cov ( λ ( X ) , R ) + E [ (cid:15) ])
The multiplication of λ ( X ) and R causes in general the variance of Z and R to not match:
Var[ Z ] (cid:54) =
Var[ R ] .
F H YPERPARAMETERS
Parameter
ResNetVGG-
Search space
Target layer conv3conv
Optimizer
Adam (Kingma & Ba (2014))
Learning Rate η ={ 0 . 03 , 0 . 1 , 0 . 3 , 1 , 3 , 10 }
Balance Factor β
= 10 /k { 0 . 001 , 0 . 01 , 0 . 1 , 1 , 10 , 100 , 300 }
Iterations
T ={ 1 , 3 , 5 , 10 , 30 , 100 }
Batch Size
B ={ 1 , 5 , 10 , 30 }
Smoothing σ s ={ 0 . 5 , 1 , 2 }
[Table]
ResNet
[Table]
VGG
[Table]
Search space
[Table]
conv
[Table]
conv
[Table]
conv2conv3conv4conv
[Table]
conv3conv4conv5conv
[Table]
[Table]
G E VALUATION M ETRICS G. S ENSITIVITY - N
Given a set T n containing n randomly selected pixel indices, Sensitivity-n measures the Pearson correlation coefﬁcient:
[Equation]
where S c ( x ) is the classiﬁer logit output for class c , R i is the relevance at pixel i and x [ x Tn =0] denotes the input with all pixels in T n set to zero.
As in the original paper, we pick the number of masked pixels n in logscale between 1 and 80% of all pixels.
For each n , we generate 100 different index sets T and test each on 1000 randomly selected images from the validation set.
The correlation is calculated over the different index sets and then averaged over all images.
G
[Equation]
The attribution maps produced by the other methods each differ in their distribution of attribution scores.
Gradient-based methods such as GuidedBP and LRP tend to produce rather sparse attribution maps, while Grad-CAM and our method yield smoother, dense maps.
Sparse methods assign the bulk of attribution mass to only a few image regions.
Since we regard only the top n scored pixels, we may sample regions outside the bounding box from the remaining low-attribution tail, since even inﬁnitesimal score differences matter in the ordering process.
An alternative metric would be to take the sum of attribution in the bounding box and compare it to the total attribution in the image.
We found this metric is not robust against extreme values.
For the ResNet-50, we found basic Gradient
Maps to be the best method as a few pixels receiving extrem scores are enough to dominate the sum.
G. S ANITY C HECK : W EIGHT R ANDOMIZATION
We use the cascading parameter randomization sanity check from Adebayo et al.
Following the original paper, we used the skimage SSIM implementation with a window of size 5.
For LRP, we found that the weight randomization ﬂips the values of the saliceny heatmap, e.g. h r ≈ −
h
o where h r is the heatmap with random weights and h o the heatmap on the original model.
Therefore for LRP, we used: max( ssim ( h o , normalize ( h r ) , ssim ( h o , normalize ( − h r ))) .
We normalized the heatmaps by ﬁrst clamping the 1-th and 99-th percentile and then rescaling the heatmap it to [0 , 1] .
We run the sanity check on 200 randomly selected samples from the ImageNet valdiation set.
[Figure]
[Figure]
H M O RF AND L E RF D EGRADATION P ATHS
[Figure]
[Figure]
o r m a li
z
e
d s o r e
[Figure]
x ) n o r m a li
z e d s c o r e
[Figure]