R ESTRICTING THE F LOW : I NFORMATION B OTTLENECKS FOR A TTRIBUTION
A BSTRACT
Attribution methods provide insights into the decision-making of machine learning models like artiﬁcial neural networks.
For a given input sample, they assign a relevance score to each individual input variable, such as the pixels of an image.
In this work, we adopt the information bottleneck concept for attribution.
By adding noise to intermediate feature maps, we restrict the ﬂow of information and can quantify (in bits) how much information image regions provide.
We compare our method against ten baselines using three different metrics on VGG-16 and
ResNet-50, and ﬁnd that our methods outperform all baselines in ﬁve out of six settings.
The method’s information-theoretic foundation provides an absolute frame of reference for attribution values (bits) and a guarantee that regions scored close to zero are not necessary for the network’s decision.
I NTRODUCTION
Deep neural networks have become state of the art in many realworld applications.
However, their increasing complexity makes it difﬁcult to explain the model’s output.
For some applications such as in medical decision making or autonomous driving, model interpretability is an important requirement with legal implications.
Attribution methods (Selvaraju et al., 2017; Zeiler & Fergus, 2014;
Smilkov et al., 2017) aim to explain the model behavior by assigning a relevance score to each input variable.
When applied to images, the relevance scores can be visualized as heatmaps over the input pixel space, thus highlighting salient areas relevant for the network’s decision.
No ground truth exists for attribution.
If an attribution heatmap highlights subjectively irrelevant areas, this might correctly reﬂect the network’s unexpected way of processing the data, or the heatmap might be inaccurate (Sixt et al., 2019).
Given an image of a railway locomotive, the attribution map might highlight the train tracks instead of the train itself.
Current attribution methods cannot guarantee that the network does not use the low scored locomotive for the prediction.
We propose a novel attribution method that estimates the amount of information an image region provides to the network’s prediction.
We use a variational approximation to upper-bound this estimate and therefore, can guarantee that areas with zero bits of information are not necessary for the prediction.
Figure 1 shows an exemplary heatmap of our method.
Up to 3 bits per pixel are available for regions corresponding to the monkeys’ faces, whereas the tree is scored with close to zero bits per pixel.
We can thus guarantee that the tree is not necessary for predicting the correct class.
To estimate the amount of information, we adapt the information bottleneck concept (Tishby et al., 2000; Alemi et al., 2017).
The bottleneck is inserted into an existing neural network and restricts the information ﬂow by adding noise to the activation maps.
Unimportant activations are replaced almost entirely by noise, removing all information for subsequent network layers.
We developed two approaches to learn the parameters of the bottleneck – either using a single sample (Per-Sample
Bottleneck), or the entire dataset (Readout Bottleneck).
We evaluate against ten different baselines.
First, we calculated the Sensitivity-n metric proposed by Ancona et al.
Secondly, we quantiﬁed how well the object of interest was localized using bounding boxes and extend the degradation task proposed by Ancona et al.
In all these metrics, our method outperforms the baselines consistently.
Additionally, we test the impact of cascading layer-wise weight randomizations on the attribution heatmaps (Adebayo et al., 2018).
For reproducibility, we share our source code and provide an easy-to-use ∗ implementation for PyTorch and TensorFlow.
We name our method IBA which stands for Information Bottleneck Attribution.
It provides a theoretic upper-bound on the used information while demonstrating strong empirical performance.
Our work improves model interpretablility and increases trust in attribution results.
To summarize our contributions:
• We adapt the information bottleneck concept for attribution to estimate the information used by the network.
Information theory provides a guarantee that areas scored irrelevant are indeed not necessary for the network’s prediction.
• We propose two ways – Per-Sample and Readout Bottleneck – to learn the parameters of the information bottleneck.
• We contribute a novel evaluation method for attribution based on bounding boxes and we also extend the metric proposed by Ancona et al.
(2017) to provide a single scalar value.
R ELATED W ORK
Attribution is an active research topic.
Gradient Maps (Baehrens et al., 2010) and Saliency Maps (Simonyan & Zisserman, 2014) are based on calculating the gradient of the target output neuron w.r.t.
to the input features.
Integrated Gradient (Sundararajan et al., 2017) and SmoothGrad (Smilkov et al., 2017) improve over gradient-based attribution maps by averaging the gradient of multiple inputs, either over brightness level interpolations or in a local neighborhood.
Other methods, such as Layerwise Relevance Propagation (LRP) (Bach et al., 2015), Deep Taylor Decomposition (DTD)
(Montavon et al., 2017), Guided Backpropagation (GuidedBP) (Springenberg et al., 2014) or DeepLIFT (Shrikumar et al., 2017) modify the propagation rule.
PatternAttribution (Kindermans et al., 2018) builds upon DTD by estimating the signal’s direction for the backward propagation.
Perturbation-based methods are not based on backpropagation and treat the model as a black-box.
Occlusion (Zeiler & Fergus, 2014) measures the importance as the drop in classiﬁcation accuracy after replacing individual image patches with zeros.
Similarly, blurring image patches has been used to quantify the importance of high-frequency features (Greydanus et al., 2018) .
Grad-Cam (Selvaraju et al., 2017) take the activations of the ﬁnal convolutional layer to compute relevance scores.
They also combine their method with GuidedBP: GuidedGrad-CAM .
Ribeiro et al.
(2016) uses image superpixels to explain deep neural networks.
High-level concepts rather input pixels are scored by TCAV (Kim et al., 2018).
Khakzar et al.
(2019) prune irrelevant neurons.
Similar to our work, Macdonald et al.
(2019) uses a rate-distortion perspective, but minimize the norm of the mask instead of the shared information.
To the best of our knowledge, we are the ﬁrst to estimate the amount of used information for attribution purposes.
Although many attribution methods exist, no standard evaluation benchmark is established.
Thus, determining the state of the art is difﬁcult.
The performance of attribution methods is highly dependent on the used model and dataset.
Often only a purely visual comparison is performed (Smilkov et al., 2017; Springenberg et al., 2014;
Montavon et al., 2017; Sundararajan et al., 2017; Bach et al., 2015).
The most commonly used benchmark is to degrade input images according to the attribution heatmap and measure the impact on the model output.
(Kindermans et al., 2018; Samek et al., 2016; Ancona et al., 2017).
The Sensitivity-n score (Ancona et al., 2018) is obtained by randomly masking the input image and then measuring the correlation between removed attribution mass and model performance.
For the ROAR score (Hooker et al., 2018), the network is trained from scratch on the degraded images.
It is computationally expensive but avoids out-of-domain inputs – an inherent problem of masking.
Adebayo et al.
(2018) proposes a sanity check: if the network’s parameters are randomized, the attribution output should change too.
Adding noise to a signal reduces the amount of information (Shannon, 1948).
It is a popular way to regularize neural networks (Srivastava et al., 2014; Kingma et al., 2015; Gal et al., 2017).
For regularization, the noise is applied independently from the input and no attribution maps can be obtained.
In Variational Autoencoders (VAEs) (Kingma & Welling, 2013), noise restricts the information capacity of the latent representation.
In our work, we construct a similar information bottleneck that can be inserted into an existing network.
Deep convolutional neural networks have been augmented with information bottlenecks before to improve the generalization and robustness against adversarial examples (Achille & Soatto, 2018;
Alemi et al., 2017).
Zhmoginov et al.
(2019) create binary masks of salient regions using an information bottleneck, but they do not estimate the amount of used information.
I NFORMATION B OTTLENECK FOR A TTRIBUTION
Instead of a backpropagation approach, we quantify the ﬂow of information through the network in the forward pass.
Given a pre-trained model, we inject noise into a feature map, which restrains the ﬂow of information through it.
We optimize the intensity of the noise to minimize the information ﬂow, while simultaneously maximizing the original model objective, the classiﬁcation score.
The parameters of the original model are not changed.
3. I NFORMATION B OTTLENECK FOR A TTRIBUTION
Generally, the information bottleneck concept (Tishby et al., 2000) describes a limitation of available information.
Usually, the labels Y are predicted using all information from the input X .
The information bottleneck limits the information to predict Y by introducing a new random variable Z .
The information the new variable Z shares with the labels Y is maximized while minimizing the information Z and X share :
[Equation]
[Equation]
where I [ X, Z ] denotes the mutual information and β controls the trade-off between predicting the labels well and using little information of X .
A common way to reduce the amount of information is to add noise (Alemi et al., 2017; Kingma & Welling, 2013).
For attribution, we inject an information bottleneck into a pretrained network.
The bottleneck is inserted into an layer which still contains local information, e.g. for the ResNet the bottleneck is added after conv3 * (after the last conv3 block).
Let R denote the intermediate representations at this speciﬁc depth of the network, i.e. R = f l ( X ) where f l is the l -th layer output.
We want to reduce information in R by adding noise.
As the neural network is trained already, adding noise should preserve the variance of the input to the following layers.
Therefore, we also damp the signal R when increasing the noise, effectively replacing the signal partly with noise.
In the extreme case, when no signal is transmitted, we replace R completely with noise of the same mean and variance as R .
For this purpose, we estimate the mean µ R and variance σ 2 R of each feature of R empirically.
As information bottleneck, we then apply a linear interpolation between signal and noise:
[Equation]
where (cid:15) ∼ N ( µ R , σ 2 R ) and λ ( X ) controls the damping of the signal and the addition of the noise.
The value of λ is a tensor with the same dimensions as R and λ i ∈ [0 , 1] .
Given λ i ( X ) = 1 at the feature map location i , the bottleneck transmits all information as Z i
= R i .
Whereas if λ i = 0 , then
Z i = (cid:15) and all information of R i is lost and replaced with noise.
It could be tempting to think that Z from equation 2 has the same mean and variance as R .
This is not the case in general as λ ( X ) and R both depend on X (for more detail see Appendix E).
In our method, we consider an area relevant if it contains useful information for classiﬁcation.
We need to estimate how much information Z still contains about R .
This quantity is the mutual information I [ R, Z ] that can be written as:
[Equation]
where P ( Z | R ) and P ( Z ) denote the respective probability distributions.
We have no analytic expression for P ( Z ) since it would be necessary to integrate over the feature map p ( z )
= (cid:82) R p ( z | r ) p ( r )d r
[Figure]
– an intractable integral.
It is a common problem that the mutual information can not be computed exactly but is instead approximated (Poole et al., 2019; Suzuki et al., 2008).
We resort to a variational approximation Q ( Z ) =
N ( µ R , σ R ) which assumes that all dimensions of Z are distributed normally and independent – a reasonable assumption, as activations after linear or convolutional layers tend to have a Gaussian distribution (Klambauer et al., 2017).
The independence assumption will generally not hold.
However, this only overestimates the mutual information as shown below.
Substituting
Q ( Z ) into the previous equation 3, we obtain:
[Equation]
The derivation is shown in appendix D and follows Alemi et al.
The ﬁrst term contains the KL-divergence between two normal distributions, which is easy to evaluated and we use it to approximate the mutual information.
The information loss function L I is therefore:
[Equation]
We know that L I overestimates the mutual information, i.e. L I ≥
I
[ R, Z ] as the second KLdivergence term D KL [ Q ( Z ) || P ( Z )] has to be positive.
If L I is zero for an area, we can guarantee that information from this area is not necessary for the network’s prediction.
Information from this area might still be used when no noise is added.
We aim to keep only the information necessary for correct classiﬁcation.
Thus, the mutual information should be minimal while the classiﬁcation score should remain high.
Let L CE be the cross-entropy of the classiﬁcation.
Then, we obtain the following optimization problem:
[Equation]
where the parameter β controls the relative importance of both objectives.
For a small β , more bits of information are ﬂowing and less for a higher β .
We propose two ways of ﬁnding the parameters λ – the Per-Sample Bottleneck and the Readout Bottleneck.
For the Per-Sample Bottleneck, we optimize λ for each image individually, whereas in the readout bottleneck, we train a distinct neural network to predict λ .
3. P ER -S AMPLE B OTTLENECK
For the Per-Sample Bottleneck , we use the bottleneck formulation described above and optimize L for individual samples – not for the complete dataset at once.
Given a sample x , λ is ﬁtted to the sample to reﬂect important and unimportant regions in the feature space.
A diagram of the Per-Sample
Bottleneck is shown in Figure 2.
Parameterization
The bottleneck parameters λ have to be in [0 , 1] .
To simplify optimization, we parametrize λ
= sigmoid ( α ) .
This parametrization allows α ∈ R d and avoids any clipping of λ to [0 , 1] during optimization.
Initialization
As when training neural networks, the initialization of parameters matters.
In the beginning, we want to retain all the information.
For all dimensions i , we initialize α i = 5 and thus λ i ≈ 0 .
993 ⇒ Z ≈ R .
At ﬁrst, the bottleneck has practically no impact on the model performance.
It then deviates from this starting point to suppress unimportant regions.
λR +(1 − λ ) (cid:15)
R input netbottleneck -work logits sigmoid λ blur network r e s i ze c onv1x c onv1x c onv1x readout net.
(cid:15) ∼ N ( µ R , σ 2 R )
Optimization
We do 10 iterations using the Adam optimizer (Kingma & Ba, 2014) with learning rate 1 to ﬁt the mask α .
To stabilize the training, we copy the single sample 10 times and apply different noise to each.
In total, we execute the model on 100 samples to create a heatmap, comparable to other methods such as SmoothGrad.
After the optimization, the model usually predicts the target with probability close to 1, indicating that only negative evidence was removed.
Measure of information
To measure the importance of each feature in Z , we evaluate D KL ( P ( Z | R )
|| Q ( Z ))
per dimension.
It shows where the information ﬂows.
We obtain a two-dimensional heatmap m by summing over the channel axis: m
[ h,w ] = (cid:80) ci =0 D KL ( P ( Z
[ i,h,w ] | R [ i,h,w ] )
|| Q ( Z [ i,h,w ] )) .
As convolutional neural networks preserve the locality in their channel maps, we use bilinear interpolation to resize the map to the image size.
For the
ResNet-50, we insert the bottleneck after layer conv3 *.
Choosing a later layer with a lower spatial resolution would increase the blurriness of the attribution maps, due to the required interpolation.
The effect of different depth choices for the bottleneck is visualized in ﬁgure 4.
Enforcing local smoothness
Pooling operations and convolutional layers with stride greater than 1 are ignoring parts of the input.
The ignored parts cause the Per-Sample Bottleneck to overﬁt to a grid structure (shown in Appendix B).
To obtain a robust and smooth attribution map, we convolve the sigmoid output with a ﬁxed Gaussian kernel with standard deviation σ s .
Smoothing the mask during training is not equivalent to smoothing the resulting attribution map, as during training also the gradient is averaged locally.
Combining everything, the parametrization for the Per-Sample
Bottleneck is:
[Equation]
3. R EADOUT B OTTLENECK
For the Readout Bottleneck , we train a second neural network to predict the mask α .
In contrast to the Per-Sample Bottleneck, this model is trained on the entire training set.
In Figure 3, the Readout
Bottleneck is depicted.
K¨ummerer et al.
(2014) introduced the readout concept for gaze prediction.
The general idea is to collect feature maps from different depths and then combine them using 1x convolutions.
In a ﬁrst forward pass, no noise is added and we collect the different feature maps.
As the spatial resolution of the feature maps differs, we interpolate them bilinearly to match the spatial dimensions of the bottleneck layer.
The readout network then predicts the information mask based on the collected feature maps.
In a second forward pass, we insert the bottleneck layer into the network and restrict the ﬂow of information.
Except for the learned importance values, the Readout Bottleneck is identical to the mechanism of the
Per-Sample Bottleneck.
The measure of information works in the same way as for the Per-Sample
Bottleneck and we also use the same smoothing.
Given a new sample, we can obtain a heatmap by merely collecting the feature maps and executing the readout network.
c onv  β = 0 .
1 /k p =
.β
= 1 /k p = 0 .β
= 10 /k p =
0 .β
= 100 /k p = 0 .β
= 1000 /k
p =
.β
= 10 / k conv conv2conv3conv4conv
The readout network consists of three 1x1 convolutional layers.
ReLU activations are applied between convolutional layers and a ﬁnal sigmoid activation yields λ ∈ [0 , 1] .
As the input consists of upscaled feature maps, the ﬁeld-of-view is large, although the network itself only uses 1x1 conv.
kernels.
E VALUATION 4. E XPERIMENTAL S ETUP
As neural network architectures, we selected the ResNet-50 (He et al., 2016) and the VGGSimonyan & Zisserman (2014), using pretrained weights from the torchvision package (Paszke et al., 2017).
These two models cover a range of concepts: dimensionality by stride or max-pooling, residual connections, batch normalization, dropout, low depth (16-weight-layer VGG), and high depth (50-weight-layer ResNet).
This variety makes the evaluation less likely to overﬁt on a speciﬁc model type.
They are commonly used in the literature concerning attribution methods.
For PatternAttribution on the VGG-16, we obtained weights for the signal estimators from Kindermans et al.
As naive baselines, we selected random attribution, Occlusion with patch sizes 8x8 and 14x14, and
Gradient Maps.
SmoothGrad and Integrated Gradients cover methods that accumulate gradients.
We include three methods with modiﬁed backpropagation rules: PatternAttribution, GuidedBP, and
LRP.
As our implementation of PatternAttribution and LRP does not support skip connections, we report no results for them on the ResNet-50.
We also include Grad-CAM and its combination with
GuidedBP, GuidedGrad-CAM.
For the compared methods, we use the hyperparameters suggested by the original authors.
For LRP (Bach et al., 2015), different rules exist.
We include the most commonly used α = 1 , β = 0 , (cid:15) =
0 rule which is also shown in the ﬁgures.
We also include α = 0 , β = - 1 , (cid:15)
= 5 as it gives better results on the bounding-box task.
For our methods, the hyperparameters are obtained using grid search with the degradation metric as objective (Appendix F).
The readout network is trained on the training set of the ILSVRC12 dataset (Russakovsky et al., 2015) for E = 20 epochs.
The optimization objective of the bottleneck is L CE + β L I as given in equation 6.
Generally, the information loss L I is larger than the classiﬁer loss by several orders of magnitude as it sums over height h , width w and channels c .
We therefore use k = hwc as a reference point to select β .
In ﬁgure 4, we display the effect of different orders of β and the effect of layer depth for ResNet- (see Appendix C for VGG-16).
A uniform uninformative heatmap is obtained for β ≥ 1000 /k
– all information gets discarded, resulting in an output probability of 0 for the correct class.
The heatmap for β = 0 .
1 /k is similar to β = 1 /k
but more information is passed, i.e. less noises is added.
In appendix F, we also compare pre- to post-training accuracy and estimate the mutual information for both Bottleneck types.
For the Per-Sample Bottleneck, we investigate β = 1 /k, 10 /k, 100 /k .
The
Readout network is trained with the best performing value β = 10 /k .
[Figure]
4. Q UALITATIVE A SSESSMENT
In Figure 5, the heatmaps of all evaluated samples are shown (for more samples, see Appendix A).
Subjectively, both the Per-Sample and Readout Bottleneck identify areas relevant to the classiﬁcation well.
While Guided Backpropagation and PatternAttribution tend to highlight edges, the Per-Sample
Bottleneck focuses on image regions.
For the Readout Bottleneck, the attribution is concentrated a little more on the object edges.
Compared to Grad-CAM, both our methods are more speciﬁc, i.e. fewer pixels are scored high.
4. S ANITY C HECK : R ANDOMIZATION OF M ODEL P ARAMETERS
Adebayo et al.
(2018) investigates the effect of parameter randomization on attribution masks.
A sound attribution method should depend on the entire network’s parameter set.
Starting from the last layer, an increasing proportion of the network parameters is re-initialized until all parameters are random.
We excluded PatternAttribution as the randomization would require to re-estimate the signal directions.
The difference between the original heatmap and the heatmap obtained from the randomized model is quantiﬁed using SSIM (Wang et al., 2004).
We discuss implementation details in the appendix G.1.
In ﬁgure 6, we display the results of the sanity check.
For our methods, we observe that randomizing the ﬁnal dense layer drops the mean SSIM by around 0.4.
The values for the Readout Bottleneck are of limited expressiveness as we did not re-train it after randomization.
For SmoothGrad and Int.
Gradients, the SSIM drops by more than 0.4.
The heatmaps of GuidedBP and LRP remain similar even if large portion of the network’s parameters are randomized – they do not explain the network prediction faithfully.
Nie et al.
(2018) provides theoretical insights about why GuidedBP fails.
Sixt et al.
(2019) analyzes why LRP and other modiﬁed BP methods fail.
4. S ENSITIVITY - N
Ancona et al.
(2018) proposed Sensitivity-n as an evaluation metric for attribution methods.
Sensitivity-n masks the network’s input randomly and then measures how strongly the amount of attribution in the mask correlates with the drop in classiﬁer score.
Given a set T n containing n randomly selected pixel indices, Sensitivity-n measures the Pearson correlation coefﬁcient:
[Equation]
where S c ( x ) is the classiﬁer logit output for class c , R i is the relevance at pixel i and x [ x Tn =0] denotes the input with all pixels in T n set to zero.
As in the original paper, we pick the number of masked pixels n in a log-scale between 1 and 80% of all pixels.
For each n , we generate 100 different index sets T and test each on 1000 randomly selected images from the validation set.
The correlation is calculated over the different index sets and then averaged over all images.
In Figure 6, the Sensitivity-n scores are shown for ResNet-50 and VGG-16.
When masking inputs pixel-wise as done in Ancona et al.
(2018), the Sensitivity-n score across methods are not well discriminable, all scores range within the lower 10% of the scale.
We ran the metric with a tile size of 8x8 pixels.
This resulted in an increase of the Sensitivity-n score to 30% of the scale.
Although not shown in the ﬁgure due to the zooming, Occlusion-8x8 performs perfectly for small n as its relevance scores correspond directly to the drop in logits per 8x8 tile.
We ﬁnd that the Per-Sample Bottlenecks β
= 10 /k perform best for both models above n = 2 · 10 3 pixels, i.e. when more then 2% of all pixels are masked.
4. B OUNDING B OX
To quantify how well attribution methods identify and localize the object of interest, we rely on bounding boxes available for the ImageNet dataset.
Bounding boxes may contain irrelevant areas, in particular for non-rectangular objects.
We restrict our evaluation to images with bounding boxes covering less than 33% of the input image.
In total, we run the bounding box evaluation on 11, images from the ImageNet validation set.
If the bounding box contains n pixels, we measure how many of the n -th highest scored pixels are contained in the bounding box.
By dividing by n , we obtain a ratio between 0 and 1.
The results are shown in Table 1.
The Per-Sample Bottleneck outperforms all baselines on both VGG-16 and
ResNet-50 each by a margin of 0.152.
An alternative metric would be to take the sum of attribution in the bounding box and compare it to the total attribution in the image.
We found this metric is not robust against extreme values.
For the
ResNet-50, we found basic Gradient Maps to be the best method as a few pixels receiving extreme scores are enough to dominate the sum.
4. I MAGE D EGRADATION
As a further quantitative evaluation, we rely on the degradation task as used by Ancona et al.
Kindermans et al.
; Hooker et al.
; Samek et al.
Given an attribution heatmap, the input is split in tiles, which are ranked by the sum of attribution values within each corresponding tile of the attribution.
At each iteration, the highest-ranked tile is replaced with a constant value, the modiﬁed input is fed through the network, and the resulting drop in target class score is measured.
A steep descent of the accuracy curve indicates a meaningful attribution map.
[Table]
When implemented in the described way, the most relevant tiles are removed ﬁrst (MoRF).
However, Ancona et al.
(2017) argues that using only the MoRF curve for evaluation is not sufﬁcient.
For the MoRF score, it is beneﬁcial to ﬁnd tiles disrupting the output of the neural network as quickly as possible.
Neural networks are sensitive to subtle changes in the input (Szegedy et al., 2013).
The tiles do not necessarily have to contain meaningful information to disrupt the network.
Ancona et al.
(2017) proposes to invert the degradation direction, removing tiles ranked as least relevant by the attribution method ﬁrst (LeRF).
The LeRF task favors methods that identify areas sufﬁcient for classiﬁcation.
We scale the network’s output probabilities to be in [0 , 1] :
[Equation]
where t 1 is the model’s average top-1 probability on the original samples and b is the mean model output on the fully degraded images.
Both averages are taken over the validation set.
A score ofcorresponds to the original model performance.
Both LeRF and MoRF degradation yield curves as visualized in Figure 7, measuring different qualities of the attribution method.
To obtain a scalar measure of attribution performance and to combine both metrics, we propose to calculate the integral between the MoRF and LeRF curves.
The results for all implemented attribution methods on the degradation task are given in Table 1.
We evaluated both models on the full validation set using 8x8 and 14x14 tiles.
In Appendix H, we show the mean LeRF and MoRF curves for 14x14 tiles.
The Per-Sample Bottleneck outperforms all other methods in the degradation benchmark except for GuidedGrad-CAM on ResNet-50 where it scores comparably (score difference of 0 . 004 ).
The Readout Bottleneck generally achieves a lower degradation scores but still perform competitively.
C ONCLUSION
We propose two novel attribution methods that return an upper bound on the amount of information each input region provides for the network’s decision.
Our models’ core functionality is a bottleneck layer used to inject noise into a given feature layer and a mechanism to learn the parametrized amount of noise per feature.
The Per-Sample Bottleneck is optimized per single data point,
whereas the
Readout Bottleneck is trained on the entire dataset.
Our method does not constrain the internal network structure.
In contrast to several modiﬁed backpropagation methods, it supports any activation function and network architecture.
To evaluate our method, we extended the degradation task to quantify model performance deterioration when removing both relevant and irrelevant image tiles ﬁrst.
We also show results on how well ground-truth bounding boxes are scored.
Our Per-Sample and Readout Bottleneck both show competitive results on all metrics used, outperforming state of the art with a signiﬁcant margin for some of the tasks.
Generally, we would advise using the Per-Sample Bottleneck over the Readout Bottleneck.
It performs better and is more ﬂexible as it only requires to estimate the mean and variance of the feature map.
The Readout Bottleneck has the advantage of producing attribution maps with a single forward pass once trained.
Images with multiple object instances provide the network with redundant class information.
The Per-Sample Bottleneck may discard some of the class evidence.
Even for single object instances, the heatmaps of the Per-Sample Bottleneck may vary slightly due to the randomness of the optimization process.
The method’s information-theoretic foundation provides a guarantee that the network does not require regions of zero-valued attribution for correct classiﬁcation.
To our knowledge, our attribution method is the only one to provide scores with units (bits).
This absolute frame of reference allows a quantitative comparison between models, inputs, and input regions.
We hope this contributes to a deeper understanding of neural networks and creates trust to use modern models in sensitive areas of application.
A CKNOWLEDGEMENT
We want to thank our anonymous reviewers for their valuable suggestions.
In particular, we thank reviewer 1 for encouraging us to include the sanity checks.
We are indebted to B. Wild, A. Elimelech,
M. Granz, and D. Dormagen for helpful discussions and feedback.
For LRP, we used the open source implementation by M. B ¨ ohle and F. Eitel (B ¨ ohle et al., 2019) and we thank AK.
Dombrowski for an implementation and patterns for PatternAttribution with us.
LS was supported by the Elsa-NeumannScholarship by the state of Berlin.
We are also grateful to Nvidia for providing us with a Titan Xp and to ZEDAT for granting us access to their HPC system.
A V ISUAL C OMPARISON OF A TTRIBUTION M ETHODS
Input
Grad-CAM GuidedBP.
Occl.-
Per
[Table]
Sample
Readout
Grad
[Table]
GuidedBP.
PatternAttr
Per-Sample
Readout
ResNet
[Table]
ResNet
[Table]
ResNet-50 ResNetVGG-
VGGVGG-
VGGVGG-
W e i m a r a n e r
B a rr o w
C a pu c h i n
M op e d
S o a pd
i s p e .
J ac k - o ’- l a n t .
Z u cc h i n i
B r oo m
B a g it c
S l o t hb
ea r on c ho ng li rf u m
R nk
B G RID A RTIFACTS WHEN NOT USING S MOOTHING
[Figure]
C E FFECTS OF DIFFERENT β AND LAYER DEPTH FOR THE VGG-
[Figure]
[Equation]
[Figure] [Equation] [Figure]
[Equation] [Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
D D ERIVATION OF THE U PPER - BOUND OF M UTUAL I NFORMATION
For the mutual information I [ R, Z ] , we have:
[Equation]
E M EAN AND VARIANCE OF Z
The λ ( X ) linearly interpolate between the feature map R and the noise (cid:15) ∼ N ( µ R , σ 2 R ) , where µ R and σ R are the estimated mean and standard derivation of R .
Both R = f l ( X ) and λ ( X ) depend on the input random variable X .
[Equation]
For the mean of Z , we have:
[Equation]
(cid:46) substituting in deﬁnition of Z =
[Equation]
independence of λ and
[Equation]
As λ ( X ) and R are multiplied together, they form a complex product distribution.
If they do not correlate, E [ Z ] ≈ E [ (cid:15) ]
≈ E
[ R ] .
A similar problem araises for the variance:
[Equation]
The multiplication of λ ( X ) and R causes in general the variance of Z and R to not match:
Var[ Z ] (cid:54) =
Var[ R ] .
F H YPERPARAMETERS
Parameter
ResNetVGG-
Search space
Target layer conv3conv
Optimizer
Adam (Kingma & Ba (2014))
Learning Rate η ={ 0 . 03 , 0 . 1 , 0 . 3 , 1 , 3 , 10 }
Balance Factor β
= 10 /k { 0 . 001 , 0 . 01 , 0 . 1 , 1 , 10 , 100 , 300 }
Iterations
T =
[Table] [Table]
[Table]
Hyperparameters for the Readout Bottleneck.
In Table 2, we provide hyperparameters for the Per-Sample Bottleneck.
In Table 3, we provide hyperparameters for the Readout Bottleneck.
In Table 4, a comparison of pre- to post-training accuracy and the estimated mutual information is shown for both Bottleneck types.
The Per-Sample Bottleneck can learn to suppress negative evidence for each sample and the accuracy is close to 1 for β ≤ 10 /k .
For β = 100 /k , also positive evidence is discarded and the accuracy decreases to 0 .
The Readout Bottleneck learns to suppress negative evidence for small β ≤ 0 .
1 /k and slightly improves the ﬁnal accuracy.
G E VALUATION M ETRICS G. S ANITY C HECK : W EIGHT R ANDOMIZATION
We use the cascading parameter randomization sanity check from Adebayo et al.
Following the original paper, we used the skimage SSIM implementation with a window of size 5.
For LRP, we found that the weight randomization ﬂips the values of the saliceny heatmap, e.g. h r ≈ −
h
o where h r is the heatmap with random weights and h o the heatmap on the original model.
Therefore for LRP, we used: max( ssim ( h o , normalize ( h r ) , ssim ( h o , normalize ( − h r ))) .
We normalized the heatmaps by ﬁrst clamping the 1-th and 99-th percentile and then rescaling the heatmap it to [0 , 1] .
We run the sanity check on 200 randomly selected samples from the ImageNet valdiation set.
H M O RF AND L E RF D EGRADATION P ATHS
[Figure] [Figure]