Paper:
In this paper, we introduce a novel method, called step-wise sensitivity analysis, which makes three contributions towards increasing the interpretability of Deep Neural Networks (DNNs). First, we are the first to suggest a methodology that aggregates results across input stimuli to gain model-centric results. Second, we linearly approximate the neuron activation and propose to use the outlier weights to identify distributed code. Third, our method constructs a dependency graph of the relevant neurons across the network to gain fine-grained understanding of the nature and interactions of DNN's internal features. The dependency graph illustrates shared subgraphs that generalise across 10 classes and can be clustered into semantically related groups. This is the first step towards building decision trees as an interpretation of learned representations.

Review:
Summary:
The paper introduces a new approach for interpreting deep neural networks called step-wise sensitivity analysis. The approach is conceptually quite simple and involves some interesting ideas, but I have some serious concerns whether the output produced by this method carries any meaning at all. If the authors were able to refute my concerns detailed below, I would raise my score substantially.
Strengths:
+ Potentially interesting heuristic to identify groups of feature channels in DNNs that encode image features in a distributed way
Weaknesses:
- Using the magnitude of the gradient in intermediate layers of ReLU networks is not indicative of importance
- No verification of the method on a simple toy example
Details:
Main issue: Magnitude of the gradient as a measure of importance.
I have trouble with the use of the gradient to identify "outliers," which are deemed important. Comparing the magnitude of activations across features does not make sense in a convnet with ReLUs, because the scale of activations in each feature map is arbitrary and meaningless. Consider a feature map h^l[i,x,y,f] (l=layer, i=images, x/y=pixels, f=feature channels), convolution kernels w^l[x,y,k,f] (k=input channels, f=output channels) and biases b^l[f]:
h^l[i,:,:,f] = ReLU(b^l[f] + \sum_k h^(l-1)[i,:,:,k] * w^l[:,:,k,f])
Assume, without loss of generality, the feature map h^l[:,:,:,f] has mean zero and unit variance, computed over all images (i) in the training set and all pixels (x,y). Let's multiply all "incoming" convolution kernels w^l[:,:,:,f] and biases b^l[f] by 10. As a result, this feature map will now have a variance of 100 (over images and pixels). Additionally, let's divide all "outgoing" kernels w^(l+1)[:,:,f,:] by 10.
Simple linear algebra suffices to verify that the next layer's features h^(l+1) -- and therefore the entire network output -- are unaffected by this manipulation. However, the gradient of all units in this feature map is 10x as high as that of the original network. Of course the gradient in layer l-1 will be unaltered once we backpropagate through w^l, but because of the authors' selection of "outlier" units, their graph will look vastly different.
In other words, it is unclear to me how any method based on gradients should be able to meaningfully assign "importance" to entire feature maps. One could potentially start with the assumption of equal importance when averaged over all images in the dataset and normalize the activations. For instance, ReLU networks with batch norm and without post-normalization scaling would satisfy this assumption. However, for VGG-16 studied here, this is not the case.
On a related note, the authors' observation in Fig. 4b that the same features are both strongly positive and strongly negative outliers for the same class suggests that this feature simply has a higher variance than the others in the same layer and is therefore picked most of the time. Similarly, the fact that vastly different classes such as shark and German Sheppard share the same subgraphs speaks to the same potential issue.
Secondary issue: No verification of the method on simple, understandable toy example.
As shown by Kindermans et al. [1], gradient-based attribution methods fail to produce the correct result even for the simplest possible linear examples. The authors do not seem to be aware of this work (at least it's not cited), so I suggest they have a look and discuss the implications w.r.t. their own work. In addition, I think the authors should demonstrate on a simple, controlled (e.g. linear) toy example that their method works as expected before jumping to a deep neural network. I suppose the issue discussed above will also surface in purely linear multi-layer networks, where the intermediate layers (and their gradients) can be rescaled arbitrarily without changing the network's function.
References:
[1] Kindermans P-J, Schütt KT, Alber M, Müller K-R, Erhan D, Kim B, Dähne S (2017) Learning how to explain neural networks: PatternNet and PatternAttribution. arXiv:170505598. Available at: http://arxiv.org/abs/1705.05598