N EURO C HAINS : E XTRACT L OCAL R EASONING C HAINS OF D EEP N EURAL N ETS
Anonymous authors
Paper under double-blind review
A BSTRACT
We study how to explain the main steps/chains of inference that a deep neural net (DNN) relies on to produce predictions in a local region of data space.
This problem is related to network pruning and interpretable machine learning but the highlighted differences are: (1) ﬁne-tuning of neurons/ﬁlters is forbidden: only exact copies are allowed; (2) we target an extremely high pruning rate, e.g., ≥ 95% ; (3) the interpretation is for the whole inference process in a local region rather than for individual neurons/ﬁlters or on a single sample.
In this paper, we introduce an efﬁcient method, NeuroChains, to extract the local inference chains by optimizing a differentiable sparse scoring for the ﬁlters and layers to preserve the outputs on given data from a local region.
Thereby, NeuroChains can extract an extremely small sub-network composed of ﬁlters exactly copied from the original DNN by removing the ﬁlters/layers with small scores.
We then visualize the sub-network by applying existing interpretation technique to the retained layer/ﬁlter/neurons and on any sample from the local region.
Its architecture reveals how the inference process stitches and integrates the information layer by layer and ﬁlter by ﬁlter.
We provide detailed and insightful case studies together with three quantitative analyses over thousands of trials to demonstrate the quality, sparsity, ﬁdelity and accuracy of the interpretation within the assigned local regions and over unseen data.
In our empirical study, NeuroChains signiﬁcantly enriches the interpretation and makes the inner mechanism of DNNs more transparent than before.
I NTRODUCTION
Deep neural networks (DNNs) greatly reshape a variety of tasks — object classiﬁcation, semantic segmentation, natural language processing, speech recognition, robotics, etc.
Despite its success on a vast majority of clean data, DNNs are also well-known to be sensitive to small amounts of adversarial noises.
The lack of sufﬁcient interpretability about their success or failure is one major bottleneck of applying DNNs to important areas such as medical diagnosis, public health, transportation systems, ﬁnancial analysis, etc.
Interpretable machine learning has attracted growing interest in a variety of areas.
The forms of interpretation vary across different methods.
For example, attribution methods (Bach et al., 2015;
Sundararajan et al., 2017; Shrikumar et al., 2017; Montavon et al., 2017; Kindermans et al., 2017;
Smilkov et al., 2017) produce the importance score of each input feature to the output prediction for an given sample, while some other methods (Zeiler & Fergus, 2014; Simonyan et al., 2013; Erhan et al., 2009) aim to explain the general functionality of each neuron/ﬁlter or an individual layer regardless of the input sample.
Another line of works (Ribeiro et al., 2016; Wu et al., 2018; Hou &
Zhou, 2018) explain DNNs in a local region of data space by training a shallow (e.g., linear) and easily interpretable model to approximate the original DNN on some locally similar samples.
Thereby, they reduces the problem to explaining the shallow model.
These methods essentially reveal the neuron to neuron correlations (e.g., input to output, intermediate layer/neuron to output, etc), but they cannot provide an overview of the whole inference process occurring inside the complicated structure of DNNs.
In this paper, we study a more challenging problem: Can we unveil the major hidden steps of inference in DNNs and present them in a succinct and human-readable form?
Solving this problem helps to answer many signiﬁcant questions, e.g., which layer(s)/neuron(s) plays the most/least im- portant role in the inference process?
Do two similar samples really share most inference steps?
Do all samples need the same number of neurons/layers to locate the key information leading to their correct predictions?
How/when/where does a failure happen during the inference on a DNN?
Which neuron(s)/layer(s)/feature(s) are shared by different samples from the same region even when their labels differ?
Are DNNs using entirely different parts for data from different local regions?
Some of them are related to other problems such as network pruning (Han et al., 2015; Li et al., 2016) and neural architecture search (NAS) (Zoph & Le, 2017).
For example, are winning tickets (Frankle &
Carbin, 2018; Liu et al., 2018)
universal over different local regions and classes?
Does the weight sharing scheme in recent NAS methods (Pham et al., 2018; Liu et al., 2019; Ying et al., 2019) limit the searching space or quality?
[Figure]
We develop an efﬁcient tool called NeuroChains to extract the underlying inference chains of a DNN for a given local data region.
Speciﬁcally, we aim to extract a much smaller sub-network composed of a subset of neurons/ﬁlters exactly copied from the original
DNN and whose output for data from the local region stays consistent with that of the original DNN.
In experiments, we assume the data from the same classes reside in a local region.
While the selected ﬁlters explain the key information captured by the original DNN when applied to data from the local region, the architecture of the sub-network stitches these information sequentially, i e., step by step and layer by layer, and thus recover the major steps of inference that lead to the ﬁnal outputs.
Despite its combinatorial nature, we parameterize the sub-network as the original DNN with an additional score multiplied to each ﬁlter/layer’s output featuremap.
Thereby, we formulate the above problem of sub-network extraction as optimizing a differentiable sparse scoring of all the ﬁlters and layers in order to preserve the outputs on all given samples.
The above problem can be solved by an efﬁcient back-propagation that only updates the scores with ﬁxed ﬁlter parameters.
The objective is built upon the Kullback–Leibler (KL) divergence between the sub-network’s output distribution and that of the original DNN, along with an (cid:96) 1 regularization for sparse scores over ﬁlters.
We further use a sigmoid gate per layer to choose whether removing the entire layer.
The gate plays an important role in reducing the sub-network size since most local regions do not rely on all the layers.
In practice, we further apply a thresholding to the sparse scores to obtain an even smaller sub-network and employ an additional ﬁne-tuning to the ﬁlter scores on the sub-network.
We illustrate the subnetwork’s architecture and visualize its ﬁlters and intermediate-layers’ featuremaps by existing methods (Mundhenk et al., 2019; Erhan et al., 2009).
NeuroChains is a novel pruning technique speciﬁcally designed for interpreting the local inference chains of DNNs.
As aforementioned, it potentially provides an efﬁcient tool to study other problems in related tasks.
However, it has several fundamental differences to network pruning and exist- ing interpretation tasks , which deter methods developed for these two problems from addressing our problem.
Comparing to network pruning : (1) ﬁne-tuning is not allowed in NeuroChains; (2) it targets a much larger pruning rate for succinct visualization, e.g., ≥ 95% for VGG-19 and ≥ 99% for ResNet-50 on ImageNet with ≤ 200 ﬁlters remained; (3) it is for data from a local region instead of the whole data distribution.
Comparing to mainstream interpretation tasks : (1) NeuroChains produces an interpretation of the entire inference process in a local region (which may contain different classes) rather than of one neuron/ﬁlter or output on/around a single sample; (2) the sub-network’s architecture provides orthogonal information to the importance of individual neurons/ﬁlters.
R ELATED W ORKS
Interpretable machine learning methods can be mainly categorized into the ones aiming to evaluate the importance of each input feature of a single sample and the ones explaining individual neurons/ﬁlters.
Approaches in the ﬁrst category usually rely on certain back-propagation from the
DNN’s output to derive an importance score for each input feature or hidden node.
Earlier works are based on the back-propagated gradients, e.g., deconvolution (Zeiler & Fergus, 2014), backpropagation (Simonyan et al., 2013) and guided back-propagation (Springenberg et al., 2014).
Sundararajan et al.
(2017) proposed to (approximately) calculate the integral of the gradients along a path between a baseline point and the input sample, which ensures the sensitivity and implementation invariance lacking in some previous methods.
More recent methods propose novel backpropagation rules to directly derive the attribution scores of neurons from output to input, e.g.,
DeepLIFT (Shrikumar et al., 2017), deep Taylor decomposition (Montavon et al., 2017), and layerwise relevance propagation (LRP) (Bach et al., 2015).
Methods in the second class treat DNNs as black boxes and seek a simple model to explain how the
DNN’s output changes in a local region.
For example, they add perturbations to different parts of the input to evaluate how the perturbations change the output (Ancona et al., 2017), which reﬂect the importance of different parts.
Zeiler & Fergus (2014) covered different parts of the input image with a gray square, which led to different prediction probabilities on the true class.
Instead, Zintgraf et al.
(2017) replaced each patch in the input image with the surrounding patch and tracked the induced changes in the output.
In LIME (Ribeiro et al., 2016), Ribeiro et al.
trained a sparse linear model on noisy input-output pairs as a local surrogate approximating the original DNNs, where the sparse weights are used to explain the importance of input features.
As mentioned before, our main difference to the above methods are we explain DNNs for a local region of multiple samples and we further explain how DNNs step by step integrate the information of important ﬁlters/neurons.
Network pruning (Han et al., 2015; Li et al., 2016) remove redundant neurons/nodes or connections/weights from a pre-trained DNN and ﬁne-tune the sub-network.
Structural pruning removes whole layers/channels/ﬁlters/neurons according to a certain norm of the associated weights (Li et al., 2016) or sparsity (Hu et al., 2016).
In contrast, Frankle & Carbin (2018); Liu et al.
(2018) prune a DNN during its training.
Luo et al.
(2017) apply pruning to two adjacent convolution layers at each time to take the dependency between the two layers into account.
Several recent works empirically verify “lottery ticket hypothesis”, i.e., there exists sub-networks (i.e., winning tickets) that can reach comparable generalization performance as the original DNN if re-trained.
In contrast, the sub-network extracted by NeuroChains cannot be fully re-trained since it has to preserve the original
DNN’s ﬁlters, and our goal is to retain the generalization performance only for a local region.
N EURO C HAINS 3. P ROBLEM : E XTRACT L OCAL I NFERENCE C HAINS AS S UB - NETWORKS
Although the DNNs widely used nowadays are usually composed of hundreds of layers and millions to billions of hidden nodes.
When applied to samples from a local region in data space, it is plausible that its inference process mainly relies on a small subset of layers/neurons/ﬁlters.
In this paper, we verify this conjecture by developing an efﬁcient and practical algorithm, i.e., NeuroChains, to extract the subset and its underlying architecture as a sub-network whose weights/ﬁlters are selected and exactly copied from the original DNN while its outputs in a given local region retain the ones produced by the original DNN.
Although DNNs are usually non-smooth in deﬁnition if using a nonsmooth piece-wise activation such as ReLU, when trained with the commonly used techniques, e.g., data augmentation, mix-up, dropout, the resulted DNNs are relatively smooth in a sufﬁciently small local region.
In order to preserve the original inference chains, we do not allow any ﬁne-tuning or re-training on any ﬁlter or the the weight vector corresponding to any neuron: they can only be exactly copied from the original DNN.
Let F ( · ; { W (cid:96) } (cid:96) =1: L ) (a mapping from input to output) denote the original DNN,
W (cid:96) represents the set of ﬁlters/weight vectors in layer- (cid:96) , and W (cid:96)
[ i ] represents the i th ﬁlter/weight vector in layer- (cid:96) .
Any sub-network fulﬁlling our above requirement can be deﬁned and parameterized by an indicator vector M (cid:96) per layer, whose each entry is a { 0 , 1 } value indicating whether retaining the associated ﬁlter/neuron in W (cid:96) .
We further deﬁne operator ◦ as
[Equation]
Thereby, { M (cid:96) } (cid:96) =1: L deﬁnes a qualiﬁed sub-network for inference chain and its weights are { W (cid:96) ◦
M (cid:96) } (cid:96) =1: L , where we extend the operator ◦ to make W ◦ M =
{ W (cid:96) ◦
M (cid:96) } (cid:96) =1: L given the original
DNN’s weights W = { W (cid:96) } (cid:96) =1: L .
Given a set of samples X drawn from a local region of data space, we can formulate the problem of ﬁnding an inference chain as the following combinatorial optimization, which aims to ﬁnd the most sparse indicator M (i.e., the sub-network with the fewest ﬁlters retained) that does not change the outputs of the original DNN for ∀ x ∈ X , i.e.,
[Equation] [Equation]
However, directly solving this combinatorial optimization is impractical since the possible choices for M (cid:96) is of exponential number.
In this paper, we relax the 0-1 indicator vector M (cid:96) to a real-valued score vector S (cid:96) of the same size.
We deﬁne an operator (cid:12) applied to W (cid:96) and its associated scores
S (cid:96) as
[Equation]
Note we do not limit entries in S (cid:96) within [0 , 1] due to the possible redundancy among ﬁlters in the original DNN, i.e., there might be ﬁlters of similar functionality for the given samples and a preferred pruning should be able to only preserve one of them and multiply it by the number of those redundant ﬁlters in the sub-network.
In addition, less constrains are easier to handle in optimization and helpful to ﬁnd sub-network whose outputs are closer to that of the original
DNN, since the class of sub-networks with parameters W (cid:12) S includes all the sub-networks with parameters W ◦ M .
Hence, we relax the challenging combinatorial optimization to the following unconstrained continuous optimization, i.e.,
[Equation]
where l ( · , · ) is a loss function aiming to minimize the distance between the original DNN’s output
F ( x ; W ) and the sub-network’s output F ( x ; W (cid:12) S ) .
In our experiments, for classiﬁcation, we use
KL-divergence between the output distributions over classes, where the two output distributions are computed by applying softmax to F ( x ; W ) and F ( x ; W (cid:12) S ) respectively, i.e.,
[Equation]
In addition, empirical evidences (Krueger et al., 2017; Singh et al., 2016) show that for most samples there exist some layers that can be entirely removed without changing the ﬁnal prediction.
Hence, only a few hard and confusing samples need more delicate features, while most other samples can be correctly classiﬁed based on simple patterns from shallower layers.
Therefore, in
NeuroChains, we apply a sigmoid function with input score α (cid:96) as a gate G (cid:96) determining whether removing the entire layer- (cid:96) during pruning, i.e.,
[Equation] [Equation]
where T is a temperature parameter.
With a gate G (cid:96) applied after each layer- (cid:96) whose input and output has the same size (which is common in many DNNs), we can recursively deﬁne the input
H (cid:96) +1 ( · ) to the next layer-( (cid:96) + 1 {+), i.e., H (cid:96) +
+}
( x ; { W (cid:96) (cid:48) (cid:12) S (cid:96) (cid:48) , α (cid:96) (cid:48) } (cid:96) (cid:48)
=1: (cid:96) )
= (cid:26) G (cid:96) · F (cid:96) ( H (cid:96) ; W (cid:96) (cid:12) S (cid:96) )
+ (1 − G (cid:96) )
· H (cid:96) ( x ; { W (cid:96) (cid:48) (cid:12) S (cid:96) (cid:48) , α (cid:96) (cid:48) } (cid:96) (cid:48)
=1: (cid:96) − 1 ) , if input size = output size
F (cid:96) ( H (cid:96) ; W (cid:96) (cid:12) S (cid:96) ) otherwise (7) where F (cid:96) ( H (cid:96) ; W (cid:96) (cid:12) S (cid:96) ) denotes the output of layer- (cid:96) .
The reason to use a gate here is that we expect to either remove the whole layer or retain it without adding an extra shortcut (which will change the original DNN’s architecture).
We apply a T < 1 to sharpen the gate values.
Since we prefer to remove non-informative layers, in the objective, we add another regularization α (cid:96) to encourage the removal of entire layers (because decreasing α reduces G (cid:96) and thus increase the chance of layer removal).
Therefore, the ﬁnal optimization for NeuroChains is
[Equation]
[-=1: L )) + λ L (cid:88) (cid:96) = (cid:107) S (cid:96) (cid:107) 
 + λ g-][Table]
Our objective above is similar to the one used in Network Slimming (Liu et al., 2017) but we optimize it for a local region (so we can consider to remove layers) and we do not allow ﬁne tuning on weights W [Table]
3. A LGORITHM
[Table]
Our algorithm is simply a standard back-propagation for the optimization problem in Eq.
(8), which produces sparse scores for ﬁlters and gate values for layers.
Note the weights in W are ﬁxed and the backpropagation only updates S .
We initialize the ﬁlter scores S =
so W (cid:12) S =
W at the beginning of optimization.
We initialize the gate score α (cid:96) = 0 for all (cid:96) = 1 : L so G (cid:96) = 0 .
5 at the beginning, i.e., the probabilities to remove or to retain a layer is equal.
For classiﬁcation, we set loss l ( · , · ) to be the KL-divergence between the output distributions of the original DNN and the sub-network.
After convergence of the optimization, we then apply a simple thresholding to these scores to further remove more ﬁlters and layers: (1) we remove the ﬁlters with score under a threshold τ ; (2) we remove layer- (cid:96) if G (cid:96) < 0 .
This yields a sufﬁciently small sub-network architecture but the scores might be sub-optimal in preserving the original DNN’s outputs within the local region.
Therefore, we further ﬁne-tune the nonzero scores in S (note ﬁlters are always ﬁxed), i.e, the scores for the retained ﬁlters/layers, by minimizing Eq.
(8) without the two regularizations, if the ﬁrst term in Eq.
(8) exceeds a threshold t .
Given a sub-network produced by NeuroChains, we then visualize its architecture and scores as the structure of the inference chains.
Moreover, we visualize the retained ﬁlters and layers on the chains by their activation patterns and featuremaps, respectively, using existing interpretation methods (Mundhenk et al., 2019; Erhan et al., 2009).
E XPERIMENTS
In experiments, we apply NeuroChains to extract the inference chains of two widely-adopted CNNs pre-trained on ImageNet, i.e., VGG-19 and ResNet-50.
We provide the basic information of the two DNNs in Table 1.
In the following, We will present three quantitative analyses over hundreds of case studies , which show that (1) NeuroChains is capable to produce sub-networks retaining only < 5% of ﬁlters and meanwhile preserve the outputs of the original DNN in most cases; (2) every ﬁlter selected by NeuroChains is important to preserving the outputs since removing one will leads to considerable drop in performance; (3) the sub-network extracted based on a ﬁnite number of samples can be generalized to unseen samples in nearby regions. To demonstrate the effectiveness of NeuroChains in local regions in the non-smooth raw-input space, we evaluate the sub-network on the adversarial examples of each sample.
We also study the ﬁring behaviour of ﬁlters in the original network and the sub-network. We will then provide several detailed and insightful case studies and visualizations of extracted sub-networks for different local regions.
[Figure]
D e c r e a s e o f p r o b a b ili t
y f o r t
h e o r
i g i n a l - D NN p r e d i c t
e d c l a ss
[Figure]
For each sub-network and every image used to extract it, we
We implement
NeuroChains by
PyTorch (Paszke et al., 2017).
In every case study, we ﬁrstly randomly sample 2 classes and then randomly sample 10 images from each class’s images that the original DNN has high conﬁdence, e.g., ≥ 95% .
Note high conﬁdence does not guarantee the correctness since it can be associated with a wrong class.
We choose images with high conﬁdence since the inference on them mostly represent the original
DNN’s inference chains in the region.
We apply inference on those 20 images and their outputs are used in solving the optimization of Eq (8) in order to extract the local inference chain in the form of a sub-network.
For models with shortcuts, e.g., ResNet-50, the sigmoid gate is applied to prune a bottleneck block rather than a layer.
A layer inside a block will be removed if the scores of all ﬁlters in the layer are nearly 0.
We use Adam optimizer for both the optimization of Eq (8) and ﬁne-tune phases for ﬁlter/layer scores.
We use a ﬁxed learning rate 0.005 in the former and 0.01 in the latter.
We set temperature T = 0 .
2 in the sigmoid gate (Eq.
(6)) to encourage the gate value G (cid:96) close to either 0 or 1, and the threshold τ to goal scores is set to 0.1 so that the outputs of sub-networks are as consistent as possible. We only tried a limited number of choices on tens of experiments, and chose the best combination balancing the ﬁdelity and sub-network size, and then applied it to all other experiments without further tuning. In particular, we tried τ ∈ { 0 . 01 , 0 . 1 , 0 . 5 } , λ ∈ { 0 . 001 , 0 . 005 , 0 . 01 , 0 . 1 } , and λ g ∈ { 1 , 2 , 5 } .
For different models, the weights of two penalties in Eq.
(8) are different.
For VGG-19, we use λ = 0 .
007 and λ g = 2 .
While we choose λ = 0 .
0065 and λ g = 0 .
9 for ResNet-50.
This choice performs consistently well and robust on all other experiments.
We ﬁne-tune S in the pruned network if the KL-divergence after training is greater than threshold t = 0 .
The iteration steps of training and ﬁne-tune is 300 and 50 respectively.
We stop training or ﬁne-tuning when the loss difference is less than 0 .
It costs only ∼ 90 s for VGG-19 and ∼ 55 s for ResNet50 to extract a sub-network on a single GPU since we only optimize a few number of scores.
4. Q UANTITATIVE A NALYSES
In Melis & Jaakkola (2018), they propose three criteria to evaluate the interpretation methods for
DNNs, i.e., faithfulness, stability and explicitness.
In this paper, we extend some of their notations and present three quantitative analyses of NeuroChains over 1500 case studies for different local regions, i.e., (1) Fidelity : does the sub-network preserve the original DNN’s outputs on the given samples?
how does it change for sub-networks of different sizes?
(2) Faithfulness : how much degeneration on the ﬁdelity will be caused if removing one ﬁlter from the sub-network?
(3) Stability : what is the ﬁdelity (the accuracy of preserving the original DNN’s prediction) of the sub-network on unseen test images from nearby regions?
In this paper, we evaluate the ﬁdelity and faithfulness by the decreasing amount of probability on the original DNN’s predicted class when using the subnetwork for inference; we evaluate the stability by evaluating the test ﬁdelity of the sub-network over an image’s K-nearest neighbor images from an validation set (details below Figure 14).
All the above metrics are averaged over 1500 sub-networks and across all the images used to extract each sub-network. [Table] L 23 _ L 23 _
L 23 _
L 25 _ L 25 _ 506 L 25 _ L 32 _ L 32 _
L 32 _ 455 L 32 _
L 34 _
L 34 _ 509 L 34 _
L
L
L
L
L L L L
C o m b i n e A LL
L
I npu t i m a g e
S c o r e C a m [ 36 ] R A P [ 37 ] B a s e li n e s The left plot of Figure 2 shows the ﬁdelity for sub-networks of different sizes (measured by the number of ﬁlters) that are extracted by NeuroChains.
Note most sub-networks only retain ≤ 1% of
ResNet-50 for succinct visualization but they preserve the outputs of ResNet-50 with high ﬁdelity.
The right plot in Figure 2 reports the faithfulness of extracted sub-networks, i e., how a subnetwork’s performance in preserving the original DNN’s output degrade if removing one ﬁlter from it, and what is the relationship between this degeneration and the score of the removed ﬁlter.
The statistics on 1500 sub-networks in Figure 2(right) show that removing even only one ﬁlter from the sub-network can signiﬁcantly degrade the explanation performance.
Hence, NeuroChains usually ﬁnd the smallest sub-networks without redundancy among retained ﬁlters/layers, i.e., every critical inference step is retained.
Moreover, the degeneration and the score are strongly and positively correlated, indicating that our optimized scores faithfully reﬂect the importance of ﬁlters in explaining the original DNN. We also present another faithfulness study based on a quotient metric deﬁned below.
Let p, q ∈ ∆ c ( ∆ c is the probability simplex for c classes) be the output probability vectors of the original neural net and the extracted sub-network respectively for a same input.
We deﬁne a quotient metric to measure the change of class prediction between p and q , i.e., [Equation] where y is the predicted class by the original neural net, and Q ( p, q ) is the quotient of two probability differences computed respectively on the original neural net and the sub-network. In particular, it computes the difference of probabilities for class y and the highest-rated other class.
The sign of
Q ( p, q ) indicates whether the predicted class changes (e.g., it changes if Q ( p, q ) < 0 ) while the magnitude of Q ( p, q ) measures the change in prediction conﬁdence.
The result is given in Figureof Appendix. We report the average stability of 1500 sub-networks extracted by NeuroChains in Figure 14 of Appendix. It shows that each sub-network can still achieve a high ﬁdelity (the accuracy of preserving the original DNN’s output) within a relatively large local region (e.g., 100 unseen samples) around the 20 samples used to extract the sub-network.
It therefore indicates that the inference chains extracted for a ﬁnite number of samples by NeuroChains can be applied and generalized to other unseen samples from the same region or even the nearby regions.
So NeuroChains is an effective solution to the problem raised in the introduction.
In order to evaluate NeuroChains on the local regions in the raw input space, we extract sub-networks for uniformly random drawn samples and then evaluate the sub-networks on these samples’ adversarial examples generated by two types of attacks: fast gradient sign method (FGSM) and projected gradient descent (PGD). Figure 5 compares the robustness of the original neural net (LEFT plot) and the extracted sub-networks (RIGHT plot) under different attacks: each of the plots show the histogram of the output probability for the ground-truth class on those samples (original and adversarial). The left plot shows that the two types of adversarial attack are very effective on the original neural net in reducing the probability of ground truth class. In contrast, the right plot shows that
the
NeuroChains extracted sub-networks are much more robust to the attacks, because the optimization in NeuroChains not only removes the irrelevant ﬁlters but also strengthens the important ﬁlters by assigning them weights > 1 . This demonstrates the effectiveness of NeuroChains when applied to local regions in the non-smooth raw-input space, and the extracted sub-networks in this case signiﬁcantly improves the robustness of the original model in defending adversarial attacks. The ﬁring behaviour of the same ﬁlter in the original network and the sub-network is usually different since NeuroChains combines similar ﬁlters by assigning scores/weights > 1 to some ﬁlters. For this reason, one ﬁlter’s featuremap produced in the sub-network is usually a combination of the featuremaps produced by multiple (pruned) ﬁlters from the original network, so the correlation between the same ﬁlter’s featuremaps in the two networks can be small.
In this way, we can better compress the networks to sub-networks with much fewer ﬁlters that are easier to interpret while preserving the original network’s inference process.
To see this phenomenon, we present two case studies in
[Figure] C ASE S TUDIES
We present three case studies of the sub-networks extracted by NeuroChains .
For Figure 1 and
Figure 4, the data points are from classes which are easy to tell apart while in Figure 7 images are always mis-classiﬁed.
More case studies are given in Appendix.
The visualization in each case study is composed of ﬁve parts: (1) the sub-network’s architecture and ﬁlter scores; (2) the 20 images from 2 classes used to extract the sub-network.
The sub-network is supposed to explain the inference chains on images from the same local region of those 20 images; (3) one image from each class and the visualization of the image’s per-layer featuremaps in the sub-network, which is produced by
SMOE (Mundhenk et al., 2019); (4) for comparison, we also present the interpretations produced by two recent works (Wang et al., 2019; Nam et al., 2019) on the two images; (5) visualization of each ﬁlter retained in the sub-network using Erhan et al.
(2009) after zoomed-in.
In the caption below each case study, we highlight some ﬁlters/featuremaps and their related class.
We also provide examples of inference chains composed of ﬁlters stitched by the sub-network’s architecture.
They show that NeuroChains considerably enrich the explanation details of DNN’s inference process in a local data region.
By connecting the important ﬁlters from different layers, the extracted subnetwork highlights the main steps leading to the output prediction.
L L
L L
L L
L
C o m b i n e
A LL
L
I npu t i m a g e
L 0 _ 52 L 0 _ 29 L 0 _
L 5 _
L 5 _ 85 L 5 _
L 10 _
L 10 _ L 10 _
L 14 _ L 14 _
L 14 _
L 19 _
L 19 _ L 19 _
L 23 _
L 23 _ 449 L 23 _
L 28 _
L 28 _ L 28 _
L 32 _ L 32 _
L 32 _
S c
o r e C a m [ 36 ] R
A P [ 37 ]
B a s e li n e s
A PPENDIX 5. Q UANTITATIVE A NALYSIS
We performed 783 experiments each using 20 samples uniformly drawn from two classes (not only the high-conﬁdence samples) and achieved 783 new sub-networks by NeuroChains on VGG-19.
We evaluated these newly generated sub-networks using the quotient metric “A quotient of ”diff to highest scoring other class (extracted)” / ”diff to highest scoring other class (original)
” Eq.
We visualized the result in Figure 8: The left plot is the histogram of the quotient computed over all the 783 × 20 samples.
The histogram shows that most samples keep the original predicted label after pruning, i.e., NeuroChains can preserve the original DNN’s outputs in most cases.
Moreover, the number of ﬁlters preserved in these sub-networks is 157( mean ) ± 43( std ) , which is small enough to explain.
The right plot reports the Faithfulness of NeuroChains in terms of the quotient’s sign.
We remove each ﬁlter from each sub-network and report how many samples’ predicted labels are changed after the removal, i.e., the quotient is negative.
Each point in the scatter plot corresponds to a sub-network, the x-axis is the score of the removed ﬁlter given by NeuroChains, and the y-axis is the proportion of samples with negative quotients.
The plot shows a strong linear correlation between the score of the removed ﬁlter and the degradation of faithfulness.
Since removing ﬁlters with high scores results in more samples with predicted class changing after pruning, the score given by NeuroChains measures the importance of ﬁlters in DNN inference.
Faithfulness of NeuroChains in terms of the quotient’s sign (RIGHT).
We evaluate the stability of NeuroChains using the nearest neighbours from the penultimate-layer representation space.
Because ReLU pattern does not provide an ideal metric to measure the distance of samples, even in the raw input space: (1) the number of ReLU linearity zones grows exponentially with the number of hidden nodes.
Most ReLU linearity zones are empty and do not contain any real sample; (2) For the few ReLU linearity zones that do contain samples, each only contains one sample and by large chance its neighboring linearity zones are empty, and this is true for most practical cases as empirical studies suggested.
So it is almost impossible to ﬁnd two samples sharing the same ReLU linearity zone or even close in their ReLU patterns of the ﬁrst layer; (3) For two ReLU linearity zones that are only different in one facet of their polyhedra (i.e., only one digit of their
ReLU patterns ﬂips), their corresponding linear models can still be very different (the linear model is an extreme case of sub-network).
Therefore, we speculate that samples close to each other in terms of their ReLU patterns do not share a sufﬁciently small sub-network preserving their original predictions.
That being said, we evaluated each NeuroChains extracted sub-network of VGG-19 using 20 samples randomly drawn from two classes on the K-th nearest neighbour (NN) of each sample by sorting the Hamming distance on their ReLU patterns.
The K-NN samples’ prediction cannot be well pre- served on the sub-networks, because the nearest neighbors in terms of ReLU patterns have very different semantic concepts or classes from the samples that the sub-networks are extracted for.
Hence, the local region of ReLU patterns is not a local region on the smooth data manifold.
To see this, in Figure 9, for each sample, we computed its L2 distance to the ReLU pattern K-NN sample’s penultimate-layer representation for K = 1 , 2 , · · · , 10 (the red curve reports mean ± std), and we compared them with the L2 distance to the K-NN in the penultimate-layer representation space (the blue curve reports mean ± std).
It shows that the ReLU pattern K-NN has a much larger L2 distance in the semantic space (i.e., penultimate-layer representation), so it is very different in concepts to the original sample.
Moreover, we show some examples of the ReLU pattern K-NN images and the penultimate-layer K-NN images for the sample in Figure 10, which show that ReLU pattern K-NN images are much less related to the original sample.
In Figure 11, we show two case studies of comparing SMOE generated heatmaps for the original network and the NeuroChains extracted sub-network.
We can see that the patterns extracted by the two networks are consistent and are all critical patterns for the class, e.g., the eyes and ﬁsts of kangaroos and the feet and face of the horse.
However, compared with the original network, these patterns are strengthened in much shallower layers of the sub-network, producing better interpretations.
This observation is also consistent with the result of analysis on adversarial attacks in Figure 5.
In Figure 12, we compare the capability of preserving the original neural network’s outputs between
NeuroChains and magnitude-based pruning (removing the ﬁlters whose output featuremaps’ average magnitude (L2 norm) over all considered samples is small).
In particular, under the same setting of each experiment in the paper, we prune the original VGG-19 and retain the ﬁlters with the largest featuremap magnitude in each layer, 180 in total (more than 157( mean ) ± 43( std ) ﬁlters for subnetworks extracted by NeuroChains), and we then ﬁne-tune the ﬁlters’ scores/weights as we did for
NeuroChains.
Figure 12 shows the histogram of the KL divergence between the original output class distribution and the one produced by the sub-networks.
For sub-networks generated by NeuroChain, the KL-divergence in most cases stays close to 0, while the output preserving capability of simple pruning is much worse.
[Figure]
g a r i t h m o f K L - d i v e r g e n c e
[Figure]
[Figure]
T
[Figure]
s t
f i d e li t y
[Figure]
D e c r e a s e o f p r o b a b ili t
y f o r t
h e o r
i g i n a l - D NN p r e d i c t
e d c l a ss
[Figure]
o
[Figure]
r i t h
[Figure]
f K L - d i v e r g e n c e
-networks (after removal) and the original sub-networks (before removal).
For both VGG-19 and Resnet-50, it shows that higher score, higher
-divergence.
5. M ORE D ETAILS ABOUT C ASE S TUDIES
On the sub-network’s architecture, we use “L0” to denote the corresponding convolution layer in
VGG-19 and “L0 1” to denote the ﬁrst ﬁlter from this layer.
For ResNet-50, we further use “L1B1” to denote the ﬁrst sub-block in the ﬁrst bottleneck block, “SC” for the shortcut connection and “C1” for the ﬁrst convolution layer in the sub-block.
The redder the node in the sub-network, the larger the scaling score, conversely, the bluer the node, the lower the score.
More case studies can be found in the Appendix.
In SMOE, Mundhenk et al.
propose to measure information at the end of every feature scale and then combined them into a saliency map.
We apply this technique to each layer of the sub-network since each layer may prefer different features.
In each of our case study, a featuremap-overlaid input image is shown for each layer and for the whole sub-network which is marked as ”Combine All”.
The visualization of each selected ﬁlter is achieved by maximizing its activation w.r.t.
the input.
Afterward, we shows the patterns that the ﬁlter aims to detect which is independent of the input image.
5. C ASE S TUDIES
[Figure]
[Figure]
[Figure]
[Figure]