Original: Distilling Critical Paths in Convolutional Neural Networks
Revised: Distilling Critical Paths in Convolutional Neural Networks

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Neural network compression and acceleration are widely demanded currently dueto the resource constraints on most deployment targets.
Revised: Neural network compression and acceleration are widely demanded currently due to the resource constraints on most deployment targets.

Original: In this paper, throughanalyzing the ﬁlter activation, gradients, and visualizing the ﬁlters’ functionality inconvolutional neural networks (CNNs), we show that the ﬁlters in higher layerslearn extremely task-speciﬁc features, which are exclusive for only a small subsetof the overall tasks, or even a single class.
Revised: In this paper, through analyzing the ﬁlter activation, gradients, and visualizing the ﬁlters’ functionality in convolutional neural networks (CNNs), we show that the ﬁlters in higher layers learn extremely task-speciﬁc features, which are exclusive for only a small subset of the overall tasks, or even a single class.

Original: Based on such ﬁndings, we revealthe critical paths of information ﬂow for different classes.
Revised: Based on such ﬁndings, we reveal the critical paths of information ﬂow for different classes.

Original: And by their intrinsicproperty of exclusiveness, we propose a critical path distillation method, which caneffectively customize the convolutional neural networks to small ones with muchsmaller model size and less computation.
Revised: And by their intrinsic property of exclusiveness, we propose a critical path distillation method, which can effectively customize the convolutional neural networks to small ones with much smaller model size and less computation.

Original:  Introduction
Revised:  Introduction

Original: Recently researchers found that the neurons in higher layers of neural networks are usually specializedto encode information relevant to a subset of classiﬁcation targets [1, 2].
Revised: Recently researchers found that the neurons in higher layers of neural networks are usually specialized to encode information relevant to a subset of classiﬁcation targets [1, 2].

Original: For example, by ablatingcertain neurons in higher layers, signiﬁcant accuracy degradation of speciﬁc classes can be observed.
Revised: For example, by ablating certain neurons in higher layers, signiﬁcant accuracy degradation of speciﬁc classes can be observed.

Original: Based on such intuition, in this paper, we interpret CNN neuron’s functionality by activation maximization visualization (AM) [3], and quantifying their contributions to speciﬁc classes throughactivation and gradients analysis.
Revised: Based on such intuition, in this paper, we interpret CNN neuron’s functionality by activation maximization visualization (AM) [3], and quantifying their contributions to speciﬁc classes through activation and gradients analysis.

Original: With these three analysis approaches, we not only cross-verify theneuron differentiation process across multiple layers, but also reveal that the class relevant informationis signiﬁcantly concentrated in a few neurons.
Revised: With these three analysis approaches, we not only cross-verify the neuron differentiation process across multiple layers, but also reveal that the class relevant information is signiﬁcantly concentrated in a few neurons.

Original: In other words, the per-class information ﬂow isdirected by a “critical path” formed by a few class-speciﬁc neurons with signiﬁcant contributions.
Revised: In other words, the per-class information ﬂow is directed by a “critical path” formed by a few class-speciﬁc neurons with signiﬁcant contributions.

Original: By identifying critical paths, we ﬁnd that the critical paths for different classes are co-existing butalmost exclusive from each other.
Revised: By identifying critical paths, we ﬁnd that the critical paths for different classes are co-existing but almost exclusive from each other.

Original: This property of exclusiveness provides the chance to distill thecritical paths and further decouple the CNN for individual class decision tasks.
Revised: This property of exclusiveness provides the chance to distill the critical paths and further decouple the CNN for individual class decision tasks.

Original: Therefore, we proposeour critical path distillation method, which could effectively select out the critical neurons in highlayers for a given task, and eliminate the unrelated ones.
Revised: Therefore, we propose our critical path distillation method, which could effectively select out the critical neurons in high layers for a given task, and eliminate the unrelated ones.

Original: The distilled neural network contains onlycritical neurons for single class and perform one-vs-all tasks.
Revised: The distilled neural network contains only critical neurons for single class and perform one-vs-all tasks.

Original: Compared to conventional compressionmethods like ﬁlter pruning [4], our method proposes a new perspective on per-class information ﬂowand thus could aggressively remove more ﬁlters without any retraining.
Revised: Compared to conventional compression methods like ﬁlter pruning [4], our method proposes a new perspective on per-class information ﬂow and thus could aggressively remove more ﬁlters without any retraining.

Original: Preliminary experimentsshow that the distilled neural network can have 90% fewer neurons in higher layers compared tooriginal one, and achieves satisfying performance with 98.6% average True Positive (TP) rate and11.0% False Positive (FP) rate in all one-vs-all tasks.
Revised: Preliminary experiments show that the distilled neural network can have 90% fewer neurons in higher layers compared to original one, and achieves satisfying performance with 98.6% average True Positive (TP) rate and 11.0% False Positive (FP) rate in all one-vs-all tasks.

Original:  Critical Path in Convolutional Neural Networks
Revised:  Critical Path in Convolutional Neural Networks

Original: 2.1 Overview of Critical Path Fig.
Revised: 2.1 Overview of Critical Path Fig.

Original: 1 (a) shows a conceptual critical path for the “bird” class(CIFAR10/VGG-16) [5].
Revised: 1 (a) shows a conceptual critical path for the “bird” class (CIFAR10/VGG-16) [5].

Original: The orange color in the layers denote the reserved neurons from the criticalpath, while all the blues ones are removed from the original network due to their insigniﬁcance to the“bird” class.
Revised: The orange color in the layers denote the reserved neurons from the critical path, while all the blues ones are removed from the original network due to their insigniﬁcance to the “bird” class.

Original: Our critical path contains two clear main components: full basic feature layers (low)and distilled class-speciﬁc neurons in high layers.
Revised: Our critical path contains two clear main components: full basic feature layers (low) and distilled class-speciﬁc neurons in high layers.

Original: In lower layers, all ﬁlters are reserved since they 
Revised: In lower layers, all ﬁlters are reserved since they usually learn basic and class-agnostic features, e.g. colors, edges [6, 7]. However, in higher layers,

Original: [Figure]
Revised: [Figure]

Original: usually learn basic and class-agnostic features, e.g. colors, edges [6, 7].

Original: However, in higher layers,ﬁlters are gradually differentiated to recognize different classes, as shown by their AM patterns.
Revised: ﬁlters are gradually differentiated to recognize different classes, as shown by their AM patterns.

Original: Therefore, only the class-related neurons are reserved in the critical path.
Revised: Therefore, only the class-related neurons are reserved in the critical path.

Original: Fig.
Revised: Fig.

Original: 1 (b) also veriﬁesthis by showing the trend of standard deviation (STD) of ﬁlters’ per-class mean activation from lowto high layers.
Revised: 1 (b) also veriﬁes this by showing the trend of standard deviation (STD) of ﬁlters’ per-class mean activation from low to high layers.

Original: High layer ﬁlters averagely have the larger standard deviation, which means theiractivation for different class inputs is highly non-uniform.
Revised: High layer ﬁlters averagely have the larger standard deviation, which means their activation for different class inputs is highly non-uniform.

Original: In our experiments, we ﬁnd that mostneurons in high layers are mostly activated by a single class, which therefore forms very exclusivecritical paths for different classes.
Revised: In our experiments, we ﬁnd that most neurons in high layers are mostly activated by a single class, which therefore forms very exclusive critical paths for different classes.

Original: Furthermore, the critical path for a single class can be decoupledfrom the original neural network and acts as a much smaller but task-speciﬁc neural network, whichwe will show in the later section.
Revised: Furthermore, the critical path for a single class can be decoupled from the original neural network and acts as a much smaller but task-speciﬁc neural network, which we will show in the later section.

Original: 2.2 Critical Path Interpretation and Veriﬁcation through Neuron Contribution Analysis
Revised: 2.2 Critical Path Interpretation and Veriﬁcation through Neuron Contribution Analysis

Original: Theidentiﬁcation of critical path is based on the neuron contribution signiﬁcance to speciﬁc classes.
Revised: The identiﬁcation of critical path is based on the neuron contribution signiﬁcance to speciﬁc classes.

Original: In this work, we include three approaches to analyze the neurons’ functionality, and show that thecross-veriﬁcation of the three approaches testiﬁes the proposed critical path.
Revised: In this work, we include three approaches to analyze the neurons’ functionality, and show that the cross-veriﬁcation of the three approaches testiﬁes the proposed critical path.

Original: Suppose a CNN’s logit layer output could be formulated as Z ( x ) with n logits, which is correspondingto n classes.
Revised: Suppose a CNN’s logit layer output could be formulated as Z ( x ) with n logits, which is corresponding to n classes.

Original: Given a speciﬁc layer l with I l ﬁlters, the n th logit output Z n is:
Revised: Given a speciﬁc layer l with I l ﬁlters, the n th logit output Z n is:

Original: [Equation]
Revised: [Equation]

Original: where W and b are layer weights and biases.
Revised: where W and b are layer weights and biases.

Original: Then we can evaluate neurons’ contribution by:(1) Mean Absolute Activation For a ﬁlter F i in layer l , its activation can be denoted by A li .
Revised: Then we can evaluate neurons’ contribution by: (1) Mean Absolute Activation For a ﬁlter F i in layer l , its activation can be denoted by A li .

Original: Theﬁrst method we use is the mean absolute activation A li with regard to different classes inputs.
Revised: The ﬁrst method we use is the mean absolute activation A li with regard to different classes inputs.

Original: When
Revised: When

Original: W l +

Revised: W l +

Original: i is non-zero, larger A li means more contribution to the ﬁnal Z n , as shown in Eq.
Revised: i is non-zero, larger A li means more contribution to the ﬁnal Z n , as shown in Eq.

Original: Therefore,the class C which produces the largest mean absolute activation could be considered as its mostimportant functionality or task since these images could mostly activate the ﬁlter.(2) Contribution Index
Revised: Therefore, the class C which produces the largest mean absolute activation could be considered as its most important functionality or task since these images could mostly activate the ﬁlter.

Original:  Meanwhile, we could also use ﬁrst-order Taylor Approximation [8] toapproximate Z n with regard to ﬁlter F i ’s activation A li :
Revised: (2) Contribution Index Meanwhile, we could also use ﬁrst-order Taylor Approximation [8] to approximate Z n with regard to ﬁlter F i ’s activation A li :

Original: [Equation]
Revised: [Equation]

Original: Here, we use the (cid:96) 1 of the coefﬁcient matrix ∂Z n ( A li ) /∂A
Revised: Here, we use the (cid:96) 1 of the coefﬁcient matrix ∂Z n ( A li ) /∂A

Original: li as the contribution index of i th ﬁlter’soutput to the speciﬁc class n : Larger value means a small δ change of A li will be ampliﬁed to causelarger inﬂuence of Z n .
Revised: li as the contribution index of i th ﬁlter’s output to the speciﬁc class n : Larger value means a small δ change of A li will be ampliﬁed to cause larger inﬂuence of Z n .

Original: One ﬁlter has different contribution indexes to different classes.
Revised: One ﬁlter has different contribution indexes to different classes.

Original: Therefore,the class C with largest contribution index could be also considered as its most important function.(3) AM Visualization Lastly, we introduce the Activation Maximization (AM) Visualization method[3].
Revised: Therefore, the class C with largest contribution index could be also considered as its most important function. (3) AM Visualization Lastly, we introduce the Activation Maximization (AM) Visualization method [3].

Original: AM is a well-established method to interpret a single ﬁlter’s activation preference.
Revised: AM is a well-established method to interpret a single ﬁlter’s activation preference.

Original: It adoptsgradient ascent algorithm to maximize a neuron’s activation with input X :
Revised: It adopts gradient ascent algorithm to maximize a neuron’s activation with input X :

Original: [Equation]
Revised: [Equation]

Original: where η is the step size, and the ﬁnal P is the visualized pattern of ﬁlter i in layer l .
Revised: where η is the step size, and the ﬁnal P is the visualized pattern of ﬁlter i in layer l .

Original: The craftedpattern P could also maximize its activation and thus could visually interpret the ﬁlter’s functionality.
Revised: The crafted pattern P could also maximize its activation and thus could visually interpret the ﬁlter’s functionality.

Original: 2.3 Critical Path Veriﬁcation
Revised: 2.3 Critical Path Veriﬁcation

Original: Next, we combine the three different metrics together to cross-verifythe ﬁlter functionality analysis and demonstrate the existence of the critical path for the single class.
Revised: Next, we combine the three different metrics together to cross-verify the ﬁlter functionality analysis and demonstrate the existence of the critical path for the single class.

Original: Neuron# Neuron#
Revised: Neuron# Neuron#

Original: Neuron# (a) Class-3:
Revised: Neuron# (a) Class-3:

Original: Cat 3.Visualization 1.MeanActivation 2.Contribution Index 纵坐标轴 ？ 纵坐标轴 ？ 纵坐标轴 ？
Revised: Cat 3.Visualization 1.MeanActivation 2.Contribution Index 纵坐标轴 ？ 纵坐标轴 ？ 纵坐标轴 ？

Original: Neuron# Neuron#
Revised: Neuron# Neuron#

Original: Neuron# (b) Class-5: Dog 3.
Revised: Neuron# (b) Class-5: Dog 3.

Original: Visualization
Revised: Visualization

Original: Classes No. 04.
Revised: Classes No. 04.

Original: Contribution Preferences
Revised: Contribution Preferences

Original: Cat
Revised: Cat

Original: Classes No. 04.
Revised: Classes No. 04.

Original: Contribution Preferences
Revised: Contribution Preferences

Original: Dog
Revised: Dog

Original: Neuron# Neuron#
Revised: Neuron# Neuron#

Original: Neuron# Neuron#
Revised: Neuron# Neuron#

Original: Neuron#
Revised: Neuron#

Original: Neuron# 1.MeanActivation 2.Contribution Index
Revised: Neuron# 1.MeanActivation 2.Contribution Index

Original: [Table]
Revised: [Table]

Original: The convolutional neural network we used here is VGG16 trained on CIFAR10 same as [4].
Revised: The convolutional neural network we used here is VGG16 trained on CIFAR10 same as [4].

Original: Wechoose the 10 th layer Conv5_1 and class 3 and 5 (cat and dog) as the example (Other classes alsoshare the same properties), and results are shown in Fig.
Revised: We choose the 10 th layer Conv5_1 and class 3 and 5 (cat and dog) as the example (Other classes also share the same properties), and results are shown in Fig.

Original: 2 (a) and (b).
Revised: 2 (a) and (b).

Original: From the mean absolute activation map and contribution index map of Fig.
Revised: From the mean absolute activation map and contribution index map of Fig.

Original: 2 (a), we could clearly seethat they point out similar ﬁlter groups with deeper colors: The ﬁlters with the largest activation alsohave the maximal contribution indexes among all the ﬁlters.
Revised: 2 (a), we could clearly see that they point out similar ﬁlter groups with deeper colors: The ﬁlters with the largest activation also have the maximal contribution indexes among all the ﬁlters.

Original: According to Taylor expansion in Eq. 2,both larger activation and coefﬁcients will lead to more contribution to the ﬁnal decision of this class.
Revised: According to Taylor expansion in Eq. 2, both larger activation and coefﬁcients will lead to more contribution to the ﬁnal decision of this class.

Original: Thus we call these ﬁlters the critical neurons, which form the critical path of this class.
Revised: Thus we call these ﬁlters the critical neurons, which form the critical path of this class.

Original: Here wealso visualize the top three ﬁlters with the largest product of mean activation and contribution index,which show exactly the cat patterns.
Revised: Here we also visualize the top three ﬁlters with the largest product of mean activation and contribution index, which show exactly the cat patterns.

Original: We also show the three ﬁlters’ contribution indexes to all classes.
Revised: We also show the three ﬁlters’ contribution indexes to all classes.

Original: The biggest contribution indexes of three ﬁlters all lie in the cat class and are much larger than otherclasses.
Revised: The biggest contribution indexes of three ﬁlters all lie in the cat class and are much larger than other classes.

Original: For generality, we also show the dog class’s critical path analysis in Fig.
Revised: For generality, we also show the dog class’s critical path analysis in Fig.

Original: 2 (b), where allsimilar conclusions could be drawn.
Revised: 2 (b), where all similar conclusions could be drawn.

Original: Next, we will introduce our critical path distillation method.
Revised: Next, we will introduce our critical path distillation method.

Original:  Critical Path Distillation
Revised:  Critical Path Distillation

Original: In this section, we will introduce how to distill the critical path of a single class.
Revised: In this section, we will introduce how to distill the critical path of a single class.

Original: First, we need tochoose layers in which ﬁlters have formed speciﬁc class functionality.
Revised: First, we need to choose layers in which ﬁlters have formed speciﬁc class functionality.

Original: The higher layers we choose,the critical paths distillation could be easier but the model size compression effect may be lower.
Revised: The higher layers we choose, the critical paths distillation could be easier but the model size compression effect may be lower.

Original: According to the experiments, we empirically choose the last six convolutional layers for distillation.
Revised: According to the experiments, we empirically choose the last six convolutional layers for distillation.

Original: Given a speciﬁc class, we ﬁrst compute the corresponding mean activation map and contributionindex map of all six layers.
Revised: Given a speciﬁc class, we ﬁrst compute the corresponding mean activation map and contribution index map of all six layers.

Original: Maps of their product are also calculated to combine these two factorsto choose reserved ﬁlters.
Revised: Maps of their product are also calculated to combine these two factors to choose reserved ﬁlters.

Original: The reserved ratio controls the “width” of the critical path, and trade-offthe accuracy and compression rate.
Revised: The reserved ratio controls the “width” of the critical path, and trade-off the accuracy and compression rate.

Original: For example, 10% reserved ratio means we only keep 52 ﬁltersour of 512 in all the chosen layers.
Revised: For example, 10% reserved ratio means we only keep 52 ﬁlters our of 512 in all the chosen layers.

Original: A ratio below can cause catastrophic accuracy drop.
Revised: A ratio below can cause catastrophic accuracy drop.

Original: Anotherimportant thing is since we modify the neural network structure, the previous ten biases of logit layerare not suitable.
Revised: Another important thing is since we modify the neural network structure, the previous ten biases of logit layer are not suitable.

Original: Therefore, we set the biases in logit layer to zeros.
Revised: Therefore, we set the biases in logit layer to zeros.

Original: Without any retraining, thedistilled model preserves the ability well to respond to the chosen class and reject all other classes, i.e.one-vs-all tasks.
Revised: Without any retraining, the distilled model preserves the ability well to respond to the chosen class and reject all other classes, i.e. one-vs-all tasks.

Original: We use TP (True Positive) rate and FP (False Positive) rate to evaluate our criticalpath distillation performance on ten one-vs-all tasks, respectively.
Revised: We use TP (True Positive) rate and FP (False Positive) rate to evaluate our critical path distillation performance on ten one-vs-all tasks, respectively.

Original: The ﬁnal results are shown in
Revised: The ﬁnal results are shown in

Original: Table.1.
Revised: Table.1.

Original: And the ﬁnal model size is compressed to 11.4 MB compared to original 59.0 MB.
Revised: And the ﬁnal model size is compressed to 11.4 MB compared to original 59.0 MB.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we propose a critical path distillation method in convolutional neural networks.
Revised: In this work, we propose a critical path distillation method in convolutional neural networks.

Original: Throughanalyzing the activation, gradients and visualization patterns, we demonstrate that higher layerﬁlters always learn class-speciﬁc features and therefore could be well distilled to customize classspeciﬁc neural networks.
Revised: Through analyzing the activation, gradients and visualization patterns, we demonstrate that higher layer ﬁlters always learn class-speciﬁc features and therefore could be well distilled to customize classspeciﬁc neural networks.

Original: Our preliminary experiments show that the customized class-speciﬁc neuralnetworks by critical path distillation could achieve average 98.6% TP rate and 11.0% FP rate inone-vs-all tasks, as well as have much smaller size and computation requirements.
Revised: Our preliminary experiments show that the customized class-speciﬁc neural networks by critical path distillation could achieve average 98.6% TP rate and 11.0% FP rate in one-vs-all tasks, as well as have much smaller size and computation requirements.

Original: In the future, wewill generalize our method to multiple classes and on the large-scale dataset, ImageNet [9].
Revised: In the future, we will generalize our method to multiple classes and on the large-scale dataset, ImageNet [9].