R ETHINKING THE P RUNING C RITERIA FOR C ONVOLUTIONAL N EURAL N ETWORK
Anonymous authors
Paper under double-blind review
A BSTRACT
Channel pruning is a popular technique for compressing convolutional neural networks (CNNs), and various pruning criteria have been proposed to remove the redundant ﬁlters of CNNs.
From our comprehensive experiments, we found some blind spots on pruning criteria: (1) Similarity: There are some strong similarities among several primary pruning criteria that are widely cited and compared.
According to these criteria, the ranks of ﬁlters’ importance in a convolutional layer are almost the same, resulting in similar pruned structures.
(2) Applicability: For a large network (each convolutional layer has a large number of ﬁlters), some criteria can not distinguish the network redundancy well from their measured ﬁlters’ importance .
In this paper, we theoretically validate these two ﬁndings with our assumption that the well-trained convolutional ﬁlters in each layer approximately follow a Gaussian-alike distribution.
This assumption is veriﬁed through systematic and extensive statistical tests.
I NTRODUCTION
Pruning (LeCun et al., 1990; Hassibi & Stork, 1993; Han et al., 2015;
He et al., 2019) a trained neural network is commonly seen in network compression.
In particular, for neural networks with convolutional ﬁlters, channel pruning refers to the pruning of the ﬁlters in the convolutional layers.
There are several critical factors for channel pruning.
Procedures .
One-shot method (Li et al., 2016):
Train a network from scratch; Use a certain criterion to calculate ﬁlters’ importance , and prune the ﬁlters which have small importance ; After additional training, the pruned network can recover its accuracy to some extent.
Iterative method (He et al., 2018; Frankle & Carbin, 2019):
Unlike One-shot methods, they prune and ﬁne-tune a network alternately.
Criteria .
The ﬁlters’ importance can be deﬁnded by a given criterion.
From different ideas, many types of pruning criteria have been proposed, such as Norm-based (Li et al., 2016), Activation-based (Hu et al., 2016;
Luo & Wu, 2017), Importance-based (Molchanov et al., 2016; 2019a), BN-based (Liu et al., 2017b) and so on.
Strategy .
Layer-wise pruning:
In each layer, we can sort and prune the ﬁlters, which have small importance measured by a given criterion.
Global pruning: Different from layer-wise pruning, global pruning sort the ﬁlters from all the layers through their importance and prune them.
[Table]
As one of the simplest and most effective channel pruning criteria, (cid:96) 1 pruning (Li et al., 2016) is widely used in the industry.
The core idea of this criterion is to sort the (cid:96) 1 norm of ﬁlters in one layer and then prune the ﬁlters, which have a small (cid:96) 1 norm.
Similarly, there is (cid:96) 2 pruning (Frankle &
Carbin, 2019; He et al., 2018).
Through the study of the distribution of norm, He et al.
(2019) demonstrates that these criteria should satisfy two conditions: (1) the variance of the norm of the ﬁlters can- not be too small; (2) the minimum norm of the ﬁlters should be small enough.
Since these two conditions do not always hold, a new criterion considering the relative importance of the ﬁlters ( (cid:96) 1 and (cid:96) 2 norm can be seen as one of algorithm which uses absolute importance of ﬁlters) is proposed (He et al., 2019).
Since this criterion uses the Fermat point ( i.e. , geometric median (Cohen et al., 2016)), we call this method Fermat .
However, due to the high calculation cost of Fermat point, He et al.
(2019) relaxed it and then got another criterion GM method.
Let F ij ∈ R N i × k × k represents the j th ﬁlter of the i th convolutional layer, where N i is the number of input channels for i th layer and k denotes the kernel size of the convolutional ﬁlter.
In i th layer, there are N i +1 ﬁlters.
The details of these criteria are shown in Table 2.
F denotes the Fermat point of F ij in Euclidean space.
These four pruning criteria are called Norm-based pruning in this paper as they include norm in their design.
[Table]
In previous works (Luo et al., 2017; Han et al., 2015; Ding et al., 2019; Dong et al., 2017; Renda et al., 2020), including the criteria mentioned above, they usually focused on (a) How much the model was compressed; (b) How much performance was restored; (c) The inference efﬁciency of the pruned network and (d) The cost of ﬁnding the pruned network.
However, there is little work to discuss two blind spots about the pruning criteria: (1) Similarity: What are the actual differences among these previous pruning criteria?
Using
VGG16 and ResNet18 on ImageNet, we show the ranks of ﬁlters’ importance under different criteria in Table 1.
It is easy to ﬁnd that they have almost the same sequence, leading to similar pruned structures.
In this situation, the criteria used absolute importance of ﬁlters ( (cid:96) 1 , (cid:96) 2 ) and the criteria used relative importance of ﬁlters ( Fermat , GM ) may not be signiﬁcantly different.
(2) Applicability: What is the Applicability of these pruning criteria to prune the CNNs?
There is a toy example w.r.t.
(cid:96) 2 criterion.
If the (cid:96) 2 norm (regarded as importance ) of the ﬁlters in one layer are 0.9, 0.8, 0.4 and 0.01, according to smaller-norm-less-informative assumption (Ye et al., 2018), it’s easy to know that we should prune the last ﬁlter.
However, if the norm are close, like 0.91, 0.92, 0.93, 0.92, it is hard to know which ﬁlter should be pruned even though the ﬁrst one is the smallest.
As shown in Fig.
1, taking Wide ResNet28-10 (WRN) as an example, this is a real and existing problem for pruning a network with the (cid:96) 2 criterion.
Some similar research methods and opinions on pruning criteria are mentioned in He et al.
(2019) and Molchanov et al.
(2019a).
We make further analysis and observation of them in this paper.
Since the criteria in Table 2 are widely cited and compared (Liu et al., 2020b; Li et al., 2020b;
He et al., 2020;
Liu et al., 2020a; Li et al., 2020a), therefore it’s important to study them for the above two issues.
In order to rigorously study them: First, we come up with an assumption about the distribution of the parameters of the convolutional ﬁlters, called Convolution Weight Distribution Assumption (CWDA), with systematic and comprehensive statistical tests (Appendix P) in Section 2.
Next, in Section 3 and Section 4, we theoretically verify the problem about Similarity and Applicability for some Norm-based criteria in layer-wise pruning.
Last but not least, in Section 5, we discuss more issues including: (1) the conditions for CWDA to be satisﬁed, (2) the Similarity and Applicability when we use other types of pruning criteria, (3) and another pruning strategy, the global pruning.
Contribution .
(1) We propose and verify an assumption called CWDA, which reveals that the well-trained convolutional ﬁlters approximately follow a Gaussian-alike distribution.
Under CWDA, we strictly validate these issues of Norm-based criteria in layer-wise pruning.
(3) Under CWDA, some Norm-based criteria using the global pruning strategy have the problem about the inconsistent magnitude of importance between different layers, which explains the phenomenon that they sometimes cut off the network.
The estimations using CWDA almost coincides to the actual results obtained from the real network, which also demonstrates the effectiveness of the
CWDA.
W EIGHT D ISTRIBUTION -A SSUMPTION
In this section, to study the Similarity and the Applicability problem between the pruning criteria shown in Table 2, we propose and verify an assumption about the distribution of the parameters of convolutional ﬁlters.
(Convolution Weight Distribution Assumption) Let F ij ∈ R N i × k × k be the j th well-trained ﬁlter of the i th convolutional layer.
In general 1 , in i th layer, F ij , j = 1 , 2 , ..., N i +1 are i.i.d and follow a
Gaussian-alike distribution:
[Equation]
where c is a constant and Σ block = diag( K 1 , K 2 , ..., K N i ) is a block diagonal matrix.
The values of the off-diagonal elements are 0 and K l ∈ R k 2 × k 2 , l = 1 , 2 , ..., N i .
This assumption is based on the observation shown in the Fig.
Let F ∈ R
( N i × k × k )
× N i +1 denotes all the parameters in i th layer and we use correlation matrix FF T to estimate the shape of Σ block .
Taking the last convolutional layers of ResNet18 and VGG16 trained on ImageNet as an example, we ﬁnd that FF T is a block diagonal matrix.
Speciﬁcally, each block is a k 2 × k 2 matrix and the off-diagonal elements are close to 0.
For j th ﬁlter F ij ∈ R N i × k × k
in i th layer as shown in Fig. 2(e), this phenomenon reveals that the parameters in the same channel of F ij tend to be linearly correlated, and the parameters of any two different channels (yellow and green channel in Fig. 2(e)) in F ij only have a low linear correlation.
Since the kernel size k usually is a small constant ( or 3) and N i (cid:29) k , the block diagonal matrix Σ block is approximately equal to a diagonal matrix.
Therefore, for the convenience of analysis, we relax the assumption, i.e. , F ij approximately follows a Gaussian-alike distribution:
[Equation]
where I N i × k × k is an identity matrix.
The statistical tests in Section 2.1 and the estimation (as shown in Fig. 10) by CWDA show the reasonability of this relaxation.
In the remaining sections, we use Eq.2 to represent CWDA unless otherwise speciﬁed.
In Fig.
3, taking VGG16 and ResNet18 on
ImageNet dataset as examples, we visualize the CWDA through the distribution of the convolutional ﬁlters.
2. S TATISTICAL TEST
In fact, CWDA is not easy to be veriﬁed.
For example, for ResNet164 (On Cifar100), the number of ﬁlters in the ﬁrst stage is only 16, which is too small to be used to estimate the statistics accurately.
More other objective reasons are shown in Section 5.1.
As these problems, we consider verifying three necessary conditions of CWDA: (1) Gaussian ( i.e. , to verify whether F ij approximately follows a Gaussian-alike distribution); (2) Standard Deviation ( i.e. , to verify whether the standard deviation of each ﬁlter in any layers is close to a constant c );
(3) Mean ( i.e. , to verify whether the mean of F ij is close to 0).
In Table 3, to illustrate that CWDA holds universally, we consider a variety of factors, such as network structure, optimizer, regularization 2 , initialization, dataset, training strategy, and other tasks in computer vision ( e.g ., semantic segmentation, detection, image matting and so on).
The details of these statistical tests are shown in Appendix P.
[Table]
[Table]
S IMILARITY
In this section, we further verify the observation that the pruning criteria in Table 2 are highly similar from two perspectives.
From an experimental point of view, we use more experiments about image classiﬁcation to investigate the similarities.
From a theoretical perspective, we rigorously show the similarities of the criteria in Table 2 in layer-wise pruning under CWDA.
Empirical Analysis .
(1) In Fig.
4, we show the test accuracy of the ResNet56 after pruning and ﬁne-tuning under using different pruning ratios and datasets.
The test accuracy curves of different pruning criteria at different stages are very close under different pruning ratios.
This phenomenon implies that those pruned networks using different pruning criteria are very similar, and there are strong similarities among these pruning criteria.
The experiments about other commonly used pruning ratio can be found in Appendix N. (2) In Fig.
5, we show the Spearman’s rank correlation coefﬁcient (Sp) 3 between different pruning criteria.
The Sp in most convolutional layers are more than 0.9, which means the network structures are almost the same after pruning.
Note that the Sp in transition layer ( i.e. , the layer where the dimensions of the ﬁlter change, like the layer between stage 1 and stage 2 of ResNet164.
The number of dimensions in stage 1 and stage 2 are 16 andrespectively.)
are relatively small.
It is interesting but will not greatly impact the structural similarity of the whole pruned network.
The reason for this phenomenon may be that the layers in these areas are sensitive.
The similar observations are shown in Fig.
2 in Ding et al.
(2019), and Fig.
6 and
Fig.
10 in Li et al.
Theoretical Analysis .
After the veriﬁcation by experiments, the similarities via using layer-wise pruning among the criteria in Table 2 are proved theoretically in this section.
Let C 1 and C 2 be two pruning criteria to calculate the importance for all convolutional ﬁlters of one layer.
If they can produce the similar ranks of importance , we deﬁne C 1 and C 2 are approximately monotonic to each other and use C 1 ∼ =
C 2 to represent this relationship.
In Section 3, we use the Sp to describe this relationship but it’s hard to be analyzed theoretically.
Therefore, we consider about a stronger condition.
Let X =
( x 1 , x 2 , ..., x k ) and Y =
( y 1 , y 2 , ..., y k ) be two given sequences.
we ﬁrst normalize their magnitude, i.e. , let (cid:98)
X
= X / E ( X ) and (cid:98) Y
=
Y / E ( Y ) .
This operation does not change the ranking sequence of the elements of X and Y , because E ( X ) and E ( Y ) are constants,
i.e. , (cid:98) X ∼ = (cid:98) Y ⇔ X
∼ =
Y .
After that, if both Var ( (cid:98) X / (cid:98) Y ) and Var ( (cid:98) Y / (cid:98) X ) are small enough, then the
Sp between X and Y is close to 1, where (cid:98) X / (cid:98) Y =
( (cid:99) x 1 / (cid:98) y 1 , .., (cid:99) x k / (cid:98) y k ) .
The reason is that in these situations, the ratio (cid:98) X / (cid:98) Y and (cid:98) Y / (cid:98) X will be close to two constants a, b .
Note that, for any 1 ≤ i ≤ k , (cid:98) x i ≈ a · (cid:98)
y i and (cid:98)
y i ≈
b · (cid:98) x i .
Then, ab ≈ 1 and a, b (cid:54) = 0 .
Therefore, there exists an approximately monotonic mapping from (cid:98)
y i to (cid:98) x
i (linear function) and it makes the Sp between X and Y close to 1.
Theorem 1.
Let X ∼ N ( 0 , c 2 · I n ) , and ( C 1 , C 2 ) is one of ( (cid:96) 1 , (cid:96) 2 ) , ( (cid:96) 1 , Fermat ) or ( Fermat , GM ) , we have
[Equation]
where (cid:98) C 1 ( X ) denotes C 1 ( X ) /
E ( C 1 ( X )) and (cid:98) C 2 ( X ) denotes C 2 ( X )
/ E ( C 2 ( X )) .
B ( n ) denotes the upper bound of left-hand side and when n is large enough, B ( n ) → 0 .
Proof.
(See Appendix G).
For i th convolutional layer of a neural network, since F ij , j = 1 , 2 , ...
N i +1 , meet CWDA and the dimensions of F ij are generally large, we can obtain (cid:96) 1 ∼ = (cid:96) 2 , (cid:96) 2 ∼ =
Fermat and Fermat ∼ = GM according to Theorem 1.
Therefore, we have (cid:96) 1 ∼ = (cid:96) 2 ∼ =
Fermat ∼ = GM , which veriﬁes the strong similarities among the criteria shown in Table 2.
A PPLICABILITY
In this section, we analyze the Applicability problem of the Norm-based criteria when we use these criteria to prune a large network (each convolutional layer has a large number of ﬁlters).
In Fig.
1, we can ﬁnd that (cid:96) 2 can not distinguish the redundancy of Wide ResNet28-10 (regarded as a large network) well from their measured ﬁlters’ importance , because their importance are very close (the distribution looks sharp).
This problem is related to the variance of importance .
He et al.
(2019) argue that a small norm deviation (the values of variance of importance are small) makes it difﬁcult to ﬁnd an appropriate threshold to select ﬁlters to prune.
However, even if the values of the variance are large, it may still have the Applicability problems.
This is because the magnitude of these importance may be much greater than the values of the variance, where we can use the mean of importance to represent their magnitude.
We believe that the following two situations may cause
Applicability problem: for the ﬁlters F i in i th convolutional layer,
(1) If the mean E ( F i ) = 0 and the variance Var ( F i ) is close to 0; (2) If E ( F i ) (cid:54) =
and Var ( F i ) / E ( F i ) is close to 0.
For a network with large number of convolutional ﬁlters, it’s easy to know that the dimensions of the ﬁlters are also large enough.
For (cid:96) 2 pruning, according to CWDA (the proof in Appendix L), we can obtain that the mean of (cid:96) 2 ( F i ) is √ 2 σ i · Γ( k i +12 ) /
Γ( k i 2 ) (cid:54) = 0 , where σ i and k i are the standard deviation and dimension of the parameters in i th layer, respectively.
And Var ( F i ) / E ( F i ) → 0 when k i is large enough.
From these reasons, the importance measured by (cid:96) 2 norm tends to be identical, i.e. , it’s hard to distinguish the network redundancy well in this situation.
Moreover, from the proof in Appendix H, we know that the Fermat point F of F i and the origin 0 approximately coincide.
According to Table 2, || F − F i ||
≈ ||
− F i ||
=
||
F i || 2 .
Therefore, the importance of Fermat and (cid:96) 2 are almost equivalent when CWDA holds.
Hence, a similar conclusion can be obtained for
Fermat criterion.
Intuitively, the (cid:96) 1 criterion should have the same Applicability problem as the (cid:96) 2 criterion.
However, from the proof in Appendix L the[-mean of (cid:96) 
-]
Var ( F i ) / E ( F i ) tends to be a none-zero constant with respect to σ i .
In other words, (cid:96) 1 criterion does not necessarily have Applicability problems unless σ i is small enough.
Except for the Norm-based criteria, we analyze another type pf pruning criteria called RePr (Prakash et al., 2019).
This criterion considers the orthogonality among the ﬁlters in one layer.
Based on the proof in Appendix L and the fact that the importance measured by (cid:96) 2 norm tend to be identical, we also know that this criterion cannot prune the network well when the number of ﬁlters is too large.
Moreover, in Section 5.2, we study the Applicability problem for more different types of pruning criteria from a numerical perspective.
D ISCUSSION 5. W HY CWDA SOMETIMES DOES NOT HOLD ?
CWDA may not always hold.
As shown in Appendix P, a small number of convolutional ﬁlters may not pass the statistical tests.
In this section, we try to analyze this phenomenon.
(1) Need to be trained well enough .
The distribution of parameters can only be discussed when the network is trained well.
If the network does not converge, it is easy to construct a scenario which does not satisfy CWDA, e.g. , in Fig.
16 (Appendix D), we train a network with uniform initialization.
Although the distribution of parameters converges with the increase of epoch more and more to a normal distribution, when only a few epochs are trained, the distribution of parameters is closer to a uniform distribution.
At this time, the distribution obviously does not satisfy CWDA.
Moreover, if a network is not trained well, e.g. , its parameters converge to bad local minima, there may be many unforeseen circumstances causing CWDA to be unsatisﬁed.
In order to eliminate these factors, the network should be trained when studying the parameter distribution.
The number of ﬁlters is insufﬁcient.
In Appendix P, the layers that can not pass the statistical tests are almost those whose position is in the front of the network.
A common feature of these layers is that they have a few ﬁlters, which may not estimate statistics well.
In Fig.
5, due to the sensibility of layers, the Sp in the transition layer are relatively small.
Taking the second convolutional layer ( ﬁlters) in VGG16 on CIFAR10 as an example, we ﬁnd that increasing the number of ﬁlters could alleviate this sensitivity.
As shown in Fig.
6, we change the number of ﬁlters in this layer from 64 to 128 or 256.
After that, the Sp increases signiﬁcantly, and it suggests that enough number of ﬁlters are essential.
(3) The dimensions of the ﬁlter are not large enough.
The dimension of a ﬁlter in the i th layer is closely related to the number of ﬁlters in the ( i − 1) th layer.
To eliminate the inﬂuence from the number of the ﬁlters, we can consider the correlation matrix of each parameter in the ﬁlter.
As the analysis in Section 2, only when the dimension of the convolutional ﬁlter is large enough will the matrix approximate a diagonal matrix.
It may also be a reason why the layer in the front of the network usually can not pass the statistical test in Appendix P.
5. W HAT ABOUT OTHER PRUNING CRITERIA ?
In this section, we study the Similarity and Applicability problem in more types of pruning criteria through numerical experiments, such as Activation-based pruning (Hu et al., 2016;
Luo & Wu, , Importance-based pruning (Molchanov et al., 2016; 2019a) and BN-based pruning (Liu et al., 2017b).
For each type, we choose two representative criteria and we call them: (1) Norm-based: (cid:96) 1 and (cid:96) 2 ; (2) Importance-based:
Taylor (cid:96) 1 and Taylor (cid:96) 2 (Molchanov et al., 2016; 2019a;b); (3) BNbased: BN γ 4 and BN β (Liu et al., 2017b); (4) Activation-based: Entropy (Luo & Wu, 2017) and
APoZ (Hu et al., 2016).
The details of these criteria can be found in Appendix M.
The Similarity for different types of pruning criteria .
In Fig.
7, we show the Sp between different types of pruning criteria, and only the Sp greater than 0.7 are shown because if Sp < 0.7, it means that there is no strong similarity between two criteria in the current layer.
According to the Sp shown in Fig.
7, we can have the following observations: (1) As veriﬁed in Section 3, (cid:96) 1 and (cid:96)maintain a strong similarity in each layer;
(2) In the layers shown in Fig. 7(a) and Fig. 7(d), the Sp between most different pruning criteria are not large in these layers, which indicates that these methods have great differences in the redundancy measurement of convolution ﬁlters.
This may lead to a phenomenon that one criterion considers a convolutional ﬁlter to be important, while another considers it redundant.
We ﬁnd a speciﬁc example which is shown in Appendix E; (3) Intuitively, the same type of criteria should be similar.
However, it can be seen from Fig. 7(b) and Fig.
7(c) that the Sp between Taylor (cid:96) 1 and Taylor (cid:96) 2 is not large, but Taylor (cid:96) 2 has strong similarity with both two Norm-based criteria.
Moreover, the Sp between BN γ and each Norm-based criteria exceeds 0.9, but it is not large in other layers (Fig. 7(b) and Fig. 7(d)).
These phenomena are worthy of further study.
The Applicability for different types of pruning criteria .
According to the analysis in Section 4, the Applicability problem depends on the mean and variance of the parameters.
Fig.
8 shows the result of the importance measured by different pruning criteria on each layer of VGG16.
VGG16 is a network whose width becomes larger gradually as the depth increases (from 64 to 512).
For this reason, we can study the Applicability problem in generally wide (shallow layers) and sufﬁciently wide (deep layers) convolutional layers.
First, we analyze the Norm-based criteria.
The mean of both (cid:96) 1 and (cid:96) 2 are relatively large, but in most layers, the variance/mean of (cid:96) 2 is much smaller than that of (cid:96) 1 , which means that the (cid:96) 2 pruning has Applicability problem, while the (cid:96) 1 does not.
This is consistent with the conclusion in Section 4.
Next, for the Activation-based criteria, the mean and variance/mean are both large, which means that these two Activation-based criteria can distinguish the network redundancy well from their measured ﬁlters’ importance .
However, for the Importance-based and BN-based criteria, their mean and variance/mean are close to 0.
According to the condition (1) shown in Section 4, these criteria have Applicability problem, especially in the deep layers.
5. W HAT ABOUT GLOBAL PRUNING ?
Compared with layer-wise pruning, global pruning is more widely (Liu et al., 2018; Molchanov et al., 2016; Liu et al., 2017b) used in channel pruning.
Similarity while using global pruning .
In
Fig.
9, we show the similarity of different types of pruning criteria using global pruning on VGG16 and ResNet56.
Comparing to the results from the layer-wise pruning shown in
Fig.
7, we can ﬁnd that the similarities of most pruning criteria are quite different in global pruning.
In particular, for the results of (cid:96) 1 and (cid:96) 2 in Fig.
9(a), the similarity between (cid:96) 1 and (cid:96)is not as strong as the one in the layer-wise case.
We argue that this phenomenon may be due to the differences between the parameter distribution in different convolutional layers.
More analysis can be found in Appendix A.
Applicability while using global pruning .
In fact, for global pruning, Norm-based criteria are not prone to Applicability problems.
From Section 4, we have the estimations for the magnitude of importance in i th layer calculated by (cid:96) 1 and (cid:96) 2 as σ i · k (cid:113) 2 π and √ 2 σ i · Γ( k i +12 ) /
Γ( k i 2 ) .
Since σ i and k i are quite different, the variance of the importance is large in this situation.
Fig.
10 shows this kind of difference of the magnitude on different convolutional layers.
Our estimation also explains a common problem in practical applications of global pruning: the network is easily pruned off.
As shown in Fig.
10, we take ResNet56 as an example.
Since the importance in ﬁrst stage is much smaller than the importance in the deeper layer, global pruning will give priority to prune the convolutional ﬁlters of the ﬁrst stage.
To solve the problem of inconsistent magnitude, we suggest that some normalization methods should be implemented or a protection mechanism should be established, e.g. , a mechanism which can ensure that each layer has at least a certain number of convolutional ﬁlters that will not be pruned.
C ONCLUSION
In this paper, we found two blind spots on pruning criteria:
Similarity and Applicability.
For Similarity, some primary pruning criteria can obtain very similar pruning results.
For Applicability, some criteria can not distinguish the redundancy of the ﬁlters well when the number of ﬁlters is large enough.
The comprehensive experiments validate these two ﬁndings, and our assumption is called CWDA.
Under CWDA, these two blind spots are also discussed when using global pruning.
A A DDITIONAL STUDY FOR CWDA A. I S NORMAL DISTRIBUTION NECESSARY ?
In this section, we use the random functions provided by NumPy 5 to generate data manually to study the inﬂuence of distribution on the similarity between Norm-based criteria.
We use these distribution functions to generate 100 vectors with 50 dimensions each time.
Each vector can be regarded as a convolutional ﬁlter with dimension 50.
As shown in the Table 4 (Left), we compare the Sp of these generated convolutional ﬁlters between (cid:96) 1 , (cid:96) 2 and GM criteria.
If Sp is greater than 0.9, we mark it as green.
Otherwise, we mark it as red.
Under these generated distribution, (cid:96) 1 and (cid:96) 2 can keep larger Sp in most cases, which indicates that they still maintain strong similarity for these distribution.
But the similarity between GM and (cid:96)(or (cid:96) 2 ) is relatively poor.
[Table]
If we use zero-mean normalization for the generated data X , i.e. , X − E X , as shown in Table4 (Right), we also ﬁnd that the Sp between these three criteria are greater than 0.9 for all distribution.
This phenomenon may indicate that the normality in CWDA is not necessary for the similarity of criteria, but the zero mean is the key to the similarity.
[Table]
Next
[Table]
we consider the mixed distribution
[Table]
i
[Table]
e. , as shown in Table 5, among the 100 vectors we generated, the ﬁrst 50 vectors are generated by Distribution
[Table]
and the last 50 vectors are generated by Distribution 2.
There are several notable phenomena in Table 5:
(1) For the mixed distribution composed of two normal distribution with zero-mean but different variances, the three criteria can maintain strong similarity;
(2) In the mixed distribution, zero-mean normalization may not make the Sp between the three criteria stronger, which is inconsistent with the situation in Table 4.
(3) For (cid:96) 1 and (cid:96) 2 , the Sp measured by the mixed distribution is lower than single distribution (Table 4) in general.
According to CWDA
[Table]
the mixed distribution in phenomenon (1) actually corresponds to global pruning strategy.
If CWDA holds, the distribution of parameters in different convolutional layers depends on the variance of parameters in each layer, and these variances are generally different.
Therefore, the global pruning strategy is actually to prune the convolutional ﬁlters that satisfy different Gaussian distribution.
Unlike layer-wise pruning, these ﬁlters cannot be represented by a general random variable F , where F ∼ N ( 0 , c 2 · I N i × k × k ) .
It needs to be described by a stochastic process, i.e. ,
F t ∼ N ( 0 , c 2 · I ( t ) N i × k × k ) .
According to the experiment of phenomenon (1) and CWDA, we expect that (cid:96) 1 and (cid:96) 2 should still maintain strong similarity while using the global pruning.
However, as shown in Fig.
9, the Sp of (cid:96) 1 and (cid:96) 2 is not very large.
This seems to contradict the experiment of phenomenon (1).
In fact, however, according to the analysis in Section 9, CWDA may not hold in the ﬁrst few layers of CNNs.
So if these convolutional ﬁlters are also considered to be globally pruned in global pruning, then the similarity between (cid:96) 1 and (cid:96) 2 may be weakened, which is consistent with the experiment in Table 5.
To verify this statement, Fig.
11 shows the result of VGG16 via using global pruning strategy.
The
Start layer means the layer that we start to use for global pruning.
For example, when Start layer is equal to 0, it is the general global pruning; When Start layer is equal to 5, the ﬁlters from the ﬁrst ﬁve layers are not be considered to be pruned.
In Fig.
11, we can see that, with the increase of start layer , the similarity between (cid:96) 1 and (cid:96) 2 becomes stronger and stronger.
This shows that the similarity between (cid:96) 1 and (cid:96) 2 is not as strong as layer-wise in global pruning, which is mainly caused by the ﬁrst few layers that do not satisfy CWDA.
Moreover, in phenomenon (2), global pruning is different from layer-wise pruning shown in Table 4.
The zero-mean normalization may not make these pruning criteria similar.
Therefore, global pruning is worthy of further studies, such as using the stochastic process to explore the similarity and the
Applicability problem of different types of pruning criteria.
B T RAINING THROUGH SLIMMING
As a representative of the BN-based pruning method, slimming pruning(Liu et al., 2017a) can not be directly compared with the criteria mentioned in the paper because it adopts a special training method.
Therefore, we use the training method in Liu et al.
(2017a) to train another ResNet56 on cifar100.
Then, the analysis of similarities between 8 different pruning criteria on such a model is shown in Fig.
In this situation, the ﬁfth criterion BN γ is the method introduced in Liu et al.
(2017a).
From
Fig.
13, there is no signiﬁcant difference in the result of the similarity between ResNet56 obtained by slimming method and resnet56 trained in general.
C T HE RESULT OF S P
[Figure]
D O THER RESULT
In Fig 15, we show the other learnable parameters ( i e. Batch normalization (BN) and fully connected neural network (FC)) in VGG16-BN.
For BN, the distribution of its parameters does not satisfy CWDA, and similar results are shown in Liu et al.
(2017a)
; Tian et al.
Moreover, the learnable parameters of fully-connected layers also do not follow a Gaussian-alike distribution, which is consistent with the conclusion in previous work Bellido & Fiesler (1993); Neal (1995); Go et al.
E A N INTERESTING CASE FOR importance MEASURED BY DIFFERENT CRITERIA
The following results are the index of pruned ﬁlters obtained by the ﬁlters’ importance from different types of pruning criteria.
We take VGG16 (2 nd ) as an example.
The 5 th ﬁlter in this layer is regarded as a redundant convolutional ﬁlter for APoZ criterion, but other criteria consider it to be almost the most important.
Taylor (cid:96) 1 : [27, 36, 25, 11, 6, 23, 24, 16, 0, 57, 48, 53, 1, 61, 18, 55, 34, 15, 51, 58, 31, 3, 12, 21, 59, 30, 7, 38, 41, 50, 10, 33, 17, 46, 62, 13, 49, 43, 42, 47, 2, 32, 44, 20, 39, 52, 56, 40, 9, 26, 37, 22, 29, 54, 60, 8, 14, 45, 4, 63, 19, 35, 28, 5 ]
Taylor (cid:96) 2 : [23, 32, 36, 11, 62, 16, 30, 59, 10, 13, 2, 50, 38, 0, 46, 43, 21, 26, 15, 22, 7, 51, 39, 33, 14, 58, 9, 40, 57, 6, 61, 44, 20, 48, 3, 53, 41, 56, 17, 12, 18, 31, 4, 1, 25, 19, 63, 24, 54, 45, 52, 37, 55, 47, 34, 35, 8, 29, 42, 27, 49, 28, 60, 5 ]
BN β : [52, 46, 32, 21, 14, 29, 17, 0, 19, 36, 1, 51, 44, 40, 41, 60, 57, 27, 22, 53, 63, 8, 30, 26, 23, 58, 39, 18, 9, 47, 31, 35, 11, 37, 55, 45, 3, 61, 6, 4, 33, 25, 15, 48, 43, 28, 56, 2, 13, 16, 34, 20, 59, 10, 7, 24, 50, 62, 12, 49, 38, 42, 5 , 54]
APoZ: [ 5 , 10, 38, 42, 62, 24, 13, 12, 7, 28, 59, 15, 23, 11, 16, 56, 34, 35, 57, 19, 2, 49, 43, 25, 6, 63, 61, 36, 9, 27, 33, 20, 48, 58, 55, 18, 51, 31, 1, 0, 53, 37, 26, 29, 47, 60, 8, 44, 41, 46, 21, 17, 14, 32, 52, 22, 39, 3, 40, 30, 4, 45, 50, 54]
F R ELATED P ROPOSITION
Proposition 1 (Amoroso distribution) .
The Amoroso distribution is a four parameter, continuous, univariate, unimodal probability density, with semi-inﬁnite range (Crooks, 2012).
And its probability density function is
[Equation]
for x, a, θ, α, β ∈ R , α >
and range x ≥ a if θ > 0 , x ≤ a if θ < 0 .
The mean and variance of
Amoroso distribution are
[Equation]
and
[Equation]
Proposition 2 (Half-normal distribution) .
Let random variable
X follow a normal distribution
N (0 , σ 2 ) , then Y = | X | follows a half-normal distribution (Pescim et al., 2010).
Moreover, Y also follows Amoroso ( x | 0 , √ 2 σ, 12 , 2) .
By Eq.
(5) and Eq.
(6), the mean and variance of half-normal distribution are
[Equation]
and
[Equation]
Proposition 3 (Scaled Chi distribution) .
Let X =
( x 1 , x 2 , ...x k ) and x i , i = 1 , ..., k are k independent, normally distributed random variables with mean 0 and standard deviation σ .
The statistic (cid:96) 2 ( X ) = (cid:113)(cid:80) ki =1 x 2 i follows Scaled Chi distribution (Crooks, 2012).
Moreover, (cid:96) 2 ( X ) also follows Amoroso ( x | 0 , √ 2 σ, k 2 , 2) .
By Eq.
(5) and Eq.
(6), the mean and variance of Scaled Chi distribution are
[Equation] [Equation]
and
Var
[Equation]
G P ROOF OF T HEOREM
Theorem 1.
Let X ∼ N ( 0 , c 2 · I n ) and ( C 1 , C 2 ) is one of ( (cid:96) 1 , (cid:96) 2 ) , ( (cid:96) 1 , Fermat ) or ( Fermat , GM ) , we have max
[Equation]
Var X (cid:32) (cid:98) C 2 ( X ) (cid:98) C 1 ( X ) (cid:33) , Var
[Equation]
(cid:32) (cid:98)
[Equation]
1 ( X
[Equation]
(cid:98) C 2 ( X ) (cid:33)(cid:41) (cid:46) B ( n ) .
(11) where (cid:98) C 1 ( X ) denotes C 1 ( X ) /
E ( C 1 ( X )) and (cid:98) C 2 ( X ) denotes C 2 ( X )
/ E ( C 2 ( X )) .
B ( n ) denotes the upper bound of left-hand side and when n is large enough, B ( n ) → 0 .
For i th layer, we use v j to represent F ij , j = 1 , 2 , ...N .
And v j meets CWDA ( i.e. , v j are i.i.d and v j ∼ N (0 , c 2 · I ) ).
Actually, from the theoretical analysis in Section 3, the fact that two criteria Cand C 2 meet Eq.11 is equivalent to C 1 ∼ = C 2 .
(1) For ( (cid:96) 2 , (cid:96) 1 ) .
In fact, (cid:96) 2 ∼ = (cid:96) 1 (their importance rankings are similar) is not trivial.
Generally speaking, for convolutional ﬁlters, dim ( v j ) is large enough.
Since v i satisﬁes CWDA, from Theorem 2, we know that the variance of ratio between (cid:98) (cid:96) 1 and (cid:98) (cid:96) 2 have a bound O ( dim ( v j ) − 1 ) , which means (cid:96) 2 and (cid:96) 1 are appropriate monotonic .
Speciﬁc numerical validation is shown in Fig.
17 of
Appendix H).
Theorem 2.
Let X ∼ N ( 0 , c 2 · I n ) , we have
[Equation]
(12) where (cid:98) (cid:96) 1 ( X ) denotes (cid:96) 1 ( X ) / E
( (cid:96) 1 ( X )) and (cid:98) (cid:96)
( X ) denotes (cid:96) 2 ( X ) / E ( (cid:96) 2 ( X )) .
Proof.
(See Appendix H).
(2) For ( (cid:96) 1 , Fermat ) .
Since v i satisﬁes CWDA, from Theorem 3, we know that the Fermat point of v i and the origin 0 approximately coincide.
According to Table 2, || Fermat − v i
||
≈ ||
− v
i ||
=
|| v i ||
Therefore, from Theorem 2, the bound B ( n ) for the ( (cid:96) 1 , Fermat ) is also 1 n .
Moreover, since CWDA, the centroid of v i is G = 1 n (cid:80) Ni
=1 v i = 0 .
Hence,
[Equation]
=
[Equation]
Theorem 3.
Let random variable v i ∈
R k
and they are i.i.d and follow normal distribution
N ( 0 , σ I k ) .
For F ∈ R k , we have argmin F (cid:8) E v i ∼ N ( 0 ,σ I k ) (cid:80) ni
=
|| F − v
i || 2 (cid:9)
= 0 .
Proof.
(See Appendix I).
(3) For ( GM , Fermat ) .
First, we show the following two theorems:
Theorem 4.
For n random variables a i ∈ R k follow N ( 0 , c 2 · I k ) .When
k is large enough, we have such an estimation:
[Equation]
where F 1 ( a i ) = (cid:80) ni
=
|| a
i || 2 / E
( (cid:80) ni
=1 || a
i || 2 ) and F 2 ( a i ) =
(cid:80) ni
=1 || a
i || 22 / E
( (cid:80) ni
=1 || a
i || 22 ) .
Proof.
(See Appendix J).
Theorem 5.
Let v 0 , v 1 , ..., v k be the k + 1 vectors in n dimensional Euclidean space E n .
For all P in E n , k (cid:88)
i
[Equation] [Equation]
where G is the centroid of v i , will hold if it satisﬁes one of the following conditions:
(1)if k ≥ n and rank ( v 1 − v 0 , v 2 − v 0 , ..., v k − v 0 ) = n .
(2)if k < n and ( v 1 − v 0 , v 2 − v 0 , ..., v k − v 0 ) are linearly independent.
(3)if v i ∼ N ( 0 , c 2 · I n ) , Eq.(15) holds with probability 1.
Proof.
(See Appendix K).
Let P ∈ { v 1 , v 2 , ..., v N } .
Since v i ∼ N ( 0 , c 2 · I ) , we can obtain that a i =
P − v i ∼ N ( 0 , 2 c 2 · I ) if P (cid:54) =
v i .
According to the analysis in Section 3 and Theorem 4, we have
[Equation]
Next, we can prove ( k + 1)
||
P − F || 22 ( Fermat ) and (cid:80) Ni =
1 || P − v
i || 2 ( GM ) are approximately monotonic , where P ∈ { v 1 , v 2 , ..., v N } .
[Equation]
Since Eq.
[Equation]
N
[Equation]
i =
[Equation]
Since Theorem
[Equation]
(cid:88) i = |
[Equation]
G
[Equation]
Since Eq.
[Equation]
(cid:88)
[Equation]
The reason for the last equation is
that (cid:80) Ni =
|| G − v
i || 22 is a constant for given v i .
H P ROOF OF T HEOREM
Proposition 4 (Stirling’s formula) .
6 For big enough x and x ∈ R
+ , we have an approximation of
Gamma function:
[Equation] [Equation]
Proposition 5 (FKG inequality) .
If f and g are increasing functions on R n (Graham, 1983), we have
[Equation]
Say that a function on R n is increasing if it is an increasing function in each of its arguments.(i.e.
, for ﬁxed values of the other arguments).
Proposition 6.
Let f ( X, Y ) is a two dimensional differentiable function.
According to Taylor theorem (Hormander, 1983), we have
[Equation]
Lemma 1.
Let X and Y are random variables.
Then we have such an estimation
[Equation]
Proof.
Let f ( X, Y ) =
X/Y , according to the deﬁnition of variance, we have
Var f
( X, Y ) =
[Equation]
X, Y )
[Equation]
X, Y )
[Equation] [Equation]
from Eq.
[Equation]
from Eq.
[Equation]
E
[Equation]
Y
[Equation]
Lemma 2.
For big enough x and x ∈ R + , we have lim x → + ∞ (cid:20) Γ( x +12 ) Γ( x 2 ) (cid:21)·x = 2 .
And lim x → + ∞ Γ( x 2 + 1) Γ( x 2 ) − (cid:20) Γ( x +12 ) Γ( x 2 ) (cid:21)=4 .
Proof.
lim x → + ∞ (cid:20) Γ( x +12 ) Γ( x 2 ) (cid:21)·x ≈ lim x
→ + ∞   (cid:113) 2 π ( x − 12 ) · ( x − 1 2 e ) x −(cid:113) 2 π ( x − 22 ) · ( x − 2 2 e ) x −  ·x from Proposition.= lim x → + ∞ (cid:18) x −x −(cid:19) · ( x − 1 2 e ) x −( x − 2 2 e ) x − 2 · (cid:18) x −2 e (cid:19) ·x = lim x → + ∞ (cid:18) 1 + x −(cid:19) x −· x −x − 2 · x −2 e ·x = on the other hand, we have lim x → + ∞ Γ( x 2 + 1) Γ( x 2 ) − (cid:20) Γ( x +12 ) Γ( x 2 ) (cid:21)= lim x → + ∞ x 2 − (cid:18) 1 + x −(cid:19) x −· x −x − 2 · x −2 e
= lim x → + ∞ x 2 e (cid:18) e − (1 + 1 x ) x
(cid:19)
= (cid:18) − 1 e ( − e ) (cid:19) =
Theorem 2 Let X ∼ N ( 0 , c 2 · I n ) , we have max (cid:40)
Var X (cid:32) (cid:98) (cid:96)
( X ) (cid:98) (cid:96) 1 ( X ) (cid:33) , Var X (cid:32) (cid:98) (cid:96)
( X ) (cid:98) (cid:96) 2 ( X ) (cid:33)(cid:41) (cid:46)n.
where (cid:98) (cid:96) 1 ( X ) denotes (cid:96) 1 ( X ) /
E ( (cid:96) 1 ( X )) and (cid:98) (cid:96) 2 ( X ) denotes (cid:96) 2 ( X ) /
E ( (cid:96) 2 ( X )) .
Proof.
For the ratio (cid:98) (cid:96) 2 ( X ) / (cid:98) (cid:96) 1 ( X ) , we have
Var (cid:32) (cid:98) (cid:96)
( X ) (cid:98) (cid:96) 1 ( X ) (cid:33) =
(cid:18) E ( (cid:96) 1 ( X ))
E ( (cid:96) 2 ( X )) (cid:19)
Var (cid:32) (cid:98) (cid:96)
( X ) (cid:98) (cid:96) 1 ( X ) (cid:33) ≈
(cid:18) E ( (cid:96) 1 ( X ))
E ( (cid:96) 2 ( X )) (cid:19) 2 (cid:18) E ( (cid:96) 2 ( X ))
E ( (cid:96) 1 ( X )) (cid:19) 2 (cid:18)
Var (cid:96) 2 ( X )
E ( (cid:96) 2 ( X )) 2 +
Var (cid:96) 1 ( X )
E ( (cid:96)
[Equation]
X )
[Equation]
Cov
[Equation]
( X )
[Equation]
(cid:96)
[Equation]
X )
[Equation]
X )
[Equation]
X )
[Equation]
from Lemma.
[Equation]
X
[Equation]
(cid:19) .
from Proposition.similarly, we also have
[Equation]
Therefore, max
[Equation]
· σ (cid:112)
[Equation]
π )from Proposition.
3 and
[Equation]
from Lemma
[Equation]
Because the approximation is widely used in the proof of Theorem 1, it is necessary to verify it numerically.
As shown in Fig.
17, we use ResNet56 on Cifar100 and ResNet110 on Cifar10 respectively to verify Theorem 1.
From Fig.
17, we ﬁnd that the estimationn of Theorem 1 is reliable, i.e. , the estimation O ( 1 n ) for max (cid:110)
Var X (cid:16) (cid:98) (cid:96)
( X ) (cid:98) (cid:96) 1 ( X ) (cid:17) , Var X (cid:16) (cid:98) (cid:96) 1 ( X ) (cid:98) (cid:96) 2 ( X ) (cid:17)(cid:111) is appropriate.
I P ROOF OF T HEOREM
Proposition 7.
Let L ( α ) p ( x ) denotes generalized Laguerre function, and it have following properties:
[Equation]
(26) and for α > 0 ,
[Equation]
Theorem 3.
Let random variable v i ∈
R k .
They are i.i.d and follow normal distribution N ( 0 , σ 2 I k ) .
For F in R k , we have argmin
[Equation]
Proof.
Let w i
=
F − v i and we have w i ∼ N ( F, σ 2 I k ) , then
[Equation]
The reason for the last equation is that || w i || 2 follows scaled noncentral chi distribution 7 when w i ∼
N ( F, σ 2 I k ) .
Let T ( x ) =
L ( k 2 − 1)
[Equation]
− x
[Equation] [Equation]
2 (cid:17) , we calculate the minimum of T ( x ) .
From Eq.
[Equation]
Since Eq.
, we ﬁnd that ddx T ( x ) > 0 when x > 0 and if x ≤ 0 , then d dx T ( x ) ≤ 0 .
It means that
T ( x ) gets the minimizer at || F ||
=
0 , i.e. , F = 0 .
J P ROOF OF T HEOREM
Lemma 3.
For two random variables X, Y ∈ R k follow N ( 0 , c 2 · I k ) and they are i.i.d.
When k is large enough, we have:
[Equation]
and
Var
[Equation]
Proof.
According to Proposition 3 and Lemma
[Equation]
it is easy to know (similar method in Eq.(86)), when k is large enough
[Equation]
that
[Equation]
and
[Equation]
Since Lemma
[Equation]
we have an estimation
Var
[Equation]
Since Eq.(31) and Eq.(32)
[Equation]
From Eq.(21) and Lemma
[Equation]
we also can obtain an estimation of E ( A / B ) , where A and B are two random variables.
i.e. ,
[Equation]
Therefore,
[Equation]
Eq
[Equation]
[Equation]
Since Eq.(31) and Eq.
[Equation]
[Equation]
Note that, the approximation is widely used in the proof of Eq.(29) and Eq.(30).
Hence, it is also necessary to verify it numerically.
As shown in Fig.
[Equation]
the estimation is appropriate.
According to
Lemma
[Equation]
the mathematical expectation and variance of the ratio of ( || X || 22 − ||
Y || 22 ) 2 and 2 ||
X
||
· || Y || 2 are both close to 0 when k is large enough and c is small enough.
that is,
[Equation]
By the way
[Equation]
the convolutional ﬁlters easily meet the condition that k is large enough.
Theorem 4.
For n random variables a i ∈ R k follow N ( 0 , c 2 · I k ) .When
k is large enough, we have such an estimation:
[Equation]
Var a
i F 2 ( a i )
F 1 ( a i )
≈ 2 nk .
where
F 1 ( a i ) =
(cid:80) ni
=1 || a
i || 2 / E
( (cid:80) ni
=1 || a
i || 2 ) and F 2 ( a i ) =
(cid:80) ni
=1 || a
i || 22 / E
( (cid:80) ni
=1 || a
i || 22 ) .
Proof.
Since Eq.
(9) and Eq.
(10), we have
[Equation]
(cid:19) .
(35) and
[Equation]
(cid:18)(cid:80)
[Equation]
(cid:19) .
According to Lagrange’s identity, we have
[Equation]
[Figure]
[Equation]
|
[Equation]
|
[Equation]
Since Eq.
[Equation]
so we have
[Equation]
By central limit theorem, we have √ n
[Equation]
[Equation]
And let g ( x ) = 1 x , we can use Delta method 8 to ﬁnd the distribution of g ( 1 n (cid:80) ni
=1 || a
i || 2 ) :
[Equation]
38) where µ and σ 2 denote the mean and variance of ||
a i || 2 respectively.
From Eq.
(37), we have
Var
[Equation]
i
[Equation]
N
[Equation]
Since Eq.
[Equation]
Since Eq.
(9) and Eq.
(10) =
[Equation]
Since Lemma.
Since Eq.
(35), we have
[Equation]
Similar to Eq.
[Equation]
n
Similar to Eq.
[Equation]
Since central limit theorem
[Equation]
Since Eq.
[Equation] [Equation]
Since Lemma.
Since Eq.
(36), we have
[Equation]
From Eq.(39) and Eq.(41), Theorem 4 holds.
In Fig.
19, we also show a numerical veriﬁcation of Theorem 4 .
K P ROOF OF T HEOREM
Proposition 8.
For a n × m random matrix ( a ij ) n × m , where a ij ∼ N (0 , σ 2 ) .
And Eq.
(8) holds with probability 1.
[Equation] [Equation]
Lemma 4.
Let v 0 , v 1 , ..., v k be the k + 1 vectors in n dimensional Euclidean space V and k ≤ n .
If rank ( v 1 − v 0 , v 2 − v 0 , ..., v k − v 0 ) = n , then ∀ x ∈ V , ∃ λ i (0 ≤ i ≤ k ) , s.t. x =
[Equation] [Equation]
(43) and (cid:80) ki =
λ i = 1 .
We call λ
= ( λ 0 , λ 1 , ..., λ k ) the generalized barycentric coordinate with respect to ( v 0 , v 1 , ..., v k ) .
(In general, barycentric coordinate is a concept in Polytope)
Proof.
Note that v i is the element of n dimensional linear space V and rank ( v 1 − v 0 , v 2 − v 0 , ..., v k − v 0 ) = n .
It means ( v 1 − v 0 , v 2 − v 0 , ..., v k − v 0 ) form a set of basis in the linear space V .
∀ x ∈ V , x − v 0 can be expressed linearly by them, i.e. , ∃ t i (1 ≤ i ≤ k ) s.t.
[Equation]
(cid:88) i =
[Equation]
=
t
[Equation]
v
[Equation]
Let λ
= (1 − (cid:80) ki
=1 t i ) and λ
i
=
t i (1 ≤ i ≤ k ) , Lemma 4 holds.
Lemma 5.
Let v 0 , v 1 , ..., v k be the k + 1 vectors in n dimensional Euclidean space V .
∀ a, b ∈
V , and the generalized barycentric coordinate of a, b with respect to ( v 0 , v 1 , ..., v k ) are λ
=
( λ 0 , λ 1 , ..., λ k ) T and µ = ( µ 0 , µ 1 , ..., µ k ) T ,respectively.
Then
[Equation] [Equation]
where D =
( − 12 d ij ) ( k +1)
× ( k +1) , and d ij =
||
v i − v j || 22 .
Proof.
Since Lemma 4, let R =
[ v 0 , v 1 , ..., v k ] n × ( k +1) , and we have a = Rλ and b =
Rµ .
Moreover,
[Equation]
Note that, for D =
( − 12 d ij ) ( k +1)
× ( k +1) ,
[Equation] [Equation]
So we have D
= R T R − 12 (cid:0) ( v Ti v i + v Tj v j ) ( k +1) ×
( k +1) (cid:1) .
It can be further simpliﬁed to D =
R T R − 12 ( V α T + αV T ) , where V =
( v T 0 v 0 , ..., v Tk v k ) T and α = (1 , ..., 1) T .
So
[Equation] [Equation]
[Equation]
therefore, we only need to prove ( λ − µ ) T ( V α T + αV T )( λ − µ )
= 0 .
From Lemma 4, we have α T ( λ − µ ) =
( λ − µ ) T α =
and the Lemma 5 holds.
Deﬁnition 1 (Ultra dimension) .
For a set U composed of vectors in a n dimensional linear space V , we deﬁne (cid:100) dim ( U ) as the Ultra dimension of U .
The deﬁnition is that if U has k linearly independent vectors and there are no more, then (cid:100) dim ( U ) = k .
In fact, if U is a linear subspace in V , then the Ultra dimension and the dimensions of the linear subspace are equivalent.
If U is a linear manifold, U = { x + v 0 | x ∈ W } , where v 0 and W are non-zero vectors and linear subspaces in V , respectively.
And dim ( W ) =
r .
Then (cid:100) dim ( U
[Equation]
=
[Equation]
In other words, (cid:100) dim ( U ) ≥ (cid:100) dim ( W ) always holds.
Lemma 6.
For arbitrary k (1 ≤ k ≤ n − 1) , let a 1 , a 2 , ..., a k be k linearly independent vectors in n dimensional linear space V .
Consider one n − 1 dimensional linear subspace W in V and a non-zero vector v 0 in V .
They form a linear manifold P = { v 0 + α | α ∈ W } .
If a 1 , a 2 , ..., a k do not all belong to P , then there must exist n − k vectors p 1 , p 2 , ..., p n − k from P , s.t ( a 1 , a 2 , ..., a k , p 1 , p 2 , ..., p n − k ) are a set of basis for the linear space V .
Proof.
we use mathematical induction.
First, show that the Lemma 6 holds for n − k = 1 .
it means we need to ﬁnd a vector p 1 ∈ P s.t.
a 1 , a 2 , ..., a k , p 1 linearly independent.
If p 1 does not exist, then ∀ p ∈ P would be linearly represented by a 1 , a 2 , ..., a k .
In other word
[Equation]
1 (cid:13) For the linear manifold P , if v 0 ∈ W .
This means that P is equal to the linear subspace W .
Since
Eq.
(54), we have W ⊂ L and (cid:100) dim ( W ) =
(cid:100) dim ( L ) .
Hence, P = W = L .
However, a 1 , a 2 , ..., a k do not all belong to P , a contradiction.
2 (cid:13) For the linear manifold P , if v 0 / ∈ W , then (cid:100) dim ( P ) = n .
Because v 0 / ∈ W , that is, v 0 cannot be represented by a set of basis of W .
In other words, v 0 and a set of basis of W are linearly independent.
However, the dimension of W is n − 1 , hence (cid:100) dim ( P )
= n .
From Eq.
(54), we have
P ⊂ L , so
[Equation] [Equation]
a contradiction.
Therefore, Lemma 6 holds for n − k = 1 .
Assume the induction hypothesis that
Lemma 6 is true when n − k = l , where 1 ≤ l . when n − k = l + 1 , i.e. , k = n − ( l + 1) , we also can ﬁnd a vector p 1 ∈ P s.t.
a 1 , a 2 , ..., a k , p 1 linearly independent.
Otherwise, ∀ p ∈ P would be linearly represented by a 1 , a 2 , ..., a k .
Similarly, we have Eq.
Note that, from Deﬁnition 1, (cid:100) dim ( P ) ≥ n − 1 , hence
[Equation]
56) a contradiction.
At this time, we have k + 1 = n − ( l + 1) + 1 = n − l vectors a 1 , a 2 , ..., a k , pwhich are not all on P .
Note that n − ( n − l ) =
l , using the induction hypothesis, the Lemma 6 also holds for n − k
= l .
In summary, Lemma 6 holds.
Theorem 5.
Let v 0 , v 1 , ..., v k be the k + 1 vectors in n dimensional Euclidean space E n .
For all P in E n ,
[Equation]
where G is the centroid of v i , will hold if it satisﬁes one of the following conditions:
(1)if k ≥ n and rank ( v 1 − v 0 , v 2 − v 0 , ..., v k − v 0 ) = n .
(2)if k < n and ( v 1 − v 0 , v 2 − v 0 , ..., v k − v 0 ) are linearly independent.
(3)if v i ∼ N ( 0 , c · I n ) , Eq.(15) holds with probability 1 where c is a constant.
Proof.
For Theorem 5 (1) .
From Lemma 4, ∀ P ∈ E n , ∃ γ = ( γ 0 , ..., γ k ) ,
s.t. P can be represented by (cid:80) ki =
γ i v i , where (cid:80) ki =
γ i = 1 .
In fact, for each v i , it also can be respresented by (cid:80) kj =0 β ij v i , where (cid:80) ki =
β ij = 1 .
We just take ( β i 0 , β i 1 , ..., β ik ) as one of the standard orthogonal basis (cid:15) i = (0 , 0 , ..., 1 i , ... 0) .
According to lemma 5,
[Equation]
The ﬁnal equation is because the diagonal elements of the matrix are all 0.
On the other hand, we have
[Equation]
where α = (cid:80) ki =0 (cid:15)
i , i.e. , α = (1 , 1 , ..., 1) .
Next, we consider || P − G || 22 .
[Equation]
In summary, we have
[Equation]
Therefore, Theorem 5 (1) holds.
For Theorem 5 (2) .
Next, we prove the case of k < n .
Obviously, Lemma 4 does not hold.
We consider about such a linear space W 1 = span ( P − G ) , i.e. , a linear space expanded by P − G , and its orthogonal complement W ⊥ 1 (in E n ).
Since dimension formula from linear space, it is easy to konw that dim ( W ⊥ 1 ) = n − 1 .
Two linear manifolds T 1 and T 2 are constructed as follows,
[Equation]
[Equation]
∀ v i ∈ T 1 , we have ( v i − G ) T ( P − G ) = 0 , Furthermore,
[Equation]
[Equation]
It is easy to know that G − v 0 is not 0.
If v 1 − v 0 , ..., v k − v 0 are all belong to T 2 , it means v 1 , .., v k are all in T 1 .
Hence, we have Eq.
By summing both sides of Eq.
(69) for i , it is obvious ﬁnd that Theorem 5 (2) holds.
If v 1 − v 0 , ..., v k − v 0 are not all belong to T 2 , since Lemma 6, there are n − k vectors p 1 − v 0 , p 2 − v 0 , .., p n − k − v 0 from T 2 s.t.
they and v 1 − v 0 , ..., v k − v 0 are linearly independent, where p i obviously belongs to manifold T 1 .
At the same time, we have 2 G − p i ∈ T 1 , we can also construct n − k new vectors 2 G − p
i − v 0 ∈
Tand calculate the rank that
[Equation]
[Equation]
The reason of the ﬁnal equation is that (cid:80) ki
=1 ( v i − v 0 )
=
( k + 1)( G − v 0 ) .
Note that there are a total of k + ( n − k )
+ ( n − k )
=
n + ( n − k ) ≥ n vectors, meets the lemma 4 condition.
For the convenience of description, we deﬁne
[Equation] [Equation]
And their centroid is
[Equation]
That is, the newly added vector does not change the centroid of v i .
On the other hand, since both
L (2) i and L (3) i are in the linear manifold T 1 , and it meets the conditions of the Eq.(69).
Similar to the derivation in the Theorem 5 (1), we have
[Equation]
The ﬁnal equation is because both L (2) i and L (3) i are in the linear manifold T 1 and satisfy Eq.
To simplify Eq.
(81), we obtain (cid:80) ki =0 (cid:0) ||
P − v
i ||
22 − ||
G − v
i || 22 (cid:1)
=
( k + 1) ||
P − G || 22 .
Therefore,
Theorem 5 (2) holds.
For Theorem 5 (3) .
When k ≥ n , from Proposition 8, we know that rank ( v 1 − v 0 , v 2 − v 0 , ..., v k − v 0 ) = n holds with probability 1.
Hence, if we use the similar deduction from Theorem 5 (1), we can ﬁnd that Theorem 5 (3) holds when k ≥ n .
On the other hand, when k < n , we can get the same result also according to Proposition 8.
The reason is that ( v 1 − v 0 , v 2 − v 0 , ..., v k − v 0 ) are linearly independent with probability 1.
L T HE GEOMETRIC STRUCTURE OF CONVOLUTIONAL FILTERS .
Theorem 6.
Let v i ∈
R k and v i ∼ N ( 0 , c 2 · I k ) .
If k → ∞ and c (cid:54) = 0 , then
[Equation]
Proof.
First, since Chebyshev inequality, for 1 ≤ i ≤ N and a given M , we have
[Equation]
from Eq.
(9), Eq.
(10) and Lemma.
(2), we can rewrite Eq.
(82) when k → ∞ :
P
[Equation]
For a small enough (cid:15) > 0 , let M = 1 /(cid:15) .
Note that (cid:113) M 2 c = c/ √ 2 (cid:15) is a constant.
When k → ∞ , √
[Equation]
/ 2) (cid:29) (cid:113) M 2 c .
Hence, for any i ∈ [1 , N ] and any small enough (cid:15) , we have
[Equation]
[Equation]
So Theorem 6(1) holds.
Let v i
= ( v i 1 , v
i 2 , ..., v ik ) and v j = ( v j 1 , v j 2 , ..., v jk ) .
So < v i , v j > = (cid:80) kp =1 v ip v jp .
Note that, v i and v j are independent, hence
[Equation]
[Equation]
since central limit theorem, we have
[Equation]
According to Eq.
(9), Lemma 2 and Eq.
(87), when k → ∞ , we have
[Equation]
So Theorem 6(2) holds.
From Theorem 6(1) and Theorem 6(2), Theorem 6(3) can be proved through
Pythagoras theorem.
For 6(4), from Proposition.
2, we have
[Equation]
As shown in Fig.
21, Theorem 6 actually reveals the geometric structure formed by the parameters of the convolutional ﬁlters in CNNs.
Speciﬁcally, from Theorem 6 (1), the convolutional ﬁlters v i of each layer locate approximately on the surface of k dimensional sphere with 0 as the origin and √ 2 c · Γ(( k +1) / 2) Γ( k/ 2) as the radius.
Then, from Theorem 6 (2), the vectors formed by any two different convolutional ﬁlters in the same layer are approximately orthogonal.
As this result, for any three different ﬁlters v 1 , v 2 and v j , we can use Pythagoras theorem and Theorem 6 (1) to prove that they are equidistant, i.e. , || v 1 − v
||
≈ ||
v 2 − v 3 ||
≈ || v 3 − v 1 || 2 .
In fact, Fig. 20 provides another view of the geometric structure of convolutional ﬁlters.
Since CWDA, E ( v i ) = 0 .
So the correlation matrix { ( Cor ( v i , v j ) } N × N = c · { ( v Ti v j } N × N , where c is a constant.
That is to say, there is only one coefﬁcient difference between correlation matrix and Gram matrix.
Therefore, the diagonal elements of the matrix are ||
v i || 22 , and
the off-diagonal elements are the dot product between the convolutional ﬁlters.
This numerical visualization also veriﬁes the conclusion of Theorem6.
M T HE DETAILS OF OTHER PRUNING CRITERIA
For notation, we denote i th convolutional ﬁlter in layer l as F li and the input feature maps in layer l as I l ∈ R N × I l × H l × W l , where N, I l , H l , W l mean the train set size, number of channels, height and width respectively, i = 1 , 2 , · · · , λ l , and l = 1 , 2 , · · · , L .
The formulation of the ﬁlters’ importance under each pruning criteria are illustrated as follows:
Norm-based criteria:
• (cid:96) 1 -Norm Li et al.
(2016): || F li || 1 ; • (cid:96) 2 -Norm Frankle & Carbin (2019): || F li || 2 ;
BN-based criteria Liu et al.
(2017b):
• BN γ : |
γ li | , where γ li is the scaling factor in the Batch Normalization layer l ; • BN β : | β li | , where β li is the shifting factor in the Batch Normalization layer l .
Activation-based criteria:
• APoZ Hu et al.
(2016): (cid:80) p,q 1 ( ( | I l ∗ F li | ) p,q >σ ) N
× I l × H l × W l , where we set σ = 0 .
0001 same as Luo & Wu (2017), and 1 ( · ) is the indicator function, ∗ is convolution operator and I l ∗ F li is the i -th output feature map; • Entropy Luo & Wu (2017): we ﬁrst prepare G li =
GAP ( I l ∗ F li ) , where G li ∈ R N × 1 and GAP ( · ) is the Global Average Pooling.
Then, we estimate statistical distribution for G li by dividing all elements in G li into m bins.
Let p j is the probability of bin j , and the the importance score is − (cid:80) mj
=1 p j log p j .
First order Taylor based criteria Molchanov et al.
(2016; 2019a;b):
• Taylor (cid:96) 1 -Norm: ||
∂loss∂F li · F li || 1 ; • Taylor (cid:96) 2 -Norm: || ∂loss∂F li · F li || 2 ;
[Table]
N A DDITIONAL EXPERIMENTS ABOUT IMAGE CLASIFICATION
[Table]
All the setting of these experiments are under can be found in https://github.com/ bearpaw/pytorch-classification .
Speciﬁcally, for pruning ratio:
VGG16 on CIFAR10, CIFAR100 and ImageNet: https://github.com/Eric-mingjie/rethinking-network-pruning/blob/ master/cifar/l1-norm-pruning/vggprune.py#L
ResNet56 on CIFAR10 and CIFAR100: https://github.com/Eric-mingjie/rethinking-network-pruning/blob/ master/cifar/l1-norm-pruning/res56prune.py#L
ResNet34 on ImageNet: https://github.com/Eric-mingjie/rethinking-network-pruning/blob/ master/imagenet/l1-norm-pruning/prune.py#L
O A BOUT WEIGHT DECAY
We train the ResNet110 and WRN-28-10 on CIFAR100 with different weight decay (1e-3, 3e-4 and 0) and use KS test to verify whether the parameters of different layers follow a normal distribution.
In Fig.
22, we can ﬁnd (1) When weight decay (wd) is non-zero, the normality is higher than that when weight decay is 0.
(2) If weight decay is 0, the p-value can still be much greater than 0.05, which means that the regularization of weight decay may not be the key reason for CWDA.
The distribution of the parameters in these two networks (weight decay is 0) are shown in Fig.
24 and Fig. 23.
P S TATISTICAL T EST
In this section, according to Table 3 and Section 2.1, we have a series of statistical tests for the necessary conditions of CWDA.
let F ij ∈ R N i × k × k represent the j th ﬁlter of the i th convolutional layer.
(1) Gaussian ( i.e. , to verify
whether
F ij approximatively follow a Gaussian-alike distribution.
In
i th layer, we use Kolmogorov–
Smirnov (
KS) test
(Lilliefors, 1967)
to
check if all the weights
in
the
same layer follow a normal distribution
(2) Standard Deviation ( i.e. , to verify whether the standard deviation of each ﬁlter in any layers tends to be a constant c .).
Let σ j denotes the standard deviation of all the weights of ﬁlter F ij in i th layer.
We use Student’s t test (Efron, 1969) to check if the variance of these σ j is small enough.
The null hypothesis H 0 and the alternative hypothesis H 1 are:
[Equation]
where N i denotes the number of the ﬁlters in i th layer and σ 0 is a given real number which is small enough, like σ 20 = 0 . 0001 .
(3) Mean ( i.e. , to verify whether the mean of F ij is 0.).
Let the mean of all the weights in the same layer is µ .
We use Student’s t test (Efron, 1969) to check if µ is close to 0.
First, we check the upper bound (Mean-Left) of µ , i.e. ,
[Equation]
where (cid:15) is a small constant, like (cid:15) = 0 .
Next, we check the lower bound (Mean-Right) and the null hypothesis H 0 and the alternative hypothesis H 1 are:
[Equation]
H 1 : µ < − (cid:15).
Of course, the p value for H 0 : µ ≥ − (cid:15) and H 0 : µ ≤ (cid:15) should be the same.
There are several
Notable points:
• In all the statistical tests, let the conﬁdence level be 0.95, (cid:15)
= 0 .
01 and σ 20 = 0 .
• we use Green color to represent that the convolutional ﬁlters in one layer can pass the statistical test.
Conversely, the Red color means that the ﬁlters can not pass the tests.
• In most layers, the convolutional ﬁlters can pass the statistical tests, except for a few layers which are in front of the network.
This phenomenon is consistent with the analysis in Section 5.1 and it does not mean CWDA is not true.
• Most of the experiments are image classiﬁcation, except for the tests in Appendix P.7.
• p-value and c-value denote p value and critical value (conﬁdence level is 0.95), respectively.
t-value is t values in Student’s t test.
If p-value is larger than c-value or t-value is smaller than c-value , we think this ﬁlter passes the tests.
• In fact, some of the experiments in Table 3 are repeated.
These experiments are shown in following tables.
The (*) means that ,for the sake of brevity, the repeat experiments are shown only once on the experiment with (*).
[Table]
P. N ETWORK S TRUCTURE
conﬁg: https://github.com/bearpaw/pytorch-classification .
[Table]
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
[Table]
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
[Table]
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0. Conv   0. Conv   0. Conv
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
[Table]
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
[Table]
[Table]
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
[Table]
[Table]
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0.
Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
[Table]
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv
[Table]
[Table]
[Table]
[Table]
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table]
https://github.com/bearpaw/pytorch-classification .
https://pytorch.org/docs/master/optim.html#torch-optim .
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table]
Conv
[Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
Conv
[Table]
[Table] [Table] [Table] [Table]
[Table]
Conv
[Table] [Table] [Table]
[Table] [Table]
Conv
[Table]
[Table] [Table] [Table] [Table]
Conv
[Table] [Table]
[Table] [Table]
Conv
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Conv
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Conv   0.
[Table] [Table] [Table] [Table]
Conv
[Table]
[Table]
[Table]
Conv
[Table]
[Table]
[Table]
Conv   0.
[Table]
Conv
[Table]
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
[Table]
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv
[Table]
[Table]
[Table]
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
[Table]
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0. Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv
[Table]
[Table]
[Table]
Passing
[Table]
Layer
[Table] [Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv
[Table] [Table]
[Table]
Conv   0.
Conv
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0.
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
Conv
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
Conv
[Table] [Table]
Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0.
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table] [Table] [Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table]
[Table] [Table]
[Table] [Table]
Conv
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
Conv
[Table]
[Table]
[Table]
[Table]
[Table]
Conv
[Table]
[Table]
[Table]
[Table]
[Table]
Conv
[Table]
[Table]
[Table]
[Table]
[Table]
Conv
[Table] [Table]
[Table] [Table]
[Table]
Conv
[Table] [Table]
[Table] [Table]
[Table]
Conv
[Table] [Table]
[Table] [Table]
[Table]
Conv
[Table]
[Table]
[Table]
[Table] [Table]
[Table]
[Table]
Conv
[Table] [Table]
[Table] [Table]
[Table]
Conv
[Table] [Table]
[Table] [Table]
[Table]
Conv
[Table] [Table]
[Table] [Table]
[Table]
Conv
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table]
Conv   0. Conv   0.
Passing rate - -
[Table] [Table]
P.
R EGULARIZATION
conﬁg: https://github.com/bearpaw/pytorch-classification .
https://github.com/LeungSamWai/Drop-Activation https://github.com/uoguelph-mlrg/Cutout https://github.com/clovaai/CutMix-PyTorch
https://github.com/DeepVoltaire/AutoAugment
[Table] [Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table]
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv
[Table] [Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table]
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv
[Table]
P. A TTENTION M ECHANISM
conﬁg
[Table]
https://github.com/moskomule/senet.pytorch https://github.com/gbup-group/DIANet https://github.com/EvgenyKashin/SRMnet https://github.com/luuuyi/CBAM.PyTorch https://github.com/gbup-group/IEBN https://github.com/implus/PytorchInsight
[Table] [Table]
Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv
[Table] [Table]
[Table]
Conv   0.
Conv
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table]
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table]
[Table] [Table]
[Table] [Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
Conv
[Table] [Table]
[Table] [Table]
[Table]
Conv   0.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
c-value p-value c-value t-value c-value
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0.
Conv   0.
Conv   0.
[Table]
[Table]
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0.
Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0. Conv   0.
Conv   0.
Conv
[Table] [Table]
[Table]
Conv
[Table]
Conv
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table]
Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table]
https://github.com/bearpaw/pytorch-classification .
https://pytorch.org/docs/master/nn.init.html#nn-init-doc .
[Table]
[Table] [Table]
Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
Conv
[Table] [Table]
[Table] [Table]
[Table] [Table]
Conv
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
Conv
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
Conv
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table]
[Table] [Table]
Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table]
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv
[Table] [Table]
[Table] [Table]
[Table] [Table]
Conv
[Table] [Table]
Conv
[Table] [Table]
[Table]
Conv
[Table]
Conv
[Table] [Table]
Conv
[Table] [Table]
Conv
[Table] [Table]
Conv   0.
[Table]
Conv   0.
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table]
Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table]
Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table]
conﬁg: https://github.com/bearpaw/pytorch-classification .
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table]
P.
O THER T ASKS
P.7.
S EGMENTATION conﬁg: https://github.com/meetshah1995/pytorch-semse https://github.com/speedinghzl/pytorch-segmentation-toolbox
[Table] [Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table]
Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0.
Conv
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table]
conﬁg
[Table]
https://github.com/jwyang/faster-rcnn.pytorch
[Table]
p
[Table]
value
[Table]
value
[Table]
value
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table] Conv [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table] Conv [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table] Conv [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] Conv
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] Conv
[Table]
[Table] [Table] [Table] [Table]
[Table] Conv [Table] [Table]
[Table]
[Table] [Table] [Table] Conv
[Table]
[Table]
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0.
Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
Conv   0.
[Table]
[Table]
Conv   0.
[Table]
Conv   0.
[Table]
Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0. Conv   0.
Conv   0.
Conv   0.
Conv   0.
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
Conv
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
Conv
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
Conv
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
Conv
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
[Table]
[Table]
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0. Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv   0.
Conv
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Conv
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Passing
[Table] [Table]
[Table]
P.7.
I MAGE M ATTING conﬁg: https://github.com/foamliu/Deep-Image-Matting-PyTorch https://github.com/CDOTAD/AlphaGAN-Matting
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
Conv   0.
[Table] [Table]
[Table] [Table]
Conv   0.
[Table]
Conv   0.
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table]
conﬁg
[Table]
https://github.com/abhiskk/fast-neural-style
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table]
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table]
://github.com/csinva/gan-pretrained-pytorch
[Table] [Table]
[Table]
B ATCH N ORMALIZATION
[Table] [Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Passing rate
[Table] [Table]
[Table] [Table]
[Table]
: http://pytorch.org/docs/master/torchvision/index.html .
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table]
Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table]
[Table] [Table]
L
[Table]
R ATE conﬁg: https://github.com/bearpaw/pytorch-classification .
[Table]
Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
Conv   0. Conv   0.
[Table] [Table]
[Table]
-value c-value
Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv   0.
Conv   0. Conv   0. Conv
[Table] Q A DDITIONAL EXPERIMENTS FOR S ECTION In this section, we use more networks (Pytorch Pretrain Models 9 to verify the obersevations in
Fig. 2 (a-d).
We show the results of three layers for each network. Q. VGG  [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
R D ETAILS OF E XPERIMENTS
[Table]
There are several additional experiments of Fig. 5 in following ﬁgures.
They show the Spearman’s rank correlation coefﬁcient (Sp) among (cid:96) 1 pruning, (cid:96) 2 pruning and GM pruning for all the experiments in Table 3.
These experiments further visualized the strong similarities of these pruning methods in different situations
[Table]