P ROBABILISTIC C ONNECTION I MPORTANCE I NFERENCE AND L OSSLESS C OMPRESSION OF D EEP N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep neural networks (DNNs) can be huge in size, requiring a considerable amount of energy and computational resources to operate, which limits their applications in numerous scenarios.
It is thus of interest to compress DNNs while maintaining their performance levels.
We here propose a probabilistic importance inference approach for pruning DNNs.
Speciﬁcally, we test the signiﬁcance of the relevance of a connection in a DNN to the DNN’s outputs using a nonparemetric scoring test and keep only those signiﬁcant ones.
Experimental results show that the proposed approach achieves better lossless compression rates than existing techniques.
I NTRODUCTION
Deep neural networks (DNNs) have many successful applications ranging from computer vision, natural language processing to computational biology.
However, large DNNs usually require signiﬁcant memory and storage overhead and sometimes a large network bandwidth, which hinges their usages on mobile devices.
Running large-size neural networks also consumes a considerable amount of energy, making their deployments on battery-constrained devices difﬁcult.
Furthermore, the over-parameterization issue in DNN architectures can impair its performances.
Recent works (Han et al.
; Ullrich et al.
; Louizos et al.
and references therein) show ways to reduce the network complexity by using proper regularization or network pruning to signiﬁcantly reduce the number of parameters.
One way to convert a dense DNN into a sparse one is by applying
L 0 /L 1 regularization on the model parameters.
The L 1 penalty is computationally efﬁcient, but it also introduces more bias on the large weights and may lead to signiﬁcant degradation in model performances (Han et al., 2015).
In contrast, L 0 regularization shows better performance, but incurs much higher computational complexity due to its combinatorial nature.
Pruning, as shown in Han et al.
(2015) and Tartaglione et al.
(2018), is another promising strategy to sparsify DNNs by only keeping network connections more relevant to the ﬁnal output.
The importance of a network connection (i.e., the connection between two network nodes) can be approximated by the magnitudes or gradients of its weights.
However, such an approximation may not be accurate since it does not consider the highly nonlinear relationships between network connections and the ﬁnal output induced by the multi-layer convolutions of DNNs.
Some available network compression methods improve the computational performance with moderate to high losses in accuracy, which can be highly undesirable in many critical missions (such as autonomous driving).
In order to achieve lossless compression, we need to correctly decipher the relationships between the network connections and the ﬁnal output.
This is a challenging task because the structural nature of DNNs makes the dependence between a network connection and the network output highly nonlinear.
In this paper, we propose a probabilistic connection importance inference (PCII) method for testing whether a connection in a DNN is relevant to the DNN output.
Speciﬁcally, we introduce a new technique called probabilistic tensor product decomposition to decompose the association of two connected network nodes into two components: one related to the DNN output and the other independent of the DNN output.
If the strength of the ﬁrst component is high, we keep the network connection.
Otherwise, we delete it.
The inference is carried out by a nonparametric score test, which is based on modeling the log-transformed joint density of the connected nodes and the ﬁnal output in a tensor product reproducing kernel Hilbert space (RKHS).
We further derive the asymptotic distribution of the proposed test statistics, thus avoiding the computationally intensive resampling step encountered in most nonparametric inference settings.
We implemented the method and tested it on image classiﬁcation tasks, in which our approach achieved the highest lossless compression rates.
Section 2 reviews relevant literature; Section 3 introduces the PCII method and algorithm; Section 4 establishes theoretical properties for using the method to infer dependence between a network connection and the DNN output; and Section 5 shows the experimental results and concludes with a short discussion.
R ELATED W ORKS
Zhu & Gupta (2017) found that a DNN can be greatly sparsiﬁed with minimal loss in accuracy.
One strategy for sparsifying DNNs is to shrink small weights to zero.
Along this line of thinking, Louizos et al.
(2017) introduced a smoothed version of L 0 regularization aiming to be both computationally feasible and beneﬁcial to generalizability.
There are also some regularization methods based on
Bayesian formulations.
Ullrich et al.
(2017) proposed to add a Gaussian mixture prior to the weights.
The sparsity is achieved by concentrating weights to cluster centers.
Molchanov et al.
(2017) proposed variational dropout, which learns individual dropout rate based on the empirical Bayes principle.
Also, as shown in Han et al.
(2015), the performance of pruning and retraining is related to choice the correct regularization, such as L 1 or L 2 regularization.
PCII offers a means to prune a DNN by keeping network connections that are the most relevant to the DNN output.
This idea goes back to the optimal brain damage work by LeCun et al.
It shows that among the many parameters in the network, many are redundant and do not contribute signiﬁcantly to the output.
Later, Hassibi et al.
(1993) proposed the optimal brain surgeon method, which leverages a second-order Taylor expansion to select parameters for deletion.
Most recently,
Han et al.
(2015) proposed a three-step pruning method.
Tartaglione et al.
(2018) proposed to prune a DNN based on the sensitivity score.
There are also several approaches targeting at sparsifying convolution layers.
For example, Anwar et al.
(2017) proposed to prune feature maps and kernels in convolution layers.
Comparing with existing pruning methods based on the magnitude or gradient of weights, our approach directly models the relationship between a network connection and the network output in a nonparametric way.
In addition, our inference is based on a statistical hypothesis testing, which outputs p -values to quantify the dependence strength of each network connection to the DNN output.
The use of p -values allows network connections to be treated in a uniﬁed way and alleviates the need of ad hoc weights normalization required in some existing approaches.
P ROBABILISTIC C ONNECTION I MPORTANCE I NFERENCE
In this section, we establish a general procedure for building the probabilistic connection structure, in which the connections are inferred by the nonparametric score test in tensor product RKHS.
As shown in our experiments (see Section 5), our technique not only sparsiﬁes the network but improves its generalization ability.
3. PCII ON FULLY CONNECTED LAYERS
Without loss of generality, we let a feed-forward acyclic DNN (Figure 1) be composed of T layers with X t being the input of the t -th network layer, where t = 0 , 1 , . . .
, T .
Let t = 0 and t =
T indicate the input and output layers, respectively, and let 0 < t < T index the hidden layers.
The collection of all the nodes is denoted as G and the collection of all network connections is denoted as E .
We use a pair of nodes to denote a connection in E .
For example, ( X t, 1 , X t +1 , 1 ) denotes an edge from the 1 st node in the t -th layer to the 1 st node in the ( t + 1 )-th layer.
For simplicity, we use
Y to denote the output layer, who takes on categorical values for a classiﬁcation task, and takes on continuous values for regression.
The output of the ( t + 1 )-th layer can be described as
[Equation]
w [Equation]
X [Equation] ,i ) , for j = 1 , . . , m t +1 ,
𝑋 𝑡, 𝑋 𝑡, 𝑋 𝑡,𝑚 𝑡 𝑋 𝑡+1, 𝑋 𝑡+1, 𝑋 𝑡+1,𝑚 𝑡+ 𝑌 𝑋 𝑡,1 (𝛼) 𝑋 𝑡+1,𝑚 𝑡+1 (𝛽) 𝑌 𝜂 𝛼,𝛽 + 𝜂 𝛼,𝛽,𝑦
Connection
Connection is associated with Y
If 𝜂 𝛼,𝛽,𝑦 is not zero where m t denotes the number of nodes in the t -th layer, and g t +1 is the activation function. It should be noted that the magnitude of weights is not necessarily the best indication of the impact of the corresponding network connection. A more preferable way is to directly model the relationship between a network connection and the ﬁnal output of a DNN. For simplicity, we use ( α, β ) to denote an arbitrary network connection.
Due to the high non-linearity of a DNN, we use the nonparametric function to model the relationship among α , β and Y .
Let us denote the joint density of α , β and Y as f ( α, β, y ) and the log-transferred density as η ( α, β, y ) .
We assume that η ( α, β, y ) can be deposed as
[Equation]
where η α , η β , and η y are marginal effects, η α,y , η β,y , and η α,β are the two-way interaction terms, η α,β,y is the three-way interaction term.
Here, we interpret the connection as the interaction of α and β , i.e. η α,β + η α,β,y .
Speciﬁcally, η α,β is the interaction effect of α and β without the impact of y , and η α,β,y characterizes the interaction of α , β impacted by y .
Our aim is to measure the signiﬁcance of the connection by how much it is related to Y .
To model this problem mathematically, we measure the association between the connection and Y by the signiﬁcance of the three-way interaction η α,β,y .
Therefore, inferring whether the connection is related to the ﬁnal output Y or not is equivalent to testing whether η α,β,y = 0 or not.
As shown in Figure 1, the connection is important for the network model if and only if the three-way interaction η α,β,y (cid:54) = 0 .
We propose a score test to qualify the signiﬁcance of this term.
The detailed construction of the statistical test is explained in Section 4.2.
3. PCII ON CONVOLUTIONAL LAYERS
For different activation functions and type of layers, we have modiﬁcations to adopt the speciﬁc structure.
Here, we generalize the proposed PCII test to handle convolutional layers, which are critical in many modern neural network architectures such as VGG16 Simonyan & Zisserman (2014).
As demonstrated in Li et al.
(2016), sparsifying the ﬁlter has little effect on reducing the computation cost.
Nevertheless, reducing the volume of ﬁlters can greatly increase computing efﬁciency.
For example, the volume of a ﬁlter is 3 × 3 × 4 .
There are four 3 × 3 slices.
PCII can be generalized to infer the importance of the slices, which can be treated as the connection between one channel in the current layer and one channel in the next layer.
Convolutional ﬁlters can be applied to transform channels in the t -th layer to channels in the ( t +1) -th layer.
We denote the j -th channel in the t -th layer as X t,j for j = 1 , . . .
, m t , where m t is the number of channels in t -th layer and t = 0 , . .
, T .
Each ﬁlter connects all channels in the t -th layer to one channel in the t + 1 -th layer.
Let h i,j t,t +1 denote a convolution operator that connects two channels
𝑿 𝑡,1 (𝛼) 𝑿 𝑡+1,𝑚 𝑡+1 (𝛽) 𝑌 𝜂 𝛼,𝛽 + 𝜂 𝛼,𝛽,𝑦
Connection
Connection is associated with Y
If 𝜂 𝛼,𝛽,𝑦 is not zero 3*3*4 Filter connected to 𝑋 𝑡+1,𝑚 𝑡+ 𝑋 𝑡, 𝑋 𝑡, 𝑋 𝑡,𝑚 𝑡 𝑋 𝑡+1,𝑚 𝑡+ 𝑋 𝑡+1, 𝑋 𝑡+1, ℎ 𝑡,𝑡+1 1,𝑚 𝑡+ ℎ 𝑡,𝑡+1 𝑚 𝑡 ,𝑚 𝑡+ α β
X t,i and X t +1 ,j .
Then, the ﬁlter connected to X t +1 ,j is { h i,jt,t +1 } m t
i
=1 where we denote each h
i,jt,t + as a slice of the ﬁlter.
For example, when we choose a 3 × 3 × 4 ﬁlter and set stride as one, this operator is the ﬁlter convolved (slided) across the width and height of the input { X t,i } m t
i
=1 and offset with the bias.
For one slice in the t + 1 -th layer, we can write its connection with the slices in the previous layer as
[Equation]
where g t +1 is an activation function and b jt,t +1 is the bias for the j -th ﬁlter.
As illustrated in Figure 2, the red arrow from X t, 1 to X t +1 ,m t +1 represents h 1 ,m t +1 t,t +1 , which is one slice of the ﬁlter { h i,m t +1 t,t +1 } m t i = connecting the channels in the t -th layer to channel X t +1 ,m t +1 in the t + 1 -th layer.
For simplicity, we denote α as one channel in the current layer and β as one channel in the next layer.
Since the relationship among the triplet ( α, β, Y ) is highly nonlinear, we model its log-transformed joint density as a nonparametric function η
( α, β, y ) .
Similar to the fully connected layers, we assume that η ( α, β, y ) has a decomposition as in (1).
As shown in Figure 2, the connection between α and β is decomposed to two parts: one unrelated to the output Y , η α,β , and another related to the output Y , η α,β,y .
We aim to select connections that have greater contributions to Y , which is mathematically translated into the task of assessing the signiﬁcance of the three-way interaction term η α,β,y against the null hypothesis η α,β,y = 0 .
3. A LGORITHM
In real applications, both fully connected layers and convolutional layers are used in a complex neural network architecture.
We integrate the PCII procedure described in the previous two subsections to simultaneously infer the connections in both fully connected and convolutional layers, as summarized in Algorithm 1.
We use p ( t ) ij to denote the p -value for testing the i -th node in t -th layer and j -th node in ( t + 1 )-th layer for the fully connected layers.
For convolutional layers, p ( t ) ij denotes the p-value for inferring the importance of the ﬁlter connecting the i -th slice in the t -th layer and the j -th slice in the t + 1 -th layer.
The calculation of the p-values are given in Section 4 based on our proposed Score test.
Input: A training dataset, a DNN architecture, and the desired model compression rate
Step 1: Use the training dataset to train a DNN of the given architecture.
Step 2: (a).
Importance inference of connections in fully connected layers: infer the three-way dependence effect of a network connection by testing the hypothesis H 0 :
η α,β,y =
v.s H 1 : η α,β,y (cid:54) = 0 , and calculate the test statistics (or p -values) for all network connections as p f = { p t,t +1 i,j , i = 1 , . . .
, m t , j = 1 , . .
, m t +1 | t and t + 1 layers are fully connected } .
(b).
Importance inference of connections in convolutional layers: infer the three-way dependence effect of a network connection by testing the hypothesis H 0 :
η α,β,y =
v.s H 1 : η α,β,y (cid:54) = 0 , and calculate the test statistics (or p -values) for all network connections as p c = { p t,t +1 i,j , i = 1 , . . .
, m t , j = 1 , . .
, m t +1 | t and t + 1 layers are convolutionally connected } .
Step 3: Rank all network connections by their test statistics (or p -values), and select a threshold ρ f for deleting network connections in fully connected layers and ρ c in convolutional layers to achieve the desired model compression rate (Strategies for choosing ρ f and ρ c are given in Section 5).
Step 4: Set the same initial value for non-zero weights and ﬁlters.
Retrain the sparsiﬁed DNN.
S CORE T EST AND T HEORETICAL P ROPERTIES 4. B ACKGROUND ON TENSOR PRODUCT RKHS
Consider two random variables α and β for fully connected layers or two random vectors α and β for convolutional layers.
Let Y be a random variable as the ﬁnal output.
The domains for α , β and Y are
A , B , and Y , respectively.
Here, we assume that the log-transformed joint density function η belongs to a tensor product RKHS H
= H (cid:104) α (cid:105) ⊗ H (cid:104) β (cid:105) ⊗ H (cid:104) Y (cid:105) where ⊗ denotes the tensor product of two linear space.
For marginal RKHS, H (cid:104) l (cid:105) with an inner product (cid:104)· , ·(cid:105) H (cid:104) l (cid:105) for l = α, β, Y , there exists a symmetric and square integrable function K l such that
[Equation]
where K l is called the reproducing kernel of H (cid:104) l (cid:105) for l = α, β, Y .
By Mercer’s theorem, any continuous kernel has the following decomposition
[Equation]
where the µ ν ’s are non-negative descending eigenvalues and the φ ν ’s are eigen-functions.
For the discrete domain { 1 , . . .
, a } , we deﬁne the kernel as K ( x, y )
=
1 { x = y } for x, y ∈ { 1 , . . .
, a } .
For a continuous domain, there are different choice of kernels such as Gaussian kernels and Sobolev kernels.
Note that the eigenvalues for different kernels on continuous domain have different decay rate.
For example, the eigenvalues of the Gaussian kernel have the exponential decay rate, i.e., there exists some c > 0 such that µ ν (cid:16) exp( − cν )
(Sollich & Williams, 2004); and the eigenvalues of the m -th
Sobolev kernels have the polynomial decay rate, i.e., µ ν (cid:16) ν − 2 m (Gu, 2013).
4. P ROBABILISTIC D ECOMPOSITION OF TENSOR PRODUCT RKHS
Next we propose the probabilistic tensor sum decomposition for each marginal RKHS, H (cid:104) l (cid:105) , for l = α, β, Y .
We ﬁrst use Euclidean space as a simple example to illustrate the basic idea of tensor sum decomposition.
Recall that the tensor sum decomposition is often called ANOVA decomposition in linear model.
For example, for the d -dimensional Euclidean space, we denote f as a vector and let f ( x ) be the x -th entry of the vector for x = 1 , . . .
, d .
We denote A as an average operator deﬁned as
A f ( x ) = (cid:10) 1 d 1 , f (cid:11) .
The tensor sum decomposition of the Euclidean space R d is
[Equation]
where the ﬁrst space is called the grand mean and the second space is called the main effect.
Then, we construct the kernel for R d 0 and R d 1 in the following lemma.
Lemma 1.
For a RKHS space H , there corresponds a unique reproducing kernel K , which is non-negative deﬁnite.
Based on the tensor sum decomposition H
= H
⊕ H 1 , where H
= { 1 /d 1 } and H 1 = { g ∈ H : E X ( g ( X ))
= 0 } , we have the kernel for H 0 as
[Equation]
x
[Equation]
and kernel for H 1 as
[Equation]
where 1 denotes the indicator function.
However, in RKHS with inﬁnite dimension, the grand mean is not a single vector.
Here, we set the average operator A as A :
= f
→ E x f ( x
) =
E x (cid:104) k x , f (cid:105) H = (cid:104)
Ek x , f (cid:105) H where k is the kernel function in H and the ﬁrst equality is due to the reproducing property.
E x k x plays the same role asd 1 in Euclidean space.
Then, we have the tensor sum decomposition in a functional space deﬁned as
H =
[Equation]
Following the same fashion, we call H
as the grand mean space and H 1 as the main effect space.
Notice that E x k x is also known as the kernel mean embedding which is well established in the statistics literature Berlinet & Thomas-Agnan (2011).
Then we introduce the following lemma to construct the kernel function for H 0 and H 1 .
Lemma 2.
For RKHS space H , there corresponds an unique reproducing kernel K , which is nonnegative deﬁnite.
Based on the tensor sum decomposition H
= H
⊕ H 1 where H {+
+}
= { E x k x } and
H 1 = { g ∈ H : E x
( g ( x ))
= 0 } , we have the kernel for H 0 as k
[Equation]
and the kernel for H 1 as
[Equation]
In neural networks, A , B are in a continuous domain.
The ﬁnal output Y can be either in continuous domain or discrete domain, which depends on the tasks.
Here, we construct the tensor sum decomposition for discrete domain and continuous domain based on Lemma 1 and Lemma 2 respectively.
Speciﬁcally, we have spaces H (cid:104) α (cid:105) , H (cid:104) β (cid:105) and
H (cid:104) Y (cid:105) decomposed as tensor sums of subspaces H (cid:104) α (cid:105)
=
H (cid:104) α (cid:105)
[Equation]
Following Gu (2013), we apply the distributive law and have the decomposition of H as
[Equation]
) ≡H 000 ⊕ H 100 ⊕ H 010 ⊕ H 001 ⊕ H 110 ⊕ H 101 ⊕ H 011 ⊕ H 111 .
(8) where H ijk = H (cid:104) α
(cid:105) i ⊕ H (cid:104) β (cid:105) j ⊕ H (cid:104) Y (cid:105) k .
Lemma 3.
Suppose K (cid:104) 1 (cid:105) is the reproducing kernel of H (cid:104) 1 (cid:105) on X 1 , and K (cid:104) 2 (cid:105) is the reproducing kernel of H (cid:104) 2 (cid:105) on X 2 .
Then the reproducing kernels of H (cid:104) 1 (cid:105) ⊗ H (cid:104) 2 (cid:105) on X
=
X 1 ×
X 2 is K ( x , y ) =
K (cid:104) 1 (cid:105) ( x (cid:104) 1 (cid:105) , y (cid:104) 1 (cid:105) )
K (cid:104) 2 (cid:105) ( x (cid:104) 2 (cid:105) , y (cid:104) 2 (cid:105) ) with x = ( x (cid:104) 1 (cid:105) , x (cid:104) 2 (cid:105) ) and y =
( y (cid:104) 1 (cid:105) , y (cid:104) 2 (cid:105) ) .
Lemmas 3 can be easily proved based on Theorems 2.6 in Gu (2013).
Lemma 3 implies that the reproducing kernels of the tensor product space is the product of the reproducing kernels.
Based on these three lemmas, we can construct kernel for each subspace deﬁned in (8).
4. S CORE T EST
Based on (8), the log-transformed density function η ∈ H can be correspondingly decomposed as (1).
Thus, η α,β,Y =
if and only if η ∗ ∈ H 0 :
=
H 000 ⊕ H 100 ⊕ H 010 ⊕ H 001 ⊕ H 110 ⊕ H 101 ⊕ Hwhere η ∗ is the underlying truth.
Hence, we focus on the following hypothesis testing problem:
[Equation]
where H\H 0 denotes set difference of H and H 0 .
We next propose a likelihood-ratio based procedure to test (9).
Suppose that t = ( α, β, y ) and t i =
( α i , β i , y i ) , i = 1 , . . . , n , are iid observations generated from T
=
( A , B , Y ) .
Let LR n be the likelihood ratio functional deﬁned as
[Equation]
where P H 0 is a projection operator from H to H 0 .
Using the reproducing property, we rewrite (10) as
[Equation]
where K is the kernel for H and K 0 is the kernel for H 0 .
Then we calculate the Fréchet derivative of the likelihood ratio functional as
[Equation]
where K 1 is the kernel for H 111 .
We deﬁne our test statistics as the squared norm of the score function of the likelihood ratio functional as
[Equation]
By the reproducing property, we can expand the left hand side of (13) as
[Equation]
We observe an interesting phenomenon that S 2 n has a similar expression with the MMD (Gretton et al., 2012) when Y is a binary variable.
When Y ∈ { 0 , 1 } , the kernel on H (cid:104) Y (cid:105) is K (cid:104) Y (cid:105)
( y i , y j )
= 1 { x = y } − 1 / 2 .
Assume that the kernel for H
[Equation]
α
[Equation]
⊗
[Equation]
is
[Equation]
( x , x ) for x ∈ A × B .
By
Lemma 3, we have K 1 ( t i , t j ) =
K (cid:104) Y (cid:105)
[Equation]
( x i , x j ) .
Then we have 8 S 2 n as
[Equation]
If we replace K (cid:104)
[Equation]
[Equation]
with K (cid:104) α,β (cid:105) , i.e., the kernel on the H (cid:104)
α (cid:105) ⊗ H (cid:104) β (cid:105) , the right hand side of (15) is the MMD measuring the distance between the joint distribution of α and β in the group with y
=and the joint distribution of α and β in the group with y = 1 .
[Equation]
Since we want to infer the importance of the connection between the α and β , we are only interested in comparing whether the interaction effect between α and β changes or not in the two groups.
The main effects of α and β only contribute to the importance of the nodes or slices and are not relevant to the connection between α and β .
4. C ALCULATION OF THE T EST S TATISTICS
We introduce a matrix form of the squared norm of score function for manifesting the computation process.
In (14), S 2 n is determined by the kernel on th H (cid:104)
α
[Equation]
⊗ H
[Equation]
⊗ H
[Equation]
Y
[Equation]
By Lemma 1 and
Lemma 2, we replace the expectation by the sample average and get the kernel function for H (cid:104) l (cid:105) l as
[Equation]
where k l is the kernel function for H (cid:104) l (cid:105) for l = α, β, Y .
Some popular choices of the kernel functions are Gaussian kernel, Laplace kernel and polynomial kernel.
Let K l be the empirical kernel matrix.
We can rewrite (14) as
[Equation]
17) where H =
I n − 1 n 11 T , I n is a n × n identity matrix and 1 n is a n × 1 vector of ones, and [ A ] ++ = (cid:80) ni =1 (cid:80) nj =1 A ij .
This test statistics is also related to three-way Lancaster’s interaction measure (Lancaster, 2002).
Fortunately, the computation of (17) only involves the matrix multiplication, the computational procedure can be executed in parallel on GPU.
In our experiment, the highly parallelized matrix operation tremendously alleviated the computing time: essentially reducing from quadratic to almost linear.
In addition, we can reduce the sample size by sub-sampling r << n data points.
Sub-sampling the dataset is a popular technique to reduce computational burden and can efﬁciently approximate the full data likelihood in a regression setting (Ma et al., 2015; Drineas et al., 2011).
However, for very large data set, the sub-sampling is not efﬁcient.
We consider a mini-batch algorithm to calculate the score in each batch and used the averaged score as the ﬁnal test statistics.
This is also related to the divide-and-conquer method which is widely used in kernel-based learning (Zhang et al., 2013; Shang & Cheng, 2017; Liu et al., 2018).
4. A SYMPTOTIC DISTRIBUTION
The asymptotic distribution of S 2 n depends on the decay rate of the kernel of the product of RKHS.
Suppose that { µ (cid:104) α (cid:105) ν , φ (cid:104) α (cid:105) ν } ∞ ν =1 is a series of eigenvalue and eigenfunction pairs for H (cid:104) α (cid:105) 1 , { µ (cid:104)
β (cid:105) ν , φ (cid:104) β
(cid:105) ν } ∞ ν = is a sequence of basis for H (cid:104) β (cid:105) 1 .
If Y is continuous, we suppose that H (cid:104) Y (cid:105) has the eigensystem, { µ (cid:104) Y (cid:105) ν , φ (cid:104)
α (cid:105) ν } ∞ ν =1 .
If Y is categorical, we suppose that H (cid:104) Y (cid:105) has the eigensystem, { µ (cid:104) Y (cid:105) ν , φ (cid:104)
α (cid:105) ν } a − 1 ν =1 .
Then we have the eigenvalue eigenfuntion pair for the tensor product space H (cid:104) α (cid:105)
[Equation]
as
[Equation]
where ν α = 1 , . . .
, ∞ , ν β = 1 , . . .
, ∞ , ν Y = 1 , . . .
, ∞ (Y is continuous), and ν Y = 1 , . .
. , a −(Y is categorical with a categories).
For simplicity, we order the pairs in the decreasing order of µ ρ , ρ = 1 , . . .
, ∞ .
The null hypothesis could be interpreted as factorization hypothesis, i.e., ( X, Y ) ⊥ Z ∨
( X, Z ) ⊥ Y ∨ ( Y, Z ) ⊥ X or X, Y, Z are mutually independent.
Theorem 1.
Suppose the kernel on H 111 is square integrable.
If Y is continuous variable, then under H 0 , we have
[Equation]
where (cid:15) ρ are i.i.d.
standard Gaussian random variables.
The proof of this theorem is shown in the Supplementary Materials A.1.
The asymptotic distribution of nS 2 n only depends on the eigenvalues of the kernel.
Theorem 1 is related to Wilks’ phenomenon demonstrated in the classic nonparametric/semiparametric regression framework (Fan et al., 2001;
Shang & Cheng, 2013), i.e., the asymptotic distribution is independent of the nuisance parameters.
In practice, we ﬁx the same kernel for the fully connected layers and the same kernel for the convolutional layers.
Thus, it provides a uniﬁed importance measure for all connections in the same type of layer, which avoids the scaling problem faced by those pruning methods that use magnitudes of weights as an importance measure.
In addition, we can use the value of the test statistics as an importance measure for pruning and bypass the effort of calculating the p-values since the order is the same according to either of these two measures.
R ESULTS
We conducted experiments to test out the PCII method in two supervised image classiﬁcation tasks:
MNIST and CIFAR10.
We used TensorFlow to train DNNs.
The back-end of PCII was implemented in Fortran and R language.
Programming interfaces were implemented to connect the back-end calculations to TensorFlow.
The experiments were run on a computer with one Nvidia Titan V GPU and 48 CPU cores.
PCII offers a convenient way to adjust compression rate by changing the p -value threshold ρ .
For other compression methods in consideration, the compression rate is usually controlled by some hyper-parameters, which need to be tuned via an ad hoc trial and error strategy.
Two types of comparisons were carried out.
In the ﬁrst type, we adjust the compression rate of a method while requiring its compressed DNN to achieve the test accuracy of the original uncompressed DNN.
We term this the lossless compression rates (LCR).
Since it is very time-consuming to obtain an exact test accuracy, we allow a 0.01% deviation from the desired test error rates.
In the second type, we compared the minimum testing error (MTE) of the compressed DNNs produced by different model compression methods.
MTE shows how a compression method can help increase the generalizability and robustness of a DNN.
We only included the results of the tested methods that we could obtain their working codes to reproduce the results reported in their original papers.
In addition, we did not include methods that are not able to achieve the LCR of the corresponding test dataset.
5. MNIST D ATASET
We tried two network architectures for MNIST (60k training images and 10k test images): a multilayer perceptron (MLP) with two hidden layers of sizes 300 and 100, respectively, and a convolutional neural network LeNet-5 (LeCun et al.
We trained LeNet-300-100 for 100 epochs to achieve a test error of 1.69%.
These approaches include two regularization based methods in Louizos et al.
(2017), as well as several pruning based methods in Han et al. (2015), and Guo et al.
The results are summarized in Table 1.
PCII achieved the lowest MTE when the compression rate was 10 x. Then, we further increased the compression rate until the error rate reached 1 . 70% .
The results show that PCII not only compressed a medium-sized MLP better than existing methods but also was able to improve generalizability of a MLP via compression (i.e., the MTE is better than the LCR).
[Table]
Interestingly, our inference results can also help in interpreting the ﬁtted neural network.
For example, through inferring the importance of the connections between input layer and the the ﬁrst hidden layer, we can visualize the importance of the features in the input layer.
Figure 3 plots the p -values of the associations between the network connections in the ﬁrst layer and the ﬁnal output.
The heatmap shows that a banded structure repeated 28 times, in which the central region tends to have smaller p -values.
The left and right margins of the heatmap show that the connections on the ﬁrst and last few channels are less relevant to the ﬁnal output (i.e., have large p -values).
This phenomenon is observed because a written digit is usually located in the central part of a image.
The LeNet-5 model ( https://goo.gl/4yI3dL ) is a modiﬁed version of LeCun et al.
It includes two convolutional layers with 20 and 50 ﬁlters, respectively, and a fully connected layer with 500 nodes.
The results are summarized in Table 2.
PCII achieved both the lowest MTE and highest LCR, again, for this model, demonstrating the broad applicability of the PCII strategy for various neural network architectures. [Table]
5. CIFAR10 D ATASET
To demonstrate the applicability of PCII to complex DNNs [Table] we applied it to VGG16 (Zagoruyko, 2015), which was adapted for the CIFAR-10 dataset (Krizhevsky & Hinton, 2009).
The network consists of 13 convolutional and two fully-connected layers.
The results are summarized in Table 3.
The test error gradually decreases as the compression rate increases from 1 x to 3 x [Table] and achieves the MTE at 6 .
When the compression rate is further increased, the test error begins to increase.
As the compression rate reaches 10 x, the test error increases to 7 .
56% that is comparable to the test error of the uncompressed VGG16.
For this dataset, we only include the result for PCII due to the limited resources.
In fact, we could not obtain the results for other methods in comparison in three days’ computing time.
[Table]
D ISCUSSION
We propose a statistically principled strategy to directly infer the importance of a connection in a neural network via PCII, a hypothesis testing framework.
The proposed PCII test provides a p-value based measure on the importance of the connection through the connection’s association with the ﬁnal output.
This type of direct quantiﬁcation cannot be easily accomplished by the magnitude-based pruning method.
Although the two examples are relatively small in size, they demonstrated the broad applicability of the PCII method and its improved power in network compression.
Last but not least we note that the PCII testing method can be easily generalized to a broad class of connection types including the skip layer connections in RNN
A S UPPLEMENTARY M ATERIALS A. P OOF OF T HEOREM
Let { µ ρ , φ ρ } ∞ ρ =1 be the eigenvalues and eigenvectors for the tensor product space H (cid:104) [Equation] [Table] [Equation]
[Table]
[Equation] (cid:104) Y [Table] By the dedomposition deﬁned (8), we have
[Equation]
(19) for ρ = 1 , . . .
, ∞ , i.e., the mean of the eigenfunction φ ρ is zero.
By simple calculation, we have ∞
[Equation]
for the exponential decayed kernels and polynomal decayed kernel with decay rate as i − m for m > 1 .
Thus, the commonly used kernels such as Gausssian kernel, Laplase Kernel and linear or qudratic
Solblev kernel satisfy this requirement.
By Mercer’s theorem, we have the decomposition as
[Equation]
where (cid:15) ρ are i.i.d.
standard normal random variables.
The last row is proved by applying the
Lindeberg–Lévy CLT to have (cid:80) ni =1 φ ( t i ) d −→ (cid:15) ρ since (1) and (2) holds.
Then, by the Kolmogorov’s inequality and (cid:80) ∞ ρ =1 µ ρ < ∞ , we have the last row holds.