U NDERSTANDING D IVERSITY B ASED N EURAL N ETWORK P RUNING IN T EACHER S TUDENT S ETUP
A BSTRACT
Despite multitude of empirical advances there is a lack of theoretical understanding of the effectiveness of different pruning methods.
We inspect different pruning techniques under the statistical mechanics formulation of a teacher-student framework and derive their generalization error (GE) bounds.
In the ﬁrst part we theoretically prove empirical observations of a recent work that showed Determinantal
Point Process (DPP) based node pruning method is notably superior to competing approaches when tested on real datasets.
In the second part, we use our theoretical setup to prove that the baseline random edge pruning method performs better than the DPP node pruning method, consistent with the ﬁnding in literature that sparse neural networks ( edge pruned ) generalize better than dense neural networks ( node pruned ) for a ﬁxed number of parameters
I NTRODUCTION
Deep neural networks have achieved impressive results in a wide variety of applications such as classiﬁcation Krizhevsky et al.
; Liu et al.
image processing Litjens et al.
Badrinarayanan et al.
(2017), natural language processing Devlin et al.
; Deng & Liu (2018);
Socher et al.
(2013), etc.
Most of these networks use millions of parameters which makes inference computationally expensive and memory intensive Devlin et al.
To address this, researchers explore pruning techniques with the primary goal of comparing performance on real datasets.
The broad scientiﬁc paradigm explored by most pruning techniques is to empirically and heuristically determine either how to prune a network or what to prune in a network (sometimes both).
In this work, we take a step towards theoretical understanding of these two prime aspects of pruning methods.
We compare the quality of different pruning methods for feedforward neural networks under the teacher-student framework Saad & Solla (1995a;b; 1997); Goldt et al.
in the thermodynamic limit (input dimension goes to inﬁnity) using generalization error bounds (GE), a theoretical measure of performance of machine learning models on unseen test data Vapnik (1999).
A fairly recent work by Mariet & Sra (2016) empirically investigates a Determinantal Point Process (DPP) based node pruning technique Macchi (1975); Kulesza et al.
In the ﬁrst part, we provide theoretical guarantees for their empirical observations thereby taking a step towards theoretical understanding of the question: how to prune?
A very recent review Blalock et al.
(2020) discusses [Table] [Table] empirical results across several papers ( 81 research articles) to conclude that sparse models obtained after edge/connection (used interchangeably) pruning outperforms dense ones obtained after node pruning for a ﬁxed number of parameters.
In the second part, we extend our theoretical setup and compare node and edge pruning techniques thereby addressing the question: what to prune?
O NLINE L EARNING IN T EACHER -S TUDENT S ETUP G OLDT ET AL . (2019):
We use a two-layer perceptron which has N input units, M hidden units and 1 output unit as the teacher network to generate labels for i.i [Table] d Gaussian input, x t = ( x t 1 , . . .
, x tN ) where x ti ∈ N (0 , 1) ∀ i ∈ { 1 , . . .
, N } .
Let θ ∗ = { w ∗ ( ∈ R M × N ) , v ∗ ∈ R M } denote the ﬁxed parameters of the teacher network.
The label y t of the input x t ( t = 1 , 2 , . . . ) is given as, y t =
(cid:80)
[Equation]
g (cid:16)
[Equation]
(cid:17) + σζ t , where ζ t ∼ N (0 , 1) is the output noise, and g is the sigmoid activation function.
The input and teacher generated labels ( { ( x 1 , y 1 ) , . . . } ) are used to train a two-layer student network with N input units, K hidden units ( K ≥ M ) and 1 output unit using online SGD learning method.
We consider the quadratic training loss, i.e., L ( θ ) =(cid:104)(cid:80)
Kk =
[Equation]
k g (cid:16) w k x t √ N (cid:17) − y t (cid:105) 2 , where θ = { w , v } denotes the parameter of the student network.
Goldt et al.
(2019) showed that GE (cid:15) ( f ) (expected error on the unseen data, for details see S31 of Goldt et al.
for the overparameterized student network is a function of the order parameters , which are Q ik = w Ti w k N ,
[Equation]
Intuitively, these order parameters measure the similarities between and within the hidden nodes of teacher and student networks.
All the assumptions of this setup are stated in the Appendix B.
N EURAL N ETWORK P RUNING IN T EACHER S TUDENT S ETUP
In this work we consider three node pruning methods and one baseline edge pruning method (DPP, random and importance node pruning and random edge pruning, see Table 1 for notations).
We ﬁrst prune the overparameterized student network using these various pruning methods.
We then compare the performance of the pruned student networks by analyzing their GE bounds following
Goldt et al.
(2019) (concept explained in Figure 1).
For node and edge pruning comparison, we choose the parameters k n and k e (see Table 1) such that the total number of parameters of the networks remain same, i.e.,
[Equation]
where c ∈ [0 , 1] is a constant.
3. C OMPARING N ODE P RUNING M ETHODS
Following Goldt et al.
we observe that the fully trained student network achieves 0 GE when there is no added output noise in the teacher network (see Sec. 2).
However, the GE of the pruned network increases due to underparameterization.
We theoretically show that this increment in GE due to DPP node pruning is less than that for random and importance node pruning methods.
This justiﬁes the empirical ﬁndings of Mariet & Sra (2016).
Our result is formalized in Threorem 1. [Table] >
D P P N o d e P r u n i n g
[Equation] [Table] Input [Table]
>
Input
[Equation]
[Table]
/izEfaR7UbpbU7fgLt+8SmqFvOvk3YezTPl60UcanIBTYAMXXIAyuAUVUAUYvIA3MAYfxqvxbnwak3k0ZSxmjsEfGF/f0s2iCg==</latexit> hidden all 3 teacher nodes which is enough to outperform DPP node pruning.
All notations follow Table
Theorem 1.
Assume ( A 1) − ( A 7) (see Appendix B).
Then for k n ≤ M we have,
[Equation]
and E [Equation] ˆ [Equation]
Rand Node k n
[Equation]
f )
[Equation]
f
[Equation]
[Equation]
and,
[Equation]
i.e., DPP node pruning outperforms random node pruning in the above setup.
Here the expectation is taken over the the subsets of hidden nodes of size k n chosen u.a.r (see Table 1 for the notations).
Proof Idea of Theorem 1: From Goldt et al.
we notice that in the overparameterized setting (i.e., K > M ), multiple student hidden nodes learn a single teacher hidden node.
This results in an equivalence relation over the set of student hidden nodes.
As a consequence, the order parameters denoting the correlation among the student and teacher hidden nodes have block structure where each block corresponds to the set of student hidden nodes which learn the same teacher hidden node (see Figure 3 in Appendix).
We ﬁrst observe that the expected kernel of the DPP node pruning is same as the order parameter Q and hence block diagonal.
Then, using property of DPP, we prove the
DPP pruning method will retain a subset of student hidden nodes with at most 1 hidden node from each block when k n ≤ M .
However, in random and importance node pruning, two student nodes from the same block may survive after pruning with non-zero probability.
Hence, more teacher nodes may remain unexplained by the student network after random or importance node pruning, resulting in increased GE (details in Appendix C).
3. C OMPARING N ODE AND E DGE P RUNING M ETHODS
In random edge pruning method, for each student hidden node, an incoming edge is kept with probability c =
lim N →∞ k e N .
Majority of empirical studies throughout literature use random edge or node pruning as a baseline for empirical comparison (see papers in Blalock et al.
making it an obvious candidate for our theoretical comparisons as well.
It has been shown empirically by Mariet & Sra (2016) and theoretically by us that DPP node pruning is an above baseline node pruning method.
In this section we show that baseline random edge pruning outperforms DPP node pruning which is consistent with the empirical observations that sparse models outperform dense models (section 3.2 of Blalock et al.
Speciﬁcally, here we show that GE after random edge pruning is less than GE after DPP node pruning which is formalized below.
Theorem 2.
Assume ( A 1) − ( A 7) (see Appendix B).
Let k n and c satisfy equation 1, and 0 ≤ c ≤ 1 Z
and Z (= KM ) ≥ 4 .
Then
[Equation]
[ f ]) , (4)
[Figure]
Proof Idea of Theorem 2: When k n ≤ M , node pruned student network leaves at least ( M − k n ) teacher nodes unexplained, whereas after random edge pruning, student network can retain at least partial information about every teacher node (see Figure 1 (B)).
After a pruning routine, the sum of partial information about all teacher nodes in an edge pruned student network dominates the sum of information for the explained subset of teacher nodes in a node pruned student network.
S IMULATIONS
We run the DPP node, random edge/node, and importance node pruning simulations under the teacher-student setup.
For all the simulations, we sampled the 800000 i.i.d input samples from
N (0 , 1) as training data and 80000 as testing data.
Following notations from Section 2, we set
M = 2 , K = 6 , N = 500 , and v ∗ = 4 .
The ﬁrst layer teacher network weights w ∗ and all the student network parameters θ = { w , v } were drawn independently from N (0 , 1) as initialization.
We choose learning rate η = 0 .
50 , and it is scaled to η √ N for w and ηN for v .
We run the simulations for both noiseless ( σ = 0 in Sec. 2) and noisy ( σ = 0 .
25 ) output labels.
For comparisons between node and edge pruning, we use the node-to-edge ratio [ 1 : 83 , 2 : 166 , 3 : 250 , 4 : 333 , 5 : 417 , 6 : 500 ] to keep the number of parameters the same, given N = 500 , K = 6 , and M = 2 .
In addition, we run the same simulation with K = 5 and M = 20 to compare random Edge Pruning with DPP node pruning.
We observe that 1) DPP node pruning outperforms random and importance node in both noisy and noiseless case (see Figure 2A), which conﬁrms Theorem 1. 2)
Random edge pruning is better than DPP node pruning for c ≤ 1 Z (= 0 . 25) with Z = 4 and M
= 5 in both noisy and noiseless cases (see Figure 2B), validating Theorem 2.
D ISCUSSION AND F UTURE W ORK
All our theoretical results have been proved on single hidden layer neural networks which gives future scope of extending them to multiple hidden layer networks.
Throughout this work, we focus only on pruning methods in which a feedforward pre-trained neural network is pruned once without retraining.
We choose this class because it is feasible to make theoretical comparisons with closed form solutions of GE.
The various existing pruning methods can be broadly subsumed into a couple of categories Blalock et al.
(2020), mainly governed by the principles of pruning heuristics.
First category is the magnitude-based approaches which are not only good and common baselines in the literature but they also give comparable performance to other methods such as the gradient-based methods Lee et al.
; Yu et al.
; Blalock et al.
Another category is the random pruning which serves as an useful baseline for showing superior performance of any other pruning technique.
We hence show all our theoretical results w.r.t these two categories.
We do not focus on any speciﬁc algorithm within these categories but explore the general concept for theoretical results.
Any speciﬁc algorithm based theoretical understanding can also be an extension of this work.
A D ETERMINANTAL P OINT P ROCESS (DPP)
DPP Macchi (1975) is a probability distribution over power set of a ground set G , here ﬁnite.
DPP is a special case of negatively associated distributions Joag-Dev et al.
(1983) which assigns higher probability mass on diverse subsets.
Formally, a DPP with a marginal kernel L ( ∈ R |G|×|G| ) is:
P [ Y = Y ] = det( L Y ) det( L + I ) , where Y ⊆ G and L Y is the principal submatrix deﬁned by the indices of
Y .
We use k -DPP to denote the probability distribution over subsets of ﬁxed size k .
Remark: DIVNET denotes DPP node pruning with reweighting as in Mariet & Sra (2016).
B A SSUMPTIONS
Our theoretical results assume Goldt et al.
(A1)
If x = ( x 1 , . .
, x N ) is an input then x i ∈ N (0 , 1) .
Also, N → ∞ .
(A2)
Both the teacher and the student networks have only one hidden layer.
(A3) K ≥ M and K =
Z · M where Z ∈ Z + .
(A4)
The activation in the hidden layer is sigmoidal for both teacher and student network.
(A5)
The output ∈ R (i
e., regression problem).
(A6)
The order parameters (see section 2) satisfy the ansatz as in (S58) - (S60) of Goldt et al.
(A7)
No noise is added to the labels generated by the teacher network, i.e., σ = 0 in Section A.
With the above assumptions, authors of Goldt et al.
(2019) gave a closed form of the GE as follows:
[Equation]
where,
[Equation]
where Q, R, T are the order parameters as deﬁned in main text.
We also have the assumption equation 1 about the relation between number of edges and nodes kept after pruning.
C P ROPERTIES OF DPP K ERNEL
In main text we see that each node in the hidden layer of a student network carries certain amount of information about the training data and it is captured in a vector form.
We create an information matrix by accumulating the information vectors of these hidden nodes.
For simplicity of theoretical analysis, we have considered the kernel as the inner product of the information matrix.
In the thermodynamic limit, the inner product is divided by the input dimension.
Formally, if h i and h j are the information at i th hidden node and j th hidden node respectively, then
[Equation]
A
B
C 1.
D    where n is the total number of training examples.
It can be seen that the analysis for the kernel deﬁned in main text is similar.
Note that all analyses are for the student network trying learn from the teacher network.
Refer to main text for details of notations.
Lemma 1.
Assume (A1) - (A7).
Then the expected kernel of DPP Node for the hidden layer is the order parameter Q .
Proof of Lemma 1.
For the two-layer teacher-student setup, the hidden layer gets information ( h 1 , . .
, h K ) from the input layer, where h i =
( h i 1 , . . .
, h in ) and h ij (=
t
Tj w i ) is the information at i th hidden node on j th input data ( t j ).
Hence, h Ti [Equation]
j
[Equation] n (cid:88) k
=
[Equation]
n (cid:88)
[Equation]
n (cid:88)
[Equation]
n (cid:88)
[Equation]
But for the given input distribution (i.i.d.
Gaussian), E
[ t k t Tk ] =
I N × N .
Hence, lim N →∞ E [ L ij ] =
lim N →∞ E [ 1 N 1 n
h Ti h j ] =
lim N →∞ 1 N w Ti w j = Q ij , and we have the lemma.
From Goldt et al.
we know that Q is a block diagonal matrix where each “block” (or “group” used interchangeably henceforth)
refers to the set of student hidden nodes that represent (explian/learn) one particular teacher hidden node.
D P ROOF OF THE T HEOREMS D. R EQUIRED L EMMAS
We use the following lemmas to prove the main theorems.
Lemma 2.
Assume ( A 1) − ( A 7) .
Let k n ≤ M nodes are selected by the DPP Node pruning method,
[Equation]
and
[Equation]
Lemma 3.
Assume (A1)-(A7).
Let t 1 , . .
, t M denote the teacher hidden nodes and l 1 , . . .
, l M denote the number of student hidden nodes in a node pruned network which learnt the corresponding teacher node.
If
[Equation]
Mm =1 l m ≤ M, then the GE of this node pruned network is,
[Equation]
Lemma 4.
Assume ( A 1) − ( A 7) .
Consider the random edge pruning method with parameter lim N →∞ k e N =
c
(here c is a constant between 0 and 1).
Then the GE (cid:15)
Rand Edge c
[Equation]
E
[Equation]
f
[Equation]
) is,
[Equation]
Lemma 5.
Let v ∗ denotes the weight of the second layer of the teacher network and { v 1 , · · · , v K } be the weights of the student network after convergence.
Then in the noiseless case for all n we have,
[Equation]
D. P ROOF OF THE T HOREMS
Proof of Theorem 1.
Now, we will prove Theorem 1.
We will show, for any network pruned by
Random Node, the GE is more than the expected GE of DPP Node pruning.
Recall the randomly pruned network f discussed in the beginning of the proof.
From Lemma 3 we can see that for node pruning the GE only depends on the number of nodes survived in each block.
From equation 16 we have,
[Equation]
where equation 12 follows from the inequality below:
[Equation]
which proves the ﬁrst part of Theorem 1.
The proof for the reweighted network is similar.
In case of importance node pruning, the nodes with lowest absolute value of outgoing edges are dropped.
Following Goldt et al.
the outgoing weights of all the hidden teacher nodes are equal (we call it v ∗ ).
Also, from Lemma 5 we see that the sum of the weights of the outgoing edges of the student nodes which learn the same teacher node add up to the outgoing edge weight of the corresponding teacher hidden node.
Moreover, we assume the ansatz v i = v j when i, j ∈ G n , where G n denotes the set of student nodes which learn the same teacher node t n .
Hence, we can see that all the outgoing edges are approximately similar.
We also verify this fact experimentally.
Therefore, this deﬁnes an approximately uniform distribution on the set of hidden nodes.
Hence, this is almost same as random node pruning and so the result follows from Theorem 1.
Proof of Theorem 2.
Lemma 2 and 4 provide the closed form of the GE after DPP node pruning and random node pruning respectively.
Using this closed form we plot (cid:15)
DPP Node k
[Equation]
We can see for Z ≥ 4 this value is ≥ 0 given 0 ≤ c ≤ 1 . 0 /Z, which proves the theorem.
Remark 1.
Our results hold for Z ≥ 4 , where Z is the number of student nodes which learn the same teacher node.
This is because in DPP node pruning at most 1 student node survives per group.
As a result for larger Z the lost information per group is higher (in the scale of (cid:0) 1 − 1 Z (cid:1) 2 ).
[Table]
+ZRgVM4g3Pw4QqacActaAMBCc/wCm9O5rw4787HonXNKWdO4A+czx/fN5GG</latexit>
[Figure]
Next we state the impossibility result as discussed inmain text.
We will show that, no reweighting scheme in the second layer for random edge pruning which is based on scaling can beat DPP node pruning after reweighting.
Formally we have the following:
Theorem 3.
Assume ( A 1) − ( A 7) .
Let k n and c satisfy equation 1, and 0 ≤ c ≤ 1 Z and Z ≥ 4 .
Assume the reweighting scheme for random edge in second layer such that, ˆ v i
=
Av i .
Then ∀
A ∈ R we have,
[Equation]
Proof of Theorem 3.
From Lemma 2 we know that the GE after rewighting the DPP node pruned network is
[Equation] [Equation]
where c satisﬁes equation 1.
Now for the given reweighting scheme in the hypothesis the GE for random edge pruning will be,
[Equation]
equation 15 can be viewed as a quadratic equation of A whose minimum correspond to the best reweighting scheme in the scaling family.
In Figure 4 B we compare this minimum with equation 14.
Formally we plotted ˆ (cid:15)
DPP Node k n
[Equation]
( E [ f ]) .
It can be seen that this value is − ve for all 0 ≤ c ≤ 1 Z , which implies GE of reweighted DPP node pruned network is always lower than reweighted random edge pruned network.
E P ROOF OF L EMMAS
Proof of Lemma 2.
Let H R = (cid:8) h i 1 , . .
, h i k n (cid:9) be the set of selected nodes by DPP Node pruning method.
Recall from Goldt et al.
(2019) that every student hidden node specializes in learning a teacher node.
Denote t ( h ) to be the teacher node learnt by h .
S m ⊆
H R be the set of selected hidden nodes of the pruned network which learnt the m th teacher node , i.e., S m = { h ∈ H R | t ( h ) =
t m } ( t m is the m th teacher node).
Hence, prn =
|{ 1 ( | S m | > 0)
| 1 ≤ m ≤ M }| is the number of teacher nodes explained by the pruned network and W.L.O.G. we can assume that t 1 , . . .
, t prn are those set of teacher nodes.
Let l 1 , . . .
, l prn be the number of student nodes in the pruned network which learn the corresponding teacher node.
Note that,
[Equation]
and l i ≤ Z (where Z is the number of student nodes dedicated to learn a single teacher node in the unpruned network) for all i .
Applying
Lemma 3 directly we can see that the GE for the pruned network is
[Equation]
The ﬁrst part of equation 16 is the GE for the group whose corresponding teacher node is partially explained and the second part accounts for the GE due to unexplained teacher nodes (number of such teacher nodes are M − prn ).
From Lemma 1 we know that the expected kernel matrix for DPP
Node pruning is the order parameter Q and it becomes a block diagonal matrix after the training converges, where size of each block is Z (which is also the number of student nodes dedicated to learn a single teacher node in the unpruned network).
Because of the block diagonal property of the
DPP kernel matrix, at most 1 student hidden node will be chosen from each block, i.e., l i = 1 ∀ i .
Hence, prn =
k n .
From Lemma 3 we can see that the GE of node pruned network only depends on the number of student node survived in each block after pruning, and, for DPP node pruning, it is always 1 (given k n ≤ M ).
This is why there is no expectation in the GE term.
So for DPP node pruning the GE is,
[Equation]
Each of the k n student nodes in the pruned network learns a different teacher node.
Consider one such teacher node and call it t i .
In the unpruned network, there are Z student hidden nodes which learn a single teacher node t i , only one of which survives after DPP node pruning.
The ﬁrst part of the error is due to the removal of student nodes ( Z − 1 student nodes for each t i ).
However, these errors can be retrieved by reweighting the survived student node.
On the contrary, there are
M − k n teacher nodes which don’t have any representative (some student hidden node from the set of student nodes which specialized in this particular teacher node) in the pruned network.
And the error (second part of the GE) due to those nodes can not be retrieved even after reweighting.
Hence, the GE after reweighting becomes,
[Equation]
Thus, we have the lemma 2.
Proof of Lemma 3.
Let G 1 , . .
, G M be the subsets of student nodes such that all student nodes in
G m learn the m th teacher node.
From the assumption we have, | G m | =
Z for all m .
After pruning, a subset P m ⊆
G m is chosen, and | P m | = l m .
Denote the order parameters of the pruned network as Q (cid:48) , R (cid:48) , T (cid:48) .
For node pruning we can see that
[Equation]
∃ m s.t. h
[Equation]
P
[Equation]
and h k ∈ P
[Equation]
otherwise
Also, for the unpruned network we have
[Equation]
if ∃ m s.t. h
[Equation]
G
[Equation]
and h k ∈ G m otherwise
Now from equation 5 we can break down the GE into three parts.
From equation 6, equation 7 and equation 8 we have,.
[Equation]
equation 17 follows from the fact that h i and h k belong to the same group G n .
So we have,
[Equation]
We can also see that equation 18 follows from Lemma 5 and the ansatz v i = v j when i, j ∈ G n .
The order parameters T nm doesn’t change after pruning, and so we have,
[Equation]
And similarly,
[Equation]
Then from equation [Equation] equation 19 and equation 20 the GE of node pruning is,
[Equation]
Hence we have the lemma.
Intuitively, this lemma states that for teacher hidden node t n if l n student hidden nodes survive after node pruning, then the fraction of information lost due to the deletion of nodes is 1 − l n Z , where Z is the number of student nodes learn a particular teacher node in the unpruned network.
Proof of Lemma 5.
From ( S 36) of Goldt et al.
we have,
[Equation]
 Hence, a ﬁxed point (in terms of v i ’s) of the ODE is,
[Equation]
Intuitively, this lemma states that the sum of the outgoing edges of the student hidden nodes which learn a particular teacher hidden node is approximately equal to the weight of the outgoing edge of that teacher hidden node.
Lemma 6.
Let Q, R, T are the order parameters of the unpruned network, and Q (cid:48) , R (cid:48) , T (cid:48) are the respective order parameters after applying the Random Edge pruning where c fraction of the edges are kept.
Then we have 1) E
[ Q (cid:48)
ik ]
= cQ ik if i =
k and c 2 Q ik otherwise
, 2) E
[ R (cid:48) st ] = cR st and 3)
T (cid:48)
mn = T mn .
Proof.
Follows directly from the pruning procedure.
Intuitively, this lemma states that the order parameters of the pruned network using random edge pruning is a scaled version of the order parameters of the unpruned networks.
However, the scaling of diagonal elements are different from that of off-diagonal elements.
Proof of Lemma 4.
In this theorem, we will give the GE of the expected network pruned by the
Random Edge method.
Pruning is performed on the edges between input layer and the hidden layer.
Hence, the order parameter changes.
From Lemma 6, we have the order parameters of the expected network (call these Q (cid:48) , R (cid:48) , T (cid:48) ).
However, the weights of the second layer remain unchanged.
Putting these values in equation 6, equation 7 and equation 8 we have,
[Equation]
and,
[Equation]
Therefore, the GE of the expected network after Random Edge pruning is,
[Equation] [Equation]
This proves the ﬁrst part of the theorem.