D IVERSITY B ASED E DGE P RUNING OF N EURAL N ETWORKS U SING D ETERMINANTAL P OINT P ROCESSES
A BSTRACT
Deep learning architectures with huge number of parameters are often compressed using pruning techniques.
Two classes of pruning techniques are node pruning and edge pruning .
A fairly recent work established that Determinantal Point Process (DPP) based node pruning empirically outperforms competing node pruning methods.
However, one prominent appeal of edge pruning over node pruning is the consistent ﬁnding in literature is that sparse neural networks (edge pruned) generalize better than dense neural networks (node pruned).
Building on these previous work and drawing motivation from synaptic diversity in the brain, we propose a novel diversity-based edge pruning technique for neural networks using DPP.
We then empirically show that DPP edge pruning for neural networks outperforms other competing methods (both edge and node) on real data.
I NTRODUCTION
The primary goal of pruning a neural network is to reduce the number of parameters without reducing the network’s performance signiﬁcantly.
A popular pruning method is node pruning where redundant neurons from the network are removed (Hanson & Pratt (1989;?), others include ‘Optimal
Brain Damage’ LeCun et al.
and ‘Optimal Brain surgeon’ Hassibi & Stork (1993)).
A fairly recent work on novel node pruning methods Mariet & Sra (2016), propose a node pruning technique where: a diverse subset of nodes are preserved in a given layer using DPP Macchi (1975); Kulesza et al.
(2012), followed by reweighting , to compensate contributions of the pruned neurons in the network.
Finally, they show that DPP node pruning with reweighting performs better than random and importance node pruning He et al.
(2014) on real datasets.
Another way of network pruning is edge pruning where the number of connections between nodes are removed to reduce redundancy (He et al.
; Han et al.
Though, node pruning methods have their own appeal, literature suggests that sparse networks obtained after edge pruning outperform dense networks obtained after node pruning (see section 3.2 of a recent review article Blalock et al.
(2020) which investigates 81 recent pruning based papers).
Edge pruning methods also draw motivation from the huge line of research in computational neuroscience on normative models of the brain ( Efﬁcient Coding models
Doya et al.
that focus on the idea of reducing redundant representations in the brain.
Interestingly, research advances in neurophysiology over the years has shown the existence of diverse synapses in the brain and established that unused synapses are eliminated by the brain to conserve energy and ensure efﬁcient information transmission Chechik et al.
In this work we build on the DPP based pruning idea from Mariet & Sra (2016), empirical observations in pruning literature and observations in the brain.
We model saliency of a particular weight (synapse) in a network as the amount of diverse information captured by it.
We then investigate empirical potentials of pruning that focus on synaptic (weight/connection/edge) diversity in feedforward neural networks via DPP (‘diversity based synaptic pruning’).
Finally, we show that DPP based edge pruning method for feedforward networks which uses synaptic diversity to remove redundant edges outperforms competing methods on real data.
We also observe that DPP edge pruning ﬁnds a sparse network which performs better than the unpruned dense network.
P RELIMINARIES 2. D ETERMINANTAL P OINT P ROCESS (DPP):
DPP Macchi (1975) is a probability distribution over power set of a ground set G , here ﬁnite.
DPP is a special case of negatively associated distributions Joag-Dev et al.
(1983) which assigns higher probability mass on diverse subsets.
Formally, a DPP with a marginal kernel L ( ∈ R |G|×|G| ) is: P [ Y = Y ] = det( L Y ) det( L + I ) , where Y ⊆ G and L Y is the principal submatrix deﬁned by the indices of Y .
We use k -DPP to denote the probability distribution over subsets of ﬁxed size k .
2. N ODE P RUNING WITH DPP (M ARIET & S RA (2016)):
Mariet & Sra (2016) uses DPP to propose a novel node pruning method for feed forward neural network.
They deﬁne information at node i of layer l as a li (= ( a li 1 , . .
. , a lin )) , where a lij is the activity of node i of layer l on j th input.
Here a li = g (cid:0) b li (cid:1) , where b li = (cid:80) n l − 1 j =1 w l − 1 ji a l − 1 j is the information at node i of layer l before activation.
A layer is pruned by choosing a subset of hidden nodes using a DPP kernel: L ( = L (cid:48)
+ (cid:15) I ), where, L (cid:48)
st = exp( − β (cid:13)(cid:13) a ls − a lt (cid:13)(cid:13) 2 ) and β is a bandwidth parameter.
The matrix L is of dimension n l × n l , as total number of nodes in layer l is n l .
By the property of DPP, this procedure will keep a diverse subset of nodes for each layer w.r.t.
information obtained from the training data.
A reweighting technique (see Section 2.2 of Mariet &
Sra (2016)) is then applied to outgoing edges of retained nodes to compensate for information lost in that layer due to node removal.
Note that DIVNET denotes DPP node pruning with reweighting as in Mariet & Sra (2016).
E DGE P RUNING USING D ETERMINANTAL P OINT P ROCESS 3. DPP K ERNEL FOR E DGE P RUNING
We follow Mariet & Sra (2016) to compute the amount of information captured by an edge about the training data.
Recall from Section
that for node v lj of layer l , the activation is given as a lj
= g (cid:0)(cid:80) n l − 1 i =1 w l − 1 ij a l − 1 i (cid:1) , where w l − 1 ij is the weight of the edge from i th node of layer
l − 1 to j th node of layer l .
Hence the information carried from layer
l − 1 to node v lj of layer l through the edge e l − 1 ij is w l − 1 ij a l − 1 i .
We choose a subset of incoming edges for node v lj such that it preserves diverse information about the input.
The DPP kernel L j for the set of incoming edges of v lj is deﬁned as
L j = L (cid:48) ( j ) + (cid:15)
I , where L (cid:48) ( j ) st = exp( − β (cid:13)(cid:13)
w l − 1 sj a l − 1 s − w l − 1 tj a l − 1 t (cid:13)(cid:13) 2 ) .
Here β is a hyperparameter and (cid:15) denotes small perturbations to the diagonal of the kernel matrix making it strictly positive deﬁnite (as in Mariet & Sra (2016)).
3. R ETRIEVING P RUNED I NFORMATION BY R EWEIGHTING
We lose information passed from one layer to the next by removing a subset of edges between them after pruning.
To maximize the total amount of information preserved, we reweight the surviving edges.
We denote S j
= { i 1 , . .
, i
k e } as the set of incoming edges chosen for the node v l j using [Figure] k e -DPP.
Denote ˆ w l − 1 ij = w l − 1 ij + δ l − 1 ij to be the new weight of the edges after reweighting.
Hence to minimize the lost information after pruning, we minimize:
[Equation]
[Equation]
This can be rewritten as a linear regression problem i.e., min x (cid:13)(cid:13) y − A l − 1 · x j (cid:13)(cid:13) 2 , where s th column of A l − 1 is a l − 1 i s , x j =
[ δ i 1 j , · · · , δ i ke j ] T , and y =
(cid:80) i ∈
¯ S j w l − 1 ij a l − [-
-]1 i .
The closed form solution of this equation is, x = (cid:0) A Tl − 1 A l − 1 (cid:1) − 1 A Tl − 1 y .
In node pruning, if a node is deleted from layer l , edges from l − 1 to l and l to l + 1 are deleted, affecting both incoming and outgoing information from layer l .
Therefore, the focus is to minimize loss in outgoing information from layer l to l +as is ensured by reweighting of Mariet & Sra (2016).
In edge pruning, edges between layer
l − 1 to l get deleted which we try to compensate in our new reweighting scheme by minimizing incoming information loss.
While DPP node pruning requires only one kernel per layer Mariet & Sra (2016), DPP edge pruning will have n l DPP kernels for l th layer.
However, DPP kernels require one-time computation and are not recomputed to predict labels for unseen data.
Note that while DPP edge pruning with reweighting inherits all beneﬁts of DIVNET discussed in Section 3.3 and 4 of Mariet & Sra (2016), it performs better than the latter on real data (Section 4).
Few important properties inherited by DPP edge kernels due to similar deﬁnitions to DIVNET are: (1) The DPP edge kernels and the reweighting coefﬁcients can also be learned from subsampled training data which causes a trade-off between performance, space complexity, and time complexity.
(2) DPP edge pruning with reweighting can be incorporated into other methods of network compression to increase diversity.
E XPERIMENT ON R EAL D ATA
In this section, we compare our proposed DPP edge pruning with reweighting, DIVNET by Mariet & Sra (2016), random edge pruning with reweighting, and importance edge/node pruning with reweighting on the MNIST LeCun et al.
(2010) and CIFAR10 Krizhevsky et al.
(2009) datasets.
We used the exact same network architectures as in Table 1 of Mariet & Sra (2016) for MNIST and
CIFAR10 , respectively.
Following Mariet & Sra (2016), we performed all pruning methods on the ﬁrst layer.
We compare the number of parameters as k e = k n ( d input + h 2 )
h− h 2 where k e is the number of edges kept for each node in edge pruning, and k n is the number of nodes kept in the hidden layer for node pruning; d input , h 1 , and h 2 represent the dimension of the input, size of the ﬁrst hidden layer, and size of the second hidden layer, respectively.
As in Mariet & Sra (2016), h 1 = h 2 .
We trained our model until the training error reaches predeﬁned thresholds (Table 1 in Mariet & Sra (2016)) and then perform the pruning.
All edge pruning methods use our reweighting scheme in 3.
while all node pruning methods use reweighting from Mariet & Sra (2016).
4. H YPERPARAMETERS
We report the hyperparameters used for the results on the MNIST and CIFAR10 .
We used that exact same experiment setup (network architectures, error thresholds, etc.) as in Mariet & Sra (2016) for fair and consistent comparisons.
We used SGD optimizers, a learning rate of 0 . 001 , and a momentum of 0 .
9 for traning on both datasets.
For MNIST , the training batch size was 1000 .
For
CIFAR10 , the training batch size was 128 .
All pruning methods were performed 10 times, and we report the means and standard deviations in Figure 1A-B (without reweighting) and Figure 1C-D (with reweighting).
The node-to-edge ratio for pruning, which keeps the number of parameters in the pruned network the same, is [397 : 614 , 472 : 921 , 548 : 1228 , 623 : 1536 , 699 : 1843 , 774 : 2150 , 849 : 2457 , 925 : 2764] for CIFAR10 and [256 : 156 , 287 : 235 , 317 : 313 , 348 : 392 , 378 : 470 , 409 : 548 , 439 : 627 , 470 : 705] for MNIST , given the network architecture in Table 1 of Mariet & Sra (2016).
These ratios correspond to 20% to 90% of the edges left for each node, as shown on the x-axis of Figure 1.
These node-to-edge ratios are calculated based on the conversion equation in
Section 4.
We used β = 10 / | T | where | T | is the size of the training dataset for all DPP node and edge kernel calculations on real data, following the choice of Mariet & Sra (2016).
4. R ESULTS
We compare the test errors for the pruned networks with respective reweighting schemes for both datasets as in Figure 1.
We observe that (1) When reweighting is not applied, importance edge pruning performs the best in both the datasets.
(2) DPP edge pruned network signiﬁcantly outperforms all other pruning methods when reweighting is added.
This happens because, in the case of importance pruning, the greedy process of choosing the edges does not keep much room for improvement, whereas choosing the edges using DPP keeps enough space for improvement after reweighting.
This shows the novelty of both DPP methods as well as the reweighting procedure.
(3) Interestingly, we ﬁnd that importance node pruning with reweighting performs better than DIVNET, which was not explored in Mariet & Sra (2016).
Furthermore, unlike other pruning methods, our sparse
DPP edge pruned network generalizes better than the unpruned dense network for both the datasets (see at 90% for MNIST and 70-90% for CIFAR10 ) in Figure 1C and D).
In literature, such sparse sub-networks of a dense network with better generalization ability is achieved via iterative pruning method Malach et al.
(2020), which is time consuming.
However, our approach gives rise to such sparse network without reinitializing and retraining but only reweighting adding to the novelty of our approach.
D ISCUSSION AND F UTURE W ORK
Our method falls under fast compression techniques that do not undergo retraining and outperforms competing methods in the same category on real data (Section 4).
While DPP edge pruning is complementary to prior work, its novelty lies in theory, literature and neuroscience motivated deﬁnition.
We consolidate the understanding of previous empirical ﬁndings and implications of observations in the brain from a theoretical perspective to propose a new pruning method.
In future work the DPP edge pruning method can be tested on large network architectures and compared to other benchmark pruning methods that undergo retraining after pruning.