C HEAP DNN P RUNING WITH P ERFORMANCE G UARANTEES . / C ONFERENCE S UBMISSIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent DNN pruning algorithms have succeeded in reducing the number of parameters in fully connected layers, often with little or no drop in classiﬁcation accuracy.
However, most of the existing pruning schemes either have to be applied during training or require a costly retraining procedure after pruning to regain classiﬁcation accuracy.
In this paper we propose a cheap pruning algorithm based on difference of convex functions (DC) optimisation.
We also provide theoretical analysis for the growth in the Generalization Error (GE) of the new pruned network.
Our method can be used with any convex regulariser and allows for a controlled degradation in classiﬁcation accuracy, while being orders of magnitude faster than competing approaches.
Experiments on common feedforward neural networks show that for sparsity levels above 90% our method achieves 10% higher classiﬁcation accuracy compared to Hard Thresholding.
I NTRODUCTION
Recently, deep neural networks have achieved state-of-the art results in a number of machine learning tasks LeCun et al.
Training such networks is computationally intensive and often requires dedicated and expensive hardware.
Furthermore, the resulting networks often require a considerable amount of memory to be stored.
Using a Pascal Titan X GPU the popular AlexNet and VGGmodels require 13 hours and 7 days, respectively, to train, while requiring 200MB and 600MB, respectively, to store.
The large memory requirements limit the use of DNNs in embedded systems and portable devices such as smartphones, which are now ubiquitous.
A number of approaches have been proposed to reduce the DNN size during training time, often with little or no degradation to classiﬁcation performance.
Approaches include introducing bayesian, sparsity-inducing priors Louizos et al.
Blundell et al.
Molchanov et al.
and binarization Hou et al.
Courbariaux et al.
(2016).Other methods include the hashing trick used in Chen et al.
(2015), tensorisation Novikov et al.
and efﬁcient matrix factorisations
Yang et al.
However, trained DNN models are used by researchers and developers that do not have dedicated hardware to train them, often as general feature extractors for transfer learning.
In such settings it is important to introduce a cheap compression method, i.e., one that can be implemented as a postprocessing step with little or no retraining.
Some ﬁrst work in this direction has been Kim et al.
Han et al.
(2015a)
Han et al.
(2015b)
although these still require a lengthy retraining procedure.
Closer to our approach recently in Aghasi et al.
the authors propose a convexiﬁed layerwise pruning algorithm termed Net-Trim.
Building upon Net-Trim, the authors in Dong et al.
(2017) propose LOBS, an algorithm for layerwise pruning by loss function approximation.
Pruning a neural network layer introduces a pertubation to the latent signal representations generated by that layer.
As the pertubated signal passes through layers of non-linear projections, the pertubation could become arbitrary large.
In Aghasi et al.
(2016) and Dong et al.
(2017) the authors conduct a theoretical analysis using the Lipschitz properties of DNNs showing the stability of the latent representations, over the training set, after pruning.
The methods employed have connections to recent work Sokolic et al.
(2017) Bartlett et al.
(2017) Neyshabur et al.
(2017) that have used the
Lipschitz properties to analyze the Generalization Error (GE) of DNNs, a more useful performance measure.
1. C ONTRIBUTIONS
In this work we introduce a cheap pruning algorithm for dense layers of DNNs.
We also conduct a theoretical analysis of how pruning affects the Generalization Error of the trained classiﬁer.
• We show that the sparsity-inducing objective proposed in Aghasi et al.
(2016) can be cast as a difference of convex functions problem, that has an efﬁcient solution.
For a fully connected layer with input dimension d 1 , output dimension d 2 and N training samples, Net-Trim and LOBS scale like O ( Nd 31 ) and O (( N + d 2 ) d 21 ) , respectively.
Our iterative algorithm scales like O ( K ( N + NkN + √ k ) log ( 1 (cid:15) )
d 1 d 2 ) , where (cid:15) is the precision of the solution, k is related to the Lipschitz and strong convexity constants, d 2 (cid:28) d 1 and K is the outer iteration number.
Emprirically, our algorithm is orders of magnitude faster than competing approaches.
We also extend our formulation to allow retraining a layer with any convex regulariser.
• We build upon the work of Sokolic et al.
(2017) to bound the GE of a DNN after pruning.
Our theoretical analysis holds for any bounded pertubation to one or multiple hidden DNN layers and provides a principled way of pruning while managing the GE.
Experiments on common feedforward architectures show that our method is orders of magnitude faster than competing pruning methods, while allowing for a controlled degradation in GE.
1. N OTATION AND D EFINITIONS
We use the following notation in the sequel:matrices ,column vectors, scalars and sets are denoted by boldface upper-case letters ( X ), boldface lower-case letters ( x ), italic letters ( x ) and calligraphic upper-case letters ( X ), respectively.
The covering number of X with d -metric balls of radius ρ is denoted by N ( X ; d, ρ ) .
A C M -regular k -dimensional manifold, where C M is a constant that captures ”intrinsic” properties, is one that has a covering number N ( X ; d, ρ )
=
( C M ρ ) k .
O UR FORMULATION 2. DC DECOMPOSITION
We consider a classiﬁcation problem, where we observe a vector x ∈ X ⊆ R N that has a corresponding class label y ∈ Y .
The set X is called the input space, Y = { 1 , 2 , ..., N Y } is called the label space and N Y denotes the number of classes.
The samples space is denoted by S =
X × Y and an element of S is denoted by s =
( x , y ) .
We assume that samples from S are drawn according to a probability distribution P deﬁned on S .
A training set of m samples drawn from P is denoted by S m =
{ s i } mi =1 = { ( x i , y i ) } mi =1 .
We start from the Net-Trim formulation and show that it can be cast as a difference of convex functions problem.
For each training signal x ∈ R N we assume also that we have access to the inputs a ∈ R d 1 and the outputs b ∈ R d 2 of the fully connected layer, with a rectiﬁer non-linearity ρ ( x ) = max (0 , x ) .
The optimisation problem that we want to solve is then
[Equation]
where λ is the sparsity parameter.
The term ||
ρ ( U T a j ) − b j || 22 ensures that the nonlinear projection remains the same for training signals.
The term λ Ω( U ) is the convex regulariser which imposes the desired structure on the weight matrix U .
The objective in Equation 1 is non-convex.
We show that the optimisation of this objective can be cast as a difference of convex functions (DC) problem.
We assume just one training sample x ∈ R N , for simplicity, with latent representations a ∈ R d and b ∈ R z
[Equation]
Notice that after the split the ﬁrst term ( b i < 0 ) is convex while the second ( b i ≥ 0 ) is concave.
We note that b i ≥ 0 by deﬁnition of the ReLu and set
[Equation]
Then by summing over all the samples we get
[Equation]
which is difference of convex functions.
The rectiﬁer nonlinearity is non-smooth, but we can alleviate that by assuming a smooth approximation.
A common choice for this task is ρ ( x )
=β log (1 + exp ( βx )) , with β a positive constant.
2. O PTIMISATION
It is well known that DC programs have efﬁcient optimisation algorithms.
We propose to use the
DCA algorithm Tao & An (1997).
DCA is an iterative algorithm that consists in solving, at each iteration, the convex optimisation problem obtained by linearizing h ( · ) (the non-convex part of f = g − h ) around the current solution.
Although DCA is only guaranteed to reach local minima the authors of Tao & An (1997) state that DCA often converges to the global minimum, and has been used succefully to optimise a fully connected DNN layer Fawzi et al.
At iteration k of DCA, the linearized optimisation problem is given by
[Equation]
where U k is the solution estimate at iteration k .
The detailed procedure is then given in algorithms 1 and 2.
We assume that the regulariser is convex but possibly non-smooth in which case the optimisation can be performed using proximal methods.
1: Choose initial point: U2: for k = 1,...,K do 3:
Compute C ← ∇ h
( U k ) .
Solve with Algorithm 2 the convex optimisation problem:
U k +1 ← arg min
U { g ( U ) + λ Ω( U ) − Tr ( U T C ) } (7) 5: end for 6: If U k
+
≈
U
k return U
k +1 .
1: Initialization : U ← U k , ρ, t2: for t = 1,...,T do 3:
Choose ( A , B ) randomly chosen minibatch.
4: ρ t ← min( ρ, ρ t 0 t ) 5:
U = prox λ Ω( · ) ( U − ρ t [ ∇ g A , B
( U )
− ∇
Tr ( U T C )]) 6: end for 7: Return U
k +1 ←
U
1: Initialization : ˜ x 0 ← U k , β, η 2: for s =
,S
do 3: ˜
u
= ∇ g (˜ x s ) 4: x 1 = y 1 = ˜ x s 5: for t = 1,2,...,T do 6:
Choose ( A , B ) randomly chosen minibatch.
7: u t = ∇ g A , B ( y t ) − ∇ g A , B (˜ x s ) + ˜ u 8: x t +1 =
prox ηh ( y t − η u t ) 9: y t
+1 = x
t +1 + β
( x t +1 − x t ) 10: end for 11: ˜ x s +1 = x T
+ 12: end for 13: Return U k +1 ← ˜ x S +
In order to solve the linearized problem we ﬁrst propose to use Proximal Stochastic Gradient Descent (Prox-SG), which we detail in algorithm 2a.
At each iteration a minibatch A and B is drawn.
The gradient for the smooth part is calculated and the algorithm takes a step in that direction with step size ρ t .
At each iteration ρ is updated as ρ t ← min( ρ, ρ t 0 t ) where t 0 is a hyperparameter.
Then the proximal operator for the non-smooth regulariser λ Ω( · ) is applied to the result.
We ﬁnd that for the outer iterations K the values 5 to 15 are usually sufﬁcient, while for the inner iterations T = 150 is usually sufﬁcient.
Although Prox-SG is very efﬁcient, it sometimes doesn’t converge to a good solution.
We therefore propose to use Accelerated Proximal SVRG (Acc-Prox-SVRG), which was presented in Nitanda (2014).
We detail this method in Algorithm 2b.
For most experiments we see a signiﬁcant improvement over Prox-SG.
The hyperparameters for Acc-Prox-SVRG are the acceleration parameter β and the gradient step η .
We have found that in our experiments, using β = 0 .
95 and η ∈ { 0 .
001 , 0 . 0001 } gives the best results.
We name our algorithm FeTa, Fast and Efﬁcient Trimming Algorithm.
G ENERALIZATION E RROR 3. G ENERALIZATION E RROR OF P RUNED L AYER
Having optimized our pruned layer for the training set we want to see if it is stable for the test set.
We denote f 1 ( · , W 1 ) the original representation and f 2 ( · , W 2 ) the pruned representation.
We assume
that after training ∀ s i ∈ S m || f 1 ( a i , W 1 ) − f 2 ( a i , W 2 ) || 22 ≤ C 1 .
Second, we assume that
∀ s ∈ S ∃ s i ∈ S
m ⇒ ||
a − a
i || 22 ≤ (cid:15) .
Third, the linear operators in W 1 , W 2 are frames with upper frame bounds B 1 , B 2 respectively.
Theorem 3.1.
For any testing point s ∈ S , the distance between the original representation f 1 ( a , W 1 ) and the pruned representation f 2 ( a , W 2 ) is bounded by || f 1 ( a , W 1 ) − f 2 ( a , W 2 )
|| 22 ≤ C 2 where C 2 = C 1 + ( B 1 + B 2 ) (cid:15) .
the detailed proof can be found in Appendix A.
3. G ENERALIZATION E RROR OF C LASSIFIER
In this section we use tools from the robustness framework Xu & Mannor (2012) to bound the generalization error of the new architecture induced by our pruning.
We consider DNN classiﬁers deﬁned as
[Equation]
) where ( f ( x )) i is the
i − th element of N y dimensional output of a DNN f : R N → R N y .
We assume that f ( x ) is composed of L layers
[Equation]
where f l ( · , W l ) represents the l − th layer with parameters W l , l = 1 , ..., L .
The output of the l − th layer is denoted z l , i.e. z l = f l ( z l − 1 , W l ) .
The input layer corresponds to z 0 = x and the output of the last layer is denoted by z
= f ( x ) .
We then need the following two deﬁnitions of the classiﬁcation margin and the score that we take from Sokolic et al.
These will be useful later for measuring the generalization error.
Deﬁnition 3.1.
( Score).
For a classiﬁer g ( x ) a training sample s i
=
( x i , y i ) has a score
[Equation]
where δ i ∈
R N y is the Kronecker delta vector with ( δ i ) i = 1 , and g ( x i ) is the output class for s i from classiﬁer g ( x ) which can also be g ( x i ) (cid:54) =
y i .
Deﬁnition 3.2.
( Training Sample Margin).
For a classiﬁer g ( x ) a training sample s i
=
( x i , y i ) has a classiﬁcation margin γ ( s i ) measured by the l 2 norm if
[Equation]
The classiﬁcation margin of a training sample s i is the radius of the largest metric ball (induced by the l 2 norm) in X centered at x i that is contained in the decision region associated with the classiﬁcation label g ( x i ) .
Note that it is possible for a classiﬁer to misclassify a training point g ( x i ) (cid:54) =
y i .
We then restate a useful result from Sokolic et al.
Corollary 3.1.1.
Assume that X is a (subset of) C M -regular k-dimensional manifold, where
N ( X ; d ; ρ ) ≤ ( C M ρ ) k .
Assume also that the DNN classiﬁer g ( x ) achieves a lower bound to the classiﬁcation score o (˜ s ) <
o ( s i ) ,
∀ s i ∈ S m and take l ( g ( x i ) , y i ) to be the 0 − 1 loss.
Then for any δ > 0 , with probability at least 1 − δ ,
[Equation]
where A =
[Equation]
and B = (cid:113)
[Equation]
can be considered constants related to the data manifold and the training sample size, and γ =
[Equation]
We are now ready to state our main result.
Theorem 3.2.
Assume that X is a (subset of) C M -regular k-dimensional manifold, where
N ( X ; d ; ρ ) ≤ ( C M ρ ) k .
Assume also that the DNN classiﬁer g 1 ( x ) achieves a lower bound to the classiﬁcation score o (˜ s ) <
o ( s i ) ,
∀ s i ∈ S m and take l ( g ( x i ) , y i ) to be the 0 − 1 loss.
Furthermore assume that we prune classiﬁer g 1 ( x ) on layer i (cid:63) using Algorithm 1, to obtain a new classiﬁer g 2 ( x ) .
Then for any δ > 0 , with probability at least 1 − δ , when ( γ − √ C 2 · (cid:81) i>i(cid:63)
[Equation]
where A =
[Equation]
and B = (cid:113)
[Equation]
m can be considered constants related to the data manifold and the training sample size, and γ =
[Equation] [Equation]
s
[Equation]
The detailed proof can be found in Appendix B.
The bound depends on two constants related to intrinsic properties of the data manifold, the regularity constant C M and the intrinsic data dimensionality k .
In particular the bound depends exponentially on the intrinsic data dimensionality k .
Thus more complex datasets are expected to lead to less robust DNNs.
This has been recently observed empirically in Bartlett et al.
The bound also depends on the spectral norm of the hidden layers
|| W
i ||
Small spectral norms lead to a larger base in ( · ) − k 2 and thus to tigher bounds.
With respect to pruning our result is quite pessimistic as the pruning error √ C 2 is multiplied by the factor (cid:81)
i>i (cid:63)
|| W
i ||
Thus in our analysis the GE grows exponentially with respect to the remaining layer depth of the pertubated layer.
This is in line with previous work Raghu et al.
Han et al.
(2015b) that demonstrates that layers closer to the input are much less robust compared to layers close to the output.
Our algorithm is applied to the fully connected layers of a DNN, which are much closer to the output compared to convolutional layers.
We can extend the above bound to include pruning of multiple layers.
Theorem 3.3.
Assume that X is a (subset of) C M -regular k-dimensional manifold, where
N ( X ; d ; ρ ) ≤ ( C M ρ ) k .
Assume also that the DNN classiﬁer g 1 ( x ) achieves a lower bound to the classiﬁcation score o (˜ s ) <
o ( s i ) ,
∀ s i ∈ S m and take l ( g ( x i ) , y i ) to be the 0 − 1 loss.
Furthermore assume that we prune classiﬁer g 1 ( x ) on all layers using Algorithm 1, to obtain a new classiﬁer g 2 ( x ) .
Then for any δ > 0 , with probability at least 1 − δ , when ( γ − (cid:80) Li = √ C i 2 (cid:81)
Lj =
i +
|| W j ||
(cid:81)
i || W
i ||) > 0 ,
GE ( g 2 ) ≤ A · ( γ − (cid:80) L i =
[Equation]
where A = (cid:113) log(2) · N y · 2 k +1 · ( C M ) k m and B = (cid:113) 2log1 /δ
m can be considered constants related to the data manifold and the training sample size, and γ =
o (˜
[Equation]
The detailed proof can be found in Appendix C.
The bound predicts that when pruning multiple layers the GE will be much greater than the sum of the GEs for each individual pruning.
We note also the generality of our result; even though we have assumed a speciﬁc form of pruning, the GE bound holds for any type of bounded pertubation to a hidden layer.
E XPERIMENTS
We make a number of experiments to compare FeTa with LOBS and NetTrim-ADMM.
We test two versions of the algorithm, FeTa 1 optimised with Prox-SGD and FeTa 2 optimized with Acc-ProxSVRG.
All experiments were run on a MacBook Pro with CPU 2.8GHz Intel Core i7 and RAM 16GB 1600 MHz DDR3.
4. T IME C OMPLEXITY
First we compare the execution time of FeTa with that of LOBS and NetTrim-ADMM.
We set Ω( U )
=
||
U || 1 and aim for 95% sparsity.
We set d 1 to be the input dimensions, d 2 to be the output dimensions and N to be the number of training samples.
Assuming that each g ( U ; x j ) is
L -Lipschitz smooth and g ( U ) is µ -strongly convex, if we optimise for an (cid:15) optimal solution and set k = L/µ , FeTa 1 scales like O ( K 1 µ(cid:15)
d 1 d 2 ) and FeTa 2 scales like O ( K ( N + NkN + √ k ) log ( 1 (cid:15) )
d 1 d 2 ) .
We obtain this by multiplying the number of outer iterations K with the number of gradient evaluations required to reach an (cid:15) good solution in inner Algorithm 2a and inner Algorithm 2b, and ﬁnally multiplying with the gradient evaluation cost.
Conversely LOBS scales like O (( N + d 2 ) d 21 ) while
NetTrim-ADMM scales like O ( Nd 31 ) due to the required Cholesky factorisation.
This gives a computational advantage to our algorithm in settings where the input dimension is large.
We validate this by constructing a synthetic experiment with d 2 = 10 , d 1 = { 2000 : 100 : 3000 } and N = 1000 .
The samples a ∈ R d 1 and b ∈ R d 2 are generated with i.i.d Gaussian entries.
We plot in Figurethe results, which are in line with the theoretical predictions.
4. C LASSIFICATION A CCURACY 4.2. S PARSE R EGULARISATION
In this section we perform experiments on the proposed compression scheme with feedforward neural networks.
We compare the original full-precision network (without compression) with the following compressed networks: (i) FeTa 1 with Ω( U ) =
||
U || 1 (ii) FeTa 2 with Ω( U ) =
||
U ||(iii) Net-Trim (
vi)
LOBS (v
Hard Thresholding.
We refer to the respective papers for Net-Trim and LOBS.
Hard Thresholding is deﬁned as F ( x ) =
x (cid:12) I ( | x | > t ) , where I is the elementwise indicator function, (cid:12) is the Hadamard product and t is a positive constant.
Experiments were performed on two commonly used datasets:
MNIST :
This contains 28 × 28 gray images from ten digit classes.
We use 55000 images for training, another 5000 for validation, and the remaining 10000 for testing.
We use the LeNet-5 model:
[Equation]
where C 5 is a 5 × 5 ReLU convolution layer, MP 2 is a 2 × 2 max-pooling layer, FC is a fully connected layer and SM is a linear softmax layer.
CIFAR-10 :
This contains 60000 32 × 32 color images for ten object classes.
We useimages for training and the remaining 10000 for testing.
The training data is augmented by random cropping to 24 × 24 pixels, random ﬂips from left to right, contrast and brightness distortions to 200000 images.
We use a smaller variant of the AlexNet model:
[Equation]
We ﬁrst prune only the ﬁrst fully connected layer (the one furthest from the output) for clarity.
Figure 2 shows the classiﬁcation accuracy vs compression ratio for FeTa 1 , FeTa 2 , NetTrim, LOBS and Hard Thresholding.
We see that Hard Thresholding works adequately up to 85% sparsity.
From this level of sparsity and above the performance of Hard Thresholding degrades rapidly and FeTa has 10% higher accuracy on average.
We also see a notable improvement of 3% − 5% for FeTaover FeTa 1 .
Finally NetTrim and LOBS also give good results for a wide range of sparsity values, with LOBS giving the best results overall.
[Figure]
Method
Networks
Original Accuracy
CR
Pruned Accuracy
Computation Time
Net-Trim
LeNet99.2% 95% 95% 455s
LOBS
LeNet99.2% 95% 97% 90s
Thresholding
LeNet99.2% 95% 83% -
FeTa
LeNet99.2% 95% 91% 32 s
FeTa
LeNet99.2% 95% 95% 18 s
Net-Trim
CifarNet 86% - - -
LOBS
CifarNet 86% 90% 83.4% 2h 47min
Thresholding
CifarNet 86% 90% 73% -
FeTa
CifarNet 86% 90% 77% 9 min
FeTa
CifarNet 86% 90% 80% 20 min
Method
Networks
Original Accuracy
CR
Pruned Accuracy
Computation Time
Net-Trim
LeNet99.2% 90% 95% 500s
LOBS
LeNet99.2%
[Table]
For the task of pruning the ﬁrst fully connected layer we also show detailed comparison results for all methods in Table 1.
For the LeNet-5 model
[Table]
FeTa achieves the same accuracy as Net-Trim while being signiﬁcantly faster.
This is expected as the two algorithms optimise a similar objective, while
FeTa exploits the structure of the objective to achieve lower complexity in optimisation.
Furthermore
FeTa achieves marginally lower classiﬁcation accuracy compared to LOBS, and is signiﬁcantly better than Thresholding.
Overall FeTa enjoys competitive accuracy results while being able to prune the dense layer 5 × to 25 × faster compared to other approaches.
For the CifarNet model Net-Trim is not feasible on the machine used for the experiments as it requires over 16GB of RAM.
Compared to LOBS FeTa again achieves marginally lower accuracy but is 8 × to 14 × faster.
Note that as mentioned in Dong et al.
(2017) and Wolfe et al.
(2017) retraining can recover classiﬁcation accuracy that was lost during pruning.
Starting from a good pruning which doesn’t allow for much degradation signiﬁcantly reduces retraining time.
Next we prune both the fully connected layers in the two architectures to the same sparsity level and plot the results in Table 2.
We lower the achieved sparsity for all methods to 90% .
The accuracy results are mostly the same as when pruning a single layer, with FeTa achieving the same or marginally worse results while enjoying signiﬁcant computation speedups for MNIST.
At the same time there is a larger degradation for the CIFAR experiment.
One signiﬁcant difference is that Thresholding achieves a notably bad result of 64% accuracy, which makes the method essentially inapplicable for multilayer pruning.
4.2. L OW R ANK R EGULARISATION
As a proof of concept for the generality of our approach we apply our method while imposing lowrank regularisation on the learned matrix U .
For low rank k we compare two methods (i) FeTa 1 with Ω( U )
=
||
U || (cid:63) and optimised with Acc-Prox-SVRG and (ii) Hard Thresholding of singular values using the truncated SVD deﬁned as U = N Σ V (cid:63) , Σ = diag ( { σ i } 1 ≤ i ≤ k ) .
We plot the results in
Figure 3.
[Figure]
In the above given U ∈ R d 1 × d 2 the Commpression Ratio (CR) is deﬁned as CR =
( k ∗ d 1 + k + k ∗ d 2 ) /
( d 1 ∗ d 2 ) .
The results are in line with the l 1 regularisation, with signiﬁcant degredation in classiﬁcation accuracy for Hard Thresholding above 85% CR.
4. G ENERALIZATION E RROR
According to our theoretical analysis the GE drops exponentially as the pruning moves away from the output layer.
To corroborate this we train a LeNet-5 to high accuracy, then we pick a single layer and gradually increase its sparsity using Hard Thresholding.
We ﬁnd that the layers closer to the input are exponentially less robust to pruning, in line with our theoretical analysis.
We plot the results in Figure 4.a.
For some layers there is a sudden increase in accuracy around 90% sparsity which could be due to the small size of the DNN.
We point out that in empirical results Raghu et al.
(2016) Han et al.
(2015b)
for much larger networks the degradation is entirely smooth.
Next we test our multilayer pruning bound.
We prune to the same sparsity levels all layers in the sets i ≥ 0 , i ≥ 1 , i ≥ 2 , i ≥ 3 .
We plot the results in Figure 4.b.
It is evident that the accuracy loss for layer groups is not simply the addition of the accuracy losses of the individual layers, but shows an exponential drop in accordance with our theoretical result.
[Figure]
We now aim to see how well our bound captures this exponential behaviour.
We take two networks g a pruned at layer 3 and g b pruned at layers 2 and 3 and make a number of simplifying assumptions.
First we assume that in Theorem 3.3 B = 0 such that GE
[Equation]
g
[Equation]
) ≤ A · ( γ −
[Equation]
|
[Equation]
[Equation] k 2 .
This is logical as B includes only log terms.
Assuming that the bounds are tight we now aim to calculate
[Equation]
We know that the value of this ratio for 90% sparsity is GE ( g a ) /GE
( g b ) ≈ 0 .
4 = 1 /
and we have managed to avoid the cumbersome A parameter.
Next we make the assumption that k ≈ 40 , this is common for the MNIST dataset and results from a simple dimensionality analysis using
PCA.
We also deviate slightly from our theory by using the average layerwise errors E s ∼ S
[ (cid:112) C ai 2 ] ,
E s ∼ S
[ (cid:112) C bi 2 ] , as well as the average scores E s ∼ S
[ o a ( x , g a ( x ))] , E s ∼ S
[ o b ( x , g b ( x ))] .
We calculate
[Equation]
which is very close to the empirical value.
C ONCLUSION
In this paper we have presented an efﬁcient pruning algorithm for fully connected layers of DNNs, based on difference of convex functions optimisation.
Our algorithm is orders of magnitude faster than competing approaches while allowing for a controlled degradation in the Generalization Error.
We provided a theoretical analysis of the degradation in GE resulting from our pruning algorithm.
This analysis validates the previously observed phenomenon that network layers closer to the input are exponentially less robust to pruning compared to layers close to the output.
Our theoretical analysis is of value by itself as it holds for any kind of bounded pertubation to one or multiple hidden DNN layers.
Experiments on common feedforward architectures validate our results.
A PPENDIX A. P ROOF OF THEOREM 3.1.
We denote by f 1 ( · , W 1 ) the original representation and by f 2 ( · , W 2 ) the pruned representation.
We assume that, after training, ∀ s ∈ S m || f 1 ( a , W 1 ) − f 2 ( a , W 2 )
||
22 ≤ C 1 .
Second, we assume that
∀ s ∈ S ∃ s i ∈ S
m ⇒ ||
a − a
i || 22 ≤ (cid:15) .
Third the linear operators in W 1 , W 2 are frames with upper frame bounds B 1 , B 2 respectively.
The following two lemmas will be useful:
Lemma 6.1.
The operator f 1 ( · , W 1 ) is Lipschitz continuous with upper Lipschitz constant B 1 .
Proof.
See Bruna et al.
(2013) for details, the derivation is not entirely trivial due to the nonsmoothness of the rectiﬁer non-linearity.
Lemma 6.2.
The operator f 2 ( · , W 2 ) is Lipschitz continuous with upper Lipschitz constant B 2 .
Proof.
We see that: ddx ρ ( x ) =
ddx 1 β log (1 + exp ( βx ))
=
exp ( βx ) 1+ exp ( βx ) ≤ 1 .
Therefore the smooth approximation to the rectiﬁer non-linarity is Lipschitz smooth with Lipschitz constant k = 1 .
Then || f 2 ( x, W 2 ) − f 2 ( y, W 2 )
||
22 ≤ k
||
W 2 x − W 2 y || 22 ≤
|| W 2 ( x − y )
||
22 ≤ B 2 || x − y
||
We drop the W i from the layer notation for clarity.
Using the triangle inequality ||
f
[Equation]
a
[Equation] [Equation]
a
[Equation] [Equation]
where we used Lemma 6.1 and Lemma 6.2 in line 5.
B. P ROOF OF THEOREM 3.2.
We will proceed as follows.
We ﬁrst introduce some prior results which hold for the general class of robust classiﬁers.
We will then give speciﬁc prior generalization error results for the case of classiﬁers operating on datapoints from C m -regular manifolds.
Afterwards we will provide prior results for the speciﬁc case of DNN clasiﬁers.
Finally we will prove our novel generalization error bound and provide a link with prior bounds.
We ﬁrst formalize robustness for generic classiﬁers g ( x ) .
In the following we assume a loss function l ( g ( x ) , y ) that is positive and bounded 0 ≤ l ( g ( x ) , y ) ≤ M .
Deﬁnition 6.1.
An algorithm g ( x ) is ( K, (cid:15) ( S m )) robust if S can be partitioned into K disjoint sets, denoted by { T t } Kt =1 , such that ∀ s i ∈
S m , ∀ s ∈ S ,
[Equation]
Now let ˆ l ( · ) and l emp ( · ) denote the expected error and the training error, i.e
[Equation]
we can then state the following theorem from Xu & Mannor (2012):
Theorem 6.3.
If S m consists of m i.i.d.
samples, and g ( x ) is ( K, (cid:15) ( S m ))
-robust, then for any δ > 0 , with probability at least 1 − δ ,
[Equation]
The above generic bound can be speciﬁed for the case of C m -regular manifolds as in Sokolic et al.
We recall the deﬁnition of the sample margin γ
( s i ) as well as the following theorem:
Theorem 6.4.
If there exists γ such that
[Equation]
then the classiﬁer g ( x ) is ( N Y · N ( X ; d, γ/ 2) , 0) -robust.
By direct substitution of the above result and the deﬁniton of a C m -regular manifold into Theorem 6.3 we get:
Corollary 6.4.1.
Assume that X is a (subset of) C M regular k − dimensional manifold, where
N ( X ; d, ρ ) ≤ ( C M ρ ) k .
Assume also that classiﬁer g ( x ) achieves a classiﬁcation margin γ and take l ( g ( x ) , y ) to be the 0 − 1 loss.
Then for any δ > 0 , with probability at least 1 − δ ,
[Equation]
Note that in the above we have used the fact that l ( g ( x ) , y ) ≤ 1 and therefore M = 1 .
The above holds for a wide range of algorithms that includes as an example SVMs.
We are now ready to specify the above bound for the case of DNNs, adapted from Sokolic et al.
Theorem 6.5.
Assume that a DNN classiﬁer g ( x ) , as deﬁned in equation 8, and let ˜ x be the training sample with the smallest score o (˜ s ) > 0 .
Then the classiﬁcation margin is bounded as
[Equation]
We now prove our main result.
We will denote by ˜ x = arg min s i ∈ S m min
j
[Equation]
x
[Equation]
v Tg
[Equation]
x
[Equation]
x i ) the training sample with the smallest score.
For this training sample we will denote j (cid:63) =
arg min j (cid:54) =
g (˜ x ) v Tg (˜ x ) j f (˜ x ) the second best guess of the classiﬁer g ( · ) .
Throughout the proof, we will use the notation v ij = √ 2( δ i − δ j ) .
First we assume the score o
(˜ x , g 1 (˜ x )) of the point ˜ x for the original classiﬁer g 1 ( x ) .
Then, for the second classiﬁer g 2 ( x ) , we take a point x (cid:63) that lies on the decision boundary between g 2 (˜ x ) and j (cid:63) such that
o 2 ( x (cid:63) , g 2 (˜ x ))
= 0 .
We assume for simplicity that, after pruning, the classiﬁcation decisions do not change such that g 1 (˜ x ) =
g 2 (˜ x ) .
We then make the following calculations
[Equation]
where we used Theorem 3.1 in line 5, since x (cid:63) is not a training sample.
From the above we can therefore write
[Equation]
By following the derivation of the margin from the original paper Sokolic et al.
and taking into account the deﬁnition of the margin we know that
[Equation] [Equation]
Therefore we can ﬁnally write γ
[Equation]
[Equation]
The theorem follows from direct application of Corollary 3.1.1.
Note that if γ −
[Equation]
0 the derived bound becomes vacuous, as by deﬁnition 0 ≤ γ 2 (˜ x ) .
C. P ROOF OF THEOREM 3.3.
We start as in theorem 3.2 by assuming the score o 1 (˜ x , g 1 (˜ x )) of the point ˜ x for the original classiﬁer g 1 ( x ) .
Then, for the second classiﬁer g 2 ( x ) , we take a point x (cid:63) that lies on the decision boundary between g 2 (˜ x ) and j (cid:63) such that
o 2 ( x (cid:63) , g 2 (˜ x ))
= 0 .
We assume as before that the classiﬁcation decisions do not change such
that g 1 (˜ x ) =
g 2 (˜ x ) .
We write
[Equation]
We can then write
[Equation]
Then as before
[Equation]
The theorem follows from direct application of Corollary 3.1.1.