I RDA M ETHOD FOR S PARSE C ONVOLUTIONAL N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a new approach, known as the iterative regularized dual averaging (iRDA), to improve the efﬁciency of convolutional neural networks (CNN) by signiﬁcantly reducing the redundancy of the model without reducing its accuracy.
The method has been tested for various data sets, and proven to be signiﬁcantly more efﬁcient than most existing compressing techniques in the deep learning literature.
For many popular data sets such as MNIST and CIFAR-10, more than 95% of the weights can be zeroed out without losing accuracy.
In particular, we are able to make ResNet18 with 95% sparsity to have an accuracy that is comparable to that of a much larger model ResNet50 with the best 60% sparsity as reported in the literature.
I NTRODUCTION
In recent decades, deep neural network models have achieved unprecedented success and state-ofthe-art performance in various tasks of machine learning or artiﬁcial intelligence, such as computer vision, natural language processing and reinforcement learning Lecun et al.
Deep learning models usually involve a huge number of parameters to ﬁt variant kinds of datasets, and the number of data may be much less than the amount of parameters
He et al.
This may implicate that deep learning models have too much redundancy.
This can be validated by the literatures from the general pruning methods Pratt (1988) to the compressing models Han et al.
(2015a).
While compressed sensing techniques have been successfully applied in many other problems, few reports could be found in the literature for their application in deep learning.
The idea of sparsifying machine learning models has attracted much attention in the last ten years in machine learning
Donoho (2006); Xiao (2010).
When considering the memory and computing cost for some certain applications such as Apps in mobile, the sparsity of parameters plays a very important role in model compression Han et al.
(2015a)
; Cheng et al.
The topic of computing sparse neural networks can be included in the bigger topic on the compression of neural networks, which usually further involves the speedup of computing the compressed models.
There are many sparse methods in machine learning models such as FOBOS method Duchi and
Singer (2009), also known as proximal stochastic gradient descent (prox-SGD) methods Mine and
Fukushima (1981), proposed for general regularized convex optimization problem, where (cid:96) 1 is a common regularization term.
One drawback of prox-SGD is that the thresholding parameters will decay in the training process, which results in unsatisfactory sparsity Xiao (2010).
Apart from that, the regularized dual averaging (RDA) method Xiao (2010), proposed to obtain better sparsity, has been proven to be convergent with speciﬁc parameters in convex optimization problem, but has not been applied in deep learning ﬁelds.
In this paper, we analyze the relation between simple dual averaging (SDA) method Nesterov (2009) and the stochastic gradient descent (SGD) method Robbins and Monro (1951), as well as the relation between SDA and RDA.
It is well-known that SGD and its variants work quite well in deep learning problems.
However, there are few literatures in applying pure training algorithms to deep CNNs for model sparsiﬁcation.
We propose an iterative RDA (iRDA) method for training sparse CNN models, and prove the convergence under convex conditions.
Numerically, we compare prox-SGD with iRDA, where the latter can achieve better sparsity results while keeping satisfactory accuracy on MNIST, CIFAR-10 and CIFAR-100.
We also show iRDA works for different CNN models such as VGG Simonyan and Zisserman (2014) and
He et al.
Finally, we compare the performance of iRDA with some other state-of-the-art compression methods.
R ELATED W ORKS
Cheng et al.
(2017) reviews the work on compressing neural network models, and categorizes the related methods into four schemes: parameter pruning and sharing, low-rank factorization, transfered/compact convolutional ﬁlters and knowledge distillation.
Among them, Liu et al.
(2015) uses sparse decomposition on the convolutional ﬁlters to get sparse neural networks, which could be classiﬁed to the second scheme.
Apart from that, Han et al.
(2015b) prunes redundant connections by learning only the important parts.
Louizos et al.
(2017) starts from a Bayesian point of view, and removes large parts of the network through sparsity inducing priors.
Yin et al.
He et al.
(2018) combines reinforcement learning methods to compression.
Li and Hao (2018) considers deep learning as a discrete-time optimal control problem, and obtains sparse weights on ternary networks.
Recently, Feng (2018) applies RDA to fully-connected neural network models on MNIST.
A LGORITHMS
Let z = ( x, y ) be an input-output pair of data, such as a picture and its corresponding label in a classiﬁcation problem, and f ( w, z ) be the loss function of neural networks, i.e. a scalar function that is differentiable w.r.t.
weights w .
We are interested in the expected risk minimization problem
[Equation]
The empirical risk minimization
[Equation]
is an approximation of (1) based on some ﬁnite given samples { z 1 , z 2 , . .
, z T } , where T is the size of the sample set.
Regularization is a useful technique in deep learning.
In general, the regularized expected risk minimization has the form
[Equation]
where Ψ( w ) is a regularization term with certain effect.
For example, Ψ( w ) =
(cid:107) w (cid:107)
may improve the generalization ability, and an (cid:96) 1 -norm of w can give sparse solutions.
The corresponding regularized empirical risk minimization we concern takes the form
[Equation]
SDA method is a special case of primal-dual subgradient method ﬁrst proposed in Nesterov (2009).
Xiao (2010) proposes RDA for online convex and stochastic optimization.
RDA not only keeps the same convergence rate as Prox-SGD, but also achieves more sparsity in practice.
In next sections, we will discuss the connections between SDA and SGD, as well as RDA and
Prox-SGD.
We then propose iRDA for (cid:96) 1 regularized problem of deep neural networks.
3. S IMPLE D UAL A VERAGING M ETHOD
As a solution of (2), SDA takes the form
[Equation]
The ﬁrst term (cid:80) tτ
=1 (cid:104)
g τ , w (cid:105) is a linear function obtained by averaging all previous stochastic gradient.
g t is the subgradient of f t .
The second term h ( w ) is a strongly convex function, and { β t } is a nonnegative and nondecreasing sequence which determines the convergence rate.
As g τ ( w τ ) , τ = 1 , . .
, t − 1 is constant in current iteration, we use g τ instead for simplicity in the following.
Since subproblem equation 5 is strongly convex, it has a unique optimal solution w t +1 .
Let w 0 be the initial point, and h ( w ) = 12 (cid:107) w − w 0 (cid:107) 22 , the iteration scheme of SDA can be written as
[Equation]
where ¯ g t = 1 t (cid:80) tτ =1 (cid:104)
g τ , w (cid:105) .
Let β t = γt α , SDA can be rewritten recursively as
[Equation]
where (cid:0) 1 − (cid:0) 1 − 1 t (cid:1) α (cid:1) → 0 and (cid:0) 1 − 1 t (cid:1) α
→ 1 as t → ∞ .
Thus, SDA can be viewed as a perturbation of SGD.
3. P ROXIMAL S TOCHASTIC G RADIENT D ESCENT AND R EGULARIZED D UAL A VERAGING M ETHODS
For the regularized problem (4), we recall the well-known Prox-SGD and RDA method ﬁrst.
At each iteration, Prox-SGD solves the subproblem
[Equation]
Speciﬁcally, α t = 1 γ √ t obtains the best convergence rate.
The ﬁrst two terms are an approximation of the original objective function.
Note that without the regularization term Ψ , equation 8 is equivalent to SGD.
It can be written in forward-backward splitting (FOBOS) scheme
[Equation]
where the forward step is equivalent to SGD, and the backward step is a soft-thresholding operator working on w t
+ 12 with the soft-thresholding parameter α t .
Different from Prox-SGD, each iteration of RDA takes the form
[Equation]
[Equation]
Similarly, taking h ( w ) = 1 2 (cid:107) w − w 0 (cid:107) 2 2 , RDA can be written as
[Equation]
or equivalently,
[Equation]
where β t = γ √ t to obtain the best convergence rate.
From equation 14, one can see that the forward step is actually SDA and the backward step is the soft-thresholding operator, with the parameter t/β t .
[Equation]
R EGULARIZATION AND THE S PARSITY
Set Ψ( w ) =
λ (cid:107) w (cid:107) 1 .
The problem (4) then becomes
[Equation]
where λ is a hyper-parameter that determines sparsity.
In this case, from Xiao’s analysis of RDA Xiao (2010), the expected cost E φ ( ¯ w t ) − φ (cid:63) associated with the random variable ¯ w t converges with rate O ( 1 √ t ) when β t = γ √ t .
This convergence rate is consistent with FOBOS Duchi and Singer (2009).
However, both results assume f to be a convex function, which can not be guaranteed in deep learning.
Nevertheless, we can still verify that RDA is a powerful sparse optimization method for deep neural networks.
We conclude the closed form solutions of Prox-SGD and RDA for equation 16 as follows.
The subproblem of Prox-SGD
[Equation]
has the closed form solution
[Equation]
The subproblem of RDA
[Equation]
has
the closed
form solution
[Equation]
The √ t -proximal stochastic gradient method has the form
[Equation]
The difference between √ t -Prox-SGD and Prox-SGD is the soft-thresholding parameter chosen to be √ t .
It has the closed form solution
[Equation]
t λ, w ( i ) t − α t g
( i ) t < − tβ t λ.
It is equivalent to
[Equation]
where the objective function is actually an approximation of
[Equation]
We can easily conclude that this iteration will converge to w =
if α t = 1 γ √ t and β t = γ √ t .
Now compare the threshold λ PG =
α t λ of PG and the threshold λ RDA = tβ t λ of RDA.
With α t = 1 γ √ t and β t = γ √ t , we have λ PG → 0 and λ RDA → ∞ as t → 0 .
It is clear that RDA uses a much more aggressive threshold, which guarantees to generate signiﬁcantly more sparse solutions.
3. I TERATIVE RDA M ETHOD FOR D EEP N EURAL N ETWORKS
Note that when Ψ = λ (cid:107) w (cid:107) 1 , RDA requires w 1 = w 0 = 0 .
However, this will make deep neural network a constant function, with which the parameters can be very hard to update.
Thus, in Algorithm 1, we modify the RDA method as Step 1, where w 1 can be chosen not equal to 0, and add an extra Step 2 to improve the performance.
We also prove the convergence rate of Step 1 for convex problem is O ( 1 √ t ) when β t
= O ( √ t ) .
Theorem 3.1 Assume there exists an optimal solution w (cid:63) to the problem (3) with Ψ( w )
=
λ (cid:107) w (cid:107)that satisﬁes h ( w (cid:63) ) ≤ D 2 for some D > 0 , and let φ (cid:63) =
φ ( w (cid:63) ) .
Let the sequences { w t } t ≥ 1 be generated by Step 1 in iRDA, and assume (cid:107) g t (cid:107) ∗ ≤ G for some constant G .
Then the expected cost
E φ ( ¯ w t ) converges to φ (cid:63) with rate O ( 1 √ t ) E φ (
[Equation]
, with ¯ w t = 1 t (cid:80) tτ
=1 w τ .
See Appendix A for the proof.
3. I NITIALIZATION
To apply iRDA, the weights of a neural network should be initialized differently from that in a normal optimization method such as SGD or its variants.
Our initialization is based on LeCun et al.
(2012), Glorot and Bengio (2010) and He et al.
(2015), with an additional re-scaling.
Let s be a scalar, the mean and the standard deviation of the uniform distribution for iRDA is zero and
[Equation]
respectively, where c is the number of channels, and k is the spatial ﬁlter size of the layer (see He
et al.
Choosing a suitable s is important when applying iRDA.
As shown in Table 5 and Table 6 in Appendix B, if s is too small or too large, the training process could be slowed down and the generalization ability may be affected.
Moreover, a small s usually requires much better initial weights, which results in too many samplings in initialization process.
In our experiments, a good s for iRDA is usually much larger than √ 2 , and unsuitable for SGD algorithms.
3. I TERATIVE R ETRAINING
Iterative retraining is a method that only updates the non-zero parameters at each iteration.
A trained model can be further updated with retraining, thus both the accuracies and sparsity can be improved.
See Table 4 for comparisons on CIFAR-10.
Input:
• A strongly convex function h (
[Equation]
) =
[Equation]
• A nonnegative and nondescreasing sequence β
[Equation]
= γ
[Equation]
t . Step 1: RDA with proper initialization
Initialize: set w 0 = 0 , ¯ g 0 =
and randomly choose w 1 with methods explained in section 3.5.
for t=1,2, ..., T do
Given the sample z i t and corresponding loss function f i t .
Compute the stochastic gradient
[Equation]
Update the average gradient:
[Equation]
Compute the next weight vector:
[Equation]
Step 2: iterative retraining for t=
T+1,T+2,T+3, ... do
Given the sample z i t and corresponding loss function f i t .
Compute the stochastic gradient
[Equation]
Set ( g t )
j =
if ( w t ) j = 0 for every j .
Update the average gradient:
[Equation]
Compute the next weight vector:
[Equation]
E XPERIMENTS
In this section, σ denotes the sparsity of a model, i.e
[Equation]
All neural networks are trained with mini-batch size 128.
4. P ARAMETERS T EST
We provide a test on different hyper-parameters, so as to give an overview of their effects on performance, as shown in Table 7.
We also show that the sparsity and the accuracy can be balanced with iRDA by adjusting the parameters λ and γ , as shown in Table 8.
Both tables are put in Appendix C.
4. M AIN R ESULTS
We compare iRDA with several methods including prox-SGD, √ t − SGD and normal SGD, on different datasets including MNIST, CIFAR-10, CIFAR-100 and ImageNet(ILSVRC2012).
The main results are shown in Table 1.
Table 2 shows the performance of iRDA on different architectures including ResNet18, VGG16 and VGG19.
Table 3 shows the performance of iRDA on different datasets including MNIST, CIFAR-10, CIFAR-100 and ImageNet(ILSVRC2012).
In all tables, SGD denotes stochastic gradient methods with momentum Ruder (2016).
4. C OMPARISON
Currently, many compression methods include human experts involvement.
Some methods try to combine other structures in training process to automatize the compression process.
For example,
He et al.
(2018) combines reinforcement learning.
iRDA, as an algorithm, requires no extra structure.
As shown above, iRDA can achieve good sparsity while keeping accuracy automatically, with carefully chosen parameters.
For CIFAR-10, we compare the performance of iRDA with some other state-of-art compression methods in Table 4.
Due to different standards, σ is referred to directly or computed from the original papers approximately.
C ONCLUSION
In comparison with many existing rule-based heuristic approaches, the new approach is based on a careful and iterative combination of (cid:96) 1 regularization and some specialized training algorithms.
We ﬁnd that the commonly used training algorithms such as SGD methods are not effective.
We thus develop iRDA method that can be used to achieve much better sparsity.
iRDA is a variant of
RDA methods that have been used for some special types of online convex optimization problems in the literature.
New elements in the iRDA mainly consist of judicious initialization and iterative retraining.
In addition, iRDA method is carefully analyzed on its convergence for convex objective functions.
Many deep neural networks trained by iRDA can achieve good sparsity while keeping the same validation accuracy as those trained by SGD with momentum on many popular datasets.
This result shows iRDA is a powerful sparse optimization method for image classiﬁcation problems in deep learning ﬁelds.
[Table] [Table]
[Table] [Table]
A P ROOF OF T HEOREM 3.
One of the differences between RDA Xiao (2010) and iRDA is that the former one takes w 1 = arg min w h ( w ) whereas the latter one chooses w 1 randomly.
In the following, we will prove the convergence of iRDA
Step 1 for convex problem.
The proofs use Lemma 9, Lemma 10, Lemmadirectly and modify Theorem 1 and Theorem 2 in Xiao (2010).
For clarity, we have some general assumptions:
• The regularization term Ψ( w ) is a closed convex function with convexity parameter σ and dom Ψ is closed.
• For each t ≥ 1 , f t ( w ) is convex and subdifferentiable on dom Ψ .
• h ( w ) is strongly convex on dom Ψ and subdifferentiable on rint(dom Ψ ) and also satisﬁes
[Equation]
Without loss of generality, assume h ( w ) has convexity parameter 1 and min w h ( w )
= 0 .
• There exist a constant G such that
[Equation]
• Require { β } t ≥ 1 be a nonnegative and nondecreasing sequence and
[Equation]
Moreover, we could always choose β 1 ≥ σ such that β 0 = β 1 .
• For a random choosing w 1 , we assume
[Equation]
First of all, we deﬁne two functions:
[Equation] [Equation]
The maximum in (37) is always achieved because F D = { w ∈ dom Ψ | h ( w ) ≤ D 2 } is a nonempty compact set.
Because of (35), we have σt + β t ≥ β 0 > 0 for all t ≥ 0 , which means t Ψ( w )+ β t h ( w ) are all strongly convex, therefore the maximum in (38) is always achieved and unique.
As a result, we have dom U t = dom V t =
E ∗ for all t ≥ 0 .
Moreover, by the assumption (33), both of the functions are nonnegative.
Let s t denote the sum of the subgradients obtained up to time t in iRDA
Step 1, that is
[Equation]
and π t ( s ) denotes the unique maximizer in the deﬁnition of V t ( s )
[Equation]
which then gives
[Equation]
Lemma A.1 For any s ∈ E ∗ and t ≥ 0 , we have
[Equation]
For a proof, see Lemma 9 in Xiao (2010).
Lemma A.
The function V t is convex and differentiable.
Its gradient is given by
[Equation]
and the gradient Lipschitz continuous with constant 1 / ( σt + β t ) , that is
[Equation]
Moreover, the following inequality holds:
[Equation]
The results are from Lemma 10 in Xiao (2010).
Lemma A.3 For each t ≥ 1 , we have
[Equation]
Since h ( w t +1 ) ≥ 0 and the sequence { β t } t ≥ 1 is nondecreasing, we have
[Equation]
To prove this lemma, we refer to the Lemma 11 in Xiao (2010).
What’s more, from the assumption 35, we could always choose β 1 ≥ σ such that β 1 = β 0 and
[Equation]
The learner’s regret of online learning is the difference between his cumulative loss and the cumulative loss of the optimal ﬁxed hypothesis, which is deﬁned by
[Equation]
and bounded by
[Equation]
Lemma A.4 Let the sequence { w t } t ≥ 1 and { g t } t ≥ 1 be generated by iRDA
Step 1, and assume (34) and (35) hold.
Then for any t ≥ 1 and any w ∈ F D = { w ∈ dom Ψ | h ( w ) ≤ D 2 } , the regret deﬁned in (50) is bounded by ∆ t
[Equation]
Proof First, we deﬁne the following gap sequence which measures the quality of the solutions w 1 , .., w t :
[Equation]
and δ t is an upper bound on the regret R t ( w ) for all w ∈ F D , to see this, we use the convexity of f t ( w ) in the following:
[Equation]
Then, We are going to derive an upper bound on δ t .
For this purpose, we subtract (cid:80) tτ
=1 (cid:104)
g τ , w 0 (cid:105) in (53), which leads to
[Equation]
the maximization term in (55) is in fact U t ( − s t ) , therefore, by applying Lemma A.1, we have
[Equation]
Next, we show that ∆ t is an upper bound for the right-hand side of inequality (56).
We consider τ ≥ 2 and τ = 1 respectively.
For any τ ≥ 2 , we have
[Equation]
where (47),(39),(45) and (43) are used.
Therefore, we have
[Equation]
For τ = 1 , we have a similar inequality by using (49)
[Equation]
Summing the above inequalities for τ = 1 , ..., t and noting that V
[Equation]
− s 0 )
=
V
= 0 , we arrive at
[Equation]
Since Ψ( w t +1 ) ≥ 0 , we subtract it from the left hand side and add Ψ( w 1 ) to both sides of the above inequality yields
[Equation]
Combing (54), (56), (57) and using assumption (34) and (36)we conclude
[Equation]
Lemma A.5 Assume there exists an optimal solution w (cid:63) to the problem (3) that satisﬁes h ( w (cid:63) ) ≤
D 2 for some D > 0 , and let φ (cid:63) =
φ ( w (cid:63) ) .
Let the sequences { w t } t ≥ 1 be generated by iRDA
Step 1, and assume (cid:107) g t (cid:107) ∗ ≤ G for some constant G .
Then for any t ≥ 1 , the expected cost associated with the random variable ¯ w t is bounded as
[Equation]
Proof First, from the deﬁnition (50), we have the regret at w (cid:63)
[Equation]
(cid:63)
[Equation]
Let z [ t ] denote the collection of i.i.d.
random variables ( z , ..., z t ) .
We note that the random variable w τ , where 1 ≤ w ≥ t , is a function of ( z 1 , ..., z τ − 1 ) and is independent of ( z τ , ..., z t ) .
Therefore
E z [ t ] ( f ( w τ , z τ ) + Ψ( w τ )) =
E z
[ τ − 1] ( E τ f ( w τ , z τ ) + Ψ( w τ ))
=
E z
[ τ − 1] φ ( w τ ) =
E z [ t ] φ ( w τ ) , and
[Equation]
Since φ (cid:63) =
φ ( w (cid:63) ) =
min w φ ( w ) , we have the expected regret
[Equation]
Then, by convexity of φ , we have
[Equation]
Finally, from (59) and (58), we have
[Equation]
Then the desired follows from that of Lemma A.4.
Proof of Theorem 3.1 From Lemma A.5, the expected cost associated with the random variable ¯ w t is bounded as
[Equation]
Here, we consider (cid:96) 1 regularization function Ψ( w )
=
λ (cid:107) w (cid:107)
and it is a convex but not strongly convex function, which means σ = 0 .
Now, we consider how to choose β t for t ≥ 1 and β 0 = β 1 .
First if β t = γt , we have 1 t · γtD 2 = γD 2 , which means the expected cost does not converge.
Then assume β t = γt α , α > 0 and α (cid:54)
= 1 , the right hand side of the inequality (60) becomes
[Equation]
t − 1 (cid:88) τ =
[Equation]
(cid:1)(cid:35)
[Equation]
From above
[Table]
we see that if 0 < α <
[Table]
the expected cost converges and the optimal convergence rate O ( t − 12 ) achieves when α = 12 .
Then we proved the Theorem 3.1.
B I NITIALIZATION
[Table] [Table]
C P ARAMETERS T EST
[Table] [Table]
λ = 10 −90.
σ 0.
λ
= 10 −90.
N/A 90.
σ 0.
N/A 0.
λ
= 10 −91 .91 .
.88 .σ 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .