S PLIT LBI FOR D EEP L EARNING : S TRUCTURAL S PARSITY VIA D IFFERENTIAL I NCLUSION P ATHS
Anonymous authors
Paper under double-blind review
A BSTRACT
Over-parameterization is ubiquitous nowadays in training neural networks to beneﬁt both optimization in seeking global optima and generalization in reducing prediction error.
However, compressive networks are desired in many real world applications and direct training of small networks may be trapped in local optima.
In this paper, instead of pruning or distilling over-parameterized models to compressive ones, we propose a new approach based on differential inclusions of inverse scale spaces , that generates a family of models from simple to complex ones by coupling gradient descent and mirror descent to explore model structural sparsity.
It has a simple discretization, called the Split Linearized Bregman Iteration (SplitLBI), whose global convergence analysis in deep learning is established that from any initializations, algorithmic iterations converge to a critical point of empirical risks.
Experimental evidence shows that SplitLBI may achieve comparable and even better performance than other training algorithms on ResNet-18 in large scale training on ImageNet-2012 dataset etc. , while with early stopping it unveils effective subnet architecture with comparable test accuracy to dense models after retraining instead of pruning well-trained ones.
I NTRODUCTION
The expressive power of deep neural networks comes from the millions of parameters, which are optimized by Stochastic Gradient Descent (SGD) (Bottou, 2010) and variants like Adam (Kingma &
Ba, 2015).
Remarkably, model over-parameterization helps both optimization and generalization.
For optimization, over-parameterization may simplify the landscape of empirical risks toward locating global optima efﬁciently by gradient descent method (Mei et al., 2018; 2019; Venturi et al., 2018;
Allen-Zhu et al., 2018; Du et al., 2018).
On the other hand, over-parameterization does not necessarily result in a bad generalization or overﬁtting (Zhang et al., 2017), especially when some weight-size dependent complexities are controlled (Bartlett, 1997; Bartlett et al., 2017; Golowich et al., 2018;
Neyshabur et al., 2019).
However, compressive networks are desired in many real world applications, e.g. robotics, selfdriving cars, and augmented reality.
Despite that (cid:96) 1 regularization has been applied to deep learning to enforce the sparsity on weights toward compact, memory efﬁcient networks, it sacriﬁces some prediction performance (Collins & Kohli, 2014).
This is because that the weights learned in neural networks are highly correlated, and (cid:96) 1 regularization on such weights violates the incoherence or irrepresentable conditions needed for sparse model selection (Donoho & Huo, 2001; Tropp, 2004;
Zhao & Yu, 2006), leading to spurious selections with poor generalization.
On the other hand, (cid:96)regularization is often utilized for correlated weights as some low-pass ﬁltering, sometimes in the form of weight decay (Loshchilov & Hutter, 2019) or early stopping (Yao et al., 2007; Wei et al., 2017).
Furthermore, group sparsity regularization (Yuan & Lin, 2006) has also been applied to neural networks, such as ﬁnding optimal number of neuron groups (Alvarez & Salzmann, 2016) and exerting good data locality with structured sparsity (Wen et al., 2016; Yoon & Hwang, 2017).
Yet, without the aid of over-parameterization, directly training a compressive model architecture may meet the obstacle of being trapped in local optima in contemporary experience.
Alternatively, researchers in practice typically start from training a big model using common task datasets like
ImageNet, and then prune or distill such big models to small ones without sacriﬁcing too much of the performance (Jaderberg et al., 2014; Han et al., 2015; Zhu et al., 2017; Zhou et al., 2017; Zhang et al., Li et al., 2017; Abbasi-Asl & Yu, 2017; Yang et al., 2018; Arora et al., 2018).
In particular, a recent study (Frankle & Carbin, 2019) created the lottery ticket hypothesis based on empirical observations: “dense, randomly-initialized, feed-forward networks contain subnetworks (winning tickets)
that – when trained in isolation – reach test accuracy comparable to the original network in a similar number of iterations".
How to effectively reduce an over-parameterized model thus becomes the key to compressive deep learning.
Yet, Liu et al.
(2019) raised a question, is it necessary to fully train a dense, over-parameterized model before ﬁnding important structural sparsity ?
In this paper, we provide a novel answer by exploiting a dynamic approach to deep learning with structural sparsity.
We are able to establish a family of neural networks, from simple to complex, by following regularization paths as solutions of differential inclusions of inverse scale spaces .
Our key idea is to design some dynamics that simultaneously exploit over-parameterized models and structural sparsity.
To achieve this goal, the original network parameters are lifted to a coupled pair, with one weight set W of parameters following the standard gradient descend to explore the over-parameterized model space, while the other set of parameters learning structure sparsity in an inverse scale space , i.e. , structural sparsity set Γ . The large-scale important parameters are learned at a fast speed while the small unimportant ones are learned at a slow speed.
The two sets of parameters are coupled in an (cid:96) 2 regularization.
The dynamics enjoys a simple discretization, i.e. the Split
Linearized Bregman Iteration (SplitLBI), with provable global convergence guarantee shown in this paper.
Here, SplitLBI is a natural extension of SGD with structural sparsity exploration: SplitLBI reduces to the standard gradient descent method when the coupling regularization is weak, while it leads to a sparse mirror descent when the coupling is strong.
Critically, SplitLBI enjoys a nice property that important subnet architecture can be rapidly learned via the structural sparsity parameter Γ following the iterative regularization path, without fully training a dense network ﬁrst.
Particularly, the support set of structural sparsity parameter Γ learned in the early stage of this inverse scale space discloses important sparse subnet architectures. Such architectures can be ﬁne-tuned or retrained to achieve comparable test accuracy as the dense, over-parameterized networks.
As a result, the structural sparsity parameter Γ may enable us to rapidly ﬁnd “winning tickets” in early training epochs for the “lottery” of identifying successful subnetworks that bear comparable test accuracy to the dense ones.
This point is empirically validated in our experiments.
Historically, the Linearized Bregman Iteration (LBI) was ﬁrstly proposed in applied mathematics as iterative regularization paths for image reconstruction and compressed sensing (Osher et al., 2005;
Yin et al., 2008), later applied to logistic regression (Shi et al., 2013).
The convergence analysis was given for convex problems (Yin et al., 2008;
Cai et al., 2009), yet remaining open for non-convex problems met in deep learning.
Osher et al.
(2016) established statistical model selection consistency for high dimensional linear regression under the same irrepresentable condition as Lasso, later extended to generalized linear models (Huang & Yao, 2018).
To relax such conditions, SplitLBI was proposed by Huang et al.
(2016) to learn structural sparsity in linear models under weaker conditions than generalized Lasso, that was successfully applied in medical image analysis (Sun et al., 2017) and computer vision (Zhao et al., 2018).
In this paper, it is the ﬁrst time that SplitLBI is exploited to train highly non-convex neural networks with structural sparsity, together with a global convergence analysis based on the Kurdyka-Łojasiewicz framework Łojasiewicz (1963).
Contributions .
(1) SplitLBI, as an extension of SGD, is applied to deep learning by exploring both over-parameterized models and structural sparsity in the inverse scale space.
(2) Global convergence of SplitLBI in such a nonconvex optimization is established based on the KurdykaŁojasiewicz framework, that the whole iterative sequence converges to a critical point of the empirical loss function from arbitrary initializations.
(3) Stochastic variants of SplitLBI demonstrate the comparable and even better performance than other training algorithms on ResNet-18 in large scale training such as ImageNet-2012, among other datasets, together with additional structural sparsity in successful models for interpretability.
(4) Structural sparsity parameters in SplitLBI provide important information about subnetwork architecture with comparable or even better accuracies than dense models before and after retraining -- SplitLBI with early stopping can provide fast “ winning tickets ” without fully training dense, over-parameterized models.
M ETHODOLOGY
The supervised learning task learns a mapping Φ W : X → Y , from input space X to output space
Y , with a parameter W such as weights in neural networks, by minimizing certain loss functions on training samples (cid:98) L n ( W ) = 1 n (cid:80) ni =1 (cid:96) ( y i , Φ W ( x i )) .
For example, a neural network of l -layer is deﬁned as Φ W ( x ) =
σ l ( W l σ l − 1 ( W l − 1 · · · σ 1 ( W 1 x )) , where W = { W i } li =1 , σ i is the nonlinear activation function of the i -th layer.
Differential Inclusion of Inverse Scale Space.
Consider the following dynamics,
[Equation]
where V is a sub-gradient of ¯Ω(Γ) : = Ω
[Equation]
Γ) + 12 κ
[Equation] Γ
[Equation] 2 for some sparsity-enforced regularization Ω λ (Γ) =
λ Ω 1 (Γ) ( λ ∈ R + ) such as Lasso or group Lasso penalties for Ω 1 (Γ) , κ > 0 is a damping parameter such that the solution path is continuous, and the augmented loss function is
[Equation]
with ν > 0 controlling the gap admitted between W and Γ .
Compared to the original loss function (cid:98) L n ( W ) , the ¯ L ( W, Γ) additionally adopt the variable splitting strategy, by lifting the original neural network parameter W to ( W, Γ) with Γ modeling the structural sparsity of W .
For simplicity, we assumed ¯ L is differentiable with respect to W here, otherwise the gradient in Eq.
(1a) is understood as subgradient and the equation becomes an inclusion.
The differential inclusion system (1), called Split Inverse Scale Space (SplitISS), can be understood as a gradient descent ﬂow of W t in the proximity of Γ t and a mirror descent ﬂow (Nemirovski & Yudin, 1983) of Γ t associated with a sparsity enforcement penalty ¯Ω .
In mirror descent ﬂow, gradient descent goes on the dual space consisting of sub-gradients V t , driving the ﬂow in sparse primal space of Γ t .
For a large enough ν , it reduces to the gradient descent method for W t .
Yet the solution path of Γ t exhibits the following property in the separation of scales: starting at the zero, important parameters of large scale will be learned fast, popping up to be nonzeros early, while unimportant parameters of small scale will be learned slowly, appearing to be nonzeros late.
In fact, taking Ω λ (Γ) = (cid:107) Γ (cid:107) 1 and κ → ∞ for simplicity, V t as the subgradient of ¯Ω t , undergoes a gradient descent ﬂow before reaching the (cid:96) ∞ -unit box, which implies that Γ t = 0 in this stage. The earlier a component in V t reaches the (cid:96) ∞ -unit box, the earlier a corresponding component in Γ t becomes nonzero and rapidly evolves toward a critical point of ¯ L under gradient ﬂow. On the other hand, the W t follows the gradient descent with a standard (cid:96) 2 -regularization. Therefore, W t closely follows dynamics of Γ t whose important parameters are selected.
Such a property is called as the inverse scale space in applied mathematics (Burger et al., 2006) and recently was shown to achieve statistical model selection consistency in high dimensional linear regression (Osher et al., 2016) and general linear models (Huang & Yao, 2018), with a reduction of bias as κ increases.
In this paper, we shall see that the inverse scale space property still holds empirically for the highly nonconvex neural network training via Eq.
For example, Fig. 1 shows a LeNet trained on MNIST by the discretized dynamics, where important sparse ﬁlters are selected in early epochs while the popular SGD returns dense ﬁlters.
Compared with directly enforcing a penalty function such as (cid:96) 1 or (cid:96) 2 regularization
[Equation]
SplitISS avoids the parameter correlation problem in over-parameterized models.
In fact, a necessary and sufﬁcient condition for Lasso or (cid:96) 1 -type sparse model selection is the incoherence or irrepresentable conditions (Tropp (2004);
Zhao & Yu (2006))
that are violated for highly correlated weight parameters, leading to spurious discoveries.
In contrast, Huang et al.
(2018) showed that equipped with such a variable splitting where Γ enjoys an orthogonal design where the restricted Hessian of the augmented loss on Γ is orthogonal, the SplitISS can achieve model selection consistency under weaker conditions than generalized Lasso, relaxing the incoherence or irrepresentable conditions when parameters are highly correlated.
For weight parameter W , instead of directly being imposed with (cid:96) 1 -sparsity, it adopts (cid:96) 2 -regularization in the proximity of the sparse path of Γ that admits simultaneously exploring highly correlated parameters in over-parameterized models and sparsity regularization.
Split Linearized Bregman Iterations.
SplitISS admits an extremely simple discrete approximation, using the Euler forward discretization of dynamics (1):
[Equation]
where V
=
Γ
= 0 , W 0 can be small random numbers such as Gaussian distribution in neural networks, for some complex networks it can be initialized as common setting.
The proximal map in
Eq.
(4c) that controls the sparsity of Γ is given by
[Equation]
We shall call such an iterative procedure as Split Linearized Bregman Iteration (SplitLBI), that was ﬁrstly coined in Huang et al.
as an iterative regularization path for sparse modeling in high dimensional statistics.
In the application to neural networks, the loss becomes highly non-convex, the
SplitLBI returns a sequence of sparse models from simple to complex ones whose global convergence condition to be shown below, while solving Eq.
(3) at various levels of λ might not be tractable except for over-parameterized models.
The sparsity-enforcement penalty used in convolutional neural networks can be chosen as follows.
Our sparsity framework aims at regularizing the groups of weight parameters using group Lasso penalty (Yuan & Lin, 2006), Ω 1 (Γ) =
(cid:80) g (cid:107) Γ g (cid:107) 2 , where (cid:107) Γ g (cid:107)
2 = (cid:113)(cid:80) | Γ g | i =
(Γ gi ) 2 and | Γ g | is the number of weights in Γ g .
Thus Eq.
(4c) has a closed form solution Γ g = κ · max (0 , 1 − 1 / (cid:107) V g (cid:107) 2 )
V g for the g -th ﬁlter.
We treat convolutional and fully connected layers in different ways.
For a convolutional layer, Γ g = Γ g ( c in , c out , size ) denote the convolutional ﬁlters where size denotes the kernel size and c in and c out denote the numbers of input channels and output channels, respectively.
When we regard each group as each convolutional ﬁlter, g = c out ; otherwise for weight sparsity, g can be every
element in the
ﬁlter
that reduces to the Lasso. For a fully connected layer,
Γ =
Γ( c in , c out
) where c
in and c out denote the numbers of inputs and outputs of the fully connected layer.
Each group g corresponds to each element ( i, j ) , and the group Lasso penalty
degenerates
to the Lasso penalty  G LOBAL C ONVERGENCE OF S PLIT LBI FOR N EURAL N ETWORKS
We present a theorem that guarantees the global convergence of SplitLBI, i.e. from any intialization, the SplitLBI sequence converges to a critical point of ¯ L .
Our treatment extends the block coordinate descent (BCD) studied in Zeng et al.
(2019), with a crucial difference being the mirror descent involved in SplitLBI.
Instead of the splitting loss in BCD (Zeng et al., 2019), a new Lyapunov function is developed here to meet the Kurdyka-Łojasiewicz property Łojasiewicz (1963).
Xue & Xin (2018) studied convergence of variable splitting method for single hidden layer networks with Gaussian inputs.
Let P :=
( W, Γ) .
Following Huang & Yao (2018), the SplitLBI algorithm in Eq.
(4a-4c) can be rewritten as the following standard Linearized Bregman Iteration,
[Equation]
where
[Equation]
p k ∈ ∂ Ψ( P k ) , and B q Ψ is the Bregman divergence associated with convex function Ψ , deﬁned by
[Equation]
Without loss of generality, consider λ = 1 in the sequel.
One can establish the global convergence of
SplitLBI under the following assumptions.
Assumption 1.
Suppose that: (a) (cid:98) L n ( W ) = 1 n (cid:80) ni =1 (cid:96) ( y i , Φ W ( x i )) is continuous differentiable and ∇ (cid:98) L n is Lipschitz continuous with a positive constant Lip ; (b) (cid:98) L n ( W ) has bounded level sets; (c) (cid:98) L n ( W ) is lower bounded (without loss of generality, we assume that the lower bound is 0 ); (d) Ω is a proper lower semi-continuous convex function and has locally bounded subgradients, that is, for every compact set S ⊂ R n , there exists a constant C > 0 such that for all Γ ∈ S and all g ∈ ∂ Ω(Γ) , there holds (cid:107) g (cid:107) ≤ C ; and (e) the Lyapunov function
[Equation]
is a Kurdyka-Łojasiewicz function on any bounded set, where B ˜ g Ω (Γ , ˜Γ) :
= Ω(Γ) − Ω(˜Γ) −(cid:104)
˜ g, Γ − ˜Γ (cid:105) , ˜Γ ∈ ∂ Ω ∗ (˜ g ) , and Ω ∗ is the conjugate of Ω deﬁned as
[Equation]
Remark 1.
Assumption 1 (a)-(c) are regular in the analysis of nonconvex algorithm (see, Attouch et al.
(2013) for instance), while Assumption 1 (d) is also mild including all Lipschitz continuous convex function over a compact set.
Some typical examples satisfying Assumption 1(d) are the (cid:96)norm, group (cid:96) 1 norm, and every continuously differentiable penalties.
By Eq.
(9) and the deﬁnition of conjugate, the Lyapunov function F can be rewritten as follows,
[Equation]
Now we are ready to present the main theorem.
Theorem 1.
[Global Convergence of SplitLBI] Suppose that Assumption 1 holds.
Let ( W k , Γ k ) be the sequence generated by SplitLBI (Eq. (4a-4c)) with a ﬁnite initialization.
If
[Equation]
then ( W k , Γ k ) converges to a critical point of ¯ L deﬁned in Eq.
(2), and { W k } converges to a critical point of (cid:98) L n ( W ) .
Applying to the neural networks, typical examples are summarized in the following corollary.
Corollary 1.
Let { W k , Γ k , g k } be a sequence generated by SLBI (16a-16c) for neural network training where (a) (cid:96) is any smooth deﬁnable loss function, such as the square loss ( t 2 ) , exponential loss ( e t ) , logistic loss log(1+ e − t ) , and cross-entropy loss; (b) σ i is any smooth deﬁnable activation, such as linear activation ( t ) , sigmoid ( 1 1+ e − t ) , hyperbolic tangent ( e t − e − t e t + e − t ) , and softplus ( 1 c log(1 + e ct ) for some c > 0 ) as a smooth approximation of ReLU; (c) Ω is the group Lasso.
Then the sequence { W k } converges to a stationary point of (cid:98) L n ( W ) under the conditions of Theorem 1.
Proofs of Theorem 1 and Corollary 1 are given in Appendix A.
E XPERIMENTS WITH S TOCHASTIC S PLIT LBI
We begin with some stochastic variants of SplitLBI and implementations, followed by four groups of experiments demonstrating the utilities of weight parameter W t and structural sparsity parameter Γ t in prediction, interpretability, and capturing effective sparse subnetworks.
Batch Split LBI .
For neural network training with large datasets, stochastic approximation of the gradients in Split LBI over the mini-batch ( X , Y ) batch t is adopted to update the parameter W , [Equation]
[Equation]
SplitLBI with momentum (Mom) .
Inspired by the variants of SGD, the momentum term can be also incorporated to the standard Split LBI that leads to the following updates of W by replacing Eq [Equation] 4a) with,
[Equation]
where τ is the momentum factor, empirically setting as 0.9 in default.
One immediate application of such stochastic algorithms of SplitLBI is to “boost networks", i.e. growing a network from the null to a complex one by sequentially applying our algorithm on subnets with increasing complexities.
SplitLBI with momentum and weight decay (Mom-Wd).
The update formulation is,
[Equation]
) where β is set as 1 e − 4 .
Implementation.
Various algorithms are evaluated over the various backbones – LeNet (LeCun et al., 2015), AlexNet (Krizhevsky et al., 2012), VGG (Simonyan & Zisserman, 2014), and ResNet (He et al., 2016) etc., respectively.
For MNIST and Cifar-10, the default hyper-parameters of Split
LBI are κ = 1 , ν = 10 and α k is set as 0 . 1 , decreased by 1/10 every 30 epochs.
In ImageNet-2012, the Split LBI utilizes κ = 1 , ν = 1000 , and α k is initially set as 0.1, decays 1/10 every 30 epochs.
We set λ = 1 in Eq.
(5) by default, unless otherwise speciﬁed.
On MNIST and Cifar-10, the batch size is set as 128; and for all methods, the batch size of ImageNet 2012 is 256.
The standard data augmentation implemented in pytorch is applied to Cifar-10 and ImageNet2012 datasets, as He et al.
The weights of all models are initialized as He et al.
In the following experiments, we deﬁne sparsity as percentage of non-zero parameters, i.e. the number of non-zero weights dividing the total number of weights in consideration, that equals to one minus the pruning rate of the network.
We also have the reproducible source codes 1 .
4. I MAGE C LASSIFICATION
In SplitLBI, the weight parameter W t explores over-parameterized models that can achieve the state-of-the-art performance in large scale training such as ImageNet-2012 classiﬁcation.
Experimental Design .
We compare different variants of SGD and Adam in the experiments.
By default, the learning rate of competitors is set as 0 .
1 for SGD and its variant and 0 .
001 for Adam and its variants, and gradually decreased by 1/10 every 30 epochs.
In particular, we have,
[Table]
[Table] with l 1 penalty (Lasso).
The l 1 norm is applied to penalize the weights of SGD by encouraging the sparsity of learned model, with the regularization parameter of the l 1 penalty term being set as 1 e − 3 (3) SGD with momentum (Mom): we utilize momentum 0.9 in SGD.
(4) SGD with momentum and weight decay (Mom-Wd): we set the momentum 0.9 and the standard l 2 weight decay with the coefﬁcient weight 1 e − 4 .
SGD with Nesterov (Nesterov): the SGD uses nesterov momentum 0.9.
Adam: (1) Naive Adam: it refers to the standard version of Adam.
We report the results of several recent variants of Adam, including (2) Adabound, (3) Adagrad, (4) Amsgrad, and (5) Radam.
SplitLBI achieves the state-of-the-art performance on ImageNet-2012, etc.
Tab. 1 shows the experimental results on ImageNet-2012, Cifar-10, and MNIST of some classical networks -- LeNet,
AlexNet and ResNet.
Our SplitLBI variants may achieve comparable or even better performance than
SGD variants in 100 epochs, indicating the efﬁcacy in learning dense, over-parameterized models.
4. S PLIT LBI L EARNS S PARSE F ILTERS FOR I MPROVED I NTERPRETATION
[Figure]
In SplitLBI, the structural sparsity parameter Γ t explores important sub-network architectures that contributes signiﬁcantly to the loss or error reduction in early training stages.
Through the (cid:96) 2 -coupling, structural sparsity parameter Γ t may guide the weight parameter to explore those sparse models in favour of improved interpretabiity.
For example, Fig. 1 visualizes some sparse ﬁlters learned by SplitLBI of LeNet-5 trained on MNIST (with κ = 10 and weight decay every 40 epochs), in comparison with dense ﬁlters learned by SGD.
The activation pattern of such sparse ﬁlters favours high order global correlations between pixels of input images.
To further reveal the insights of learned patterns of SplitLBI, we visualize the ﬁrst convolutional layer of ResNet-18 on ImageNet-2012 along the training path of our SplitLBI as in Fig.
The left ﬁgure compares the training and validation accuracy of SplitLBI and SGD.
The right ﬁgure compares visualizations of the ﬁlters learned by
SplitLBI and SGD using Springenberg et al.
Implementation.
To be speciﬁc, denote the weights of an l -layer network as { W 1 , W 2 , · · · , W l } .
For the i − th layer weights W i , denote the j − th channel W ij .
Then we compute the gradient of the sum of the feature map computed from each ﬁlter W ij with respect to the input image (here a snake image).
We further conduct the min-max normalization to the gradient image, and generate the ﬁnal visualization map.
The right ﬁgure compares the visualized gradient images of ﬁrst convolutional layer of 64 ﬁlters with 7 × 7 receptive ﬁelds.
We visualize the models parameters at 20 (purple),(green), and 60 (black) epochs, respectively, which corresponds to the bounding boxes in the right ﬁgure annotated by the corresponding colors, i.e. , purple, green, and black.
We order the gradient images produced from 64 ﬁlters by the descending order of the magnitude ( (cid:96) 2 - images are ordered visualized
Filters learned by ImageNet prefer to non-semantic texture rather than shape and color.
The ﬁlters of high norms mostly focus on the texture and shape information, while color information is with the ﬁlters of small magnitudes.
This phenomenon is in accordance with observation of AbbasiAsl & Yu (2017) that ﬁlters mainly of color information can be pruned for saving computational cost.
Moreover, among the ﬁlters of high magnitudes, most of them capture non-semantic textures while few pursue shapes.
This shows that the ﬁrst convolutional layer of ResNet-18 trained on ImageNet learned non-semantic textures rather than shape to do image classiﬁcation tasks, in accordance with recent studies (Geirhos et al., 2019).
How to enhance the semantic shape invariance learning, is arguably a key to improve the robustness of convolutional neural networks.
4. G LOBAL C ONVERGENCE AND S TRUCTURAL S PARSITY OF S PLIT LBI
We conduct ablation studies based on Cifar-10 dataset with VGG-16 and ResNet-56 to evaluate (i) global convergence of ¯ L ; and (ii) the structural sparsity learned by Γ t via exploring test accuracies of sparse models obtained by projecting W t onto the support set of Γ t (mask), by varying two key hyper-parameters κ and ν .
Implementation.
We choose SplitLBI with momentum and weight decay, since it achieves very good performance on large-scale experiments.
Speciﬁcally, we have two set of experiments, where each experiment is repeated for 5 times: (1) we ﬁx ν = 100 and vary κ = 1 , 2 , 5 , 10 , where sparsity of Γ t and validation accuracies of sparse models are shown in top row of Fig.
Note that we keep κ · α k = 0 .
1 in Eq (1a), to make comparable learning rate of each variant, and also consistent with
SGD.
Thus the learning rate α k will be adjusted by different κ values.
(2) we ﬁx κ = 1 , and validate the results of SplitLBI with ν = 10 , 20 , 50 , 100 , 200 , 500 , 1000 , 2000 in the second row of Fig.with the learning rate
α k = 0 .
Moreover, rather than using sparse models associated with Γ t , Fig.
6 in Appendix shows the validation accuracies of full models learned by W t .
SplitLBI converges to Critical Point . Figure 3 shows the curves of training loss ( (cid:98) L n ) and accuracies, with each point representing the average and variance bar over 5 times. As shown, both training loss and training accuracy will converge, which validates our theoretical result in Theorem 1. Besides, larger κ brings in slower convergence, which agrees with the analysis the convergence rate is inversely scale to κ in Lemma A.5.
Sparse subnetworks achieve comparable performance to dense models without ﬁne-tuning or retraining .
From the experiments above, the sparsity of Γ grows as κ and ν increase.
While large κ may cause a small number of important parameters growing rapidly, large ν will decouple W t and Γ t such that the growth of W t does not affect Γ t that may over-sparsify and deteriorate model accuracies.
Thus a moderate choice of κ and ν is preferred in practice.
In all cases, one can see that moderate sparse models can achieve comparable predictive power to dense models, even without ﬁne-tuning or retraining.
This shows that the structural sparsity parameter Γ t can indeed capture important weight parameter W t through their coupling.
4. S PLIT LBI WITH E ARLY S TOPPING AND R ETRAIN F INDS E FFECTIVE S UB -N ETWORKS
Equipped with early stopping, Γ t in early epochs may learn effective subnetworks ( i e. “winning tickets” (Frankle & Carbin, 2019; Liu et al., 2019)) that achieve comparable or even better performance after retraining than existing pruning strategies by SGD. [Figure]
Experimental Design.
We adopt a comparison baseline as the one-shot pruning strategy in Frankle & Carbin (2019), which ﬁrstly trains a dense over-parameterized model by SGD for T = 160 epochs and ﬁnd the sparse structure by pruning weights or ﬁlters (Liu et al., 2019), then secondly retrains the structure from the scratch with T epochs from the same initialization as the ﬁrst step.
For SplitLBI, instead of pruning weights/ﬁlters from dense models, we directly utilize the structural sparsity Γ t at different training epochs to deﬁne the subnet architecture, followed by retrain-from-scratch (Fine-tune is shown in Appendix Sec.
D with preliminary results).
Experiments are conducted on Cifardataset where we still use VGG–16, ResNet-50, and ResNet-56 as the networks to make direct comparisons to previous works.
SplitLBI uses momentum and weight decay with hyperparameters shown in Tab. 10 in Appendix.
In particular, we set λ = 0 .
1 , and 0 .
for VGG-16, and ResNetrespectively, since ResNet-56 has less parameters than VGG-16.
Furthermore, we introduce another variant of our SplitLBI by using Lasso rather than group lasso penalty for Γ t to sparsify the weights of convolutional ﬁlters; and the corresponding models are denoted as VGG-16 (Lasso) and ResNet- (Lasso).
Every experiment is repeated for ﬁve times and the results are shown in Fig.
Note that in different runs of SplitLBI, the sparsity of Γ t slightly varies.
Sparse subnets found by early stopping of SplitLBI achieve remarkably good accuracy after retrain from scratch.
In Fig.5 (a-b), sparse ﬁlters discovered by Γ t at different epochs are compared against the methods of Network Slimming (Liu et al., 2017),
Soft Filter Pruning (Yang et al., 2018),
Scratch-B, and Scratch-E, whose results are reported from Liu et al.
At similar sparsity levels, SplitLBI can achieve comparable or even better accuracy than competitors, even with sparse architecture learned from very early epochs (e.g. t = 20 or 10 ).
Moreover in Fig.5 (c-d), we can draw the same conclusion for the sparse weights of VGG-16 (Lasso) and ResNet-50 (Lasso), against the results reported in Liu et al.
These results shows that the structural sparsity parameter Γ t found by early stopping of SplitLBI already discloses important subnetwork architecture that may achieve remarkably good accuracy after retrain from scratch.
Therefore, it is not necessary to fully train a dense model to ﬁnd a successful sparse subnet architecture with comparable performance to the dense ones -- one can early stop SplitLBI properly where the structural parameter Γ t unveils “ winning tickets ” (Frankle & Carbin, 2019).  C ONCLUSION
In this paper, a parsimonious deep learning method is proposed based on differential inclusions of inverse scale spaces.
Implemented by a variable splitting scheme, such a dynamics system can exploit over-parameterized models and structural sparsity simultaneously.
Besides, its simple discretization, i.e., the SplitLBI, has a proven global convergence and hence can be employed to train deep networks. We have experimentally shown that it can achieve the state-of-the-art performance on many datasets including ImageNet-2012, with better interpretability than SGD. What’s more, equipped with early stopping, such a structural sparsity can unveil the ”winning tickets” – the architecture of sub-networks which after re-training can achieve comparable and even better accuracy than original dense networks. A PPENDIX TO SplitLBI for
deep
learning:
structural
sparsity
via
differential inclusion paths A P ROOF OF T HEOREM
First of all, we reformulate Eq.
(6) into an equivalent form.
Without loss of generality, consider Ω = Ω 1 in the sequel.
Denote R ( P ) :
= Ω(Γ) , then Eq.
(6) can be rewritten as,
[Equation]
where p k =
[0 , g k ] T ∈ ∂R ( P k ) and g k ∈ ∂ Ω(Γ k ) .
Thus SplitLBI is equivalent to the following iterations,
[Equation]
Exploiting the equivalent reformulation (16a-16c), one can establish the global convergence of ( W k , Γ k , g k ) based on the Kurdyka-Łojasiewicz framework.
In this section, the following extended version of Theorem 1 is actually proved.
Theorem 2.
[Global Convergence of SplitLBI] Suppose that Assumption 1 holds.
Let ( W k , Γ k , g k ) be the sequence generated by SplitLBI (Eq. (16a-16c)) with a ﬁnite initialization.
If
[Equation]
then ( W k , Γ k , g k ) converges to a critical point of F .
Moreover, { ( W k , Γ k ) } converges to a stationary point of ¯ L deﬁned in Eq. 2, and { W k } converges to a stationary point of (cid:98) L n ( W ) .
A. K URDYKA -Ł OJASIEWICZ P ROPERTY
To introduce the deﬁnition of the Kurdyka-Łojasiewicz (KL) property, we need some notions and notations from variational analysis, which can be found in Rockafellar & Wets (1998).
The notion of subdifferential plays a central role in the following deﬁnitions.
For each x ∈ dom( h ) := { x ∈ R p : h ( x ) < + ∞} , the Fr´echet subdifferential of h at x , written (cid:98)
∂h
( x ) , is the set of vectors v ∈ R p which satisfy
[Equation]
When x / ∈ dom( h ) , we set (cid:98) ∂h ( x ) = ∅ .
The limiting-subdifferential (or simply subdifferential ) of h introduced in Mordukhovich (2006), written ∂h ( x ) at x ∈ dom( h ) , is deﬁned by
[Equation]
A necessary (but not sufﬁcient) condition for x ∈ R p to be a minimizer of h is 0 ∈ ∂h
( x ) .
A point that satisﬁes this inclusion is called limiting-critical or simply critical .
The distance between a point x to a subset S of R p , written dist( x , S ) , is deﬁned by dist( x , S )
=
inf {(cid:107) x
− s (cid:107) : s ∈ S} , where (cid:107) · (cid:107) represents the Euclidean norm.
Let h : R p
→
R ∪ { + ∞} be an extended-real-valued function (respectively, h : R p ⇒ R q be a point-to-set mapping), its graph is deﬁned by
[Equation]
resp.
Graph
[Equation]
and its domain by dom( h ) :
= { x ∈ R p : h ( x ) <
+ ∞} (resp.
dom( h ) :
= { x ∈ R p : h ( x ) (cid:54) = ∅ } ).
When h is a proper function, i.e., when dom( h ) (cid:54) = ∅ , the set of its global minimizers (possibly empty) is denoted by arg min h := { x ∈ R p : h ( x ) =
inf h } .
The KL property (Łojasiewicz, 1963; 1993; Kurdyka, 1998; Bolte et al., 2007a;b) plays a central role in the convergence analysis of nonconvex algorithms (Attouch et al., 2013; Wang et al., 2019).
The following deﬁnition is adopted from Bolte et al.
(2007b).
Deﬁnition 1.
[Kurdyka-Łojasiewicz property] A function h is said to have the Kurdyka-Łojasiewicz (KL) property at ¯ u ∈ dom( ∂h ) :
= { v ∈ R n | ∂h ( v ) (cid:54) = ∅} , if there exists a constant η ∈ (0 , ∞ ) , a neighborhood N of ¯ u and a function φ : [0 , η ) → R + , which is a concave function that is continuous at 0 and satisﬁes φ (0) = 0 , φ ∈ C 1 ((0 , η )) , i.e., φ is continuous differentiable on (0 , η ) , and φ (cid:48) ( s ) > 0 for all s ∈ (0 , η ) , such that for all u ∈ N ∩ { u ∈ R n | h (¯ u ) <
h ( u ) <
h (¯ u ) + η } , the following inequality holds
[Equation]
If h satisﬁes the KL property at each point of dom( ∂h ) , h is called a KL function.
KL functions include real analytic functions, semialgebraic functions, tame functions deﬁned in some o-minimal structures (Kurdyka, 1998; Bolte et al., 2007b), continuous subanalytic functions (Bolte et al., 2007a) and locally strongly convex functions.
In the following, we provide some important examples that satisfy the Kurdyka-Łojasiewicz property.
Deﬁnition 2.
[Real analytic]
A function h with domain an open set U ⊂ R and range the set of either all real or complex numbers, is said to be real analytic at u if the function h may be represented by a convergent power series on some interval of positive radius centered at u : h ( x ) = (cid:80) ∞
j =
α j ( x − u ) j , for some { α j } ⊂ R .
The function is said to be real analytic on V ⊂
U
if it is real analytic at each u ∈ V (Krantz & Parks, 2002, Deﬁnition 1.1.5).
The real analytic function f over R p for some positive integer p > 1 can be deﬁned similarly.
According to Krantz & Parks (2002), typical real analytic functions include polynomials, exponential functions, and the logarithm, trigonometric and power functions on any open set of their domains.
One can verify whether a multivariable real function h ( x ) on R p is analytic by checking the analyticity of g ( t ) :
=
h ( x + t y ) for any x , y ∈ R p .
Deﬁnition 3.
[Semialgebraic]
(a) A set D ⊂ R p is called semialgebraic (Bochnak et al., 1998) if it can be represented as
[Equation]
where P ij , Q ij are real polynomial functions for 1 ≤ i ≤ s, 1 ≤ j ≤ t. (b) A function h :
R p
→ R ∪ { + ∞} (resp.
a point-to-set mapping h : R p ⇒ R q ) is called semialgebraic if
its graph Graph( h ) is semialgebraic.
According to (Łojasiewicz, 1965; Bochnak et al., 1998) and (Shiota, 1997, I.2.9, page 52), the class of semialgebraic sets are stable under the operation of ﬁnite union, ﬁnite intersection, Cartesian product or complementation.
Some typical examples include polynomial functions, the indicator function of a semialgebraic set, and the Euclidean norm (Bochnak et al., 1998, page 26).
A. KL P ROPERTY IN D EEP L EARNING AND P ROOF OF C OROLLARY
In the following, we consider the deep neural network training problem.
Consider a l -layer feedforward neural network including l − 1 hidden layers of the neural network.
Particularly, let d i be the number of hidden units in the i -th hidden layer for i = 1 , . . .
, l − 1 .
Let d 0 and d l be the number of units of input and output layers, respectively.
Let W i ∈
R
d i × d i − 1 be the weight matrix between the ( i − 1) -th layer and the i -th layer for any i = 1 , . . .
l 2 .
According to Theorem 2, one major condition is to verify the introduced Lyapunov function F deﬁned in (9) satisﬁes the Kurdyka-Łojasiewicz property.
For this purpose, we need an extension of semialgebraic set, called the o-minimal structure (see, for instance Coste (1999), van den Dries (1986), Kurdyka (1998), Bolte et al.
(2007b)).
The following deﬁnition is from Bolte et al.
(2007b).
Deﬁnition 4.
[o-minimal structure]
An o-minimal structure on ( R , + , · ) is a sequence of boolean algebras O n of “deﬁnable” subsets of R n , such that for each n ∈ N
(i) if A belongs to O n , then A × R and R ×
A belong to O n +1 ; (ii) if Π : R n +1 → R n is the canonical projection onto R n , then for any A in O n +1 , the set Π( A ) belongs to O n ; (iii) O n contains the family of algebraic subsets of R n , that is, every set of the form { x ∈ R n : p ( x ) = 0 } , where
p : R n → R is a polynomial function
(iv) the elements of O 1 are exactly ﬁnite unions of intervals and points.
Based on the deﬁnition of o-minimal structure, we can show the deﬁnition of the deﬁnable function .
Deﬁnition 5.
[Deﬁnable function] Given an o-minimal structure O (over ( R , + , · ) ), a function f : R n → R is said to be deﬁnable in O if its graph belongs to O n +1 .
According to van den Dries & Miller (1996)
; Bolte et al. (2007b), there are some important facts of the o-minimal structure, shown as follows.
(i)
The collection of semialgebraic sets is an o-minimal structure.
Recall the semialgebraic sets are Bollean combinations of sets of the form
[Equation]
[Equation] [Equation]
[Equation] [Equation]
m
[Equation]
x ) < 0 } , where p and q i ’s are polynomial functions in R n .
(ii) There exists an o-minimal structure that contains the sets of the form
[Equation]
where f is real-analytic around [ − 1 , 1] n .
(iii)
There exists an o-minimal structure that contains simultaneously the graph of the exponential function R (cid:51) x
(cid:55)→ exp( x ) and all semialgebraic sets.
(iv) The o-minimal structure is stable under the sum, composition, the inf-convolution and several other classical operations of analysis.
The Kurdyka-Łojasiewicz property for the smooth deﬁnable function and non-smooth deﬁnable function were established in (Kurdyka, 1998, Theorem 1) and (Bolte et al., 2007b, Theorem 11), respectively.
Now we are ready to present the proof of Corollary 1.
Proof.
[Proof of Corollary 1] To justify this corollary, we only need to verify the associated Lyapunov function F satisﬁes Kurdyka-Łojasiewicz inequality.
In this case and by (10), F can be rewritten as follows
[Equation]
Because (cid:96) and σ i ’s are deﬁnable by assumptions, then (cid:98) L n ( W, Γ) are deﬁnable as compositions of deﬁnable functions.
Moreover, according to Krantz & Parks (2002), (cid:107) W − Γ (cid:107) 2 and (cid:104) W, g (cid:105) are semi-algebraic and thus deﬁnable.
Since the group Lasso Ω(Γ) = (cid:80) g (cid:107) Γ (cid:107) 2 is the composition of (cid:96) 2 and (cid:96) 1 norms, and the conjugate of group Lasso penalty is the maximum of group (cid:96) 2 -norm, i.e.
Ω ∗ (Γ) = max g (cid:107) Γ g (cid:107) 2 , where the (cid:96) 2 , (cid:96) 1 , and (cid:96) ∞
norms are deﬁnable, hence the group Lasso and its conjugate are deﬁnable as compositions of deﬁnable functions.
Therefore, F is deﬁnable and hence satisﬁes Kurdyka-Łojasiewicz inequality by (Kurdyka, 1998, Theorem 1).
The veriﬁcations of other cases listed in assumptions can be found in the proof of (Zeng et al., 2019,
Proposition 1).
This ﬁnishes the proof of this corollary.
A. P ROOF OF T HEOREM
Our analysis is mainly motivated by a recent paper (Benning et al., 2017), as well as the inﬂuential work (Attouch et al., 2013).
According to (Attouch et al., 2013, Lemma 2.6), there are mainly four ingredients in the analysis, that is, the sufﬁcient descent property , relative error property , continuity property of the generated sequence and the Kurdyka-Łojasiewicz property of the function.
More speciﬁcally, we ﬁrst establish the sufﬁcient descent property of the generated sequence via exploiting the Lyapunov function F (see, (9)) in Lemma A.4 in Section A.4, and then show the relative error property of the sequence in Lemma A.5 in Section A.5.
The continuity property is guaranteed by the continuity of ¯ L ( W, Γ) and the relation lim k →∞ B g k Ω
(Γ k +1 , Γ k )
= 0 established in Lemma 1(i) in
Section A.4.
Thus, together with the Kurdyka-Łojasiewicz assumption of F , we establish the global convergence of SLBI following by (Attouch et al., 2013, Lemma 2.6).
Let ( ¯ W, ¯Γ , ¯ g ) be a critical point of F , then the following holds
[Equation]
By the ﬁnal inclusion and the convexity of Ω , it implies ¯ g ∈ ∂ Ω(¯Γ) .
Plugging this inclusion into the second inclusion yields αν − 1 (¯Γ − ¯ W ) = 0 .
Together with the ﬁrst equality imples ∇
[Equation] [Equation]
[Equation]
This ﬁnishes the proof of this theorem.
A. S UFFICIENT D ESCENT P ROPERTY ALONG L YAPUNOV F UNCTION
Let P k :=
( W k , Γ k ) , and Q k :=
( P k , g k − 1 ) , k ∈ N .
In the following, we present the sufﬁcient descent property of Q k along the Lyapunov function F .
Lemma.
Suppose that (cid:98) L n is continuously differentiable and ∇ (cid:98) L n is Lipschitz continuous with a constant Lip > 0 .
Let { Q k } be a sequence generated by SLBI with a ﬁnite initialization.
If 0 < α < 2 κ ( Lip + ν − 1 ) , then
[Equation]
where ρ := 1 κ − α
( Lip + ν − 1 ).
Proof.
By the optimality condition of (15a) and also the inclusion p k =
[0 , g k ] T ∈ ∂R ( P k ) , there holds
[Equation]
which implies
[Equation]
where
[Equation]
Noting that ¯ L ( P )
= (cid:98) L n ( W ) + 12 ν (cid:107) W − Γ (cid:107) 22 and by the Lipschitz continuity of ∇ (cid:98) L n ( W ) with a constant Lip > 0 implies ∇ ¯ L is Lipschitz continuous with a constant Lip + ν − 1 .
This implies ¯ L (
[Equation]
Substituting the above inequality into (20) yields
[Equation]
Adding some terms in both sides of the above inequality and after some reformulations implies
[Equation]
where the ﬁnal equality holds for D (Γ k +1 , Γ k ) − B g k Ω (Γ k +1 , Γ k )
=
B g k +1 Ω (Γ k , Γ k
[Equation]
That is,
[Equation]
where the ﬁnal inequality holds for B g k +1 Ω
[Equation]
B
[Equation]
Thus, we ﬁnish the proof of this lemma.
Based on Lemma A.4, we directly obtain the following lemma.
Lemma 1.
Suppose that assumptions of Lemma A.4 hold.
Suppose further that Assumption 1 (b)-(d) hold.
Then
(i) both α { ¯ L ( P k ) } and { F ( Q k ) } converge to the same ﬁnite value, and lim k →∞ B g k Ω (Γ k +1 , Γ k )
= 0 .
(ii) the sequence { ( W k , Γ k , g k ) } is bounded, (iii) lim k →∞ (cid:107) P k
+1 − P k (cid:107) 22 =
and lim k →∞ D (Γ k +1 , Γ k ) = 0 , (iv) 1 K (cid:80)
Kk =0 (cid:107)
P k
+1 − P k (cid:107)
→ 0 at a rate of O (1 /K ) .
Proof.
By (21), ¯ L ( P k ) is monotonically decreasing due to D (Γ k +1 , Γ k ) ≥ 0 .
Similarly, by (24),
F ( Q k ) is also monotonically decreasing.
By the lower boundedness assumption of (cid:98) L n ( W ) , both ¯
L ( P ) and F ( Q ) are lower bounded by their deﬁnitions, i.e., (2) and (9), respectively.
Therefore, both { ¯ L ( P k ) } and { F ( Q k ) } converge, and it is obvious that lim k →∞ F ( Q k ) ≥ lim k →∞ α ¯ L
( P k ) .
By (23),
[Equation]
By the convergence of F ( Q k ) and the nonegativeness of B g k − 1 Ω (Γ k , Γ k − 1 ) , there holds
[Equation]
By the deﬁnition of F ( Q k )
=
α ¯ L ( P k ) + B g k − 1 Ω (Γ k , Γ k − 1 ) and the above equality, it yields
[Equation]
Since (cid:98) L n ( W ) has bounded level sets, then W k is bounded.
By the deﬁnition of ¯ L ( W, Γ) and the ﬁniteness of ¯ L ( W k , Γ k ) , Γ k is also bounded due to W k is bounded.
The boundedness of g k is due to g k ∈ ∂ Ω(Γ k ) , condition (d), and the boundedness of Γ k .
By (24), summing up (24) over k = 0 , 1 , . . .
, K yields
[Equation]
Letting K → ∞ and noting that both (cid:107) P k +1 − P k (cid:107)
and D (Γ k +1 , Γ k ) are nonnegative, thus lim
[Equation]
lim
[Equation]
Again by (25),
[Equation]
which implies 1 K (cid:80)
Kk =0 (cid:107)
P k
+1 − P k (cid:107) 2 → 0 at a rate of O (1 /K ) .
A. R ELATIVE E RROR P ROPERTY
In this subsection, we provide the bound of subgradient by the discrepancy of two successive iterates.
By the deﬁnition of F (9),
[Equation]
Lemma.
Under assumptions of Lemma 1, then
[Equation]
where
ρ 1 := 2 κ − 1 + 1 + α ( Lip + 2 ν − 1 ) .
Moreover, 1 K (cid:80) Kk
=1 (cid:107) H k
(cid:107) 2 → 0 at a rate of O (1 /K ) .
Proof.
Note that
[Equation]
By the deﬁnition of ¯ L (see (2)),
[Equation]
where the last inequality holds for the Lipschitz continuity of ∇ (cid:98) L n with a constant Lip > 0 , and by (16a),
[Equation]
Substituting the above (in)equalities into (27) yields
[Equation]
Thus,
[Equation] [Equation]
By (16c), it yields
[Equation]
Noting that ∇ Γ ¯ L ( W k , Γ k )
=
ν − 1 (Γ k − W k ) , and after some simpliﬁcations yields
[Equation]
where the last inequality holds for the triangle inequality and κ − 1 > αν − 1 by the assumption.
By (28), (29), and the deﬁnition of H k +1 (26), there holds
[Equation]
By (30) and Lemma 1(iv), 1 K (cid:80) Kk
=1 (cid:107) H k
(cid:107) 2 → 0 at a rate of O (1 /K ) .
This ﬁnishes the proof of this lemma.
[Table]
To further study the inﬂuence of hyperparameters
[Table]
we record performance of W t for each epoch t with different combinations of hyperparameters.
The experiments is conducted 5 times each, we show the mean in the table, the standard error can be found in the corresponding ﬁgure.
We perform experiments on Cifar10 and two commonly used network VGG16 and ResNet56.
On κ , we keep ν = 100 and try κ = 1 , 2 , 5 , 10 , the validation curves of models W t are shown in
Fig. 6 and Table 2 summarizes the mean accuracies. Table 3 summarizes best validation accuracies achieved at some epochs, together with their sparsity rates.
These results show that larger kappa leads to slightly lower validation accuracies, where the numerical results are shown in Table 2 .
We can ﬁnd that κ = 1 achieves the best test accuracy.
On ν , we keep κ = 1 and try ν = 10 , 20 , 50 , 100 , 200 , 500 , 1000 , 2000 the validation curve and mean accuracies are show in Fig.
6 and Table 4.
Table 5 summarizes best validation accuracies achieved at some epochs, together with their sparsity rates.
By carefully tuning ν we can achieve similar or even better results compared to SGD.
Different from κ , ν has less effect on the generalization performance.
By tuning it carefully, we can even get a sparse model with slightly better performance than SGD trained model.
C C OMPUTATIONAL C OST OF S PLIT LBI
We further compare the computational cost of different optimizers: SGD (Mom), SplitLBI (Mom) and Adam (Naive). We test each optimizer on one GPU, and all the experiments are done on one GTX2080. For computational cost, we judge them from two aspects : GPU memory usage and time needed for one batch.
The batch size here is 64, experiment is performed on VGG-16 as shown in
Table 6. [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
ν =0.
[Table]
Term
[Table]
Acc
[Table]
Acc
[Table]
Acc ν =99.
[Table]
=99.
[Table]
[Table]
[Table]
ν =74.
[Table] [Table]
=0.
[Table] [Table]
D F INE - TUNING OF SPARSE SUBNETWORKS
We design the experiment on MNIST, inspired by Frankle & Carbin (2019).
Here, we explore the subnet obtained by Γ T after T = 100 epochs of training.
As in Frankle et al.
(2019), we adopt the “rewind” trick: re-loading the subnet mask of Γ 100 at different epochs, followed by ﬁne-tuning.
In particular, along the training paths, we reload the subnet models at Epoch 0, Epoch 30, 60, 90, and 100, and further ﬁne-tune these models by SplitLBI (Mom-Wd).
All the models use the same initialization and hence the subnet model at Epoch 0 gives the retraining with the same random initialization as proposed to ﬁnd winning tickets of lottery in Frankle & Carbin (2019).
We will denote the rewinded ﬁne-tuned model at epoch 0 as (Lottery), and those at epoch 30, 60, 90, and 100, as F-epoch30, F-epoch60, F-epoch90, and F-epoch100, respectively.
Three networks are studied here – LeNet-3, Conv-2, and Conv-4.
LeNet-3 removes one convolutional layer of LeNet-5; and it is thus less over-parameterized than the other two networks.
Conv-2 and Conv-4, as the scaled-down variants of VGG family as done in Frankle & Carbin (2019), have two and four fully-connected layers, respectively, followed by max-pooling after every two convolutional layer.
The whole sparsity for Lenet-3 is 0.055, Conv-2 is 0.0185, and Conv-4 is 0.1378.
Detailed sparsity for every layer of the model is shown in Table 7, 8, 9.
We ﬁnd that fc-layers are sparser than conv-layers.
We compare SplitLBI variants to the SGD (Mom-Wd) and SGD (Lottery) (Frankle & Carbin, 2019) in the same structural sparsity and the results are shown in Fig.
In this exploratory experiment, one can see that for overparameterized networks – Conv-2 and Conv-4, ﬁne-tuned rewinding subnets –
F-epoch30, F-epoch60, F-epoch90, and F-epoch100, can produce better results than the full models; while for the less over-parameterized model LeNet-3, ﬁne-tuned subnets may achieve less yet still comparable performance to the dense models and remarkably better than the retrained sparse subnets from beginning (i.e. SplitLBI/SGD (Lottery)).
These phenomena suggest that the subnet architecture disclosed by structural sparsity parameter Γ T is valuable, for ﬁne-tuning sparse models with comparable or even better performance than the dense models of W T .
E R ETRAINING OF SPARSE SUBNETS FOUND BY S PLIT LBI (L OTTERY )
Here we provide more details on the experiments in Fig.
Table 10 gives the details on hyperparameter setting.
Moreover, Figure 8 provides the sparsity variations during SplitLBI training in Fig.
[Table]
for every layer of Conv-2.
Here sparsity is deﬁned in Sec. 4, number of weights denotes the total number of parameters in the designated layer. The sparsity is more signiﬁcant in fully connected (FC) layers than convolutional layers.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]