L EARNING S PARSE S TRUCTURED E NSEMBLES WITH SG-MCMC AND N ETWORK P RUNING
Anonymous authors
Paper under double-blind review
A BSTRACT
An ensemble of neural networks is known to be more robust and accurate than an individual network, however usually with linearly-increased cost in both training and testing.
In this work, we propose a two-stage method to learn Sparse Structured Ensembles (SSEs) for neural networks.
In the ﬁrst stage, we run SG-MCMC with group sparse priors to draw an ensemble of samples from the posterior distribution of network parameters.
In the second stage, we apply weight-pruning to each sampled network and then perform retraining over the remained connections.
In this way of learning SSEs with SG-MCMC and pruning, we not only achieve high prediction accuracy since SG-MCMC enhances exploration of the modelparameter space, but also reduce memory and computation cost signiﬁcantly in both training and testing of NN ensembles.
This is thoroughly evaluated in the experiments of learning SSE ensembles of both FNNs and LSTMs.
For example, in LSTM based language modeling (LM), we obtain 21% relative reduction in
LM perplexity by learning a SSE of 4 large LSTM models, which has only 30% of model parameters and 70% of computations in total, as compared to the baseline large LSTM LM.
To the best of our knowledge, this work represents the ﬁrst methodology and empirical study of integrating SG-MCMC, group sparse prior and network pruning together for learning NN ensembles.
I NTRODUCTION
Recently there are increasing interests in using ensembles of Deep Neural Networks (DNNs) (Ju et al.
; Huang et al.
(2017)), which are known to be more robust and accurate than individual networks.
An explanation stems from the fact that learning neural networks is an optimization problem with many local minima (Hansen & Salamon (1990)).
Multiple models obtained from applying stochastic optimization, e.g. the widely used Stochastic Gradient Descent (SGD) and its variants, converge to different local minima and tend to make different errors.
Due to this diversity, the collective prediction produced by an ensemble is less likely to be in error than individual predictions.
The collective prediction is usually performed by averaging the predictions of the multiple neural networks.
On the other hand, the improved prediction accuracy of such model averaging can be understood from the principled perspective of Bayesian inference with Bayesian neural networks.
Speciﬁcally, for each test point ~ x , we consider the predictive distribution P (~ y | ~ x, D ) = ∫ P (~ y | ~ x, θ ) P ( θ |D ) dθ , by integrating the model distribution P (~ y | ~ x, θ ) with the posterior distribution over the model parameters P ( θ |D ) given training data D .
The predictive distribution is then approximated by Monte
Carlo integration P (~
[Equation]
|
[Equation] [Equation]
) ≈
[Equation]
P
[Equation]
y | ~ x, θ ( m ) ) , where θ ( m ) s P ( θ |D ) , m = 1 , · · · , M , are posterior samples of model parameters.
It is well known that such Bayesian model averaging is more accurate in prediction and robust to over-ﬁtting than point estimates of model parameters (Balan et al.
; Li et al.
(2016); Gan et al.
Despite the obvious advantages as seen from both perspectives, a practical problem that hinders the use of DNN ensembles in real-world tasks is that an ensemble requires too much computation in both training and testing.
Traditionally, multiple neural networks are trained, e.g. with different random initialization of model parameters.
Recent studies in (Loshchilov & Hutter (2016); Huang et al.
(2017)) propose to learn an ensemble which consists of multiple snapshot models along the op- timization path within a single training process, by leveraging a special cyclic learning rate schedule.
This reduces the training cost, but the testing cost is still high.
In this paper we also aim at learning an ensemble within a single training process, but by leveraging the recent progress in Bayesian posterior sampling, namely the Stochastic Gradient Markov Chain
Monte Carlo (SG-MCMC) algorithms.
Moreover, we apply group sparse priors in training to enforce group-level sparsity on the network’s connections.
Subsequently we can further use model pruning to compress the networks so that the testing cost is reduced with no loss of accuracy.
Figure 1 presents a high-level overview of our two-stage method to learn Sparse Structured Ensembles (SSEs) for DNNs.
Speciﬁcally, in the ﬁrst stage, we run SG-MCMC with group sparse priors to draw an ensemble of samples from the posterior distribution of network parameters.
In the second stage, we apply weight-pruning to each sampled network and then perform retraining over the remained connections as ﬁne-tuning.
In this way of learning SSEs with SG-MCMC and pruning, we reduce memory and computation cost signiﬁcantly in both training and testing of NN ensembles, while maintaining high prediction accuracy.
This is empirically veriﬁed in our experiments of learning SSE ensembles of both FNNs and LSTMs.
We evaluate the performance of the proposed method on two experiments with different types of neural networks.
The ﬁrst is an image classiﬁcation experiment, which uses Feed-forward Neural
Networks (FNNs) on the well-known MNIST dataset ( Deng (2012)).
Second, we experiment with the more challenging task of Long Short-term Memory (LSTM, Hochreiter & Schmidhuber (1997))
based language modeling, which is conducted on the Penn Treebank dataset (Marcus et al.
It is found that the proposed method works well across both tasks.
For example, we obtain 12% relative reduction (from 78.4 to 68.6) in LM perplexity by learning a SSE of 4 large LSTM models, which has only 40% of model parameters and 90% of computations in total, as compared to the large LSTM LM in Zaremba et al.
Furthermore, when the embedding weights of input and output are shared as in Inan et al.
(2016), we obtain a perplexity of 62.1 (achieving 21% reduction from 78.4) by 4 large LSTMs with only 30% of model parameters and 70% of computations in total.
R ELATED W ORK
This work draws inspiration from three recent research ﬁndings, namely running SG-MCMC for efﬁcient and scalable Bayesian posteriori sampling, applying group sparse priors to enforce network sparsity, and network pruning.
To the best of our knowledge, this work represents the ﬁrst methodology and empirical study of integrating these three techniques and demonstrates its usefulness to learning and using ensembles.
In the following, more discussions are given on related studies.
SG-MCMC sampling: SG-MCMC represents a family of MCMC sampling algorithms developed in recent years, e.g. Stochastic Gradient Langevin Dynamics (SGLD)
(Welling & Teh (2011), Stochastic Gradient Hamiltonian Monte Carlo (SGHMC)
Chen et al.
(2014), mainly for Bayesian learning from large scale datasets.
SG-MCMC has the following favorable properties for learning ensembles.
(i) SG-MCMC works by adding a scaled gradient noise during training, and thus enhances exploration of the model-parameter space.
This is beneﬁcial for ﬁnding diverse sample models for ensembles.
(ii) Scalable and simple: the basic SG-MCMC algorithm, e.g. SGLD, is just a noisy Stochastic Gradient Descent (SGD), which means the same training cost as SGD on large is
but without considering model pruning to reduce the cost of averaging.
Sparse structure learning: Group Lasso penalty (Yuan & Lin (2006)) has been widely used to regularize likelihood function to learn sparse structures.
It was applied with SGD in (Wen et al.
; Alvarez & Salzmann (2016)) and in Wen et al.
(2017) to learn structurally sparse DNNs and LSTMs respectively.
But all focus on point estimates and are not in the context of learning ensembles.
Group Lasso idea has been studied in Bayesian learning, which is known as applying group sparse priors (Marlin et al.
; Babacan et al.
; but these previous works use variational method.
Applying group sparse priors with SG-MCMC has not been explored.
Model compression: Model pruning and retraining (Han et al.
(2015a), Hu et al.
(2016)) has been studied to compress CNNs.
Recently, Han et al.
(2017) and Narang et al.
(2017) apply model pruning to LSTM models for automatic speech recognition task.
We use similar model pruning and retraining method in the experiments.
We ﬁnd that model averaging can enable the ensemble with heavily-pruned networks to be more robust in prediction.
Learning ensembles: Some efforts have been made to reduce the training and testing cost for ensembles.
For reducing the training time cost of ensembles, a special cyclic learning rate schedule is developed in (Loshchilov & Hutter (2016);
Huang et al.
(2017)), which restarts the learning rate periodically to attempt to visit multiple local minima along its optimization path and saves snapshot models.
In contrast to relying on such empirical setting of the learning rate to explore model space, theoretical consistency properties of SG-MCMC methods in posterior sampling have been established (Teh et al.
For reducing the testing time cost of ensembles, Hinton et al.
and
Balan et al.
(2015) distill the knowledge of an ensemble into a single model, but still require large training cost.
L EARNING ENSEMBLES WITH SG-MCMC AND NETWORK PRUNING
We consider the classiﬁcation problem under Bayesian inference framework.
Given training data
D , { ( x i , y i ) } Ni =1 with input feature x i ∈ R D and class label y i ∈ Y , where Y is the set of classes.
We view a neural network as a conditional probabilistic model P ( y i | x i , θ ) .
Denote the network parameters by θ , with P ( θ ) a prior distribution.
We compute the posterior distribution over the model parameters, P ( θ |D )
∝ P ( θ ) ∏ Ni =1 P ( y i | x i , θ ) .
For testing, given a test input ~ x , the Bayesian predictive distribution for its label ~ y is given by P (~ y | ~ x, D ) =
E P ( θ jD )
[ P (~ y | ~ x, θ )] , which can be viewed as model averaging across parameters with distribution P ( θ |D ) .
However, the integration over the posterior is analytically intractable for deep neural networks (DNNs).
Thus it is approximated by Monte Carlo integration as in the following:
[Equation]
where { θ ( m ) } Mm =1 is a set of posterior samples drawn from P ( θ |D ) , e.g. by the popular Markov
Chain Monte Carlo (MCMC) methods.
Traditional MCMC methods either have low-efﬁciency for high dimensional sampling or scale poorly with dataset.
Fortunately, the recently developed
SG-MCMC methods work on stochastic gradients over small mini-batches, which alleviate these problems and can be applied for posterior sampling for DNNs.
3. S AMPLING VIA S TOCHASTIC G RADIENT L ANGEVIN D YNAMICS
Speciﬁcally, we choose the simplest and most widely used SG-MCMC algorithm - Stochastic Gradient Langevin Dynamics (SGLD)
(Welling & Teh (2011)) as the sampling method in our ﬁrst stage of learning ensembles.
Extension by using other high-order SG-MCMC algorithms is straightforward.
SGLD calculates a stochastic gradient of negative log posterior based on S t , small mini-batch of training data:
[Equation]
where U ( θ ) , − log P ( D| θ ) − log P ( θ ) is known as the potential energy in SG-MCMC sampling and e U t ( θ ) is its approxmation over the t -th mini-batch.
The updating rule of SGLD is as simple as
SGD with an additional Gaussian noise ξ s N (0 , ϵ t I) as following:
[Equation]
where ϵ t is the learning rate or step size.
By using gradient information and stochastic mini-batch updating, SGLD overcomes the problems in traditional MCMC methods and thus leads to efﬁcient posterior sampling.
In the following, we provide three discussions about applying SGLD to learning ensembles.
First, note that SGLD is proposed with the use of annealing learning rates since SGLD does not have a Metropolis-Hastings correction step; discretization error goes to zero only when learning rates annealed to zero.
In spite of that, some studies suggest to use constant learning rates in practice (Sato & Nakagawa (2014), Chaudhari et al.
(2016)), which is found to give better mixing rate and make more extensive exploration of parameter space.
This is also compatible with our aim of learning a good ensemble, since we want to collect diverse models.
We test both annealing and constant learning rates in our experiments and ﬁnd that using constant learning rates performs better, as expected.
Hence, we only report the results of using constants learning rate in this work.
Second, we need to consider how to sample θ from the parameter updating sequence { θ t } Tt =1 , where T is the total number of iterations.
Firstly, a burn-in process is desired.
Secondly, a thinned collection of samples { θ
kTM } Mk =1 performs better than other strategies like backward collection { θ t } Tt =
T (cid:0) M +1 , since there are lower correlations between samples.
Our preliminary results as well as the results from Gan et al.
both hold for that, so we take thinned collection as the default setting in this work.
Finally, we need to consider how long to run the sampling algorithm and how many models are used for model averaging.
The ﬁxed-scale additional noise in SGLD generally reduces overﬁtting, thus longer running can be allowed in order to better explore the parameter space.
As shown by the empirical result in Fig. 3(b), the SGLD learning method indeed can improve performance by averaging more models than other traditional methods of learning ensembles.
3. P RUNING AND R ETRAINING
After all the models are collected, we come to the second stage of learning DNN ensembles - network pruning and retraining.
We use a simple pruning rule, i.e. ﬁnding the network connections whose weights are below certain threshold and removing them away, as did in (Han et al.
(2015b)).
The threshold is determined by the conﬁgured overall sparsity or pruning ratio, e.g. 90%, after sorting the weights by their absolute values.
Once the network is pruned, the posterior changes from P ( θ |D ) to the reduced posterior
Q ( m ) ( ϕ ( m ) |D ) , where m is the index of the pruned network.
Retraining is then performed for each pruned network:
[Equation]
We thus obtain an ensemble of networks { ^ ϕ ( m ) }
Mm =1 , which are in fact maximum a posterior (MAP) estimates under the reduced posteriors.
The effect of pruning is to reduce the model size as well as the computation cost.
Interestingly, it is found in our experiments that retraining of the sampled models, whether being pruned or not, signiﬁcantly improve the performance of the ensemble.
There are two justiﬁcations for the retraining phase.
First, theoretically (namely with inﬁnite samples), model averaging according to Equ. (1) does not need retraining.
However, the actual number of samples used in practice is rather small for computational efﬁciency.
So retraining essentially compensates for the limited size of samples for model averaging.
Second, if we denote by (cid:22) ϕ ( m ) the network obtained just after pruning but before retraining, it can be seen that the MAP estimate ^ ϕ ( m ) is more likely than (cid:22) ϕ ( m ) under the reduced posterior.
Note that the probability of (cid:22) ϕ ( m ) under the reduced posterior is close to the probability of (cid:22) ϕ ( m ) under the original posterior, since we only prune small network weights.
So retraining increases the posteriori probabilities of the networks in the ensemble and hopefully improves the prediction performance of the networks in the ensemble.
S PARSE S TRUCTURED E NSEMBLES
The main computation for training or testing a DNN is the large amount of matrix calculations, which are commonly accelerated by using a GPU hardware.
However, a randomly pruned network is not friendly for GPUs to handle, since the randomly positioned zeros in the weight matrices still require ﬂoating-point operations (FLOPs) without special treatment.
In our Sparse Structured
Ensembles (SSEs), we take this into consideration and aim at learning structures for reducing FLOPs in the sense of matrix calculations.
4. G ROUP S PARSE P RIOR
In optimization, a regularization term is often used as a penalty to the objective function to do tradeoff between minimizing a loss function and choosing a desirable model with certain constraints.
The group Lasso regularization Yuan & Lin (2006) proposes to do feature selection in group level, which means keeping or removing all the parameters in a group simultaneously to achieve structured sparsity corresponding to grouping strategy.
It can be formulated as:
[Equation]
where θ g is a group of weights in θ , G is the number of groups, dim ( θ g ) denotes the number of weights in θ g and ∥ · ∥ 2 denotes the l 2 norm.
The term √ dim ( θ g ) ensures that each group gets regularized uniformly corresponding to its dimension.
The coefﬁcient λ , called GSP strength coefﬁcient, is a hyperparameter to do trade off between gaining group sparsity and minimizing the loss function.
While in training, the gradient of each component can be calculated by
[Equation]
A small constant could be added to ∥ θ g ∥ 2 in order to avoid the denominator being zero.
In our experiments, we ﬁnd ∥ θ g ∥ 2 ﬂuctuate near zero and thus do not add the constant.
In Bayesian inference framework, the regularization term corresponds to the negative log prior term − log P ( θ ) in the potential energy
U ( θ ) , thus the group Lasso regularization can be converted into a speciﬁc prior as follows:
[Equation]
where Z is a normalization constant.
The gradient term −∇ θ log P ( θ ) in Equ. (2) for SGLD parameter updating can be directly calculated via Equ. (7), without the use of complex hierarchical decomposition form for the prior as the variational methods do (Marlin et al.
; Babacan et al.
We call it a Group Sparse Prior (GSP) as named in Marlin et al.
4. G ROUPING S TRATEGIES
To learn sparse structured networks for our SSE, it is necessary to specify grouping strategy according to the characteristics of different types of neural networks.
In this paper, we show how to learn
SSE for both FNN and LSTM.
Their grouping strategies are described separately in the following.
Feed-forward Neural Network:
For FNN, we group all the outgoing connections from a single neuron (input or hidden) together following Scardapane et al.
Since FNN’s simple hierarchical structure, if a neuron’s outputs are all zeros, it makes no contribution to the next layer and can be removed.
This leads to node pruning instead of random connection pruning, which reduces the rows and columns of weight matrices between layers, thus leading to lower matrix-level FLOPs as
[Table]
[Figure]
expected.
We can also group the incoming connections of a neuron, but the neurons with no incoming weights are still required to shift their biases to the next layer, which is a bit more complex than the above strategy we choose.
Long Short-term Memory:
The case is not that simple for LSTM, since the input and hidden units are used four times when calculating the input gate, forget gate, cell updates and output gate as follows:
[Equation]
(8) where all the vectors are row vectors, σ ( · ) is the sigmoid function, [ · , · ] denotes concatenating horizontally and ⊙ is element-wise multiplication.
Removing an input or hidden unit is difﬁcult for
LSTM since every unit affects all the updating steps.
However, note that the weight matrix between units and each gate is fully-connected, it is still beneﬁcial to reduce the matrix size by removing a row or column.
Speciﬁcally, we keep two index lists during pruning to record the remained rows and columns for each weight matrix.
When doing computations, we just use partial units to update partial dimensions of the gates according to the index lists.
This is ﬂexible for different units to provide updating for different gate dimensions.
Thus, our ﬁrst grouping strategy is to group each row and each column for the four weight matrices separately in Equ.
Note that the group sparse prior generally selects or removes a certain group, it is allowed to make groups overlapped for reducing matrix size.
We consider this untied strategy since the most basic implementation of LSTM cell conducts calculation as in Equ.
Alternatively, the LSTM updating formulas can be written as in Gal & Ghahramani (2016): 
[Equation]
where W is a matrix of dimension 2 n by 4 n ( n being the unit number for a hidden state), which is the horizontally concatenation of the four weight matrices in Equ.
Since acceleration has been reported by concatenating matrix (Appleyard et al.
, we also try to group each row and column of W as a second grouping strategy.
This strategy is simpler since only two index lists are kept instead of eight, and we call it tied W strategy.
In a concurrent work of learning structurally sparse LSTMs Wen et al.
(2017) using SGD, a grouping strategy, called Intrinsic Sparse Structures (ISS), is proposed to reduce the hidden size by grouping all the weights associated with a certain hidden unit together and removing them simultaneously.
LSTMs learned by ISS can be reconstructed easily with the pruned smaller hidden size, without the need to keep original model size and index lists.
However, the embedding size is not reduced in (Wen et al.
(2017)), which leads to high cost in computing the input for the 1st LSTM layer.
To overcome this, there are two schemes.
(a) Each column of the input embedding matrix is grouped to further reduce the input size of the 1st LSTM layer; (b) The weights from the embedding layer and the softmax layer are shared, as proposed in Inan et al.
(2016), thus the embedding size is the same as hidden size of the last LSTM layer.
An illustration of these strategies are shown in Fig. 2(c).
E XPERIMENTS
In our experiments, we implemented the proposed method in TensorFlow, and present the results in two parts: (i) learn SSE of FNNs for image classiﬁcation task on MNIST; (ii) learn SSE of
LSTM models, which is more challenging, for word-level language modeling task on Penn TreeBank corpus.
The sparsity of a network in this paper means the percentage of the pruned weights from the total parameters, FLOPs for a matrix W is calculated as the size of the smallest sub-matrix formed by such rows and columns in W that contain all non-zero elements in W , and FLOPs for a network is the sum of FLOPs for all its weight matrices.
The parameters and FLOPs presented in the following tables are the total size considering all the models in an ensemble, unless otherwise indicated.
PR,
GSP and SSE denotes Pruning and Retraining, Group Sparse Prior and Sparse Structured Ensemble respectively.
5. C LASSIFICATION ON MNIST
First we use our method on FNNs for classiﬁcation on the well-known MNIST dataset.
We choose a commonly used network structure of 2 hidden layers with 300 and 100 hidden neurons respectively [Table] [Table] and ReLU activations, denoted as FNN-784-300-100-10.
We run our experiments without any additional tricks such as dropout, batch normalization etc.
Such basic setting allows easy reproduction of the results.
All the results reported in table 1 are averaged results from 10 independent runs.
The detailed model structure information for one arbitrary model taken is shown in table 2.
The baseline FNN-784-300-100-10 is trained by Stochastic Gradient Descent (SGD).
Speciﬁcally it is trained for 100 epochs with an annealing learning rate of 0.5 which decays by a factor of 2 every 10 epoch.
The baseline obtains 1.66% test error rate, and an ensemble of 18 independently trained
FNN-784-300-100-10 networks decrease the error to 1.49%.
In the ﬁrst group of experiments with SGLD learning, we use Laplace priors, similar to adding
L1 regularization.
We train also for 100 epochs but with a constant learning rate of 0.5.
Network samples are collected every 5 epoch after a 10 epoch burn in.
The ensemble learned by SGLD gives 1.53% test error, which is slightly worse than the independently trained ensemble, since each sample drawn by constant-step-size SGLD sampling is not as accurate as the sample trained through
SGD optimization.
Adding L1 could enforce sparse structure learning and allows us to prune the network weights.
After pruning, we retrain each network in the ensemble for 20 epochs with a small learning rate of 0.01 which decay by factor 1.15 every epoch.
The resulting ensembles are denoted by SGLD+L1+PR in Table 1.
For these ensembles, the highest sparsity without losing accuracy is 90%.
When 96% of the parameters are pruned, the performance is worsened obviously.
In the second group of experiments with SGLD learning, our new method, SGLD with group sparse prior (GSP) is applied.
The resulting ensembles are denoted by SGLD+GSP+PR in Table 1.
When compared to SGLD+L1+PR, the new method achieves larger sparsity (up to 96%) and FLOP reduction without losing accuracy, presumably because applying GSP forces the pruned connections to be aligned, thus removes more neurons.
When compared to the baseline FNN, the SSE of 18 networks learned by the new method decreases test error from 1.66% to 1.29% with 70% of parameters and 2 . 2 × computational cost.
5. L ANGUAGE M ODELING
Next, we experiment with the more challenging task of learning ensembles of LSTMs, which represent a widely used type of recurrent neural networks (RNNs) for sequence learning tasks.
Speciﬁcally, we study the task of LSTM-based language modeling (LM), which basically is to predict the next word given previous words.
The prediction performance is measured by perplexity (PPL), which is deﬁned as the exponential of negative log-probability per token.
A popular LM benchmarking dataset - Penn TreeBank (PTB) corpus (Marcus et al.
(1993)) is used, with a vocabulary of 10K words and 929K/73K/10K words in training, development and test sets respectively.
We use Zaremba et al.
(2014) as the baseline
and follow their LSTM architectures to make comparable results.
We test different methods on the medium (2 layers with 650 hidden units each) and large (2 layers with 1500 hidden units each) LSTM models as used in Zaremba et al.
The dimension of word embedding as input is the same as the size of hidden units.
All the models are trained with the dropout technique introduced in Zaremba et al.
The experiments without GSP just follow their dropout keep ratio which are 0.5 and 0.35 for medium and large model respectively.
It is found in our experiments that when applying GSP, a higher dropout keep ratio is desired, which are 0.7 for medium and 0.5 for large model.
This is presumably because that both GSP and dropout are some form of regularization and regularizing too much will lead to underﬁtting.
For both untied and tied cases of LSTM weight matrices, the GSP strength coefﬁcients are λ = 4 .
2 × 10 (cid:0) 5 and λ = 2 .
3 × 10 (cid:0) 5 respectively for medium and large model; for the ISS case, the GSP strength coefﬁ- [Table] [Figure] cients are λ = 3 .
× 10 (cid:0) 5 and λ = 1 . 5 × 10 (cid:0) 5 respectively.
For both medium and large models, the learning rates are ﬁxed to 1.5 and 1.0 respectively for SGLD training, and decay by a factor of 1. for retraining by SGD.
All the hyperparameter settings above are found empirically via grid search on the validation set.
The results are organized into four parts.
(1) For ablation analysis, we study the contribution of each component in the new method SGLD+GSP+PR through a series of comparison experiments, as shown in Table 3.
(2) We show the effects of the number of model samples and sampling strategies for the method SGLD+GSP+PR, as given in Fig.
(3) We display in Fig.
4 the sparse structures, obtained from applying the method SGLD+GSP+PR.
(4) Main results are summarized and compared in Table 5.
Table 3 lists the model ablation results of SGLD training with different combinations of pruning, retraining and GSP (untied grouping strategy as default).
It is found that applying PR or GSP alone lead to worse performance than vanilla SGLD for learning LSTM ensembles.
When they are applied together, GSP forces the network to learn group sparse structures which are highly robust to pruning, thus leading to a better result.
With GSP, applying pruning only leads to negligible loss of performance but greatly reduces the model size, as can been seen from comparing SGLD+GSP+PR with
SGLD+GSP+R; pruning without retraining produces inferior result.
The three grouping strategies perform close to each other.
Remarkably, compared to the medium LSTM ensemble obtained by multiple training, the ensemble learned by the new method SGLD+GSP+PR reduces the PPL from 72.0 to 69.5, and with only 10% parameters and 40% FLOPs in total.
SGLD indeed provides a good approach to ﬁnding diverse sample models for ensembles.
Fig.3(a) presents the PPL curves along the learning of LSTM ensemble by the new method SGLD+GSP+PR over the development set.
It clearly shows the performance gains brought by model averaging and PR.
The relationship between the performance of an ensemble and the number of models in an ensemble is examined in Fig.3(b), together with a comparison between different sampling strategies.
We test the performances of different ensembles on the PTB development set, each consisting of 2 to 35 medium LSTMs.
The blue curve shows the result of running SGLD+GSP+PR for 80 epochs and sampling uniformly after a 10-epoch burn in process, e.g. sampling every 2 epoch to collect 35 models and sampling every 35 epoch to get 3 models.
The orange curve is obtained by sampling every 6 epochs, which means that the more model collected the longer run of SGLD.
It can be seen from comparing the two curves that it is better to sample with larger interval with relatively small number of models.
It is also clear from Fig.3(b) that the traditional ensemble learning method by SGD training of multiple models is inferior to the SGLD learning method.
L S T M l a y e r L S T M l a y e r L S T M l a y e r
L S T M l a y e r L S T M l a y e r
L S T M l a
[Figure]
[Table] Fig. 4 shows the sparse structured patterns of the weight matrices from a single LSTM model sample in the ensemble trained by applying SGLD with GSP, separately for three different grouping strategies of weight matrices.
Table 4 shows the FLOPs for each layer for these LSTM model samples.
Note that word embedding is not included for FLOP calculation, since it is a table lookup process instead of matrix calculation in practice, but we still prune it to reduce model size.
Models learned by ISS have desirable homogeneously-sparse structures and thus fewer FLOPs.
Comparison of various models based on LSTMs on PTB dataset are summarized in Table 5.
We investigate to use small number of models to achieve trade off between cost and performance.
An attractive model is the SSE of 4 large LSTMs, which only requires 40% of parameters and 90% of computational cost in total, compared to the baseline large LSTM Zaremba et al.
(2014), but decrease the perplexity from 78.4 to 68.7.
This result is also better than those obtained by Zaremba et al.
(2014) (38 independently trained large LSTMs) and Gal & Ghahramani (2016) (10 independently trained large LMs with costly MC dropout for testing), not only in terms of PPL reduction but also in term of reducing memory and computing costs.
As suggested by a referee, we compare SGD (1 model)+GSP+PR with SGLD (ensemble)+GSP+PR.
SGD+GSP+PR represents the SGD training with group sparse prior and model pruning/retraining.
SGD (1 model)+GSP+PR can reduce the model size but the PPL is much worse than the ensemble, which clearly shows the improvement provided by the ensemble.
Additionally, we compare SGLD (4 models)+GSP+PR with SGD (4 models)+GSP+PR, namely the classic ensemble training method by multiple independent runs with different initializations.
The two ensembles achieve close PPLs.
However, SGD ensemble learning requires 30 × 4 epochs training and 15 × 4 epochs retraining,
SGLD ensemble learning takes 80 epochs training plus 15 × 4 epochs retraining, which reduces about [Table]
Note that a number of better model architectures (Inan et al.
; Merity et al.
; Yang et al.
(2017)) have emerged than the baseline large LSTM LM, which we used as a baseline.
Our
SGLD+GSP+PR in principle can be applied to those new model architectures.
As an example, we apply SGLD+GSP+PR to models that share input and output embeddings Inan et al.
With shared embeddings, we further reduce the perplexity to 62.1 by using the SSE of 4 large LSTMs, which can be regarded as an ensemble version of Inan et al.
(2016) without variational dropout (VD) and augmented loss (AL).
As a side note, without shared embeddings, the lowest perplexity achieved is 66.4 by the SSE of 20 large LSTMs, which is also among the top models obtained with standard
LSTMs to the best of our knowledge.
[Table]
PR 1 large 0.1 × 0.2 × 81. SGD+GSP+PR 4 large 0.4 × 0.9 × 69. SGLD+GSP+PR 20 large 2.0 × 4.5 × 68. SGLD+GSP+PR 4 large 0.4 × 0.9 × 70.
SGLD+GSP+PR+SE (ISS) 4 large 0.3 × (20.4M) 0.7 × 64.
C ONCLUSION AND F UTURE W ORKS
In this work we propose a novel method of learning NN ensembles efﬁciently and cost-friendly by integrating three mutually enhanced techniques: SG-MCMC sampling, group sparse prior and network pruning.
The resulting SGLD+GSP+PR method is easy to implement, yet surprisingly effective.
This is thoroughly evaluated in the experiments of learning SSE ensembles of both FNNs and LSTMs.
The Sparse Structured Ensembles (SSEs) learned by our method gain better prediction performance with reduced training and test cost when compared to traditional methods of learning
NN ensembles.
Moreover, by proper controlling the number of models used in the ensemble, the method can also be used to produce SSE, which outperforms baseline NN signiﬁcantly without increasing the model size and computation cost.
Some interesting future works: (1) interleaving model sampling and model pruning; (2) application of this new method, as a new powerful tool of learning ensembles, to more tasks