L EARNING I NTRINSIC S PARSE S TRUCTURES WITHIN L ONG S HORT -T ERM M EMORY
Anonymous authors
Paper under double-blind review
A BSTRACT
Model compression is signiﬁcant for the wide adoption of Recurrent Neural Networks (RNNs) in both user devices possessing limited resources and business clusters requiring quick responses to large-scale service requests.
This work aims to learn structurally-sparse Long Short-Term Memory (LSTM) by reducing the sizes of basic structures within LSTM units, including input updates, gates, hidden states, cell states and outputs .
Independently reducing the sizes of basic structures can result in inconsistent dimensions among them, and consequently, end up with invalid LSTM units.
To overcome the problem, we propose Intrinsic Sparse
Structures (ISS) in LSTMs.
Removing a component of ISS will simultaneously decrease the sizes of all basic structures by one and thereby always maintain the dimension consistency.
By learning ISS within LSTM units, the obtained LSTMs remain regular while having much smaller basic structures.
Based on group Lasso regularization, our method achieves 10 .
59 × speedup without losing any perplexity of a language modeling of Penn TreeBank dataset.
It is also successfully evaluated through a compact model with only 2 .
69 M weights for machine Question
Answering of SQuAD dataset.
Our approach is successfully extended to nonLSTM RNNs, like Recurrent Highway Networks (RHNs)
I NTRODUCTION
Model Compression (Jaderberg et al.
(2014), Han et al.
(2015a), Wen et al.
, Louizos et al.
(2017)) is a class of approaches of reducing the size of Deep Neural Networks (DNNs) to accelerate inference.
Structure Learning (Zoph & Le (2017), Philipp & Carbonell (2017), Cortes et al.
(2017)) emerges as an active research area for DNN structure exploration, potentially replacing human labor with machine automation for design space exploration.
In the intersection of both techniques, an important area is to learn compact structures in DNNs for efﬁcient inference computation using minimal memory and execution time without losing accuracy.
Learning compact structures in
Convolutional Neural Networks (CNNs) have been widely explored in the past few years.
Han et al.
(2015b)
proposed connection pruning for sparse CNNs.
Pruning method also works successfully in coarse-grain levels, such as pruning ﬁlters in CNNs (Li et al.
and reducing neuron numbers (Alvarez & Salzmann (2016)).
Wen et al.
(2016) presented a general framework to learn versatile compact structures (neurons, ﬁlters, ﬁlter shapes, channels and even layers) in DNNs.
Learning the compact structures in Recurrent Neural Networks (RNNs) is more challenging.
As a recurrent unit is shared across all the time steps in sequence, compressing the unit will aggressively affect all the steps.
A recent work by Narang et al.
(2017) proposes a pruning approach that deletes up to 90% connections in RNNs.
Connection pruning methods sparsify weights of recurrent units but cannot explicitly change basic structures , e.g. , the number of input updates, gates, hidden states, cell states and outputs .
Moreover, the obtained sparse matrices have an irregular/nonstructured pattern of non-zero weights, which is unfriendly for efﬁcient computation in modern hardware systems (Lebedev & Lempitsky (2016)).
Previous study (Wen et al.
on sparse matrix multiplication in GPUs showed that the speedup 1 was either counterproductive or ignorable.
More speciﬁc, with sparsity 2 of 67 .
6 % and 94 .
3 % in weight matrices of
AlexNet , the speedup was 0 .
25 × , 0 .
52 × , 1 . 38 × , 1 . 04 × , and 1 .
36 × , respectively.
This problem also
Deﬁned the
[Figure]
exists in CPUs.
Fig.
1 shows that non-structured pattern in sparsity limits the speedup.
We only starts to observe speed gain when the sparsity is beyond 80% and the speedup is only around 2 .
5 × even when the sparsity is 95% .
In this work, we focus on learning structurally sparse LSTMs for computation efﬁciency.
More speciﬁc, we aim to reduce the number of basic structures simultaneously during learning, such that the obtained LSTMs have the original schematic with dense connections but with smaller sizes of these basic structures.
Such compact models have structured sparsity, with columns and rows in weight matrices removed, whose computation efﬁciency is shown in Fig.
Moreover, off-the-shelf libraries in deep learning frameworks can be directly utilized to deploy the reduced LSTMs.
Details should be explained.
There is a vital challenge originated from recurrent units: as the basic structures interweave with each other, independently removing these structures can result in mismatch of their dimensions and then inducing invalid recurrent units.
The problem does not exist in CNNs, where neurons (or ﬁlters) can be independently removed without violating the usability of the ﬁnal network structure.
One of our key contributions is to identify the structure inside RNNs that shall be considered as a group to most effectively explore sparsity in basic structures.
More speciﬁc, we propose Intrinsic Sparse Structures (ISS) as groups to achieve the goal.
By removing weights associated with one component of ISS, the sizes/dimensions (of basic structures) are simultaneously reduced by one.
We evaluated our method by LSTMs and RHNs in language modeling of Penn Treebank dataset (Marcus et al.
and machine Question Answering of SQuAD dataset (Rajpurkar et al.
Our approach works both in ﬁne-tuning and in training from scratch.
In a RNN with two stacked LSTM layers with hidden sizes of 1500 ( i.e. , 1500 components of ISS) for language modeling (Zaremba et al.
, our method learns that the sizes of 373 and 315 in the ﬁrst and second LSTMs, respectively, are sufﬁcient for the same perplexity.
It achieves 10 .
59 × speedup of inference time.
The result is obtained by training from scratch with the same number of epochs.
Directly training LSTMs with sizes of 373 and 315 cannot achieve the same perplexity, which proves the advantage of learning ISS for model compression.
Encouraging results are also obtained in more compact and state-of-the-art models – the RHN models (Zilly et al.
and BiDAF model (Seo et al.
Our source code is publicly available ( URL is removed for double blind review ).
R ELATED W ORK
A major approach in DNN compression is to reduce the complexity of structures within DNNs.
The studies can be categorized to three classes: removing redundant structures in original DNNs, approximating the original function of DNNs (Denil et al.
(2013), Jaderberg et al.
(2014), Hinton et al.
(2015), Lu et al.
(2016), Prabhavalkar et al.
(2016), Molchanov et al.
(2017)), and designing
[Equation]
⊗ ⊗ ⊕ inputupdates
[Figure]
DNNs with inherently compact structures (Szegedy et al. (2015), He et al.
(2016), Wu et al.
Bradbury et al.
Our method belongs to the ﬁrst category.
Research on removing redundant structures in Feed-forward Neural Networks (FNNs), typically in
CNNs, has been extensively studied.
Based on (cid:96) 1 regularization (Liu et al.
, Park et al.
or connection pruning (Han et al.
(2015b), Guo et al.
, the number of connections/parameters can be dramatically reduced.
Group Lasso based methods were proved to be effective in reducing coarse-grain structures ( e.g., neurons, ﬁlters, channels, ﬁlter shapes, and even layers) in CNNs (Wen et al.
(2016), Alvarez & Salzmann (2016), Lebedev & Lempitsky (2016), Yoon & Hwang (2017)).
For instance, Wen et al.
(2016) reduced the number of layers from 32 to 18 in ResNet without any accuracy loss for CIFAR-10 dataset.
A recent work by Narang et al.
(2017) advances connection pruning techniques for RNNs.
It compresses the size of Deep Speech 2 (Amodei et al.
from 268 MB to around 32 MB.
However, to the best of our knowledge, little work has been carried out to reduce coarse-grain structures beyond ﬁne-grain connections in RNNs.
To ﬁll this gap, our work targets to develop a method that can learn to reduce the number of basic structures within LSTM units.
After learning those structures, ﬁnal LSTMs are still regular LSTMs with the same connectivity, but have the sizes reduced.
Another line of related research is Structure Learning of FNNs or CNNs.
Zoph & Le (2017) uses reinforcement learning to search good neural architectures.
Philipp & Carbonell (2017) dynamically adds and eliminates neurons in FNNs by using group Lasso regularization.
Cortes et al.
(2017) gradually adds sub-networks to current networks to incrementally reduce the objective function.
All these works focused on ﬁnding optimal structures in FNNs or CNNs for classiﬁcation accuracy.
In contrast, this work aims at learning compact structures in LSTMs for model compression.
L EARNING I NTRINSIC S PARSE S TRUCTURES 3. I NTRINSIC S PARSE S TRUCTURES
The computation within LSTMs is (Hochreiter & Schmidhuber (1997))
[Equation]
where (cid:12) is element-wise multiplication, σ ( · ) is sigmoid function, and tanh ( · ) is hyperbolic tangent function.
Vectors are row vectors.
W s are weight matrices, which transform the concatenation (of hidden states h t − 1 and inputs x t ) to input updates u t and gates ( i t , f t and o t ).
Fig.
2 is the schematic of LSTMs in the layout of Olah (2015). The transformations by W s and the corresponding nonlinear
W xf
W xi
W xu
W xo
W hf
W hi
W hu
W ho x h
[Figure]
functions are illustrated in rectangle blocks.
Our goal is to reduce the size of this sophisticated structure within LSTMs, meanwhile maintaining the original schematic.
Because of element-wise operators (“ ⊕ ” and “ ⊗ ”), all vectors along the blue band in Fig.
2 must have the same dimension.
We call this constraint as “ dimension consistency ”.
The vectors required to obey the dimension consistency include input updates, all gates, hidden states, cell states, and outputs.
Note that hidden states are usually outputs connected to classiﬁer layer or stacked LSTM layers.
As can be seen in Fig. 2, vectors (along the blue band) interweave with each other so removing an individual component from one or a few vectors independently can result in the violation of dimension consistency.
To overcome this, we propose Intrinsic Sparse Structures (ISS) within LSTMs as shown by the blue band in Fig.
One component of ISS is highlighted as the white strip.
By decreasing the size of ISS ( i.e. , the width of the blue band), we are able to simultaneously reduce the dimensions of basic structures.
To learn sparse ISS, we turn to weight sparsifying.
There are totally eight weight matrices in Eq.
We organize them in the form of Fig.
3 as basic LSTM cells in TensorFlow.
We can remove one component of ISS by zeroing out all associated weights in the white rows and white columns in
Fig.
Why?
Suppose the k -th hidden state of h is removable, then the k -th row in the lower four weight matrices can be all zeros (as shown by the left white horizontal line in Fig. 3), because those weights are on connections receiving the k -th useless hidden state.
Likewise, all connections receiving the k -th hidden state in next layer(s) can be removed as shown by the right white horizontal line.
Note that next layer(s) can be an output layer, LSTM layers, fully-connected layers, or a mix of them.
ISS overlay two or more layers, without explicit explanation, we refer to the ﬁrst LSTM layer as the ownership of ISS.
When the k -th hidden state turns useless, the k -th output gate and k -th cell state generating this hidden state are removable.
As the k -th output gate is generated by the k -th column in W xo and W ho , these weights can be zeroed out (as shown by the fourth vertical white line in Fig. 3).
Tracing back against the computation ﬂow in Fig.
2, we can reach similar conclusions for forget gates, input gates and input updates, as respectively shown by the ﬁrst, second and third vertical line in Fig.
For convenience, we call the weights in white rows and columns as an “ ISS weight group ”.
Although we propose ISS in LSTMs, variants of ISS for vanilla RNNs, Gated
Recurrent Unit (GRU) (Cho et al.
(2014)), and Recurrent Highway Networks (RHNs) (Zilly et al.
(2017)) can also be realized based on the same philosophy.
For even a medium-scale LSTM, the number of weights in one ISS weight group can be very large.
It seems to be very aggressive to simultaneously slaughter so many weights to maintain the original recognition performance.
However, the proposed ISS intrinsically exists within LSTMs and can even be unveiled by independently sparsifying each weight using (cid:96) 1 -norm regularization.
The experimental result is covered in Appendix A. It unveils that sparse ISS intrinsically exist in LSTMs and the learning process can easily converge to the status with a high ratio of ISS removed.
In
Section 3.2, we propose a learning method to explicitly remove much more ISS than the implicit (cid:96) 1 -norm regularization.
3. L EARNING M ETHOD
Suppose w ( n ) k is a vector of all weights in the k -th component of ISS in the n -th LSTM layer ( 1 ≤ n ≤ N and 1 ≤ k ≤ K ( n ) ), where N is the number of LSTM layers and K ( n ) is the number of ISS components ( i.e. , hidden size) of the n -th LSTM layer.
The optimization goal is to remove as many “ISS weight groups” w ( n ) k as possible without losing accuracy.
Methods to remove weight groups (such as ﬁlters, channels and layers) have been successfully studied in CNNs as summarized in Section 2.
However, how these methods perform in RNNs is unknown.
Here, we extend the group
Lasso based methods (Yuan & Lin (2006)) to RNNs for ISS sparsity learning.
More speciﬁc, the group Lasso regularization is added to the minimization function in order to encourage sparsity in
ISS.
Formally, the ISS regularization is
[Equation]
where w is the vector of all weights and || · || 2 is (cid:96) 2 -norm ( i.e. , Euclidean length).
In Stochastic
Gradient Descent (SGD) training, the step to update each ISS weight group becomes
[Equation]
where E ( w ) is data loss, η is learning rate and λ > 0 is the coefﬁcient of group Lasso regularization to trade off recognition accuracy and ISS sparsity.
The regularization gradient, i.e. , the last term in
Eq.
(3), is a unit vector.
It constantly squeezes the Euclidean length of each w ( n ) k to zero, such that, a high portion of ISS components can be enforced to fully-zeros after learning.
To avoid division by zero in the computation of regularization gradient, we can add a tiny number (cid:15) in || · || 2 , that is,
[Table] [Table]
where w ( n ) kj is the j -th element of w ( n ) k .
We set (cid:15) = 1 .
e − 8 .
The learning method can effectively squeeze many groups near zeros, but it is very hard to exactly stabilize them as zeros because of the always-present ﬂuctuating weight updates.
Fortunately, the ﬂuctuation is within a tiny ball centered at zero.
To stabilize the sparsity during training, we zero out the weights whose absolute values are smaller than a pre-deﬁned threshold τ .
The process of thresholding is applied per mini-batch.
E XPERIMENTS
Our experiments use published models as baselines.
The application domains include language modeling of Penn TreeBank and machine Question Answering of SQuAD dataset.
For more comprehensive evaluation, we sparsify ISS in LSTM models with both a large hidden size of 1500 and a small hidden size of 100 .
We also extended ISS approach to state-of-the-art Recurrent Highway
Networks (RHNs) (Zilly et al.
to reduce the number of units per layer.
We maximize threshold τ to fully exploit the beneﬁt.
For a speciﬁc application, we preset τ by cross validation.
The maximum τ which sparsiﬁes the dense model (baseline) without deteriorating its performance is selected.
The validation of τ is performed only once and no training effort is needed.
τ is 1 .
e −for the stacked LSTMs in Penn TreeBank, and it is 4 . 
 e − 4 for the RHN and the BiDAF model.
We used HyperDrive by Rasley et al.
(2017) to explore the hyperparameter of λ .
More details can be found in our source code.
To measure the inference speed, the experiments were run on a dual socket Intel Xeon CPU E52673 v3 @ 2 .
40 GHz processor with a total of 24 cores ( 12 per socket) and 128 GB of memory.
Intel
MKL library 2017 update 2 was used for matrix-multiplication operations.
OpenMP runtime was utilized for parallelism.
We used Intel C++ Compiler 17.0 to generate executables that were run on
Windows Server 2016.
Each of the experiments was run for 1000 iterations, and the execution time was averaged to ﬁnd the execution latency.
4. L ANGUAGE M ODELING 4.1. S TACKED LSTM S
A RNN with two stacked LSTM layers for language modeling (Zaremba et al.
(2014)) is selected as the baseline.
It has hidden sizes of 1500 ( i.e. , 1500 components of ISS) in both LSTM units.
The output layer has a vocabulary of 10000 words.
The dimension of word embedding in the input layer is 1500 .
Word embedding layer is not sparsiﬁed because the computation of selecting a [Table] vector from a matrix is very efﬁcient.
The same training scheme as the baseline is adopted to learn ISS sparsity, except a larger dropout keep ratio of 0 . 6 versus 0 .
35 of the baseline because group
Lasso regularization can also avoid over-ﬁtting.
All models are trained from scratch for 55 epochs.
The results are shown in Table 1.
Note that, when trained using dropout keep ratio of 0 .
6 without adopting group Lasso regularization, the baseline over-ﬁts and the lowest validation perplexity is 97 .
The trade-off of perplexity and sparsity is controlled by λ .
In the second row, with tiny perplexity difference from baseline, our approach can reduce the number of ISS in the ﬁrst and second LSTM unit from 1500 , down to 373 and 315 , respectively.
It reduces the model size from 66 .
0 M to 21 .
8 M and achieves 10 .
59 × speedup.
Remarkably, the practical speedup ( 10 . 59 × ) even goes beyond theoretical mult-add reduction ( 7 . 48 × ) as shown in Table 1 —which comes from the increased computational efﬁciency.
When applying structured sparsity, the underlying weight matrices become smaller so as to ﬁt into the L3 cache with good locality, which improves the FLOPS (ﬂoating point operations per second).
This is a key advantage of our approach over non-structurally sparse RNNs generated by connection pruning (Narang et al.
(2017)), which suffers from irregular memory access pattern and inferior-theoretical speedup.
At last, when learning a compact structure, our method can perform as structure regularization to avoid overﬁtting.
As shown in the third row in Table 1, lower perplexity is achieved by even a smaller ( 25 . 2 M) and faster ( 7 . 10 × ) model.
Its learned weight matrices are visualized in Fig.
4, where 1119 and 965 ISS components shown by white strips are removed in the ﬁrst and second LSTM, respectively.
A straightforward way to reduce model complexity is to directly design a RNN with a smaller hidden size and train from scratch.
Compare with direct design approach, our ISS method can automatically learn optimal structures within LSTMs.
More importantly, compact models learned by ISS method have lower perplexity, comparing with direct design method.
To evaluate it, we directly design a RNN with exactly the same structure of the second RNN in Table 1 and train it from scratch instead of learning ISS from a larger RNN.
The result is included in the last row of Table 1.
We tuned dropout keep ratio to get best perplexity for the directly-designed RNN.
The ﬁnal test perplexity is 85 . 66 , which is 7 .
01 higher that our ISS method.
4.1. E XTENSION TO R ECURRENT H IGHWAY N ETWORKS
Recurrent Highway Networks (RHN) (Zilly et al.
(2017)) is a class of state-of-the-art recurrent models, which enable “step-to-step transition depths larger than one”.
In a RHN, we deﬁne the number of units per layer as RHN width .
Speciﬁcally, we select the “Variational RHN + WT” model in Table 1 of Zilly et al.
(2017) as the baseline.
It has depth 10 and width 830 , with totally 23 .
5 M parameters.
In a nutshell, our approach can reduce the RHN width from 830 to 525 without losing perplexity.
[Table]
Following the same idea of identifying the “ISS weight groups” to reduce the size of basic structures in LSTMs
[Table]
we can identify the groups in RHNs to reduce the RHN width.
In brief, one group include corresponding columns/rows in weight matrices of the H nonlinear transform, of the T and C gates, and of the embedding and output layers.
The group size is 46520 .
The groups are indicated by JSON ﬁles in our source code.
By learning ISS in RHNs, we can simultaneously reduce the dimension of word embedding and the number of units per layer.
Table 2 summarizes results.
All experiments are trained from scratch with the same hyperparameters in the baseline, except that smaller dropout ratios are used in ISS learning.
Larger λ , smaller RHN width but higher perplexity.
More importantly, without losing perplexity, our approach can learn a smaller model with RHN width 517 from an initial model with RHN width 830 .
This reduces the model size to 11 .
1 M, which is 52 .
8% reduction.
Moreover, ISS learning can ﬁnd a smaller RHN model with width 726 , meanwhile improve the state-of-the-art perplexity as shown by the second entry in Table 2.
We evaluate ISS method by state-of-the-art dataset (SQuAD) and model (BiDAF).
SQuAD (Rajpurkar et al.
(2016)) is a recently released reading comprehension dataset, crowdsourced from
100 , 000+ question-answer pairs on 500+ Wikipedia articles.
ExactMatch (EM) and F1 scores are two major metrics for the task 3 .
The higher those scores are, the better the model is.
We adopt BiDAF (Seo et al.
to evaluate how ISS method works in small LSTM units.
BiDAF is a compact machine Question Answering model with totally 2 .
69 M weights.
The ISS sizes are only 100 in all LSTM units.
The implementation of BiDAF is made available by its authors 4 .
BiDAF has character, word and contextual embedding layers to extract representations from input sentences, following which are bi-directional attention layer, modeling layer, and ﬁnal output layer.
LSTM units are used in contextual embedding layer, modeling layer, and output layer.
All LSTMs are bidirectional (Schuster & Paliwal (1997)).
In a bidirectional LSTM, there are one forward plus one backward LSTM branch.
The two branches share inputs and their outputs are concatenated for next stacked layers.
We found that it is hard to remove ISS components in contextual embedding layer, because the representations are relatively dense as it is close to inputs and the original hidden size ( 100 ) is relatively small.
In our experiments, we exclude LSTMs in contextual embedding layer and sparsify all other LSTM layers.
Those LSTM layers are the computation bottleneck of BiDAF.
We proﬁled the computation time on CPUs, and ﬁnd those LSTM layers (excluding contextual embedding layer) consume 76 .
47% of total inference time.
There are three bi-directional LSTM layers we will sparsify, two of which belong to the modeling layer, and one belongs to the output layer.
More details of BiDAF are covered by Seo et al.
For brevity, we mark the forward (backward) path of the 1 st bi-directional LSTM in the modeling layer as ModFwd1 ( ModBwd1 ).
Similarly, ModFwd2 and ModBwd2 are
[Table]
[Table] As discussed in Section 3. [Table]
multiple parallel layers can receive the hidden states from the same LSTM layer and all connections (weights) receive those hidden states belong to the same ISS.
For instance
[Table]
ModFwd2 and ModBwd2 both receive hidden states of ModFwd1 as inputs, therefore the k -th “ISS weight group” includes the k -th rows of weights in both ModFwd2 and ModBwd2 , plus the weights in the k -th ISS component within ModFwd1 .
For simplicity, we use “ISS of ModFwd1 ” to refer to the whole group of weights.
Structures of six ISS are included in Table 5 in Appendix B.
We learn ISS sparsity in BiDAF by both ﬁne-tuning the baseline and training from scratch.
All the training schemes keep as the same as the baseline except applying a higher dropout keep ratio.
After training, we zero out weights whose absolute values are smaller than 0 .
This does not impact EM and F1 scores, but increase sparsity.
Table 3 shows the EM, F1, the number of remaining ISS components, model size, and inference speed.
The ﬁrst row is the baseline BiDAF.
Other rows are obtained by ﬁne-tuning baseline using
ISS regularization.
In the second row by learning ISS, with small EM and F1 loss, we can reduce ISS in all LSTMs except ModFwd1 .
For example, almost half of the ISS components are removed in OutBwd .
By increasing the strength of group Lasso regularization ( λ ), we can increase the ISS sparsity by losing some EM/F1 scores.
The trade-off is listed in Table 3.
With 2 .
63 F1 score loss, the sizes of OutFwd and OutBwd can be reduced from original 100 to 15 and 12 , respectively.
At last, we ﬁnd it hard to reduce ISS sizes without losing any EM/F1 score.
This implies that BiDAF is compact enough and its scale is suitable for both computation and accuracy.
However, our method can still signiﬁcantly compress this compact model under acceptable performance loss.
At last, instead of ﬁne-tuning baseline, we train BiDAF from scratch with ISS learning.
The results are summarized in Table 4.
Our approach also works well when training from scratch.
Overall, training from scratch balances the sparsity across all layers better than ﬁne-tuning, which results in even better compression of model size and speedup of inference time.
The histogram of vector lengths of “ISS weight groups” is plotted in Appendix C.
C ONCLUSION
We proposed Intrinsic Sparse Structures (ISS) within LSTMs and its learning method to simultaneously reduce the sizes of input updates, gates, hidden states, cell states and outputs within the sophisticated LSTM structure.
By learning ISS, a structurally sparse LSTM can be obtained, which essentially is a regular LSTM with reduced hidden dimension.
Thus, no software or hardware speciﬁc customization is required to get storage saving and computation acceleration.
Though ISS is proposed with LSTMs, it can be easily extended to vanilla RNNs, Gated Recurrent Unit (GRU) (Cho et al.
(2014)), and Recurrent Highway Networks (RHNs) (Zilly et al.
A PPENDIX A ISS U NVEILED BY (cid:96) 1 - NORM R EGULARIZATION
[Figure]
We take the large stacked LSTMs by Zaremba et al.
(2014) for language modeling as the example.
The network has two stacked LSTM layers whose dimensions of inputs and states are both 1500 , and it has an output layer with a vocabulary of 10000 words. The sizes of “ISS weight groups” of two LSTM layers are 24000 and 28000 . The perplexities of validation set and test set are respectively 82 .
57 and 78 .
We ﬁne-tune this baseline LSTMs with (cid:96) 1 -norm regularization.
The same training hyper-parameters as the baseline are adopted, except a bigger dropout keep ratio of 0 .
6 (original 0 . 35 ).
A weaker dropout is used because (cid:96) 1 -norm is also a regularization to avoid overﬁtting.
A too strong dropout plus (cid:96) 1 -norm regularization can result in underﬁtting.
The weight decay of (cid:96) 1norm regularization is 0 . 0001 .
The sparsiﬁed network has validation perplexity and test perplexity of 82 .
40 and 78 .
60 , respectively, which is approximately the same with the baseline.
The sparsity of weights in the ﬁrst LSTM layer, the second LSTM layer and the last output layer is 91 . 32% and 90 . 22% , respectively. Fig.
5 plots the learned sparse weight matrices.
The sparse matrices in the top row reveal some interesting patterns: there are lots of all-zero columns and rows, and their positions are highly correlated.
Those patterns are proﬁled in the bottom row.
Much to our surprise, sparsifying individual weight independently can converge to sparse LSTMs with many ISS removed— 504 and 220 ISS components in the ﬁrst and second LSTM layer are all-zeros.
A PPENDIX B ISS IN B I DAF
[Table]
A PPENDIX C H ISTOGRAM OF VECTOR LENGTHS OF ISS WEIGHT GROUPS IN B I DAF
[Figure]