L EARNING I NTRINSIC S PARSE S TRUCTURES WITHIN L ONG S HORT -T ERM M EMORY
Anonymous authors
Paper under double-blind review
A BSTRACT
Model compression is signiﬁcant for the wide adoption of Recurrent Neural Networks (RNNs) in both user devices possessing limited resources and business clusters requiring quick responses to large-scale service requests.
This work aims to learn structurally-sparse Long Short-Term Memory (LSTM) by reducing the sizes of basic structures within LSTM units, including input updates, gates, hidden states, cell states and outputs.
Independently reducing the sizes of basic structures can result in inconsistent dimensions among them, and consequently, end up with invalid LSTM units.
To overcome the problem, we propose Intrinsic
Sparse Structures (ISS) in LSTMs.
Removing a component of ISS will decrease the sizes of all basic structures by one simultaneously and thereby always maintain the dimension consistency.
By learning ISS within LSTM units, the obtained
LSTMs remain regular while having much smaller basic structures.
Our method achieves 10 .
59 × speedup in state-of-the-art LSTMs, without losing any perplexity of language modeling of Penn TreeBank dataset.
It is also successfully evaluated through a compact model with only 2 .
69 M weights for machine Question Answering of SQuAD dataset.
I NTRODUCTION
Model Compression (Jaderberg et al.
(2014), Han et al.
(2015a), Wen et al.
, Louizos et al.
(2017)) is a class of approaches of reducing the size of Deep Neural Networks (DNNs) to accelerate inference.
Structure Learning (Zoph & Le (2017), Philipp & Carbonell (2017), Cortes et al.
(2017)) emerges as an active research area for DNN structure exploration, potentially replacing human labor with machine automation for design space exploration.
In the intersection of both techniques, an important area is to learn compact structures in DNNs for efﬁcient inference computation using minimal memory and execution time without losing accuracy.
Learning compact structures in
Convolutional Neural Networks (CNNs) have been widely explored in the past few years.
Han et al.
(2015b)
proposed connection pruning for sparse CNNs.
Pruning method also works successfully in coarse-grain levels, such as pruning ﬁlters in CNNs (Li et al.
and reducing neuron numbers (Alvarez & Salzmann (2016)).
Wen et al.
(2016) presented a general framework to learn versatile compact structures (neurons, ﬁlters, ﬁlter shapes, channels and even layers) in DNNs.
Learning the compact structures in Recurrent Neural Networks (RNNs) is more challenging.
As a recurrent unit is shared across all the time steps in sequence, compressing the unit will aggressively affect all the steps.
A recent work by Narang et al.
(2017) proposes a pruning approach that deletes up to 90% connections in RNNs.
Connection pruning methods sparsify weights of recurrent units but cannot explicitly change basic structures, e.g. , the number of input updates, gates, hidden states, cell states and outputs.
Moreover, the obtained sparse matrices have an irregular pattern of nonzero weights, which is unfriendly for efﬁcient computation in modern hardware systems (Lebedev & Lempitsky (2016)).
Previous study (Wen et al.
on sparse matrix multiplication in GPUs showed that the speedup 1 was either counterproductive or ignorable.
More speciﬁc, with sparsityof 67 .
6 % and 94 .
3 % in weight matrices of AlexNet , the speedup was 0 .
25 × , 0 .
52 × , 1 . 38 × , 1 . 04 × , and 1 .
36 × , respectively.
In this work, we focus on learning structurally sparse LSTMs for computation efﬁciency.
More speciﬁc, we aim to reduce the number of input updates, gates, hidden states, cell states and outputs simultaneously during learning, such that the obtained LSTMs can maintain the original schematic with dense connections and smaller sizes of these basic structures.
Such compact models help save memory and execution time.
Moreover, off-the-shelf libraries in deep learning frameworks can be directly utilized to deploy the reduced LSTMs.
There is a vital challenge originated from recurrent units: as the structures of inputs, gates, states and outputs interweave with each other, independently removing these structures can result in mismatch of their dimensions and then inducing invalid recurrent units.
In essence, the dimension consistency is introduced by the element-wise operations in recurrent units, e.g. , element-wise multiplication/ addition among input updates, gates, states and outputs in LSTMs.
The problem does not exist in
CNNs, where neurons (or ﬁlters) can be independently removed without violating the usability of the ﬁnal network structure.
we propose Intrinsic Sparse Structures (ISS) to overcome the unique challenge in LSTMs.
By removing weights associated with one component of ISS, the sizes/dimensions (of input updates, gates, hidden states and cell states in current layer, and outputs to the next stacked layers) are simultaneously reduced by one.
Therefore, learning ISS can directly reduce the size of structures in LSTMs meanwhile maintains the dimension consistency.
We evaluated our method by LSTMs in language modeling of Penn Treebank dataset (Marcus et al.
and machine Question Answering of SQuAD dataset (Rajpurkar et al.
Our approach works both in ﬁne-tuning and in training from scratch.
In a RNN with two stacked LSTM layers with hidden sizes of 1500 ( i.e. , 1500 components of ISS) for language modeling (Zaremba et al.
, our method learns that the sizes of 373 and 315 in the ﬁrst and second LSTMs, respectively, are sufﬁcient for the same perplexity.
It achieves 10 .
59 × speedup of inference time.
The result is obtained by training from scratch with the same number of epochs.
Directly training LSTMs with sizes of 373 and 315 cannot achieve the same perplexity, which proves the advantage of learning ISS for model compression.
Promising results are also obtained in a compact model – the state-of-theart BiDAF model (Seo et al.
for Question Answering in SQuAD dataset.
BiDAF has only 2 .
69 M weights.
Our source code is public available ( URL is removed for double blind review ).
R ELATED W ORK
A major approach in DNN compression is to reduce the complexity of structures within DNNs.
The studies can be categorized to three classes: removing redundant structures in original DNNs, approximating the original function of DNNs (Denil et al.
(2013), Jaderberg et al.
(2014), Hinton et al.
(2015), Lu et al.
(2016), Prabhavalkar et al.
(2016), Molchanov et al.
(2017)), and designing
DNNs with inherently compact structures (Szegedy et al. (2015), He et al.
(2016), Wu et al.
Bradbury et al.
Our method belongs to the ﬁrst category.
Research on removing redundant structures in Feed-forward Neural Networks (FNNs), typically in
CNNs, has been extensively studied.
Based on (cid:96) 1 regularization (Liu et al.
, Park et al.
or connection pruning (Han et al.
(2015b), Guo et al.
, the number of connections/parameters can be dramatically reduced.
Group Lasso based methods were proved to be effective in reducing coarse-grain structures ( e.g., neurons, ﬁlters, channels, ﬁlter shapes, and even layers) in CNNs (Wen et al.
(2016), Alvarez & Salzmann (2016), Lebedev & Lempitsky (2016), Yoon & Hwang (2017)).
For instance, Wen et al.
(2016) reduced the number of layers from 32 to 18 in ResNet without any accuracy loss for CIFAR-10 dataset.
A recent work by Narang et al.
(2017) advances connection pruning techniques in CNNs and extends it to RNNs.
It compresses the size of Deep Speech(Amodei et al.
from 268 MB to around 32 MB.
However, to the best of our knowledge, little work has been carried out to reduce coarse-grain structures beyond ﬁne-grain connections in RNNs, possibly because of the sophisticated structures of RNNs, especially in LSTMs.
To ﬁll this gap, our work targets to develop a method that can learn to reduce the number of input updates, gates, hidden states, cell states and outputs within LSTM units.
After learning those structures, compact LSTM units remain original structural schematic but have the sizes reduced.
Another line of related research is Structure Learning of FNNs or CNNs.
Zoph & Le (2017) uses reinforcement learning to search good neural architectures.
Philipp & Carbonell (2017) dynamically adds and eliminates neurons in FNNs by using group Lasso regularization.
Cortes et al.
(2017) gradually adds sub-networks to current networks to incrementally reduce the objective function.
All
[Equation]
[Figure]
these works focused on ﬁnding optimal structures in FNNs or CNNs for classiﬁcation accuracy.
In contrast, this work aims at learning compact structures in LSTMs for model compression.
L EARNING I NTRINSIC S PARSE S TRUCTURES 3. I NTRINSIC S PARSE S TRUCTURES
The computation within LSTMs is (Hochreiter & Schmidhuber (1997))
[Equation]
where (cid:12) is element-wise multiplication, σ ( · ) is sigmoid function, and tanh ( · ) is hyperbolic tangent function.
Vectors are row vectors. W s are weight matrices, which transform the concatenation (of hidden states h t − 1 and inputs x t ) to input updates u t and gates ( i t , f t and o t ).
Fig.
1 is the schematic of LSTMs in the layout of Olah (2015).
The transformations by W s and the corresponding nonlinear functions are illustrated in rectangle blocks.
Our goal is to reduce the size of this sophisticated structure within LSTMs, meanwhile maintaining the original schematic.
Because of element-wise operators (“ ⊕ ” and “ ⊗ ”), all vectors along the blue band in Fig.
1 must have the same dimension.
We call this constraint as “ dimension consistency ”. The vectors required to obey the dimension consistency include input updates, all gates, hidden states, cell states, and outputs.
Note that hidden states are usually outputs connected to classiﬁer layer or stacked LSTM layers.
Due to the dimension consistency, it is difﬁcult to reduce the structures in LSTMs: as can be seen in Fig. 1, vectors (along the blue band) interweave with each other so removing an individual component from one or a few vectors independently can result in the violation of dimension consistency.
Manually hacking the model by ﬁlling zeros can force dimension match. However, unnecessary computation involved with zeros hinders the full exploitation of computation efﬁciency within LSTMs.
To overcome the difﬁculty, we propose Intrinsic Sparse Structures (ISS) within LSTMs as shown by the blue band in Fig.
One component of ISS is highlighted as the white strip.
By decreasing the size of ISS ( i.e. , the width of the blue band), we are able to simultaneously reduce the dimensions of input updates, gates, hidden states, cell states and outputs.
To learn sparse ISS, we ﬁrst need to determine how to sparsify weights.
In LSTMs, input updates, gates, states and outputs are dynamically activated by stochastic inputs.
To permanently remove a component of ISS, the associated weights of those activations should be all zeros.
As such, the problem is converted to how to sparsify weight matrices when learning ISS.
There are totally eight
W xf
W xi
W xu
W xo
W hf
W hi
W hu W ho
[Figure]
h
Weights matrices in LSTM
Weights in next layer(s)
[Figure]
weight matrices in Eq. We organize them in the form of Fig. 2 as basic LSTM cells in TensorFlow.
We can remove one component of ISS by zeroing out all weights in the white rows and white columns in Fig.
Here, biases are omitted for simple discussion without loss of generality.
Suppose the k -th hidden state of h is removable, then the k -th row in the lower four weight matrices can be all zeros (as shown by the left white horizontal line in Fig. 2), because those weights are on connections receiving the k -th useless hidden state.
Likewise, all connections receiving the kth hidden state in next layer(s) can be removed as shown by the right white horizontal line.
Note that next layer(s) can be an output layer, LSTM layers, fully-connected layers, or a mix of them.
ISS overlay two or more layers, without explicit explanation, we refer to the ﬁrst LSTM layer as the ownership of ISS.
Furthermore, when the k -th hidden state turns useless, the k -th output gate and k -th cell state generating this hidden state are removable.
As the k -th output gate is generated by the k -th column in W xo and W ho , these weights can be zeroed out (as shown by the fourth vertical white line in Fig. 2).
Tracing back against the computation ﬂow in Fig.
1, we can reach similar conclusions for forget gates, input gates and input updates, as respectively shown by the ﬁrst, second and third vertical line in Fig.
For convenience, we call the weights in white rows and columns as an “ ISS weight group ”.
Although we focus on learning ISS in LSTMs, variants of ISS for vanilla RNNs, Gated Recurrent Unit (GRU) (Cho et al.
(2014)), and Recurrent Highway
Networks (RHNs) (Zilly et al.
(2017)) can also be realized based on the same philosophy.
For even a medium-scale LSTM, the number of weights in one ISS weight group can be very large. It seems to be very aggressive to simultaneously slaughter so many weights to maintain the original recognition performance.
For example, the weight number can be 3200 for an ISS weight group of the ﬁrst layer in a network with two stacked LSTM layers, whose input and hidden dimensions are both 200 .
However, the proposed ISS intrinsically exists within LSTMs and can be unveiled by even independently sparsifying each weight using (cid:96) 1 -norm regularization.
We take the large stacked LSTMs by Zaremba et al. (2014) for language modeling as the example of sparsifying weights by (cid:96) 1 -norm regularization.
The network has two stacked LSTM layers whose dimensions of inputs and states are both 1500 , and an output layer with a vocabulary of 10000 words.
The sizes of “ISS weight groups” of the two stacked LSTM layers are as large as 24000 and 28000 .
The perplexities of validation set and test set are respectively 82 .
57 and 78 .
We ﬁne-tune this baseline LSTMs with (cid:96) 1 -norm regularization.
The same training hyper-parameters with the baseline are adopted, except a bigger dropout keep ratio of 0 .
6 (original 0 . 35 ).
A weaker dropout is used because (cid:96) 1 -norm is also a regularization to avoid overﬁtting. A too strong dropout plus (cid:96) 1 -norm regularization can result in underﬁtting.
The weight decay of (cid:96) 1 -norm regularization is 0 . 0001 .
The sparsiﬁed network has validation perplexity and test perplexity of 82 .
40 and 78 .
60 , respectively, which is approximately the same with the baseline.
The sparsity of weights in the ﬁrst LSTM layer, the second LSTM layer and the last output layer is 91 .
32% and 90 .
22% , respectively.
Fig.
3 plots the learned sparse weight matrices.
The sparse matrices in the top row reveal some interesting patterns: there are lots of all-zero columns and rows, and their positions are highly correlated.
Those patterns are proﬁled in the bottom row.
Much to our surprise, sparsifying individual weight independently can converge to sparse LSTMs with many ISS removed— 504 and 220 ISS components in the ﬁrst and second LSTM layer are all-zeros.
It unveils that sparse ISS intrinsically exist in LSTMs and the learning process can easily converge to the status with a high ratio of ISS removed.
In Section 3.2, we propose a learning method to explicitly remove much more ISS than the implicit (cid:96) 1 -norm regularization.
3. L EARNING M ETHOD
Suppose w ( n ) k is a vector of all weights in the k -th component of ISS in the n -th LSTM layer ( 1 ≤ n ≤ N and 1 ≤ k ≤ K ( n ) ), where N is the number of LSTM layers and K ( n ) is the number of ISS components ( i.e. , hidden size) of the n -th LSTM layer.
The optimization goal is to remove as many weight groups w ( n ) k as possible without losing accuracy.
Methods to remove weight groups (such as ﬁlters, channels and layers) have been successfully studied in CNNs as summarized in Section 2. However, how these methods perform in RNNs is unknown.
Here, we extend the group Lasso based methods (Yuan & Lin (2006)) to RNNs for ISS sparsity learning.
More speciﬁc, the group Lasso regularization is added to the minimization function in order to encourage sparsity in ISS.
Formally, the ISS regularization is
[Equation]
where w is the vector of all weights and || · || 2 is (cid:96) 2 -norm ( i.e. , Euclidean length).
In Stochastic
Gradient Descent (SGD) training, the step to update each ISS weight group becomes
[Equation]
where E ( w ) is data loss, η is learning rate and λ > 0 is the coefﬁcient of group Lasso regularization to trade off recognition accuracy and ISS sparsity.
The regularization gradient, i.e. , the last term in
Eq.
(3), is a unit vector. It constantly squeezes the Euclidean length of each w ( n ) k to zero, such that, a high portion of ISS components can be enforced to fully-zeros after learning.
To avoid division by zero in the computation of regularization gradient, we can add a tiny number (cid:15) in || · || 2 , that is,
[Equation]
where w ( n ) kj is the j -th element of w ( n ) k . We set (cid:15) = 1 .
e − 8 in this work.
The learning method can effectively squeeze many groups near zeros, but it is very hard to exactly stabilize them as zeros because of the always-present ﬂuctuating weight updates.
Fortunately, the ﬂuctuation is within a tiny ball centered at zero.
To stabilize the sparsity during training, we zero out the weights whose absolute values are smaller than a pre-deﬁned threshold τ .
The process of thresholding is applied after each iteration.
E XPERIMENTS
In our experiments, we chose published models as baselines.
The application domains include language modeling of Penn TreeBank and machine Question Answering of SQuAD dataset.
For
[Table]
more comprehensive evaluation , we sparsify ISS in models with both a large hidden size ofand a small hidden size of 100 .
Threshold τ
= 1 .
e − 4 works well in general, however, we can fully exploit the beneﬁt by maximizing it.
For a speciﬁc application, we preset τ by cross validation.
We select the maximum τ which sparsiﬁes the baseline model without deteriorating its performance.
The validation of τ is performed only once, and no training efforts are needed.
τ is selected as 1 .
e − 4 and 4 .
e − 4 for Penn TreeBank and SQuAD, respectively.
We used HyperDrive by Rasley et al.
(2017) to explore the hyperparameter of λ .
More details are in our source code.
To measure inference speed, the experiments were run on a dual socket Intel Xeon CPU E5v3 @ 2 .
40 GHz processor with a total of 24 cores ( 12 per socket) and 128 GB of memory.
Intel
MKL library 2017 update 2 was used for matrix-multiplication operations.
OpenMP runtime was utilized for parallelism.
We used Intel C++ Compiler 17.0 to generate executables that were run on
Windows Server 2016.
Each of the experiments was run for 1000 iterations, and the execution time was averaged to ﬁnd the execution latency.
4. L ANGUAGE M ODELING
A RNN with two stacked LSTM layers for language modeling (Zaremba et al.
(2014)) is selected as the baseline.
It has hidden sizes of 1500 ( i.e. , 1500 components of ISS) in both LSTM units.
The output layer has a vocabulary of 10000 words.
The dimension of word embedding in the input layer is 1500 .
Word embedding layer is not sparsiﬁed, because the computation of selecting a vector from a matrix is very efﬁcient.
Exactly the same training scheme of the baseline is adopted to learn ISS sparsity, except a larger dropout keep ratio of 0 . 6 versus 0 .
35 in the baseline.
The reason is explained in Section 3.1.
All models are trained from scratch for 55 epochs.
The results are shown in Table 1.
The trade-off between perplexity and sparsity is controlled by λ .
In the second row, with tiny perplexity difference from baseline, our approach can reduce the number of ISS in the ﬁrst and second
LSTM unit from 1500 , down to 373 and 315 , respectively.
It reduces the model size from 66 .
0 M to 21 .
8 M and achieves 10 .
59 times speedup without losing perplexity.
Remarkably, the practical speedup ( 10 . 59 × ) even goes beyond theoretical Mult-add reduction ( 7 . 48 × ) as shown in Table— which comes from increased computational efﬁciency.
When we add structured sparsity, the underlying weight matrices become smaller and they ﬁt into the L3 cache with good locality, which improves the FLOPS (ﬂoating point operations per second).
This is a key advantage of our approach over non-structurally sparse RNNs generated by connection pruning (Narang et al.
(2017)), which can result in irregular memory access pattern and speedup inferior to theoretical one.
At last, when our method is learning a compact structure, it can perform as structure regularization to avoid overﬁtting, such that lower perplexity is achieved by even a smaller ( 25 . 2 M) and faster ( 7 . 10 × ) model, as shown by the third row.
Its learned weight matrices are visualized in Fig.
4, where 1119 and
ISS components shown by white strips are removed in the ﬁrst and second LSTM, respectively.
A straightforward way to reduce model complexity is to directly design a smaller RNN and train from scratch.
Compare with the direct design approach, our ISS method can automatically determine appropriate structures within LSTMs while maintaining an expected perplexity.
More importantly, compact models learned by ISS method have lower perplexity, comparing with direct design method.
To evaluate this, we directly design a RNN with exactly the same structure of the second RNN in
Table 1, but train it from scratch instead of learning ISS from a larger RNN.
The result is included in the last row of Table 1.
We tuned dropout keep ratio to get best perplexity for the directly-designed
RNN.
The ﬁnal test perplexity is 85 . 66 , which is 7 .
01 worse that our ISS method.
[Figure] [Figure]
4. M ACHINE R EADING C OMPREHENSION
In this section
[Figure]
we evaluate ISS method by state-of-the-art dataset (SQuAD) and state-of-the-art model (BiDAF) with small ISS sizes.
SQuAD (Rajpurkar et al.
(2016)) is a recently released reading comprehension dataset, crowdsourced from
100 , 000+ question-answer pairs on 500+ Wikipedia articles.
ExactMatch (EM) and F1 scores are two major metrics for the task.
Higher those scores are, better the model is.
The model is BiDAF (Seo et al.
[Figure]
which is a machine Question
Answering model.
It is a compact model with totally 2 .
69 M weights, and the ISS sizes are onlyin all LSTM units.
We adopt BiDAF to evaluate how ISS method works in small LSTM units.
The implementation of BiDAF is made available by its authors 3 .
BiDAF has character, word and contextual embedding layers to extract representations from input sentences, following which are bi-directional attention layer, modeling layer
[Figure]
and ﬁnal output layer.
LSTM units are used in contextual embedding layer, modeling layer, and output layer.
All LSTMs are bidirectional (Schuster & Paliwal (1997)).
In a bidirectional LSTM, there are one forward plus one backward LSTM branch.
Two branches share inputs and their outputs are concatenated for next stacked layers.
We found it is hard to remove ISS components in contextual embedding layer, because its representations are relatively dense when it is close to inputs and the original hidden size ( 100 ) is relatively small.
In our experiments, we exclude LSTMs in contextual embedding layer and sparsify all other LSTM layers.
Those LSTM layers are computation bottleneck of BiDAF.
We proﬁled the computation time on CPUs, and ﬁnd those LSTM layers (excluding contextual embedding layer) consume 76 .
47% total inference time.
There are totally three bi-directional LSTM layers we will sparsify, two of which belong to the modeling layer, and one belongs to the output layer.
More details of BiDAF are covered by Seo et al.
For brevity, we mark the forward (backward) path of the 1 st bi-directional LSTM in the modeling layer as ModFwd1 ( ModBwd1 ).
Similarly, ModFwd2 and ModBwd2 are for the 2 nd bi-directional LSTM.
Forward (backward) LSTM path in the output layer are marked as OutFwd and OutBwd .
As discussed in Section 3.1 and Fig.
2, there can be multiple parallel layers receiving the hidden states from the same LSTM layer, and all connections (weights) receiving those hidden states belong
[Table]
[Table]
to ISS of the shared LSTM.
For instance, ModFwd2 and ModBwd2 both receive hidden states of
ModFwd1 as inputs, therefore the k -th “ISS weight group” includes the k -th rows of weights in both ModFwd2 and ModBwd2 , plus the weights in the k -th ISS component within ModFwd1 .
For simplicity, we use “ISS of ModFwd1 ” to refer to the whole group of weights.
Structures of six ISS are included in Table 2.
We learn ISS sparsity in BiDAF by both ﬁne-tuning the baseline and training from scratch.
All training schemes are the same with the baseline, but using a higher dropout keep ratio.
After training, we zero out weights whose absolute values are smaller than 0 .
This does not impact EM and F1 scores, but increase sparsity.
Table 3 shows the EM, F1, the number of remaining ISS components, model size, and inference speed.
The ﬁrst row is the baseline without ISS learning, and ISS sizes are all 100 .
The validation
EM and F1 is 67 .
98 and 77 . 85 , respectively.
Other rows are obtained by ﬁne-tuning baseline using
ISS regularization.
By learning ISS, with small EM and F1 loss, we can reduce ISS in all LSTMs except ModFwd1 (in the second row).
For example, almost half of the ISS components are removed in OutBwd .
By increasing the strength of group Lasso regularization ( λ ), we can increase the ISS sparsity by losing some EM/F1 scores.
The trade-off is listed in Table 3.
With only 2 .
63 F1 score loss, the sizes of OutFwd and OutBwd can be reduced from 100 of the original to 15 and 12 , respectively.
At last, we ﬁnd it hard to reduce ISS sizes without losing any EM/F1 score.
This implies that BiDAF is compact enough and its scale is suitable for both computation and performance (EM and F1 scores).
However, our method can still signiﬁcantly compress this compact model under acceptable performance loss.
At last, instead of ﬁne-tuning baseline, we train BiDAF from scratch with ISS learning.
The results are shown in Table 4.
Our approach also works well when training from scratch.
Overall, training from scratch balances the sparsity across all layers better than ﬁne-tuning, which even results in better compression of model size and speedup of inference time.
C ONCLUSION
We proposed Intrinsic Sparse Structures (ISS) within LSTMs and its learning method to simultaneously reduce the sizes of input updates, gates, hidden states, cell states and outputs within the sophisticated LSTM structure.
By learning ISS, a structurally sparse LSTM can be obtained, which essentially is a regular LSTM but with hidden dimension reduced.
Therefore, no software or hardware speciﬁc customization is required to get storage saving and computation acceleration.
Although, ISS is proposed with LSTMs, it can be easily extended to vanilla RNNs, Gated Recurrent
Unit (GRU) (Cho et al.
(2014)), and Recurrent Highway Networks (RHNs) (Zilly et al.