G AUSSIAN - GATED LSTM: I MPROVED CONVERGENCE BY REDUCING STATE UPDATES
Anonymous authors
Paper under double-blind review
A BSTRACT
Recurrent neural networks can be difﬁcult to train on long sequence data due to the well-known vanishing gradient problem
Some architectures incorporate methods to reduce RNN state updates thereby allowing the network to preserve memory over long temporal intervals.
We propose a timing-gated LSTM RNN model, called the Gaussian-gated LSTM (g-LSTM) for reducing state updates.
The time gate controls when a neuron can be updated during training, enabling longer memory persistence and better error-gradient ﬂow.
This model captures long temporal dependencies better than an LSTM on very long sequence tasks and the time gate parameters can be learned even from a non-optimal initialization.
Because the time gate limits the updates of the neuron state the number of computes needed for the network update is also reduced.
By adding a computational budget term to the training loss , we obtain a network which further reduces the number of computes by at least 10 ×
Finally , we propose a temporal curriculum learning schedule for the g LSTM that helps speed up the convergence time of the equivalent
LSTM on long sequences.
I NTRODUCTION
Numerous methods and architectures have been proposed to mitigate the vanishing gradient problem in RNNs, with LSTMs (Hochreiter & Schmidhuber, 1997) as one of the ﬁrst prominent solutions doing so by including gating structures in the computation.
Although the LSTM has excelled at handling many tasks (Schmidhuber, 2015; Lipton, 2015), it still has difﬁculties in learning complex and long time dependencies (Neil et al., 2016;
Chang et al., 2017; Trinh et al., 2018).
In the last few years, various methods which reduce the state updates of an RNN (LSTM) have been explored to better learn long time dependencies from data.
Clockwork RNNs (Koutnik et al., 2014) group the hidden units of the RNN into “modules,” where each module is executed at pre-speciﬁed time steps thereby skipping time steps which helps learn longer time dependencies.
Recently, various other methods have been proposed which can be characterized by the use of additional “time gates,” k n , that control the information ﬂow from one time step to the next (Krueger et al., 2016;
Campos et al., 2017).
Phased LSTM (PLSTM) (Neil et al., 2016) learns a parameterized function, k n , from the the time input of the current state and was proven to be successful at learning over long sequences.
The PLSTM time gate, parameterized by period, phase, and ratio parameters for each hidden unit, is deﬁned through a modulo function with an ill-deﬁned gradient.
Furthermore, with periodic functions being hard to learn using gradient-based methods (Shamir, 2016) and with k n being periodic, the
PLSTM was unable to learn the time gate parameters and hence relied on careful initialization.
In order to offset these difﬁculties, this work proposes a new LSTM variant called the Gaussian-gated
LSTM (g-LSTM).
Similar to the PLSTM, it is an LSTM model with a parameterized k n but with only two parameters per hidden unit.
Unlike the PLSTM which uses a periodic formulation for k n , the g-LSTM uses a Gaussian function.
We show in this work that the g-LSTM can provide a number of possible advantages over the LSTM, in particular, on long sequence tasks that pose convergence problems during training:
• The g-LSTM network can process very long sequences by reducing the time over which the neurons can be updated.
It converges faster than the LSTM, especially on sequences that are over 500 steps.
• The “openness” of the neuron for an update can be adapted according to the task during training, even for extreme, non-optimal initializations of the time gate parameters.
• By introducing a computational budget term into the loss function during training, the “openness” of the neuron can be optimized for a reduced computational budget.
This reduction can be achieved with little or no degradation to the network performance and is useful for network pruning.
• A “temporal curriculum” training schedule can be set up for the g-LSTM so that it helps to speed up the convergence of a normal LSTM.
The paper is structured as follows:
In section 2, we discuss brieﬂy the related work.
Then, in section 3, we present the formulation of the g-LSTM, the datasets used in this work and details about the experimental hyperparameters.
In section 4, we present experiments demonstrating the usefulness of the g-LSTM with respect to the four claims listed above.
We provide gradient analysis in section 5 to further explain the faster convergence results of the g-LSTM.
Finally in section 6, we conclude with a brief discussion of the results.
R ELATED W ORK
There have been a multitude of proposed methods to improve the training of RNNs, especially for long sequences.
Apart from incorporating additional gating structures, for example the LSTM and the GRU (Cho et al., 2014), more recently various techniques were proposed to further increase the capabilities of recurrent networks to learn on sequences of length over 1000.
Proposed initialization techniques such as the orthogonal initialization of kernel matrices (Cooijmans et al., 2016) and diagonal recurrent kernel matrices ( e.g. Li et al.
(2018)) have demonstrated success.
Trinh et al.
(2018) propose using truncated backpropagation with an additional auxiliary loss to reconstruct previous events.
Methods that enable more efﬁcient learning on long temporal sequences use solutions that preserve memory over longer timescales. Such solutions were ﬁrst explored by Koutnik et al.
(2014) in the Clockwork RNN (CW-RNN).
This network skips state updates by allowing different neurons to be “activated” on different, modulated clock cycles. More recently proposed models for skipping updates include the Phased LSTM (PLSTM) (Neil et al., 2016) which uses a modulo-periodic timing gate to limit state updates; the Zoneout network (Krueger et al., 2016) which skips state updates in a random manner; and the Skip RNN (Campos et al., 2017) which learns a state skipping scheme from the data to shorten the effective sequence length for the task.
Additionally, the LSTM-Jump (Yu et al., 2017) uses a reinforcement learning algorithm to learn when to skip state updates, showing a method to more quickly process (long) sequential data with an RNN while maintaining an accuracy comparable to a baseline LSTM.
It has been suggested but not yet demonstrated in the literature that the parameters of the CW-RNN clock cycle and PLSTM timing gate could be learned in training.
Currently, the implementation of these networks requires a careful initialization of these parameters.
With the Gaussian-gated LSTM (g-LSTM) in this work we present a time gated RNN network that converges on long sequence tasks and also has the ability to learn its time gate parameters even when initialized in a nonoptimal way.
M ETHODS 3. G -LSTM
The g-LSTM is an LSTM model with an additional time gate (Fig. 1).
This time gate is used to regulate the information ﬂow in time.
Equations 1 - 3 describe the update equations for the hidden and cell states of the LSTM.
Equations 4 and 5 describe the gating mechanism of the time gate, k n .
x n tanh h nf o i tanh x x x + c nc n h n (a) LSTM x n tanh
h
nf o
i tanh x x x + c nc n h n
K
K t n (b) g-LSTM
[Equation] [Equation] [Equation] In a standard LSTM, the gating functions i n , f n , o n , represent the input , forget , and output gates respectively at sequence index n .
c n is the cell activation vector, and x n and h n represent the input feature vector and the hidden output vector, respectively.
The cell state c n is updated with a fraction of the previous cell state that is controlled by f n , and a new input state created from the element-wise (Hadamard) product, denoted by (cid:12) , of i n and the candidate cell activation as in Eq. 2.
In the g-LSTM, we further control the cell state and the output hidden state through the k n gate which is independent of the input data and hidden states, and is purely dependent on the time input corresponding to the sequence index n .
The use of the Hadamard product ensures that each hidden unit is independently controlled by the corresponding time gate unit, thus enabling the different units in the layer to process the input at different time scales.
The time gate k n is deﬁned based on a Gaussian function as: k n = e − ( t n − µ ) 2 /σ 2 where the mean parameter, µ , deﬁnes the time when the hidden unit is “open” and the standard deviation, σ , controls the openness of the time gate for each unit around its corresponding µ .
The time inputs t = { t 1 , t 2 , ..., t n , ..., t N } for the sequence x = { x 1 , x 2 , ..., x n , ..., x N } can correspond to the physical notion of time at the respective sequence input.
In the absence of a standard notion of time, we use the sequence indices as the time input, i.e. t = { 1 , 2 , ..., n, ..., N } .
In this work, we assume this notion of time by default.
The “openness” of k n for a neuron is deﬁned by the parameterization of its Gaussian function.
3. B ACK P ROPAGATION FOR G -LSTM
An important characteristic of the g-LSTM is reduced gradient ﬂow in back propagation training methods.
By having the gating structure as in Eqs.
4 and 5 there are fewer gradient product terms, which reduces the likelihood of vanishing or exploding gradients.
In a gradient descent learning scheme for a given loss function, L , when training the recurrent parameters, W h (from Eqs.
1 - 3), the gradient as in Eq.
6 is used.
[Equation]
[Equation] By the chain rule ∂h N ∂W h expands for all time steps of the sequence, n ∈ { 1 , ..., N } .
Because each output state is gated by the time gate, k n , the gradient terms in the expansion of ∂h N ∂W h are scaled by k n .
When the time gate is open less often, i.e. with a small σ value, then there are fewer inﬂuential gradient terms.
More details are given in Appendix A. [Table]
The performance metric is the ﬁnal mean squared error (MSE) loss for the adding task, and the label error rate for both sMNIST and sCIFAR-10.
3. D ATASETS
The experiments described in the paper are carried out on the adding task and two standard long sequence datasets: the sequential MNIST and the sequential CIFAR-10 datasets.
Adding task:
In order to test the long sequence learning capability of the g-LSTM, we use the adding task (Hochreiter & Schmidhuber, 1997).
In this task, the network is presented with two sequences of length N , x = { x 1 , ..., x N } , x n ∼ U (0 , 1) ) and m = { m 1 , ..., m N } , m n ∈ { 0 , 1 } , (cid:80)
Nt =1 m n = 2 .
The sequence m has exactly two values of 1 and the remaining values of the sequence are 0 .
The indices of the “1” values are chosen at random.
For each pair in the sequence ( x , m ) , the associated label value, y , is the sum of the two values in x corresponding to the “1” values of m .
The objective of this task is to minimize the mean squared error between the predicted sum from the network, ˆ y , and the labeled sum, y .
A new training set of 5000 sequence samples is presented in every epoch during training in order to avoid overﬁtting.
The test set consists of a separate 5000 samples.
For N > 1000 , it is known that LSTMs have difﬁculty learning the task and hence we focus on values of N > 1000 in this work.
sMNIST: The sequential MNIST dataset is widely used to analyze the performance of a recurrent model.
This dataset consists of 60,000 training samples and 10,000 test samples, each a single vector sequence of length 784 corresponding to the 28 × 28 pixel images in the MNIST dataset (LeCun et al., 1998).
We also use permuted MNIST (pMNIST), a permuted variant of the sMNIST dataset where the sequences are processed with a ﬁxed random permutation, making the task harder.
sCIFAR-10:
The sequential CIFAR-10 dataset is another long sequence dataset based on CIFAR10 (Krizhevsky et al., 2014) with 10 classes.
The 32 × 32 RGB pixel images are reshaped into sequences of length 1024 with 3 dimensional features corresponding the RGB channels at every time step.
Like in the sMNIST dataset, the dataset consists of 60,000 training samples and 10, test samples.
3. E XPERIMENTAL H YPERPARAMETERS
For the adding task, a mean squared error (MSE) loss was used with the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 10 − 3 .
The g-LSTM time gate parameters were trained using a learning rate of 10 0 .
For both sMNIST and sCIFAR-10 datasets, the cross-entropy loss function was used along with the RMSProp optimizer (Tieleman & Hinton, 2012) with a learning rate of 10 − 3 .
Decay parameters of 0 .
5 and 0 .
9 were used for sMNIST and sCIFAR-10, respectively.
The bias of the forget gate is initialized following (Gers et al., 2000; Jozefowicz et al., 2015).
R ESULTS
Section 4.1 presents results that demonstrate the faster convergence properties of the g-LSTM on long sequence tasks.
Section 4.2 shows the trainability of the time gate parameters of the g-LSTM even when the parameters are initialized in a non-optimal way.
Section 4.3 presents a modiﬁed loss function used during training to reduce the number of computes for the network update and Section 4.4 presents a new “temporal curriculum” learning schedule that allows g-LSTMs to help LSTMs converge faster. [Figure]
s t l a b e l e rr o r r a t e
[Figure]
4. F AST CONVERGENCE PROPERTIES OF G -LSTM
First, we look at the convergence properties of the g-LSTM on the long-sequence adding task, the sMNIST task and the sCIFAR-10 task.
Table 1, above [Figure] details the network architectures used in the experiments in this section.
Similar to the architecture from Trinh et al.
(2018), the recurrent layer of the sCIFAR-10 network is followed by two 256 unit fully-connected (FC) layers [Figure] where DropConnect (Wan et al.
( p = 0 . 5 ) is applied to the second FC layer.
The kernel matrices in the LSTM networks were initialized in an orthogonal manner as described in (Cooijmans et al., 2016)
[Figure]
The test performances of these networks during the course of the training on different datasets are shown in Fig. 2, while the corresponding ﬁnal performance metrics at the end of training are shown in Table 1.
From Fig. 2, it is evident that the test loss of the g-LSTM decreases faster in training than the LSTM across all datasets.
Further experiments show that this trend is maintained with different training optimizers, LSTM initializations, and network sizes as shown in Appendix D.
Table 2 compares the performance of various networks including the g-LSTM and the baseline
LSTM on sMNIST and sCIFAR-10 (from Table 1).
The results show that the g-LSTM consistently performs better than the LSTM and has a similar performance to other state-of-the-art networks.
Different network sizes were also investigated for the sMNIST task, see Appendix D.
[Table]
4. T RAINABILITY OF THE TIME GATE PARAMETERS OF G -LSTM
To demonstrate that the g-LSTM can be trained even with non-optimal initializations, we look at the performance of the g-LSTM on the adding task with different time gate parameter initializations.
We concern ourselves with sequences of length 1000 that are difﬁcult for the LSTM.
The time gate parameters are initialized in a way to temporally constrain the network so that it can only process for a short period of time.
For example, a network with time gate parameters initialized with µ =and σ = 40 as in Figure 3 (a) can only process a short period of time around the middle of sequence.
It follows that the network would be unable to learn with these parameters because in the adding task the input data is distributed equally across the sequence length ( T = 1000 ).
Therefore, in order to learn the task from this initialization, the time gate parameters must learn a distribution such that the gates over all hidden units are open across the entirety of the sequence.
We observe that the time gate parameters do learn, as shown in Figure 3 (b), thereby enabling the network to solve the task.
Independent of various time gate initializations, the network reaches an
MSE of around 3 .
9 × 10 − 5 at the end of 700 epochs; details of which could be found in Appendix B.
The ability of the network to learn the time gate parameters necessary to cover the entire sequence is especially signiﬁcant because it shows that even with this narrow time window initialization that requires learning of the time gates, the g-LSTM learns the task, whereas the PLSTM does not learn the task as well.
An example of this is shown in Appendix C.
[Figure]
4. R EDUCTION IN COMPUTATION
Although the formulation of the g-LSTM appears to require more computes, it offers substantial speedup as a large proportion of the neurons can be skipped in a timestep at runtime.
We can set a threshold on the time gate so that we skip all corresponding computations for time steps where k n is below this threshold.
To further reduce the number of operations, it is preferred that the σ of the k n for different neurons should be small but the network performance should not be signiﬁcantly degraded.
To achieve this goal, we included a “computational budget” loss term during the optimization of the gate parameters, µ and σ .
The loss equation for updating the k n parameters is given by:
[Equation]
Similar to the Skip RNN network (Campos et al., 2017), a budget loss term which minimizes the average openness of the time gate over time is applied:
[Equation]
for every neuron j of the g-LSTM.
The study was carried out on sMNIST using a network withunits, σ initialized to 50, µ initialized uniformly at random between 1 to 784, and a λ value of 1.
The network’s performance of 2 . 2% LER was comparable to the network’s performance of 1 . 3% when no additional budget constraint was imposed.
The ﬁnal σ range for the budgeted g-LSTM is much smaller compared to that of the g-LSTM as shown in Fig. 4.
There is only a slight increase in LER for the budgeted g-LSTM versus the g-LSTM (see Table 2), even though there is a signiﬁcant decrease in the average time gate openness across all hidden units
[Figure]
[Figure] [Figure]
In order to reduce the number of computes
[Figure]
we set a threshold, v T for k n so that the update steps are carried out only if k n > v T , if k n < v T the previous neuron state can be copied over to the current state.
By increasing v T , the number of computes decreases as shown in Fig.
In the case of v T = 0 . 01 , only 8 .
2% of the time gates are open on average across all hidden units and all time steps.
Furthermore, the LER increased only slightly to 2 .
3% from 2 .
We give a quantitative estimate for the number of operations (Ops) corresponding to the number of update equations for a g-LSTM.
In the estimate, we count a multiply and an add operation as 1 Op and non-linear functions as 5 Ops.
For an LSTM, the number of operations is given by
[Equation]
where T is the number of time steps, H is the number of hidden units, and D is the dimension of the input data.
For a g-LSTM, the number of operations is given by
[Equation]
where N gate = 13 T H is the total number of operations for computing the time gate.
The total number of operations for the g-LSTM network on the sMNIST dataset is around 80 MOps for N = 110 and T = 784 , after thresholding the budgeted g-LSTM this number is reduced to 7. MOps. Additional λ hyperparameters were also investigated for the sMNIST task, see Appendix D.
4. T EMPORAL CURRICULUM TRAINING SCHEDULE FOR LSTM S
We demonstrate that it is possible to train an LSTM network to converge faster on a difﬁcult task by using a “temporal curriculum” training schedule for the equivalent g-LSTM network.
According to this schedule, the initial σ values of the g-LSTM network are increased continuously throughout the training period ending up with high values by the end of training.
With such high values, the time gates are essentially open, resulting in an LSTM network.
At every training epoch, the lowest ρ % of the σ values, ˆ σ in the layer are updated as: ˆ σ −→ (1 + α ) · ˆ σ .
We analyze the impact of this training schedule for training an LSTM network on sCIFAR-10.
For the equivalent g-LSTM network with 110 units, µ is initialized uniformly at random between 1 and 1024 and σ is initialized to 50.
An α value of 1/6 and ρ value of 15% are chosen.
To ensure that the time gate is fully open by the end of training, σ is set to 5000 across all units during the last 10 epochs of training.
The learning rate of the time gate parameters is set to 0 , i.e. µ and σ are no longer updated.
Figure 6 shows that the temporal curriculum training schedule allows for faster convergence of an LSTM network.
The ﬁnal weights of the trained g-LSTM network can then be copied over to a LSTM network for inference.
G RADIENT FLOW
We present results regarding backpropagation ﬂow through the LSTM and g-LSTM networks.
Following the hypothesis presented in Section 3.2 on the reduced likelihood of vanishing or exploding gradients in the g-LSTM, we investigate the average gradient norms across time steps, similar to the work in (Krueger et al., 2016). We compute the gradient norms of the loss with respect to the hidden activations, the exact deﬁnition is given in Appendix E. Comparing the error propagation of the g-LSTM and LSTM networks on the SMNIST task (as in Sec. 4.1), Figure 7 shows the gradient norms at each time step after training for two different epochs. [Figure]
Interpreting the gradient ﬂow from higher to lower time steps (right to left), the gradients of the g-LSTM shown in Fig. 7 show higher gradient values in earlier time steps than the LSTM. It is possible that one of the reasons the g-LSTM converges more quickly (as in Fig. 2 (c)) is that this back-propagated gradient information is more consistent across time steps and does not vanish at early time steps.
This work proposes a novel RNN variant with a time gate which is parameterized by the input in time. The convergence speeds of the g-LSTM and LSTM are similar for short sequence tasks but the g-LSTM shows faster convergence and produces higher accuracies than LSTM networks on long sequence tasks, as demonstrated for adding task sequences which are longer than 1000 timesteps; and on the sMNIST and sCIFAR-10 datasets.
We also demonstrate that the time gate parameters of the g-LSTM (unlike those of the PLSTM) are learnable even when the time gates are initialized in an extreme non-optimal manner for the adding task.
The time gate of the g-LSTM can reduce the number of computes that is needed for the updates of the LSTM equations and with an additional loss term to reduce the compute budget, the σ values of the time gate are reduced leading to a 10 × decrease in the number of actual computes and with little loss in network accuracy, for the sMNIST dataset.
The observation that the budgeted g-LSTM has neurons which are closed by the timing gate suggests that this method can be used to prune a network.
We also show that our proposed temporal curriculum training schedule for the g-LSTM can help a corresponding LSTM network to converge during training on long sequence tasks.
For future work, it will be of interest to investigate whether these properties carry over to larger or domain-speciﬁc datasets.
A B ACK PROPAGATION IN G AUSSIAN - GATED RNN
For ease of illustration we analyze the gradient of a plain RNN with a Gaussian time gate (Eqs.and 8).
[Equation]
where ∂ ˜ h
[Equation]
From Eq.
9 we can deduce some information about the advantages of the Gaussian time gate in gradient ﬂow for two simple cases of the function k n .
In Case 1 we choose a timing gate openness which corresponds to a very small σ for the Gaussian gate, i.e. the gate is only open for 1 time step.
[Equation]
In Case 2 we choose a timing gate openness which corresponds to a slightly larger σ for the Gaussian gate, i.e. it is open for 5 time steps.
[Equation]
[Table]
[Table]
B C OMPARING VARIOUS G -LSTM INITIALIZATIONS
[Table]
[Figure]
n
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
h i dd e n
[Figure]
i
[Figure]
C C OMPARING TIME GATE PARAMETER TRAINABILITY IN G -LSTM AND PLSTM Network
[Figure] [Table] [Figure] [Table]
[Figure]
[Figure]
D H YPERPARAMETER I NVESTIGATION
We look at the network performance for different hyperparameter values, focusing on the sMNIST task.
Network Initialization and Optimizer
In Section 4.1 in Fig.
2 (c) we show that the g-LSTM network converges faster than the LSTM for the SMNIST task using the RMSProp optimizer and with an orthogonal initialization of LSTM kernels of both networks, as in (Cooijmans et al., 2016).
In addition to using this initializer and optimizer we include results using the ADAM initializer (learning rate of 10 − 3 ) and a random weight initialization, “Xavier” as in (Glorot & Bengio, 2010).
Across all of these different training techniques we consistently observe that the g-LSTM converges more quickly than the LSTM.
[Figure]
Network size Aside from the network size of 110 hidden units, we investigated the training convergence for two additional network sizes: 25 and 220 hidden units.
Note that the LSTM for network size 25 is trained for 100 additional epochs until convergence was observed.
Across all different network sizes the g-LSTM converges much faster than the LSTM network.
With fewer hidden units, as seen in Fig.
12 (a), an even more dramatic speed-up in convergence is seen for the g-LSTM compared with the LSTM.
The ﬁnal LERs (g-LSTM, LSTM) for each network size are: 25 units ( 2 . 79% , 3 .
58% ), 110 units ( 1 . 35% , 1 . 81% ) , 220 units ( 1 . 10% , 1 . 34% ).
[Figure]
t l a
[Figure]
l
[Figure]
rr o
[Figure]
a t
[Figure]
Budgeted g-LSTM We provide additional results of the budgeted network (subsection 4.3) for 2 additional λ values, λ = 0 .
1 and λ = 10 , comparing with the original result, for λ = 1 .
The ﬁnal LERs: 2 . 4% ( λ = 0 . 1 ), 2 . 2% ( λ = 1 ), 2 . 8% ( λ = 10 ).
The number of computes used by the network trained with λ = 10 is signiﬁcantly lower than the network that was trained for both λ = 0 .
1 and 1 .
[Figure] [Figure] i [Figure]
n
[Figure]
i t s
[Figure]
E A VERAGE G RADIENT N ORM D EFINITION
The average gradient norm in Section 5 is deﬁned as:
[Equation]
where N is the number of time steps of the sequence (for SMNIST, N = 784 ).
Γ = 1 K E  (cid:88) ( k ) (cid:13)(cid:13)(cid:13)(cid:13) ∂L
∂h ( k ) n (cid:13)(cid:13)(cid:13)(cid:13) ≈
LK (cid:88) ( l,k ) (cid:13)(cid:13)(cid:13)(cid:13) ∂L
∂h ( k ) n (cid:13)(cid:13)(cid:13)(cid:13) summing over all L samples of the training set and all K hidden units.