C AN RECURRENT NEURAL NETWORKS WARP TIME ?
Anonymous authors
Paper under double-blind review
A BSTRACT
Successful recurrent models such as long short-term memories (LSTMs) and gated recurrent units (GRUs) use ad hoc gating mechanisms.
Empirically these models have been found to improve the learning of medium to long term temporal dependencies and to help with vanishing gradient issues.
We prove that learnable gates in a recurrent model formally provide quasiinvariance to general time transformations in the input data.
We recover part of the LSTM architecture from a simple axiomatic approach.
This result leads to a new way of initializing gate biases in LSTMs and GRUs.
Experimentally, this new chrono initialization is shown to greatly improve learning of long term dependencies, with minimal implementation effort.
Recurrent neural networks (e.g. (Jaeger, 2002)) are a standard machine learning tool to model and represent temporal data; mathematically they amount to learning the parameters of a parameterized dynamical system so that its behavior optimizes some criterion, such as the prediction of the next data in a sequence.
Handling long term dependencies in temporal data has been a classical issue in the learning of recurrent networks.
Indeed, stability of a dynamical system comes at the price of exponential decay of the gradient signals used for learning, a dilemma known as the vanishing gradient problem (Pascanu et al., 2012; Hochreiter, 1991; Bengio et al., 1994).
This has led to the introduction of recurrent models speciﬁcally engineered to help with such phenomena.
Use of feedback connections (Hochreiter & Schmidhuber, 1997) and control of feedback weights through gating mechanisms (Gers et al., 1999) partly alleviate the vanishing gradient problem.
The resulting architectures, namely long short-term memories (LSTMs (Hochreiter & Schmidhuber, 1997; Gers et al., 1999)) and gated recurrent units (GRUs (Chung et al., 2014)) have become a standard for treating sequential data.
Using orthogonal weight matrices is another proposed solution to the vanishing gradient problem, thoroughly studied in (Saxe et al., 2013; Le et al., 2015; Arjovsky et al., 2016; Wisdom et al., 2016;
Henaff et al., 2016).
This comes with either computational overhead, or limitation in representational power.
Furthermore, restricting the weight matrices to the set of orthogonal matrices makes forgetting of useless information difﬁcult.
The contribution of this paper is threefold:
∙ We show that postulating invariance to time transformations in the data (taking invariance to time warping as an axiom) necessarily leads to a gate-like mechanism in recurrent models (Section 1).
This provides a clean derivation of part of the popular LSTM and GRU architectures from ﬁrst principles.
In this framework, gate values appear as time contraction or time dilation coefﬁcients, similar in spirit to the notion of time constant introduced in (Mozer, 1992).
∙ From these insights, we provide precise prescriptions on how to initialize gate biases (Section 2) depending on the range of time dependencies to be captured.
It has previously been advocated that setting the bias of the forget gate of LSTMs to 1 or 2 provides overall good performance (Gers & Schmidhuber, 2000; Jozefowicz et al., 2015).
The viewpoint here explains why this is reasonable in most cases, when facing medium term dependencies, but fails when facing long to very long term dependencies.
∙ We test the empirical beneﬁts of the new initialization on both synthetic and real world data (Section 3).
We observe substantial improvement with long-term dependencies, and slight gains or no change when short-term dependencies dominate.
F ROM TIME WARPING INVARIANCE TO GATING
When tackling sequential learning problems, being resilient to a change in time scale is crucial.
Lack of resilience to time rescaling implies that we can make a problem arbitrarily difﬁcult simply by changing the unit of measurement of time.
Ordinary recurrent neural networks are highly nonresilient to time rescaling: a task can be rendered impossible for an ordinary recurrent neural network to learn, simply by inserting a ﬁxed, small number of zeros or whitespaces between all elements of the input sequence.
An explanation is that, with a given number of recurrent units, the class of functions representable by an ordinary recurrent network is not invariant to time rescaling.
Ideally, one would like a recurrent model to be able to learn from time-warped input data 𝑥 ( 𝑐 ( 𝑡 )) as easily as it learns from data 𝑥 ( 𝑡 ) , at least if the time warping 𝑐 ( 𝑡 ) is not overly complex.
The change of time 𝑐 may represent not only time rescalings, but, for instance, accelerations or decelerations of the phenomena in the input data.
We call a class of models invariant to time warping , if for any model in the class with input data 𝑥 ( 𝑡 ) , and for any time warping 𝑐 ( 𝑡 ) , there is another (or the same) model in the class that behaves on data 𝑥 ( 𝑐 ( 𝑡 )) in the same way the original model behaves on 𝑥 ( 𝑡 ) .
(In practice, this will only be possible if the warping 𝑐 is not too complex.)
We will show that this is deeply linked to having gating mechanisms in the model.
Invariance to time rescaling
Let us ﬁrst discuss the simpler case of a linear time rescaling.
Formally, this is a linear transformation of time, that is
[Equation]
with 𝛼 > 0 .
For instance, receiving a new input character every 10 time steps only, would correspond to 𝛼 = 0 . 1 .
Studying time transformations is easier in the continuous-time setting.
The discrete time equation of a basic recurrent network with hidden state ℎ 𝑡 ,
[Equation]
can be seen as a time-discretized version of the continuous-time equation
[Equation]
namely, (2) is the Taylor expansion ℎ ( 𝑡 + 𝛿𝑡 )
≈ ℎ ( 𝑡 ) + 𝛿𝑡 d ℎ ( 𝑡 ) d 𝑡 with discretization step 𝛿𝑡 = 1 .
Now imagine that we want to describe time-rescaled data 𝑥 ( 𝛼𝑡 ) with a model from the same class.
Substituting 𝑡 ← 𝑐 ( 𝑡 ) =
𝛼𝑡 , 𝑥 ( 𝑡 ) ← 𝑥 ( 𝛼𝑡 ) and ℎ ( 𝑡 ) ← ℎ ( 𝛼𝑡 ) and rewriting (3) in terms of the new variables, the time-rescaled model satisﬁes
[Equation]
However, when translated back to a discrete-time model, this no longer describes an ordinary RNN but a leaky RNN (Jaeger, 2002, §8.1).
Indeed, taking the Taylor expansion of ℎ ( 𝑡 + 𝛿𝑡 ) with 𝛿𝑡 =in (4) yields the recurrent model
[Equation]
Thus, a straightforward way to ensure that a class of (continuous-time) models is able to represent input data 𝑥 ( 𝛼𝑡 ) in the same way that it can represent input data 𝑥 ( 𝑡 ) , is to take a leaky model in which 𝛼 > 0 is a learnable parameter, corresponding to the coefﬁcient of the time rescaling.
Namely, the class of ordinary recurrent networks is not invariant to time rescaling, while the class of leaky RNNs (5) is.
Learning 𝛼 amounts to learning the global characteristic timescale of the problem at hand.
More precisely, 1 /𝛼 ought to be interpreted as the characteristic forgetting time of the neural network.
Invariance to time warpings
In all generality, we would like recurrent networks to be resilient not only to time rescaling, but to all sorts of time transformations of the inputs, such as variable accelerations or decelerations.
An eligible time transformation, or time warping , is any increasing differentiable function 𝑐 from R + to R + .
This amounts to facing input data 𝑥 ( 𝑐 ( 𝑡 )) instead of 𝑥 ( 𝑡 ) .
Applying a time warping 𝑡 ← 𝑐 ( 𝑡 ) to the model and data in equation (3) and reasoning as above yields
[Equation]
Ideally, one would like a model to be able to learn from input data 𝑥 ( 𝑐 ( 𝑡 )) as easily as it learns from data 𝑥 ( 𝑡 ) , at least if the time warping 𝑐 ( 𝑡 ) is not overly complex.
To be invariant to time warpings, a class of (continuous-time) models has to be able to represent Equation (6) for any time warping 𝑐 ( 𝑡 ) .
Moreover, the time warping is unknown a priori, so would have to be learned.
Ordinary recurrent networks do not constitute a model class that is invariant to time rescalings, as seen above.
A fortiori, this model class is not invariant to time warpings either.
For time warping invariance, one has to introduce a learnable function 𝑔 that will represent the derivative 4 of the time warping, d 𝑐 ( 𝑡 ) d 𝑡 in (6).
For instance 𝑔 may be a recurrent neural network taking the 𝑥 ’s as input.
Thus we get a class of recurrent networks deﬁned by the equation
[Equation]
where 𝑔 belongs to a large class (universal approximator) of functions of the inputs.
The class of recurrent models (7) is quasi -invariant to time warpings.
The quality of the invariance will depend on the learning power of the learnable function 𝑔 : a function 𝑔 that can represent any function of the data would deﬁne a class of recurrent models that is perfectly invariant to time warpings; however, a speciﬁc model for 𝑔 (e.g., neural networks of a given size) can only represent a speciﬁc, albeit large, class of time warpings, and so will only provide quasi-invariance.
Heuristically, 𝑔 ( 𝑡 ) acts as a time-dependent version of the ﬁxed 𝛼 in (4).
Just like 1 /𝛼 above, 1 /𝑔 ( 𝑡 0 ) represents the local forgetting time of the network at time 𝑡 0 : the network will effectively retain information about the inputs at 𝑡 0 for a duration of the order of magnitude of 1 /𝑔 ( 𝑡 0 ) (assuming 𝑔 ( 𝑡 ) does not change too much around 𝑡 0 ).
Let us translate back this equation to the more computationally realistic case of discrete time, using a Taylor expansion with step size 𝛿𝑡 = 1 , so that d ℎ ( 𝑡 ) d 𝑡 [Equation] · · · becomes ℎ 𝑡 +
= ℎ 𝑡 + · · · .
Then the model (7) becomes
[Equation]
is ℎ ( 𝑡 ) =
𝑒 − 𝛼 ( 𝑡 − 𝑡 0 )
ℎ ( 𝑡 0 ) , and so the network retains information from the past 𝑡 < 𝑡 0 during a time proportional to 1 /𝛼 .
It is, of course, algebraically equivalent to introduce a function 𝑔
that learns the derivative of 𝑐 , or to introduce a function 𝐺 that learns 𝑐 .
However, only the derivative of 𝑐 appears in (6).
Therefore the choice to work with d 𝑐 ( 𝑡 ) d 𝑡 is more convenient.
Moreover, it may also make learning easier, because the simplest case of a time warping is a time rescaling, for which d 𝑐 ( 𝑡 ) d 𝑡 = 𝛼 is a constant.
Time warpings 𝑐 are increasing by deﬁnition: this translates as 𝑔 > 0 .
The time warping has to be learned only based on the data seen so far.
where 𝑔 𝑡 itself is a function of the inputs.
This model is the simplest extension of the RNN model that provides invariance to time warpings.
It is a basic gated recurrent network, with input gating 𝑔 𝑡 and forget gating (1 − 𝑔 𝑡 ) .
Here 𝑔 𝑡 has to be able to learn an arbitrary function of the past inputs 𝑥 ; for instance, take for 𝑔 𝑡 the output of a recurrent network with hidden state ℎ 𝑔 :
[Equation]
with sigmoid activation function 𝜎 (more on the choice of sigmoid below).
Current architectures just reuse for ℎ 𝑔
the states ℎ of the main network (or, equivalently, relabel ℎ ← ( ℎ, ℎ 𝑔 ) to be the union of both recurrent networks and do not make the distinction).
The model (8) provides invariance to global time warpings, making all units face the same dilation/contraction of time.
One might, instead, endow every unit 𝑖 with its own local contraction/dilation function 𝑔 𝑖 .
This offers more ﬂexibility (gates have been introduced for several reasons beyond time warpings (Hochreiter, 1991)), especially if several unknown timescales coexist in the signal: for instance, in a multilayer model, each layer may have its own characteristic timescales corresponding to different levels of abstraction from the signal.
This yields a model
[Equation]
with ℎ 𝑖 and ( 𝑊 𝑖𝑥 , 𝑊 𝑖ℎ , 𝑏 𝑖 ) being respectively the activation and the incoming parameters of unit 𝑖 , and with each 𝑔 𝑖 a function of both inputs and units.
Equation 10 deﬁnes a simple form of gated recurrent network, that closely resembles the evolution equation of cell units in LSTMs, and of hidden units in GRUs.
In (10), the forget gate is tied to the input gate ( 𝑔 𝑖𝑡 and 1 − 𝑔 𝑖𝑡 ).
Such a setting has been successfully used before (e.g. (Lample et al., 2016)) and saves some parameters, but we are not aware of systematic comparisons.
Below, we initialize LSTMs this way but do not enforce the constraint throughout training.
Continuous time versus discrete time
Of course, the analogy between continuous and discrete time breaks down if the Taylor expansion is not valid.
The Taylor expansion is valid when the derivative of the time warping is not too large, say, when 𝛼 (cid:46) 1 or 𝑔 𝑡 (cid:46) 1 (then (8) and (7) are close).
Intuitively, for continuous-time data, the physical time increment corresponding to each time step 𝑡 → 𝑡 + 1 of the discrete-time recurrent model should be smaller than the speed at which the data changes, otherwise the situation is hopeless.
So discrete-time gated models are invariant to time warpings that stretch time (such as interspersing the data with blanks or having long-term dependencies), but obviously not to those that make things happen too fast for the model.
Besides, since time warpings are monotonous, we have d 𝑐 ( 𝑡 ) d 𝑡 > 0 , i.e., 𝑔 𝑡 > 0 .
The two constraints 𝑔 𝑡 > 0 and 𝑔 𝑡 < 1 square nicely with the use of a sigmoid for the gate function 𝑔 .
T IME WARPINGS AND GATE INITIALIZATION
If we happen to know that the sequential data we are facing have temporal dependencies in an approximate range [ 𝑇 min , 𝑇 max ] , it seems reasonable to use a model with memory (forgetting time) lying approximately in the same temporal range.
As mentioned in Section 1, this amounts to having values of 𝑔 in the range [︁ 1 𝑇 max , 1 𝑇 min ]︁ .
The biases 𝑏 𝑔 of the gates 𝑔 greatly impact the order of magnitude of the values of 𝑔 ( 𝑡 ) over time.
If the values of both inputs and hidden layers are centered over time, 𝑔 ( 𝑡 ) will typically take values centered around 𝜎 ( 𝑏 𝑔 ) .
Values of 𝜎 ( 𝑏 𝑔 ) in the desired range [︁ 1 𝑇 max , 1 𝑇 min ]︁ are obtained by choosing the biases 𝑏 𝑔 between − log( 𝑇 max − 1) and − log( 𝑇 min − 1) .
This is a loose prescription: we only want to control the order of magnitude of the memory range of the neural networks.
Furthermore, we don’t want to bound 𝑔 ( 𝑡 ) too tightly to some value forever: if rare events occur, abruplty changing the time scale can be useful.
Therefore we suggest to use these values as initial values only.
This suggests a practical initialization for the bias of the gates of recurrent networks such as (10): when characteristic timescales of the sequential data at hand are expected to lie between 𝑇 min and 𝑇 max , initialize the biases of 𝑔 as − log( 𝒰 ([ 𝑇 min , 𝑇 max ]) − 1) where 𝒰 is the uniform distribution 7 .
For LSTMs, using a variant of (Graves et al., 2013):
[Equation]
the correspondence between between the gates in (10) and those in (13) is as follows: 1 − 𝑔 𝑡 corresponds to 𝑓 𝑡 , and 𝑔 𝑡 to 𝑖 𝑡 .
To obtain a time range around 𝑇 for unit 𝑖 , we must both ensure that 𝑓 𝑖𝑡 lies around 1 − 1 /𝑇 , and that 𝑖 𝑡 lies around 1 /𝑇 .
When facing time dependencies with largest time range 𝑇 max , this suggests to initialize LSTM gate biases to
[Equation]
with 𝒰 the uniform distribution and 𝑇 max the expected range of long-term dependencies to be captured.
Hereafter, we refer to this as the chrono initialization .
E XPERIMENTS
First, we test the theoretical arguments by explicitly introducing random time warpings in some data, and comparing the robustness of gated and ungated architectures.
Next, the chrono LSTM initialization is tested against the standard initialization on a variety of both synthetic and real world problems.
It heavily outperforms standard LSTM initialization on all synthetic tasks, and outperforms or competes with it on real world problems.
The synthetic tasks are taken from previous test suites for RNNs, speciﬁcally designed to test the efﬁciency of learning when faced with long term dependencies (Hochreiter & Schmidhuber, 1997;
Le et al., 2015; Graves et al., 2014; Martens & Sutskever, 2011; Arjovsky et al., 2016).
In addition (Appendix A), we test the chrono initialization on next character prediction on the Text8 (Mahoney, 2011) dataset, and on next word prediction on the Penn Treebank dataset (Mikolov et al., 2012). Single layer LSTMs with various layer sizes are used for all experiments, except for the word level prediction, where we use the best model from (Zilly et al., 2016), a 10 layer deep recurrent highway network (RHN). Pure warpings and paddings.
To test the theoretical relationship between gating and robustness to time warpings, various recurrent architectures are compared on a task where the only challenge comes from warping. The unwarped task is simple: remember the previous character of a random sequence of characters. Without time warping or padding, this is an extremely easy task and all recurrent architectures are successful. The only difﬁculty will come from warping; this way, we explicitly test the robustness of various architectures to time warping and nothing else.
[Table]
[Table]
[Figure]
1 to maximum_warping . (For uniform padding/warpings, the leaky RNN and gated RNN curves overlap, with loss 0 .)
Lower is better.
Unwarped task example:
Input:
All human beings are born free and equal
Output:
All human beings are born free and equa
Uniform warping example (warping × 4 ):
Input:
AAAAllllllll hhhhuuuummmmaaaannnn
Output:
AAAAllllllll hhhhuuuummmmaaaa
Variable warping example (random warping × 1 – × 4 ):
Input:
Allllll hhhummmmaannn bbbbeeiiingssss
Output:
AAAlllll huuuummaaan bbeeeingggg
Uniformly time-warped tasks are produced by repeating each character maximum_warping times both in the input and output sequence, for some ﬁxed number maximum_warping .
Variably time-warped tasks are produced similarly, but each character is repeated a random number of times uniformly drawn between 1 and maximum_warping .
The same warping is used for the input and output sequence (so that the desired output is indeed a function of the input).
This exactly corresponds to transforming input 𝑥 ( 𝑡 ) into 𝑥 ( 𝑐 ( 𝑡 )) with 𝑐 a random, piecewise afﬁne time warping.
Fig.
2 gives an illustration.
For each value of maximum_warping , the train dataset consists of 50 , 000 length- 500 randomly warped random sequences, with either uniform or variable time warpings.
The alphabet is of size 10 (including a dummy symbol).
Contiguous characters are enforced to be different.
After warping, each sequence is truncated to length 500 .
Test datasets of 10 , 000 sequences are generated similarily.
The criterion to be minimized is the cross entropy in predicting the next character of the output sequence.
[Figure] [Figure]
A similar, slightly more difﬁcult task uses padded sequences instead of warped sequences, obtained by padding each element in the input sequence with a ﬁxed or variable number of 0 ’s (in continuoustime, this amounts to a time warping of a continuous-time input sequence that is nonzero at certain points in time only).
Each time the input is nonzero, the network has to output the previous nonzero character seen.
We compare three recurrent architectures: RNNs (Eq.
(2), a simple, ungated recurrent network), leaky RNNs (Eq.
(5), where each unit has a constant learnable “gate” between 0 and 1 ) and gated RNNs, with one gate per unit, described by (10).
All networks contain 64 recurrent units.
The point of using gated RNNs (10) (“LSTM-lite” with tied input and forget gates), rather than full LSTMs, is to explicitly test the relevance of the arguments in Section 1 for time warpings.
Indeed these LSTM-lite already exhibit perfect robustness to warpings in these tasks.
RMSprop with an 𝛼 parameter of 0 .
9 and a batch size of 32 is used.
For faster convergence, learning rates are divided by 2 each time the evaluation loss has not decreased after 100 batches.
All architectures are trained for 3 full passes through the dataset, and their evaluation losses are compared.
Each setup is run 5 times, and mean, maximum and minimum results among the ﬁve trials are reported.
The results are summarized in Fig.
Gated architectures signiﬁcantly outperform RNNs as soon as moderate warping coefﬁcients are involved.
As expected from theory, leaky RNNs perfectly solve uniform time warpings, but fail to achieve optimal behavior with variable warpings, to which they are not invariant.
Gated RNNs, which are quasi invariant to general time warpings, achieve perfect performance in both setups for all values of maximum_warping .
Synthetic tasks.
For synthetic tasks, optimization is performed using RMSprop (Tieleman & Hinton, 2012) with a learning rate of 10 − 3 and a moving average parameter of 0 .
No gradient clipping is performed; this results in a few short-lived spikes in the plots below, which do not affect ﬁnal performance.
C OPY TASKS .
The copy task checks whether a model is able to remember information for arbitrarily long durations.
We use the setup from (Hochreiter & Schmidhuber, 1997; Arjovsky et al., 2016), which we summarize here.
Consider an alphabet of 10 characters.
The ninth character is a dummy character and the tenth character is a signal character.
For a given 𝑇 , input sequences consist of 𝑇 + 20 characters.
The ﬁrst 10 characters are drawn uniformly randomly from the ﬁrst 8 letters of the alphabet.
These ﬁrst characters are followed by 𝑇 − 1 dummy characters, a signal character, whose aim is to signal the network that it has to provide its outputs, and the last 10 characters are dummy characters.
The target sequence consists of 𝑇 + 10 dummy characters, followed by the ﬁrst 10 characters of the input.
This dataset is thus about remembering an input sequence for exactly 𝑇 timesteps.
We also provide results for the variable copy task setup presented in (Henaff et al., 2016), where the number of characters between the end of the sequence to copy and the signal character is drawn at random between 1 and 𝑇 .
The best that a memoryless model can do on the copy task is to predict at random from among possible characters, yielding a loss of 10log(8)
𝑇 +
(Arjovsky et al., 2016).
On those tasks we use LSTMs with 128 units.
For the standard initialization (baseline), the forget gate biases are set to 1 .
For the new initialization, the forget gate and input gate biases are chosen according to the chrono initialization (16), with 𝑇 max = 3 𝑇 2 for the copy task, thus a bit larger than input length, and 𝑇 max = 𝑇 for the variable copy task.
The results are provided in Figure 3.
Importantly, our LSTM baseline (with standard initialization) already performs better than the LSTM baseline of (Arjovsky et al., 2016), which did not outperform random prediction.
This is presumably due to slightly larger network size, increased training time, and our using the bias initialization from (Gers & Schmidhuber, 2000).
On the copy task, for all the selected 𝑇 ’s, chrono initialization largely outperforms the standard initialization.
Notably, it does not plateau at the memoryless optimum.
On the variable copy task, chrono initialization is even with standard initialization for 𝑇 = 500 , but largely outperforms it for 𝑇 = 1000 .
A DDING TASK .
The adding task also follows a setup from (Hochreiter & Schmidhuber, 1997; Arjovsky et al., 2016).
Each training example consists of two input sequences of length 𝑇 .
The ﬁrst one is a sequence of numbers drawn from 𝒰 ([0 , 1]) , the second is a sequence containing zeros everywhere, except for two locations, one in the ﬁrst half and another in the second half of the sequence.
The target is a single number, which is the sum of the numbers contained in the ﬁrst sequence at the positions marked in the second sequence.
The best a memoryless model can do on this task is to predict the mean of 2 × 𝒰 ([0 , 1]) , namely 1 (Arjovsky et al., 2016).
Such a model reaches a mean squared error of 0 . 167 .
LSTMs with 128 hidden units are used.
The baseline (standard initialization) initializes the forget biases to 1 .
The chrono initialization uses 𝑇 max = 𝑇 .
Results are provided in Figure 4.
For all 𝑇 ’s, chrono initialization signiﬁcantly speeds up learning.
Notably it converges 7 times faster for 𝑇 = 750 .
C ONCLUSION
The self loop feedback gating mechanism of recurrent networks has been derived from ﬁrst principles via a postulate of invariance to time warpings.
Gated connections appear to regulate the local time constants in recurrent models.
With this in mind, the chrono initialization, a principled way of initializing gate biases in LSTMs, has been introduced.
Experimentally, chrono initialization is shown to bring notable beneﬁts when facing long term dependencies.
A A DDITIONAL EXPERIMENTS
[Figure]
[Figure]
Pixel level classiﬁcation: MNIST and pMNIST.
This task, introduced in (Le et al., 2015), consists in classifying images using a recurrent model.
The model is fed pixels one by one, from top to bottom, left to right, and has to output a probability distribution for the class of the object in the image.
We evaluate standard and chrono initialization on two image datasets: MNIST (LeCun et al., 1999) and permuted MNIST, that is, MNIST where all images have undergone the same pixel permutation.
LSTMs with 512 hidden units are used.
Once again, standard initialization sets forget biases to 1 , and the chrono initialization parameter is set to the length of the input sequences, 𝑇 max = 784 .
Results on the validation set are provided in Figure 5.
On non-permuted MNIST, there is no clear difference, even though the best validation error is obtained with chrono initialization.
On permuted MNIST, chrono initialization performs better, with a best validation result of 96 . 3% , while standard initialization obtains a best validation result of 95 . 4% .
Next character prediction on text8.
Chrono initialization is benchmarked against standard initialization on the character level text8 dataset (Mahoney, 2011).
Text8 is a 100 M character formatted text sample from Wikipedia.
(Mikolov et al., 2012)’s train-valid-test split is used: the ﬁrst 90 M characters are used as training set, the next 5 M as validation set and the last 5 M as test set.
The exact same setup as in (Cooijmans et al., 2016) is used, with the code directly taken from there.
Namely: LSTMs with 2000 units, trained with Adam (Kingma & Ba, 2014) with learning rate 10 − 3 , batches of size 128 made of non-overlapping sequences of length 180 , and gradient clipping at 1 . 0 .
Weights are orthogonally initialized, and recurrent batch normalization (Cooijmans et al., 2016) is used.
Chrono initialization with 𝑇 max = 8 is compared to standard 𝑏 𝑓 = 1 initialization.
Results are presented in Figure 6.
On the validation set, chrono initialization uniformly outperforms standard initialization by a small margin.
On the test set, the compression rate is 1 .
37 with chrono initialization, versus 1 . 38 for standard initialization.
This same slight difference is observed on two independent runs.
Our guess is that, on next character prediction, with moderately sized networks, short term dependencies dominate, making the difference between standard and chrono initialization relatively small.
Next word prediction on Penn Treebank.
To attest for the resilience of chrono initialization to more complex models than simple LSTMs, we train on word level Penn Treebank (Mikolov et al., 2012) using the best deep RHN network from (Zilly et al., 2016).
All hyperparameters are taken from of (Zilly et al., 2016).
For the chrono bias initialization, a single bias vector 𝑏 is sampled according to 𝑏 ∼ log( 𝒰 (1 , 𝑇 max )) , the carry gate bias vectors of all layers are initialized to − 𝑏 , and the transform gate biases to 𝑏 .
𝑇 max is chosen to be 11 (because this gives an average bias initialization close to the value 2 from (Zilly et al., 2016)).
Without further hyperparameter search and with a single run, we obtain test results similar to (Zilly et al., 2016), with a test perplexity of 6 . 54 .