Paper:
A RECURRENT NEURAL NETWORK WITHOUT CHAOS
We introduce an exceptionally simple gated recurrent neural network (RNN) that achieves performance comparable to well-known gated architectures, such as LSTMs and GRUs, on the word-level language modeling task. We prove that our model has simple, predicable and non-chaotic dynamics. This stands in stark contrast to more standard gated architectures, whose underlying dynamical systems exhibit chaotic behavior.
1 INTRODUCTION
Gated recurrent neural networks, such as the Long Short Term Memory network (LSTM) introduced by Hochreiter & Schmidhuber (1997) and the Gated Recurrent Unit (GRU) proposed by Cho et al. (2014), prove highly effective for machine learning tasks that involve sequential data. We propose an exceptionally simple variant of these gated architectures. The basic model takes the form
ht = θt tanh(ht−1) + ηt tanh(Wxt), (1)
where stands for the Hadamard product. The horizontal/forget gate (i.e. θt) and the vertical/input gate (i.e. ηt) take the usual form used in most gated RNN architectures. Specifically
θt := σ (Uθht−1 + Vθxt + bθ) and ηt := σ (Uηht−1 + Vηxt + bη) (2)
where σ(x) := (1 + e−x)−1 denotes the logistic sigmoid function. The network (1)–(2) has quite intuitive dynamics. Suppose the data xt present the model with a sequence
(Wxt)(i) = { 10 if t = T 0 otherwise,
(3)
where (Wxt)(i) stands for the ith component of the vector Wxt. In other words we consider an input sequence xt for which the learned ith feature (Wxt)(i) remains off except at time T . When initialized from h0 = 0, the corresponding response of the network to this “impulse” in the ith feature is
ht(i) ≈  0 if t < T ηT if t = T αt if t > T
(4)
with αt a sequence that relaxes toward zero. The forget gate θt control the rate of this relaxation. Thus ht(i) activates when presented with a strong ith feature, and then relaxes toward zero until the data present the network once again with strong ith feature. Overall this leads to a dynamically simple model, in which the activation patterns in the hidden states of the network have a clear cause and predictable subsequent behavior.
Dynamics of this sort do not occur in other RNN models. Instead, the three most popular recurrent neural network architectures, namely the vanilla RNN, the LSTM and the GRU, have complex, irregular, and unpredictable dynamics. Even in the absence of input data, these networks can give rise to chaotic dynamical systems. In other words, when presented with null input data the activation patterns in their hidden states do not necessarily follow a predictable path. The proposed network (1)–(2) has rather dull and minimalist dynamics in comparison; its only attractor is the zero state,
and so it stands at the polar-opposite end of the spectrum from chaotic systems. Perhaps surprisingly, at least in the light of this comparison, the proposed network (1) performs as well as LSTMs and GRUs on the word level language modeling task. We therefore conclude that the ability of an RNN to form chaotic temporal dynamics, in the sense we describe in Section 2, cannot explain its success on word-level language modeling tasks.
In the next section, we review the phenomenon of chaos in RNNs via both synthetic examples and trained models. We also prove a precise, quantified description of the dynamical picture (3)–(4) for the proposed network. In particular, we show that the dynamical system induced by the proposed network is never chaotic, and for this reason we refer to it as a Chaos-Free Network (CFN). The final section provides a series of experiments that demonstrate that CFN achieve results comparable to LSTM on the word-level language modeling task. All together, these observations show that an architecture as simple as (1)–(2) can achieve performance comparable to the more dynamically complex LSTM.
2 CHAOS IN RECURRENT NEURAL NETWORKS
The study of RNNs from a dynamical systems point-of-view has brought fruitful insights into generic features of RNNs (Sussillo & Barak, 2013; Pascanu et al., 2013). We shall pursue a brief investigation of CFN, LSTM and GRU networks using this formalism, as it allows us to identify key distinctions between them. Recall that for a given mapping Φ : Rd 7→ Rd, a given initial time t0 ∈ N and a given initial state u0 ∈ Rd, a simple repeated iteration of the mapping Φ
ut+1 = Φ(ut) t > t0,
ut0 = u0 t = t0,
defines a discrete-time dynamical system. The index t ∈ N represents the current time, while the point ut ∈ Rd represents the current state of the system. The set of all visited states O+(u0) := {ut0 , ut0+1, . . . , ut0+n, . . .} defines the forward trajectory or forward orbit through u0. An attractor for the dynamical system is a set that is invariant (any trajectory that starts in the set remains in the set) and that attracts all trajectories that start sufficiently close to it. The attractors of chaotic dynamical systems are often fractal sets, and for this reason they are referred to as strange attractors.
Most RNNs generically take the functional form
ut = Ψ(ut−1,W1xt,W2xt, . . . ,Wkxt), (5)
where xt denotes the tth input data point. For example, in the case of the CFN (1)–(2), we have W1 = W , W2 = Vθ and W3 = Vη . To gain insight into the underlying design of the architecture of an RNN, it proves usefull to consider how trajectories behave when they are not influenced by any external input. This lead us to consider the dynamical system
ut = Φ(ut−1) Φ(u) := Ψ(u, 0, 0, . . . , 0), (6)
which we refer to as the dynamical system induced by the recurrent neural network. The timeinvariant system (6) is much more tractable than (5), and it offers a mean to investigate the inner working of a given architecture; it separates the influence of input data xt, which can produce essentially any possible response, from the model itself. Studying trajectories that are not influenced by external data will give us an indication on the ability of a given RNN to generate complex and sophisticated trajectories by its own. As we shall see shortly, the dynamical system induced by a CFN has excessively simple and predictable trajectories: all of them converge to the zero state. In other words, its only attractor is the zero state. This is in sharp contrast with the dynamical systems induced by LSTM or GRU, who can exhibit chaotic behaviors and have strange attractors.
The learned parameters Wj in (5) describe how data influence the evolution of hidden states at each time step. From a modeling perspective, (6) would occur in the scenario where a trained RNN has learned a weak coupling between a specific data point xt0 and the hidden state at that time, in the sense that the data influence is small and so all Wjxt0 ≈ 0 nearly vanish. The hidden state then transitions according to ut0 ≈ Ψ(ut0−1, 0, 0, . . . , 0) = Φ(ut0−1). We refer to Bertschinger & Natschläger (2004) for a study of the chaotic behavior of a simplified vanilla RNN with a specific statistical model, namely an i.i.d. Bernoulli process, for the input data as well as a specific statistical model, namely i.i.d. Gaussian, for the weights of the recurrence matrix.
2.1 CHAOTIC BEHAVIOR OF LSTM AND GRU IN THE ABSENCE OF INPUT DATA
In this subsection we briefly show that LSTM and GRU, in the absence of input data, can lead to dynamical systems ut = Φ(ut−1) that are chaotic in the classical sense of the term (Strogatz, 2014). Figure 1 depicts the strange attractor of the dynamical system:
ut = [ ht ct ] u 7→ Φ(u) = [ o tanh (f c+ i g) f c+ i g ] (7)
i := σ(Wih+ bi) f := σ(Wfh+ bf ) o := σ(Woh+ bo) g := tanh(Wgh+ bg) (8)
induced by a two-unit LSTM with weight matrices
Wi = [ −1 −4 −3 −2 ] Wo = [ 4 1 −9 −7 ] Wf = [ −2 6 0 −6 ] Wg = [ −1 −6 6 −9 ] (9)
and zero bias for the model parameters. These weights were randomly generated from a normal distribution with standard deviation 5 and then rounded to the nearest integer. Figure 1(a) was obtained by choosing an initial state u0 = (h0, c0) uniformly at random in [0, 1]2 × [0, 1]2 and plotting the h-component of the iterates ut = (ht, ct) for t between 103 and 105 (so this figure should be regarded as a two dimensional projection of a four dimensional attractor, which explain its tangled appearance). Most trajectories starting in [0, 1]2 × [0, 1]2 converge toward the depicted attractor. The resemblance between this attractor and classical strange attractors such as the Hénon attractor is striking (see Figure 5 in the appendix for a depiction of the Hénon attractor). Successive zooms on the branch of the LSTM attractor from Figure 1(a) reveal its fractal nature. Figure 1(b) is an enlargement of the red box in Figure 1(a), and Figure 1(c) is an enlargement of the magenta box in Figure 1(b). We see that the structure repeats itself as we zoom in.
The most practical consequence of chaos is that the long-term behavior of their forward orbits can exhibit a high degree of sensitivity to the initial states u0. Figure 2 provides an example of such behavior for the dynamical system (7)–(9). An initial condition u0 was drawn uniformly at random in [0, 1]2 × [0, 1]2. We then computed 100, 000 small amplitude perturbations û0 of u0 by adding a small random number drawn uniformly from [−10−7, 10−7] to each component. We then iterated (7)–(9) for 200 steps and plotted the h-component of the final state û200 for each of the 100, 000 trials on Figure 2(a). The collection of these 100, 000 final states essentially fills out the entire attractor, despite the fact that their initial conditions are highly localized (i.e. at distance of no more than 10−7) around a fixed point. In other words, the time t = 200 map of the dynamical system will map a small neighborhood around a fixed initial condition u0 to the entire attractor. Figure 2(b) additionally illustrates this sensitivity to initial conditions for points on the attractor itself. We take an initial condition u0 on the attractor and perturb it by 10−7 to a nearby initial condition û0. We then plot the distance ‖ût − ut‖ between the two corresponding trajectories for the first 200 time steps. After an initial phase of agreement, the trajectories strongly diverge.
The synthetic example (7)–(9) illustrates the potentially chaotic nature of the LSTM architecture. We now show that chaotic behavior occurs for trained models as well, and not just for synthetically generated instances. We take the parameter values of an LSTM with 228 hidden units trained on the
Penn Treebank corpus without dropout (c.f. the experimental section for the precise procedure). We then set all data inputs xt to zero and run the corresponding induced dynamical system. Two trajectories starting from nearby initial conditions u0 and û0 were computed (as before û0 was obtained by adding to each components of u0 a small random number drawn uniformly from [−10−7, 10−7]). Figure 3(a) plots the first component h(1) of the hidden state for both trajectories over the first 1600 time steps. After an initial phase of agreement, the forward trajectories O+(u0) and O+(û0) strongly diverge. We also see that both trajectories exhibit the typical aperiodic behavior that characterizes chaotic systems. If the inputs xt do not vanish, but come from actual word-level data, then the behavior is very different. The LSTM is now no longer an autonomous system whose dynamics are driven by its hidden states, but a time dependent system whose dynamics are mostly driven by the external inputs. Figure 3(b) shows the first component h(1) of the hidden states of two trajectories that start with initial conditions u0 and û0 that are far apart. The sensitivity to initial condition disappears, and instead the trajectories converge toward each other after about 70 steps. The memory of this initial difference is lost. Overall these experiments indicate that a trained LSTM, when it is not driven by external inputs, can be chaotic. In the presence of input data, the LSTM becomes a forced system whose dynamics are dominated by external forcing.
Like LSTM networks, GRU can also lead to dynamical systems that are chaotic and they can also have strange attractors. The depiction of such an attractor, in the case of a two-unit GRU, is provided in Figure 6 of the appendix.
2.2 CHAOS-FREE BEHAVIOR OF THE CFN
The dynamical behavior of the CFN is dramatically different from that of the LSTM. In this subsection we start by showing that the hidden states of the CFN activate and relax toward zero in a predictable fashion in response to input data. On one hand, this shows that the CFN cannot produce non-trivial dynamics without some influence from data. On the other, this leads to an interpretable model; any non-trivial activations in the hidden states of a CFN have a clear cause emanating from
data-driven activation. This follows from a precise, quantified description of the intuitive picture (3)–(4) sketched in the introduction.
We begin with the following simple estimate that sheds light on how the hidden states of the CFN activate and then relax toward the origin. Lemma 1. For any T, k > 0 we have
|hT+k(i)| ≤ Θk |hT (i)|+ H
1−Θ
( max
T≤t≤T+k |(Wxt)(i)| ) where Θ and H are the maximum values of the ith components of the θ and η gate in the time interval [T, T + k], that is:
Θ = max T≤t≤T+k θt(i) and H = max T≤t≤T+k ηt(i).
This estimate shows that if during a time interval [T1, T2] one of
(i) the embedded inputs Wxt have weak ith feature (i.e. maxT≤t≤T+k |(Wxt)(i)| is small), (ii) or the input gates ηt have their ith component close to zero (i.e. H is small),
occurs then the ith component of the hidden state ht will relaxes toward zero at a rate that depends on the value of the ith component the the forget gate. Overall this leads to the following simple picture: ht(i) activates when presented with an embedded input Wxt with strong ith feature, and then relaxes toward zero until the data present the network once again with strong ith feature. The strength of the activation and the decay rate are controlled by the ith component of the input and forget gates. The proof of Lemma 1 is elementary —
Proof of Lemma 1. Using the non-expansivity of the hyperbolic tangent, i.e. | tanh(x)| ≤ |x|, and the triangle inequality, we obtain from (1)
|ht(i)| ≤ Θ |ht−1(i)|+H max T≤t≤T+k |(Wxt)(i)|
whenever t is in the interval [T, T + k]. Iterating this inequality and summing the geometric series then gives |hT+k(i)| ≤ Θk|hT (i)|+ ( 1−Θk
1−Θ
) H max
T≤t≤T+k |(Wxt)(i)|
from which we easily conclude.
We now turn toward the analysis of the long-term behavior of the the dynamical system
ut = ht, u 7→ Φ(u) := σ (Uθu + bθ) tanh(u). (10) induced by a CFN. The following lemma shows that the only attractor of this dynamical system is the zero state. Lemma 2. Starting from any initial state u0, the trajectory O+(u0) will eventually converge to the zero state. That is, limt→+∞ ut = 0 regardless of the the initial state u0.
Proof. From the definition of Φ we clearly have that the sequence defined by ut+1 = Φ(ut) satisfies −1 < ut(i) < 1 for all t and all i. Since the sequence ut is bounded, so is the sequence vt := Uθut + bθ. That is there exists a finite C > 0 such that (Uθut)(i) + bθ(i) < C for all t and i. Using the non-expansivity of the hyperbolic tangent, we then obtain that |ut(i)| ≤ σ(C)|ut−1(i)|, for all t and all i. We conclude by noting that 0 < σ(C) < 1.
Lemma 2 remains true for a multi-layer CFN, that is, a CFN in which the first layer is defined by (1) and the subsequent layers 2 ≤ ` ≤ L are defined by:
h (`) t = θ (`) t tanh(h (`) t−1) + η (`) t tanh(W (`)h (`−1) t ).
Assume that Wxt = 0 for all t > T , then an extension of the arguments contained in the proof of the two previous lemmas shows that
|h(`)T+k| ≤ C(1 + k) (`−1)Θk (11)
where 0 < Θ < 1 is the maximal values for the input gates involved in layer 1 to ` of the network, and C > 0 is some constant depending only on the norms ‖W (j)‖∞ of the matrices and the sizes |h(j)T | of the initial conditions at all previous 1 ≤ j ≤ ` levels. Estimate (11) shows that Lemma 2 remains true for multi-layer architectures.
Inequality (11) shows that higher levels (i.e. larger `) decay more slowly, and remain non-trivial, while earlier levels (i.e. smaller `) decay more quickly. We illustrate this behavior computationally with a simple experiment. We take a 2-layer, 224-unit CFN network trained on Penn Treebank and feed it the following input data: The first 1000 inputs xt are the first 1000 words of the test set of Penn Treebank; All subsequent inputs are zero. In other words, xt = 0 if t > 1000. For each of the two layers we then select the 10 units that decay the slowest after t > 1000 and plot them on Figure 4. The figure illustrates that the second layer retains information for much longer than the first layer. To quantify this observation we define the relaxation time (or half-life) of the ith unit as the smallest T such that |h1000+T (i)| < 0.5|h1000(i)|. Using this definition yields average relaxation times of 2.2 time steps for the first layer and 23.2 time steps for the second layer. The first layer has a standard deviations of approximately 5 steps while the second layer has a standard deviation of approximately 75 time steps. A more fine-grained analysis reveals that some units in the second layer have relaxation times of several hundred steps. For instance, if instead of averaging the relaxation times over the whole layer we average them over the top quartile (i.e. the 25% units that decay the most slowly) we get 4.8 time steps and 85.6 time steps for the first and second layers, respectively. In other words, by restricting attention to long-term units the difference between the first and second layers becomes much more striking.
Overall, this experiment conforms with the analysis (11), and indicates that adding a third or fourth layer would potentially allow a multi-layer CFN architecture to retain information for even longer.
3 EXPERIMENTS
In this section we show that despite its simplicity, the CFN network achieves performance comparable to the much more complex LSTM network on the word level language modeling task. We use two datasets for these experiments, namely the Penn Treebank corpus (Marcus et al., 1993) and the Text8 corpus (Mikolov et al., 2014). We consider both one-layer and two-layer CFNs and LSTMs for our experiments. We train both CFN and LSTM networks in a similar fashion and always compare models that use the same number of parameters. We compare their performance with and without dropout, and show that in both cases they obtain similar results. We also provide results published in Mikolov et al. (2014), Jozefowicz et al. (2015) and Sukhbaatar et al. (2015) for the sake of comparison.
For concreteness, the exact implementation for the two-layer architecture of our model is
h (0) t = W (0)xt
ĥ (0) t = Drop(h (0) t , p)
h (1) t = θ (1) t tanh(h (1) t−1) + η (1) t tanh(W (1)ĥ (0) t )
ĥ (1) t = Drop(h (1) t , p)
h (2) t = θ (2) t tanh(h (2) t−1) + η (2) t tanh(W (2)ĥ (1) t )
ĥ (2) t = Drop(h (2) t , p)
yt = LogSoftmax(W (3)ĥ (2) t + b)
where Drop(z, p) denotes the dropout operator with a probability p of setting components in z to zero. We compute the gates according to
θ (`) t := σ
( U
(`) θ h̃ (`) t−1 + V (`) θ h̃ (`−1) t + bθ ) and η(`)t := σ ( U (`)η h̃ (`) t−1 + V (`) η h̃ (`−1) t + bη ) where h̃(`)t−1 = Drop(h (`) t−1, q) and h̃ (`−1) t = Drop(h (`−1) t , q),
and thus the model has two dropout hyperparameters. The parameter p controls the amount of dropout between layers; the parameter q controls the amount of dropout inside each gate. We use a similar dropout strategy for the LSTM, in that all sigmoid gates f, o and i receive the same amount q of dropout.
To train the CFN and LSTM networks, we use a simple online steepest descent algorithm. We update the weights w via w(k+1) = w(k) − lr · ~p where ~p = ∇wL‖∇wL‖2 , (12) where lr is the learning rate and ∇wL denotes the approximate gradient of the loss with respect to the weights as estimated from a certain number of presented examples. We use the usual backpropagation through time approximation when estimating the gradient: we unroll the net T steps in the past and neglect longer dependencies. In all experiments, the CFN and LSTM networks are unrolled for T = 35 steps and we take minibatches of size 20. As all search directions ~p have Euclidean norm ‖~p‖2 = 1, we perform no gradient clipping during training. We initialize all the weights in the CFN, except for the bias of the gates, uniformly at random in [−0.07, 0.07]. We initialize the bias bθ and bη of the gates to 1 and −1, respectively, so that at the beginning of the training θt ≈ σ(1) ≈ 0.73 and ηt ≈ σ(−1) ≈ 0.23. We initialize the weights of the LSTM in exactly the same way; the bias for the forget and input gate are initialized to 1 and −1, and all the other weights are initialized uniformly in [−0.07, 0.07]. This initialization scheme favors
the flow of information in the horizontal direction. The importance of a careful initialization of the forget gate was pointed out in Gers et al. (2000) and Jozefowicz et al. (2015). Finally, we initialize all hidden states to zero for both models.
Dataset Construction. The Penn Treebank Corpus has 1 million words and a vocabulary size of 10,000. We used the code from Zaremba et al. (2014) to construct and split the dataset into a training set (929K words), a validation set (73K words) and a test set (82K words). The Text8 corpus has 100 million characters and a vocabulary size of 44,000. We used the script from Mikolov et al. (2014) to construct and split the dataset into a training set (first 99M characters) and a development set (last 1M characters).
Experiments without Dropout. Tables 1 and 2 provide a comparison of various recurrent network architectures without dropout evaluated on the Penn Treebank corpus and the Text8 corpus. The last two rows of each table provide results for LSTM and CFN networks trained and initialized in the manner described above. We have tried both one and two layer architectures, and reported only the best result. The learning rate schedules used for each network are described in the appendix.
We also report results published in Jozefowicz et al. (2015) were a vanilla RNN, a GRU and an LSTM network were trained on Penn Treebank, each of them having 5 million parameters (only the test perplexity was reported). Finally we report results published in Mikolov et al. (2014) and Sukhbaatar et al. (2015) where various networks are trained on Text8. Of these four networks, only the LSTM network from Mikolov et al. (2014) has the same number of parameters than the CFN and LSTM networks we trained (46.4M parameters). The vanilla RNN, Structurally Constrained Recurrent Network (SCRN) and End-To-End Memory Network (MemN2N) all have 500 units, but less than 46.4M parameters. We nonetheless indicate their performance in Table 2 to provide some context.
Experiments with Dropout. Table 3 provides a comparison of various recurrent network architectures with dropout evaluated on the Penn Treebank corpus. The first three rows report results published in (Jozefowicz et al., 2015) and the last four rows provide results for LSTM and CFN networks trained and initialized with the strategy previously described. The dropout rate p and q are chosen as follows: For the experiments with 20M parameters, we set p = 55% and q = 45% for the CFN and p = 60% and q = 40% for the LSTM; For the experiments with 50M parameters, we set p = 65% and q = 55% for the CFN and p = 70% and q = 50% for the LSTM.
4 CONCLUSION
Despite its simple dynamics, the CFN obtains results that compare well against LSTM networks and GRUs on word-level language modeling. This indicates that it might be possible, in general, to build RNNs that perform well while avoiding the intricate, uninterpretable and potentially chaotic dynamics that can occur in LSTMs and GRUs. Of course, it remains to be seen if dynamically simple RNNs such as the proposed CFN can perform well on a wide variety of tasks, potentially requiring longer term dependencies than the one needed for word level language modeling. The experiments presented in Section 2 indicate a plausible path forward — activations in the higher layers of a multi-layer CFN decay at a slower rate than the activations in the lower layers. In theory, complexity and long-term dependencies can therefore be captured using a more “feed-forward” approach (i.e. stacking layers) rather than relying on the intricate and hard to interpret dynamics of an LSTM or a GRU.
Overall, the CFN is a simple model and it therefore has the potential of being mathematically wellunderstood. In particular, Section 2 reveals that the dynamics of its hidden states are inherently more interpretable than those of an LSTM. The mathematical analysis here provides a few key insights into the network, in both the presence and absence of input data, but obviously more work is needed before a complete picture can emerge. We hope that this investigation opens up new avenues of inquiry, and that such an understanding will drive subsequent improvements.

Review:
This paper poses an interesting idea: removing chaotic behavior or RNNs.
While many other papers on new RNN architecture usually focus too much on the performance improvement and leave the analysis part on their success as a black-box, this paper does a good job on presenting why its method may work well.
Although, the paper shows lots of comparison between the chaotic systems (GRUs & LSTMs) and the stable system (proposed CFN model), the reviewer is not fully convinced by the main claim of this paper, the nuance that chaotic behaviour makes dynamic system to have rich representation power but makes the system too unstable. In the paper, the LSTM shows a very sensitive behaviour, even when a very small amount of noise is added to the input. However, it still performs surprisingly well with this chaotic behaviour. 
Measuring the model complexity is a very difficult task, therefore, many papers manage to use either same number of hidden units or choose approximately close model sizes. In this paper, the experiments were carried by using the same amount of parameters for both the LSTM and CFN. However, I think the CFN may have much more simpler computational graph. Taking the idea of this work, can we develop a stable dynamic system, but which does not only have one attractor?
It is also interesting to see that the layers of CFNs are updated in different timescales in a sense that the decaying speed decreases when the layer gets higher. Could you provide more statistics on this? For example, what is the average relaxation time of the whole hidden units at each layer?
Batch normalization and layer normalization can be helpful to make the training of RNNs become more stable. How would the behaviour of batch normalized or perhaps layer normalized LSTM look like? Also, it is often not trivial to make batch normalization or layer normalization to work on a new architecture. I think it may be useful to compare batch normalized or layer normalized versions of the LSTM and CFN.
The quality of the work is good, explanation is clear enough along with nice analyses and proofs. Overall, the performance is not any better than LSTMs, but it is still interesting when thinking of simplicity of this model. I am a bit concerned if this model might not work that well in more harder task, e.g., translation. Figure 4 of this paper is very interesting, where the proposed architecture shows that the hidden units at the second layer tends to keep its information longer than the first layer ones.