H OW TO TRAIN RNN S ON CHAOTIC DATA ?
Anonymous authors
Paper under double-blind review
A BSTRACT
Recurrent neural networks (RNNs) are wide-spread machine learning tools for modeling sequential and time series data.
They are notoriously hard to train because their loss gradients backpropagated in time tend to saturate or diverge during training.
This is known as the exploding and vanishing gradient problem.
Previous solutions to this issue either built on rather complicated, purpose-engineered architectures with gated memory buffers, or - more recently - imposed constraints that ensure convergence to a fixed point or restrict (the eigenspectrum of) the recurrence matrix.
Such constraints, however, convey severe limitations on the expressivity of the RNN.
Essential intrinsic dynamics such as multistability or chaos are disabled.
This is inherently at disaccord with the chaotic nature of many, if not most, time series encountered in nature and society.
Here we offer a comprehensive theoretical treatment of this problem by relating the loss gradients during
RNN training to the Lyapunov spectrum of RNN-generated orbits.
We mathematically prove that RNNs producing stable equilibrium or cyclic behavior have bounded gradients, whereas the gradients of RNNs with chaotic dynamics always diverge.
Based on these analyses and insights, we adapt the old idea of teacher forcing to yield an effective yet simple training technique for chaotic data, and offer guidance on how to choose relevant hyperparameters according to the Lyapunov spectrum.
I NTRODUCTION
Recurrent neural networks (RNNs) are widely used across various fields in engineering and science for learning sequential tasks or modeling and predicting time series (Lipton et al., 2015).
Yet, they struggle when long-term temporal dependencies, very slow, or hugely varying time scales are involved (Hochreiter, 1991; Bengio et al., 1994; Schmidt et al., 2021; Li et al., 2018; Rusch & Mishra, 2021a).
Time series or sequential data with such properties are, however, very common in fields like climate physics (Thomson, 1990), neuroscience (Fusi et al., 2007; Russo & Durstewitz, 2017), ecology (Turchin & Taylor, 1992), or language processing (Cho et al., 2014b).
Training RNNs on such data is hard because the loss gradients backpropagated in time easily saturate or diverge in this process.
This is commonly referred to as the exploding and vanishing gradient problem (EVGP)
(Hochreiter, 1991; Bengio et al., 1994; Pascanu et al., 2013).
One solution to the EVGP is based on specifically designed RNN architectures with gating mechanisms, such as long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997) or gated recurrent units (GRU) (Cho et al., 2014a).
These architectures allow states at earlier time steps to more easily influence activity much later through a kind of protected memory buffer, thus alleviating the
EVGP by structural design.
In practice, such models need to be backed up by further techniques like gradient clipping to keep the gradients in check (Pascanu et al., 2013).
The relatively complex architectural design of these networks impedes their mathematical analysis and requires reverse engineering after training (Maheswaranathan et al., 2019; Monfared & Durstewitz, 2020a;b; Schmidt et al., 2021).
Partly to forego these complications, a variety of other solutions has been proposed recently, imposing restrictions on the recurrence matrix to bound the gradients (Arjovsky et al., 2016;
Chang et al., 2019), or enforcing global stability by design or regularization (Erichson et al., 2021;
Kolter & Manek, 2019).
Often these procedures dramatically curtail the expressivity of the RNN (Kerg et al., 2019; Orhan & Pitkow, 2020; Schmidt et al., 2021); in particular, they rule out chaotic dynamics (see further discussion below).
This is at odds with the plethora of chaotic phenomena in nature, engineering, and society.
Chaotic dynamics are commonplace, almost default in any complex physical or biological system.
This includes scientific areas as diverse as neuroscience (Durstewitz & Gabriel, 2007; van Vreeswijk &
Sompolinsky, 1996), physiology (Kesmia et al., 2020), geophysics (Sivakumar, 2004), climate systems (Tziperman et al., 1997), astrophysics (Laskar & Robutel, 1993), ecology (Duarte et al., 2010), chemical reactions (Field & Gy¨orgyi, 1993), cell (Olsen & Degn, 1977) or population (May, 1987) biology.
Chaotic phenomena are also crucial for the understanding of societal and epidemiological processes, such as the spread of diseases (Mangiarotti et al., 2020), or in economics (Faggini, 2014).
They are further relevant in purely technical contexts such as electrical engineering (Tchitnga et al., 2019; Kamdjeu Kengne et al., 2021) or laser optics (Kantz et al., 1993).
They have even been suggested to play an up to now largely neglected, but potentially very significant role in speech recognition (Sabanal & Nakagawa, 1996) and natural language processing (Inoue et al., 2021).
Hence, in almost any practical setting, chaotic phenomena abound.
They cannot, in general, be ignored when devising RNN training algorithms.
Here we offer a comprehensive theoretical treatment of the relation between RNN dynamics and the behavior of the loss gradients during training.
We find a close connection between an RNN’s loss gradients and the largest Lyapunov exponent of its freely generated orbits.
We mathematically prove that RNNs producing stable fixed point or cyclic behavior have bounded gradients.
Crucially, however, the loss gradients of RNNs producing chaotic dynamics always diverge.
Hence, the chaotic nature of many time series data induces a principle problem, and, despite significant efforts in the past to solve the EVGP, training RNNs on such data remains an open issue.
We illustrate the implications of our theory for RNN training on several simulated and empirical chaotic time series, and adapt the old idea of sparsely forced BPTT as a simple yet effective remedy that enables to learn the underlying dynamics despite exploding gradients.
R ELATED WORKS
Exploding and vanishing gradients.
While ‘classical’ remedies of the EVGP (Hochreiter, 1991; Bengio et al., 1994; Pascanu et al., 2013) rest on purpose-tailored architectures with gating mechanisms, which safeguard information flow across longer temporal distances (Hochreiter & Schmidhuber, 1997; Cho et al., 2014a), the focus has recently shifted to simpler RNNs that address the EVGP by restricting the recurrence matrix to be orthogonal (Henaff et al., 2016;
Helfrich et al., 2018;
Jing et al., 2019), unitary (Arjovsky et al., 2016), or antisymmetric (Chang et al., 2019), or by ensuring globally stable fixed point solutions (Kag et al., 2020; Kag & Saligrama, 2021a), for example through co-trained Lyapunov functions (Kolter & Manek, 2019).
However, all these approaches impose strong limitations on the dynamical repertoire of the RNN, enforcing global convergence to fixed points or simple cycles.
In doing so, they drastically reduce the expressiveness of these models (Kerg et al., 2019; Orhan & Pitkow, 2020).
To address this problem, Erichson et al.
(2021) somewhat relaxed the constraints on the recurrence matrix by introducing a skew-symmetric decomposition combined with a Lipschitz condition on the activation function.
Another recent approach discretizes oscillator ODEs to arrive at a stable system of coupled (Rusch & Mishra, 2021a) or independent (Rusch & Mishra, 2021b) oscillators which increase the RNN’s expressiveness while bounding its gradients.
By design (and as acknowledged by the authors), neither of these architectures is capable of producing chaotic dynamics, however, as the underlying ODEs do not allow for exponential divergence of close-by trajectories (a prerequisite for chaos).
Given these often principle limitations of parametrically or dynamically strongly constrained models, a fruitful direction may be to modify the training process itself, e.g. through modified or auxiliary loss functions (Trinh et al., 2018;
Schmidt et al., 2021), or special procedures for parameter updating (Kag & Saligrama, 2021b) or loss truncation (Williams & Zipser, 1989; Menick et al., 2021).
Our empirical evaluation will follow up on such ideas, but also highlight that simple loss truncation, windowing, or architectural solutions like LSTMs are not sufficient.
Learning dynamical systems.
Surprisingly disconnected from the work on the EVGP and learning long-term dependencies, a huge and long-standing literature deals with training RNNs on nonlinear dynamical systems (DS) (Pearlmutter, 1990; Trischler & D’Eleuterio, 2016; Vlachas et al., 2020), including chaotic systems like the famous Lorenz equations (Lorenz, 1963) or chaotic turbulence in fluid dynamics (Li et al., 2021).
Teacher forcing (TF; Williams & Zipser (1989), Pearlmutter (1990),
Doya (1992), see also Goodfellow et al.
(2016)) is one of the earliest techniques introduced to keep
RNN trajectories on track while training.
The idea behind TF is to simply replace RNN states by observations when available, thereby also effectively cutting off the gradients.
TF essentially derives from ideas in dynamical control theory, and adaptive schemes that increasingly hand over control to the RNN throughout training have been devised (Abarbanel, 2013;
Abarbanel et al., 2018; Bengio et al., 2015).
A related technique from the control theory literature is “multiple shooting” (Voss et al., 2004):
Here the whole observed time series is chopped into chunks, and for each chunk of trajectory a new initial condition is estimated.
Explicit constraints ensure continuity between the separate trajectory bits during optimization.
State space models and the Expectation-Maximization algorithm became popular particularly in the 90es for uncovering the latent dynamics underlying a set of time series observations (Ghahramani & Roweis, 1999), and remain an important tool until today (Durstewitz, 2017; Koppe et al., 2019).
Most recently, approaches based on variational inference and the reparameterization trick (Kingma & Welling, 2014), like sequential variational autoencoders (SVAE), gained in popularity for DS approximation (Hernandez et al., 2020).
“Deterministic” RNNs (i.e., with latent states not treated as random variables), like conventional LSTMs (Vlachas et al., 2018), remain top choices for DS reconstruction, however.
Although connections between DS ideas and loss gradients have been drawn early on (Bengio et al., 1994), so far only particular scenarios (like fixed point attractors) have been considered.
Closest to our work is recent work by Schmidt et al.
(2021), where non-divergence of loss gradients is established when RNNs converge to fixed points or cycles.
However, this was done only for the particular class of piecewise-linear RNNs (PLRNNs), more restrictive conditions for cycles were imposed than assumed here, and - most importantly - the chaotic case on which we focus here was not considered.
Another recent study (Engelken et al., 2020) also points out the general connections between Lyapunov exponents and loss gradients that we develop in sect.
3.1, but does not provide any in-depth theoretical treatment, proofs, or empirical evaluation, as we do here.
Thus, a systematic theoretical framework that relates RNN dynamics more generally, and across a range of different
RNN architectures, to the behavior of its training gradients, is still lacking so far.
T HEORETICAL ANALYSIS : R ELATION BETWEEN RNN DYNAMICS AND LOSS GRADIENTS
In our analysis, we will cover all major types of system dynamics (fixed points, cycles, chaos, and quasi-periodicity), and mathematically investigate their implications for the loss gradients.
We will do this for all major classes of RNNs, including standard RNNs with largely arbitrary activation function, LSTMs, GRUs, and PLRNNs.
The next section will first develop and illustrate the basic intuition behind the relations between RNN dynamics and loss gradients.
3. P RELIMINARIES : RNN DYNAMICS AND LOSS GRADIENTS
Formally, all popular RNN architectures, including LSTMs, GRUs, or PLRNNs, are discrete time
DS, defined by a (first-order-Markovian) recursive prescription for the temporal evolution of the latent states z t ∈ R M of the form
[Equation]
where s t ∈ R N is the input at time t and θ are RNN parameters.
For instance, for standard RNNs we have F θ ( z t − 1 , s t ) =
f (cid:0) W z t − 1 + Bs t + h (cid:1) , where f is an element-wise activation function like tanh or a rectified linear unit (ReLU).
Assuming we start at some initial value z 1 ∈ R M , and given a sequence of external inputs S = { s t } , we can recursively rewrite eq.
(1) as
[Equation]
In DS theory, we characterize the long-term behavior of such sequences by its spectrum of Lyapunov exponents.
The Lyapunov exponents estimate the exponential growth rates in different local directions of the system’s state space, and the largest Lyapunov exponent gives the dominant exponential behavior.
Let us denote the system’s Jacobian at time t by
[Equation]
For instance, for standard RNNs we would have J t =
W diag (cid:0) f ′ (cid:0)
W z t − 1 + Bs t
+ h (cid:1)(cid:1) , where diag denotes a diagonal matrix for which the i -th diagonal entry is the derivative of f w.r.t.
z i,t − 1 .
Then, the maximal Lyapunov exponent along an RNN trajectory { z 1 , z 2 , · · · , z T , · · · } is defined as
[Equation]
where ∥ · ∥ denotes the spectral norm (or any subordinate norm) of a matrix.
If λ max < 0 nearby trajectories will ultimately converge to a fixed point or cycle, while for λ max > 0 (a necessary condition for chaos) initially nearby trajectories will exponentially separate, i.e. we will have divergence along one (or more) directions in state space.
This accounts for the sensitive dependence on initial conditions in chaotic systems.
Now let L ( W , B , h ) be some loss function employed for RNN training that decomposes in time as L = (cid:80)
Tt =
1 L t .
Suppose we fancy BPTT as our training algorithm (similar derivations could be performed for RTRL), we recursively develop the loss gradients w.r.t.
some RNN parameter θ in time (i.e., across layers of the RNN unrolled in time) as
[Equation]
with
[Equation]
and
[Equation]
where ∂ + denotes the immediate derivative.
Now observe that the behavior of the loss gradients crucially depends on the product series of Jacobians in eqn.
If the maximum absolute eigenvalues of the Jacobians J t will, in the geometric mean, be larger than 1 (i.e., (cid:13)(cid:13)(cid:13)(cid:81) T − 2 r =
J T − r (cid:13)(cid:13)(cid:13) 1 /T > 1 ), gradients will explode as T → ∞ , while they will saturate if (cid:13)(cid:13)(cid:13)(cid:81) T − 2 r =
J T − r (cid:13)(cid:13)(cid:13) 1 /T < 1 .
Thus, the key point to note is that the same terms that occur in the definition of the Lyapunov spectrum, eqn.
(4), resurface in the loss gradients, eqn.
This accounts for the tight links between system dynamics and gradients.
3. F IXED POINTS AND CYCLIC DYNAMICS
Let us start by considering the simplest types of dynamics that can occur in RNNs (or any discretetime DS): fixed points and cycles.
In fact, by far most of the literature on global stability in RNNs and on loss gradients focused on just fixed points (Chang et al., 2019; Kolter & Manek, 2019;
Erichson et al., 2021), with only few authors who recently started to also connect cyclic behavior to loss gradients (Schmidt et al., 2021; Rusch & Mishra, 2021a).
Recall that a fixed point of a recursive map z t =
F ( z t − 1 ) is defined as a point z ∗ for which we have z ∗ =
F ( z ∗ ) .
Likewise, a k -cycle ( k > 1 ) is a set of temporally consecutive periodic points P k :=
{ z t 1 , z t 2 , . . .
, z t k } =
{ z t 1 , F ( z t 1 ) , . . .
, F k − 1 ( z t 1 ) } that we obtain from recursive application of the map such that each of the cyclic points z t r ∈
P
k is a fixed point of the k times iterated map F k (with k being the smallest positive integer for which this holds).
To simplify the subsequent treatment, we will collectively refer to fixed points and cycles as k -cycles ( k ≥ 1 ).
Further recall that a fixed point or k -cycle is called stable if the maximum absolute eigenvalue of the Jacobian evaluated at that point is smaller than 1 , neutrally stable if exactly 1 , and unstable otherwise.
Although the results we develop in this and the following sections will hold more widely, we will restrict our attention to recursive maps F θ from the class of RNNs R = { standardRNN , LSTM , GRU , PLRNN } (see Appx.
A.1 for details).
Based on the observations made in the previous sections we can state the following theorem that links RNN dynamics and loss gradients:
Theorem 1.
Consider an RNN F θ ∈ R parameterized by θ , and assume that it converges to a stable fixed point or k -cycle Γ k ( k ≥ 1) with B Γ k as its basin of attraction 3 .
Then for every z 1 ∈ B Γ k (i) the Jacobian ∂ z T ∂ z 1 exponentially vanishes as T → ∞ ; (ii) for Γ k
the tangent vectors ∂ z T ∂θ and thus the gradient of the loss function, ∂ L T ∂ θ , will be bounded from above, i.e. will not diverge for T → ∞ ; and (iii) for the PLRNN (27) both (cid:13)(cid:13)
∂
z T ∂θ (cid:13)(cid:13) and (cid:13)(cid:13) ∂
L T ∂ θ (cid:13)(cid:13) will remain bounded for every z 1 ∈ B Γ k as
T → ∞ .
Proof.
( i ) Assume that Γ k is a stable k -cycle ( k ≥ 1) denoted by
[Equation]
Then, the largest Lyapunov exponent of Γ k is given by
[Equation]
By assumption of stability of Γ k we have λ Γ k < 0 and also ρ (cid:0) (cid:81)
k − 1 s =
J t ∗ k − s (cid:1) < 1 , which implies
[Equation]
Now suppose that O z 1 is an orbit of (1) converging to Γ k , i.e. z 1 ∈ B Γ k .
Since O z 1 and Γ k have the same largest Lyapunov exponent, we have
[Equation]
and hence for z 1 ∈ B Γ k
[Equation]
( ii ) & ( iii ) See Appx.
A.2.1.
Remark 1.
The result of Theorem 1 part ( i ) will be generally true for any first-order-Markovian recursive map (1), but the conclusions in part ( ii ) may hinge on its specific definition.
Remark 2.
None of the results above and throughout sect.
3 require the dynamics to be autonomous, the theory applies whether there is external input or not.
In fact, mathematically, non-autonomous (externally forced) systems can always be rewritten as autonomous dynamical systems (Alligood et al., 1996; Perko, 2001; Zhang et al., 2009), see Appx.
A.1.1 for details.
The results above ensure that loss gradients will not diverge (explode) as T → ∞ in RNNs that are “well-behaved” in the sense that they converge to a fixed point or cycle.
This is a generalization of the results given in Theorem 1 in Schmidt et al.
(2021), where this was shown only a) for the specific class of PLRNNs and b) for specific constraints imposed on the eigenvalue spectrum of the RNN’s
Jacobians which were relaxed in our theorem above.
While our treatment above is centered on the “exploding-gradients” case, various architectural modifications or regularization techniques can ensure that gradients do not vanish either, i.e. remain bounded from below as well.
This was established, for instance, in Schmidt et al.
(2021) for
PLRNNs using ’manifold attractor regularization’.
In Appx.
A.2.1 we show that the results from
Theorem 2 from Schmidt et al.
(2021) on doubly bounded (from below and above) loss gradients can indeed be extended to the more general case covered by Theorem 1 above.
3. C HAOTIC DYNAMICS
We will now consider the all-important chaotic case.
Let F be a recursive map and O z 1 = { z 1 , z 2 , z 3 , · · · } be an orbit of F .
The orbit is chaotic if (i) it is not asymptotically periodic and (ii) has at least one positive Lyapunov exponent (Glendinning & Simpson, 2021; Meiss, 2007).
If the system’s invariant set is bounded , condition (ii) is considered a standard signature of chaos, as in this case two nearby orbits separate exponentially fast, but at the same time their mutual separation cannot go to infinity so that there are also folds.
The following theorem states the sufficient condition for exploding gradients:
Theorem 2.
Suppose that an RNN F θ ∈ R (parameterized by θ ) has a chaotic attractor Γ ∗ with
B Γ ∗ as its basin of attraction.
Then, for every orbit with z 1 ∈ B Γ ∗ , (i) the Jacobians connecting temporally distal states z T and z t ( T ≫ t ), ∂ z T ∂ z t , will exponentially explode for T → ∞ , and (ii) the tangent vector ∂
z T ∂θ
and so the gradients of the loss function, ∂ L T ∂θ , will diverge as T → ∞ .
Proof.
Let the RNN F θ ∈ R have a chaotic orbit denoted by Γ ∗ =
{ z ∗ 1 , z ∗ 2 , · · · , z ∗ T , · · · } .
Then, denoting by J ∗
T the Jacobian of (1) at z ∗ T ∈ Γ ∗ , the largest Lyapunov exponent of Γ ∗ is given by
[Equation]
Since Γ ∗ is chaotic, so λ > 0 .
Hence, from (12), it is concluded that
[Equation]
Now, according to Oseledec’s multiplicative ergodic Theorem, nearly all the points in the basin of attraction of Γ ∗ have the same largest Lyapunov exponent λ .
Thus, (13) holds for every z 1 ∈ B Γ ∗ .
( ii ) See Appx.
A.2.2.
Remark 3.
The first part of Theorem 2 holds for all first-order-Markovian recursive maps (1).
Note that for LSTMs, ∂ z T ∂ z t ( z :=
( h , c ) T ) denotes the full Jacobian of both hidden and cell states.
We collect some further mathematical results and remarks related to Theorem 2 in Appx.
A.3.1.
Hence, the essential result is that for all popular RNNs R and activation functions, loss gradients will inevitably diverge if the RNN latent states converge to a chaotic attractor.
3. Q UASI - PERIODICITY
Quasi-periodicity is a long-term behavior which occurs on a torus and, superficially, bears some similarity to chaos in the sense that, strictly speaking, orbits are also aperiodic .
That is, as T → ∞ , trajectories will never close up with themselves.
Moreover, every trajectory becomes arbitrarily close to any point on the torus, that is, it is dense.
One important difference between quasi-periodic and chaotic systems is, however, that in a quasi-periodic system, as time passes, two close initial conditions are linearly diverging, while in a chaotic system the divergence is exponential.
Theorem 3.
Assume that an RNN F θ ∈ R (parameterized by θ ) has a quasi-periodic attractor Γ with B Γ as its basin of attraction.
Then, for every z 1 ∈ B Γ
[Equation]
Proof.
See Appx.
A.2.3.
According to Theorem 3, for every orbit converging to a quasi-periodic attractor, the Jacobians ∂ z T ∂ z t may diverge or vanish as T → ∞ , but this will not occur exponentially fast as T → ∞ .
Thus, even for bounded non-chaotic RNNs we may sometimes stumble into the problem of diverging gradients.
Although this may be a less common scenario, we point out it may occur if we train RNNs on real data from oscillatory systems with incommensurate frequencies, as for instance encountered in electronic engineering.
In Appx.
A.3.2 we have collected further mathematical results on the connection between RNN dynamics and loss gradients that hold regardless of the RNN’s limiting behavior.
E MPIRICAL EVALUATION
Our theoretical results imply that chaotic time series pose a principle challenge for RNN training that cannot easily be circumvented through specifically designed architectures, constraints, or regularization criteria.
If the underlying DS we aim to capture is chaotic, loss gradients propagated back in time will inevitably explode.
Here we will work out some implications for RNN training and potential remedies empirically.
4. T RAINING ON SYSTEMS WITH EXPLODING GRADIENTS BY SPARSE TEACHER FORCING
To illustrate the connections between theory and RNN training, we revive the old idea of TF (Williams & Zipser, 1989) as a mechanism for truncating error gradients while training.
However, we would like to do this such that important information about the system dynamics does not get lost, for which Lyapunov theory offers some guidance.
Specifically, we should not force the system back onto the true trajectory all or most of the time (as in “classical TF”), but should effectively “re-calibrate” it only at certain time points chosen wisely according to the system’s local divergence rates.
This procedure will be referred to as sparsely forced BPTT in the following.
Assume we want to train an RNN with hidden states z t ∈ R M and linear (or affine) output layer on a time-series { x 1 , x 2 , · · · , x T } generated by a chaotic system.
The linear output layer ˆ x t =
Bz t , B ∈ R N × M , maps the RNN hidden states into the observation space.
This allows us to modify the original TF procedure by constructing a control series { ˜ z 1 , ˜ z 2 , · · · , ˜ z T } from the observations by “inverting” the linear output mapping
[Equation]
The idea is to supply this control signal only sparsely, separated by the learning interval τ between consecutive forcings.
Hence, defining T = { nτ + 1 } n ∈ N 0 as the set of all time points at which we force the RNN onto the ‘true’ values, the RNN updates can be written as
[Equation]
This forcing is applied after calculation of the loss, such that L t =
∥ x t − Bz t ∥ 22 irrespective of whether t is in
T or not (and of course it is applied only during training, not at test time!).
Replacing hidden states z t with their teacher-forced signals ˜
z t simply breaks divergence between true and predicted trajectories at time points t ∈ T , and also cuts off the Jacobians by breaking the temporal contingency (for details see Appx.
A.7).
The learning interval τ hence controls how many time steps are included in the gradient calculation and has to be chosen with care such as to balance the effects of exploding gradients vs. those of loosing relevant time scales and long-term dependencies.
While it is general wisdom that an optimal batch size will facilitate training, the point here is thus much more specific: Ideally τ should be chosen in accordance with the system’s
Lyapunov spectrum, for instance based on the predictability time (Bezruchko & Smirnov, 2010)
[Equation]
We emphasize that such a simple recipe for addressing the exploding gradient problem is based on modifying the training routine, and is thus in principle applicable to any model architecture.
4. E XAMPLE 1: L ORENZ SYSTEM AND EXTERNALLY FORCED D UFFING OSCILLATOR IN CHAOTIC REGIME
Let us illustrate these ideas on two classical textbook examples of chaotic DS, the chaotic Lorenz attractor as an autonomous system, and the chaotically forced Duffing oscillator as an example with explicit external input (see Appx.
A.4 for details).
Trajectories were repeatedly drawn from these systems, on which we trained a PLRNN, a vanilla RNN with tanh activation function, and a
LSTM by stochastic gradient descent (SGD) to minimize the MSE loss between predicted and actual observations.
As optimizer we used Adam (Kingma & Ba, 2015) from PyTorch (Paszke et al., 2017) with a learning rate of 0 .
For all models, training proceeded solely by sparsely forced BPTT and did not employ gradient clipping or any other technique that may interfere with optimal loss truncation.
In nonlinear DS reconstruction, we are mainly interested in reproducing invariant properties of the underlying system such as the attractor geometry (or topology; Takens (1981); Sauer et al.
or the frequency composition (i.e., time-averaged properties), while measures like ahead-prediction errors are less meaningful especially on chaotic time series (Wood, 2010; Koppe et al., 2019).
Thus, in evaluating training performance, here we follow Koppe et al.
in using a Kullback-Leibler divergence D stsp to quantify the agreement between observed and generated probability distributions across state-space to asses the overlap in attractor geometry (Appx. A.5).
Moreover, we employ a dimension-wise frequency correlation measure (PSC) to quantify the agreement of power-spectra of the observed and generated time-series (Appx. A.5).
[Figure]
Fig.
1 shows the dependence of the reconstruction quality on the learning interval τ for all RNN architectures on (a) the Lorenz and (b) the externally forced Duffing system.
Fig.
2 provides particular examples of reconstructions for τ chosen too small, too large, or about right.
For all models we find a system-dependent range for the optimal learning interval that agrees well with the predictability time defined in eqn.
(17), where estimates for the maximal Lyapunov exponent were taken from the literature (Rosenstein et al., 1993; Gilpin, 2021).
As a reference, dashed lines represent the reconstruction performance for all architectures when trained with classical BPTT and gradient clipping.
The training procedure was the same as for sparsely forced BPTT, except that instead of supplying a control-signal, gradients were normalized to 1 prior to each parameter update.
As evidenced by the much worse performance, gradient clipping does not effectively address the EVGP, even for LSTMs .
As further shown in Fig.
9 in Appx.
A.6.4, using the optimal window length τ but resetting the initial condition to zero (instead of its control value ˜
z t ) for each chunk equally destroys performance.
This suggests that neither mere gradient normalization nor simple windowing are sufficient, but will wipe out essential information about the dynamics.
In Appx.
A.6 we collect further results on the chaotic R¨ossler attractor (Fig. 5), high-dimensional
Mackey-Glass equations (Fig. 7), and the Lorenz attractor with partial observations (Fig.
4. E XAMPLE 2: C HAOTIC WEATHER DATA
As for an empirical example
[Figure]
we trained all RNNs (vanilla RNN, PLRNN, LSTM) on a temperature time series recorded at the Weather Station at the Max Planck Institute for Biogeochemistry in Jena, Germany.
To expose the chaotic behavior and obtain a robust estimate of the maximal
Lyapunov exponent, trends and yearly cycles were removed
[Figure]
and nonlinear noise-reduction was performed (Kantz et al.
; Appx.
A.4).
The maximal Lyapunov exponent was determined with the TISEAN package (Hegger et al., 1999), as shown in Figure 3 (a).
The value obtained is in close agreement with the literature (Mill´an et al., 2010).
Figure 3 shows that also for these empirical data the optimal training interval τ agrees well with the predictability time, eqn.
(17), for all trained RNNs.
Furthermore, as was the case for the DS benchmarks, gradient clipping was not able to satisfactorily tackle the EVGP, even when paired with architectures like LSTMs explicitly designed for alleviating this problem.
Similar results are reported for another real-world dataset, electrophysiological (local field potential) recordings, in
Appx.
A.***.
[Figure]
D ISCUSSION AND CONCLUSIONS
In this paper we proved that RNN dynamics and loss gradients are intimately related for all major types of RNNs and activation functions.
If the RNN is “well behaved” in the sense that its dynamics converges to a fixed point or cycle, loss gradients will remain bounded, and established remedies (Hochreiter & Schmidhuber, 1997; Schmidt et al., 2021) can be used to refrain them from vanishing.
However, if the dynamics are chaotic, gradients will always explode.
This constitutes a principle problem in RNN training that cannot easily be mastered through architectural design or gradient clipping.
It is furthermore a practically highly relevant one, as most time series we encounter in nature, and many from man-made systems as well, are inherently chaotic.
While we do not offer a full solution to this problem here, we suggest it might be tackled in training by taking a system’s local divergence rates as measured through the Lyapunov spectrum into account.
Hence, rather than conquering the EVGP by structural design or specific constraints or regularization terms, we recommend to put the focus more on the training process itself.
We illustrated this point empirically using sparsely forced BPTT , a training technique that pulls trajectories back on track at times determined by the maximal Lyapunov exponent.
Doing so leads to optimal reconstruction results for a variety of simulated and real-world benchmarks, regardless of the specific RNN architecture employed in training.
We stress that our goal above all was to provide a mathematically grounded perspective on the problem, with the empirical section focused on elucidating the practical implications of the theoretical results.
Empirically, for instance, precise Lyapunov exponents may sometimes be hard to obtain, although our empirical examples confirm that estimates based on log-divergence plots may work sufficiently well.
A CKNOWLEDGMENTS E THICS STATEMENT
The current work deals with theoretical-mathematical aspects of RNN training and performs basic research on limitations of training algorithms.
As such it has no direct ethical implications.
Since many real world applications depend on accurate forward predictions of time series, however, this paper may raise awareness for an important issue that will also be relevant in practical settings, including sensitive domains like medical time series, weather forecasts, or traffic control.
R EPRODUCIBILITY STATEMENT
All theoretical results in this paper were carefully and thoroughly proven, with all proofs and detailed derivations available in the Appendix.
Likewise, we will make available all code used in the empirical section in a way that will allow others to easily reproduce the results from this paper.
This means we will include everything, starting with the code for benchmark simulations and simulated time series data used for evaluation, code for our model and training algorithms, up to the metafiles that produce the actual figures in this work on our lab github site.
All of this will be clearly documented.
A A PPENDIX A. T HEOREMS : P RELIMINARIES A.1. T RANSFORMING NON - AUTONOMOUS INTO AUTONOMOUS DISCRETE - TIME DS
Following (Zhang et al., 2009), and based on similar reasoning as for continuous time (ODE-based)
DS (Alligood et al., 1996; Perko, 2001), let us consider the non-autonomous discrete-time DS
[Equation]
Defining z t = ( x t , t ) T and G ( z t )
=
( F ( x t , t ) , t + 1) T , system (18) can be rewritten as the autonomous system
[Equation]
Hence, in all our theoretical treatment we can confine our attention to systems of the form eqn.
A.1. RNN DERIVATIVES
Considering the loss function L = (cid:80) T t =1 L t of an RNN F θ ∈ R parameterized by θ , we have
[Equation]
(20) where
[Equation]
The tangent vector ∂
z T ∂θ has the form
[Equation]
z T
[Equation]
where ∂ + denotes the immediate partial derivative.
Since for an RNN F θ ∈ R the activation function is element-wise, with θ the m -th element of a parameter vector θ (or belonging to the m -th row of a parameter matrix θ ), we have
[Equation]
For instance, let θ =
W be a weight matrix, then
[Equation]
In this case, for the standard RNN we have
[Equation]
where ξ mk ( z T − 1 ) = f ′ w m,k (cid:0) (cid:80)
Mj
=
w mj z j,T − 1 + (cid:80) Mj
=
b mj s j,T + h m (cid:1) , and f ′ w m,k stands for the derivative of f with respect to w m,k .
Therefore, for standard RNNs, (22) becomes
[Equation]
[Equation]
A.1. P IECEWISE - LINEAR RNN (PLRNN)
The PLRNN has the generic form (Koppe et al., 2019; Schmidt et al., 2021)
[Equation]
where ϕ ( z t − 1 ) =
max( z t − 1 , 0) is the element-wise rectified linear unit (ReLU) function, z t ∈ R M is the neural state vector, A ∈ R M × M is a diagonal matrix of auto-regression weights,
W ∈ R M
× M is a matrix of connection weights, h ∈ R M is the bias vector, s t ∈ R K the external input weighted by C ∈ R M × K , and ε t ∼ N (0 , Σ ) a Gaussian noise term with diagonal covariance matrix Σ .
Equation (27) can be rewritten as
[Equation]
where D Ω( t ) :
= diag( d Ω( t ) ) is a diagonal matrix with d Ω( t ) :
= ( d 1 , d 2 , · · · , d M ) an indicator vector such that d m ( z m,t )
=:
d m = 1 whenever z m,t > 0 , and zeros otherwise.
For the PLRNN (28) we have
[Equation]
and
[Equation]
Furthermore, the derivatives (22) for the PLRNN (28) are
[Equation]
A.1. L ONG S HORT -T ERM M EMORY (LSTM)
The LSTM is defined by the equations
[Equation]
[Equation]
where { s t } is the input sequence, W denotes weight matrices, b bias terms, i t , f t , g t , o t demonstrate the input, forget, cell, and output gates, h t and c t are the hidden and cell states at time t respectively, σ is the sigmoid activation function, and ⊙ represents the element-wise (Hadamard) product (see (Hochreiter & Schmidhuber, 1997; Graves et al., 2016;
Vlachas et al., 2018) for further information on LSTMs).
Defining z t :
= ( h t , c t ) T , the LSTM (31) can be represented as the first-order recursive map
[Equation]
The term ∂ L t ∂θ in (20) for some LSTM parameter θ can be written as
[Equation]
A necessary condition for LSTMs to have a chaotic orbit is given by:
Proposition 1.
Let the LSTM given by (31) have a chaotic attractor Γ ∗ with B Γ ∗ as its basin of attraction.
Then for every z 1 = ( h 1 , c 1 ) T ∈ B Γ ∗
[Equation] [Equation]
Proof.
The Jacobian matrix of (32) for t > 1 can be written in the block form
[Equation]
[Equation]
Further, due to the chain rule, we have
[Equation]
 
[Equation]
36) and by induction we obtain
[Equation]
Now assume that (32) has a chaotic orbit given by
[Equation]
According to (37), the largest Lyapunov exponent of Γ ∗ is given by
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
T
[Equation]
c
[Equation]
∂ c
[Equation]

 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) .
Since Γ ∗ is chaotic, so λ Γ ∗ > 0 , which gives lim
[Equation]
(cid:118) (cid:117)
(cid:117)(cid:117)(cid:116)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)  
[Equation]
h
[Equation]
T
[Equation]
h ∗
[Equation]
h
[Equation]
T ∂ c ∗∂ c
[Equation]
T ∂ h ∗∂ c ∗
[Equation]
Based on Oseledec’s multiplicative ergodic Theorem, (39) holds for every z 1 ∈ B Γ ∗ .
This completes the proof.
A.1. G ATED R ECURRENT U NIT (GRU)
A GRU network is defined by the equations
[Equation]
where r t represents the reset gate, z t the update gate, s t and h t denote the inputs and the hidden state respectively, W z , W r , W h ∈ R M × N and U z , U r , U h ∈ R M × M are weight matrices, b z , b r , b h ∈ R M are bias vectors, and σ is the element-wise logistic sigmoid function (for more details about
GRUs see Cho et al.
(2014a)).
A.1. U NITARY EVOLUTION RNN ( U RNN)
The uRNN, proposed in (Arjovsky et al., 2016), is defined as the nonlinear DS
[Equation] [Equation]
for which W ∈ U ( M ) is an unitary matrix, V ∈ C M × N , b ∈ R M is the bias parameter, s t is the real- or complex-valued input of dimension N , and
[Equation]
Proposition 2.
The uRNN given by (41) cannot have any chaotic orbit.
Proof.
For any arbitrary orbit O z 1 of (41) we have
[Equation] [Equation]
43) where D t = diag (cid:16) σ ′ b (cid:0) W z t − 1 + V s t (cid:1) (cid:17) .
Since W is unitary and so a norm preserving matrix, it is concluded that
[Equation]
which implies
[Equation]
This rules out the existence of chaos (since λ max > 0 is a necessary condition for O z 1 to be chaotic).
Note that, more generally, any RNN which is constrained such as to exhibit global convergence to a fixed point or cycle, by definition must have a maximum Lyapunov exponent λ max ≤ 0 (in accordance with Theorem 1), hence cannot exhibit chaotic behavior by definition.
A. T HEOREMS : P ROOFS A.2. P ROOF OF THEOREM 1, PARTS ( II ) & ( III )
Proof.
( ii )
If J is the Jordan normal form of (cid:81) k − 1 s =
J t ∗ k − s , then (cid:81)
k − 1 s =
J t ∗ k − s =
P J P − 1 , where
J =  
J m 1 ( λ 1 )  · · ·
J m 2 ( λ 2 ) · · · ...
J m p − 1 ( λ p − 1 )  . . .
J m p ( λ p ) 
 , (46) and m i is the algebraic multiplicity of each eigenvalue
λ
i .
Since ρ (cid:0) (cid:81) k − 1 s =
J t ∗ k − s (cid:1) < 1 , so the eigenvalue λ i associated with each Jordan block satisfies | λ i | < 1 ( i = 1 , · · · , p ) .
Moreover, every m i × m i Jordan block has the form
J m i
( λ i ) = 
 λ i  · · ·  λ i · · · ... λ i   . . .
λ i 
 .
Accordingly (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:18) k − 1 (cid:89) s =
J t ∗ k − s (cid:19) j (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) =
[Equation]
P J j P
[Equation]
[Equation]
in which p = ∥ P ∥ (cid:13)(cid:13)
P − 1 (cid:13)(cid:13) .
Furthermore, for j ∈ N , J j is a block diagonal matrix of the form
[Equation]
in which every m i ×
m i Jordan block has the form
[Equation] [Equation]
In addition, for every block J jm i
( λ i ) , we have
[Equation]
Moreover, for any 1 < ˜ r i < 1 | λ i | , there exists some l i such that j m i <
˜ r ji for j ≥ l i .
This means for j ≥ l i
[Equation]
52) such that | ˜
r
i λ i | =
˜ r i | λ i | < 1 .
Besides, for J j = J jm 1 ( λ 1 )
[Equation]
2 ( λ
[Equation] [Equation]
[Equation]
Hence, from (48), (52) and (53), it is deduced that for j ≥ l
[Equation]
54) in which r = | ˜ r λ | < 1 .
Furthermore, let for Γ k
[Equation]
(cid:111)
=
max 0 ≤ s ≤ k −(cid:110) (cid:13)(cid:13)(cid:13)(cid:13) ∂
+ z t ∗ k − s ∂θ (cid:13)(cid:13)(cid:13)(cid:13) (cid:111)
=
ξ, max T ≥(cid:110)
∥
z T ∥ (cid:111) = max 0 ≤ s ≤ k −(cid:110) ∥
z t ∗ k − s ∥ (cid:111) =
¯ q.
[Table] [Table]
[Table] [Table]
(cid:13)(cid:13)(cid:13)(cid:13) ∂
z T ∂θ (cid:13)(cid:13)(cid:13)(cid:13) =
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ∂
+ z T ∂θ +
T − 2 (cid:88) t
= (cid:18) t − 1 (cid:89) r =
J ∗ T − r (cid:19) ∂
+ z T − t ∂θ (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) =
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ∂
+ z T ∂θ +
T − 1 (cid:88) t
= (cid:18) t − 1 (cid:89) r =
J ∗ T − r (cid:19) ∂
+ z T − t ∂θ (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ ¯ q ξ (cid:18) 1 +
T − 1 (cid:88) t
= (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) t − 1 (cid:89) r =
J ∗ T − r (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:19) .
On the other hand, for T = kj , from (54) and (55) we have
T − 1 (cid:88) t
= (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) t − 1 (cid:89) r =
J ∗ T − r (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) =
kj − 1 (cid:88) t =
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) t − 1 (cid:89) r =
J ∗ kj − r (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
=
k − 1 (cid:88) t
= (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) t − 1 (cid:89) r =
J ∗ kj − r (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) + 2 k − 1 (cid:88) t = k (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) t − 1 (cid:89) r = J ∗ kj − r (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) + 3 k − 1 (cid:88) t =2 k (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) t − 1 (cid:89) r =
J ∗ kj − r (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) + · · · + ( j − 1) k − 1 (cid:88) t =( j − 2)
[Equation]
Thus, considering (cid:0) ¯ m + ¯ m 2 + · · · + ¯ m k −
[Equation]
= M , it is deduced that
[Equation]
which, by (21), implies ∂ L T ∂θ will be bounded for T → ∞ .
( iii ) Consider the PLRNN given by (27), where for simplicity we ignore the external inputs and noise terms.
Let { z t 1 , z t 2 , z t 3 , . . .
} be an orbit which converges to Γ k .
Hence
[Equation]
which implies there exists a neighborhood U of Γ k and k sub-sequences { z t km } ∞ m =1 , { z t km +1 } ∞ m =1 , · · · , { z t km +( k − 1)
} ∞ m =1 of the sequence { z t n } ∞ n =1 such that all these sub-sequences belong to U and
a) z t km + s = F k
( z t k ( m − 1)+ s ) , s = 0 , 1 , 2 , · · · , k − 1 , b) lim
m →∞ z t km + s =
z t ∗ k − s , s = 0 , 1 , 2 , · · · , k − 1 , c) for every z t n ∈ U there is some s ∈ { 0 , 1 , 2 , · · · , k − 1 } such that z t n ∈ { z t km + s } ∞ m =1 .
In this case, for every z t n ∈ U with z t n ∈ { z t km + s } ∞ m =1 , there exists some ˜ n ∈ N such that z t n = z t k ˜ n + s and lim ˜ n →∞ z t k ˜ n + s =
z t ∗ k − s .
Therefore, continuity of F results in
[Equation] [Equation]
and so by (28)
[Equation]
which implies
[Equation]
Assuming lim ˜ n →∞ W Ω
( t k ˜ n + s ) =
L , since (62) holds for every z t ∗ k − s , substituting z t ∗ k − s = e T 1 = (1 , 0 , · · · , 0) T in (62), we can prove that the first column of L equals the first column of W Ω( t ∗ k − s ) .
Performing the same procedure for z t ∗ k − s = e T i , i = 2 , 3 , · · · , M , yields
[Equation]
According to (59), U contains an infinite number of terms of the sequence { z t n } ∞ n =1 , i.e.
[Equation]
Suppose that z t n ∈ U for some n ≥ N .
Thus, there exists some s ∈ { 0 , 1 , 2 , · · · , k − 1 } such that z t n ∈ { z t km + s } ∞ m =1 .
Without loss of generality let s = 0 .
Hence, there is some ˜ n ∈ N such that
[Equation]
this case, moving forward in time gives
[Equation] [Equation]
[Equation]
Consequently, for n ≥ N and j ∈ N , we can write
[Equation]
On the other hand, in equation (28), there are different configurations for matrix D Ω( t − 1) and hence different forms for matrix W Ω( t k ˜ n + s ) .
In this case, the phase space of the system is divided into different sub-regions by some borders; see (Monfared & Durstewitz, 2020a;b) for more details.
Also, since the system (28) is a linear map in each sub-region, the k periodic points of Γ k must belong to different sub-regions (at least two different sub-regions).
Accordingly, based on (63) and (65), there exists some ˜ N ∈ N such that for every ˜ n ≥ ˜ N both z t k ˜ n + s and z t ∗ k − s belong to the same sub-region and so the matrices W
[Equation]
∗ k − s ) ( s ∈ { 0 , 1 , 2 , · · · , k − 1 } ) are identical.
Hence, for n ≥ N , ˜ n ≥ ˜ N and j ∈ N , equation (66) becomes
[Equation]
Therefore, similar to the part ( ii ) , we can prove for every z 1 ∈ B Γ k , ∂
z T ∂θ and ∂ L T ∂θ will also remain bounded.
A.2. P ROOF OF THEOREM 2, PART ( II )
Proof.
( ii ) Let for every T >
[Equation] [Equation]
{ L T } T ∈ N ,T> 2 is a sequence of matrices L T
[Equation]
and, due to (13),
[Equation]
= ∞ .
Hence, there is at least one sub-sequence { l ( T n ) mk } T n ∈ N ,T n > 2 (for some m, k ∈ { 1 , 2 , · · · , M } ) such that lim T n →∞ l
( T n ) mk = ∞ .
On the other hand
[Equation]
Moreover, there exists some N > 2 such that (for t = T − N + 1 )
[Equation]
For θ as the k -th element of a parameter vector θ (or belonging to the k -th row of a parameter matrix θ ), the term
[Equation]
is a vector in which the i -th element is l ( T ) ik ∂ + z ∗ k,N − 1 ∂θ .
Since lim T n →∞ l ( T n ) mk = ∞ , due to (70)
[Equation]
= ∞ , which implies ∂ z ∗ T ∂θ will diverge as T → ∞ .
Similarly, by (21), we can prove ∂ L ∗ T ∂θ is divergent for T → ∞ .
By Oseledec’s multiplicative ergodic Theorem, the results also hold for every z 1 ∈ B Γ ∗ .
A.2. P ROOF OF THEOREM
Proof.
Let Γ = { z 1 , z 2 , . . .
z T , · · · } be a quasi-periodic attractor.
Then, the largest Lyapunov exponent of Γ is
[Equation]
We prove for every 0 < ϵ <
[Equation]
For this purpose
[Equation]
we show ∀ 0 < ϵ <
(I) lim T →∞ (1 − ϵ )
T − 1 < lim T →∞ (cid:13)(cid:13)(cid:13)
∂
z T ∂ z 1 (cid:13)(cid:13)(cid:13) , and (II) lim T →∞ (cid:13)(cid:13)(cid:13) ∂
z T ∂ z 1 (cid:13)(cid:13)(cid:13) <
lim T →∞ (1 + ϵ ) T −
[Equation]
Assume for the sake of contradiction that (I) does not hold.
Then there exists some 0 < ϵ < 1 such that
[Equation]
Therefore
[Equation]
and so
[Equation]
Consequently, due to (72), for T → ∞ we have ln(1 − ϵ ) ≥ 0 .
This implies ϵ ≤ 0 , which is a contradiction.
Similarly if we assume (II) is not true, then there exists some 0 < ϵ < 1 such that
[Equation]
Thereby
[Equation]
78) and thus
[Equation]
This means ln(1 + ϵ ) ≤ 0 as T → ∞ , i.e. ϵ ≤ 0 , which is a contradiction.
Therefore (14) holds for Γ and also, according to Oseledec’s multiplicative ergodic Theorem, for every z 1 in the basin of attraction of Γ .
A. A DDITIONAL RESULTS ON RELATION BETWEEN DYNAMICS AND GRADIENTS A.3. F URTHER RESULTS AND REMARKS RELATED TO T HEOREM
Remark 4.
The result of Theorem 2 also holds for unstable orbits { z 1 , z 2 , z 3 , · · · } with positive largest Lyapunov exponent.
Trivially, for such orbits that diverge to infinity (unbounded latent states) gradients of the loss function will explode as T → ∞ .
Remark 5.
For RNNs with ReLU activation functions there are finite compartments in the phase space each with a different functional form.
In such a case, to define the largest Lyapunov exponent of Γ ∗ , in the proof of Theorem 2 we assume that Γ ∗ never maps to the points of the borders.
Based on Theorem 2, we can also formulate the necessary conditions for chaos and diverging gradients in standard RNNs with particular activation functions by considering the norms of their recurrence matrix, for which the following Corollary provides the basis:
Corollary 1.
Let for a standard RNN
[Equation]
If the RNN is chaotic, then ∥ W ∥ γ > 1 .
Proof.
Assume for the sake of contradiction that ∥ W ∥ γ ≤ 1 .
From
[Equation]
it is concluded that lim T
[Equation]
2 <t ≤ T W diag (cid:0) f ′ (cid:0)
[Equation]
Bs
[Equation]
(cid:13) (cid:13) < ∞ , which contradicts (13).
This means ∥ W ∥ γ > 1 is a necessary condition for the standard RNN to be chaotic.
Remark 6.
For RNN with the tanh and sigmoid activation functions γ = 1 and γ = 14 , respectively.
Thus, by Corollary 1, the necessary conditions for chaos in these two cases are ∥ W ∥ > 1 and ∥ W ∥ > 4 , respectively.
A.3. O THER CONNECTIONS BETWEEN DYNAMICS AND GRADIENTS
As sect.
3 elucidated, there is a direct link between the norms of the Jacobians of the RNN along trajectories and the EVGP.
By observing this link, we can formulate some general conditions that will have implications for the behavior of the gradients regardless of the limiting behavior of the
RNN, as collected in the following theorem:
Theorem 4.
Let O z 1 = { z 1 , z 2 , . . .
z T , · · · } be a sequence (orbit) generated by an RNN F θ ∈ R parameterized by θ , and P
[Equation]
(i) Assume that O z 1 is an orbit for which (cid:13)(cid:13)(cid:13) ∂
+ z T ∂θ (cid:13)(cid:13)(cid:13) ≤ ξ ∀ t .
If (cid:80) ∞ T =2 ∥ J T ∥ < ∞ , then the Jacobian ∂ z T ∂ z 1 , the tangent vector ∂
z T ∂θ and thus the gradient of the loss function, ∂ L T ∂θ , will be bounded for T → ∞ .
(ii) If (cid:80) ∞
T =2 ∥ P T ∥ < ∞ , then the Jacobian ∂ z T ∂ z 1 will neither vanish nor explode as T → ∞ .
Proof.
Let ∥ .
∥ be any matrix norm satisfying ∥
A 1 A 2 ∥ ≤ ∥ A 1 ∥ ∥
A 2 ∥ .
( i )
By boundedness of
∂
+ z T ∂θ
we
have
[Equation] [Equation]
Moreover,
[Equation] [Equation]
Since (cid:80) ∞ T =2 ∥ J T ∥ converges, according to (Wedderburn, 1964), the infinite products (cid:81) ∞ T =2 (cid:0) 1 + ∥ J T ∥ (cid:1) in (83) converge to a finite number ˜
K ̸
= 0 .
Consequently, by (82) and (83)
[Equation]
84) which implies ∂ L T ∂θ will be bounded for T → ∞ .
Furthermore
[Equation] [Equation]
which completes the proof.
( ii ) Since (cid:80) ∞
T =1 ∥ P T ∥ < ∞ , due to (Wedderburn, 1964) the infinite product
[Equation]
converges to a matrix K ̸
= O , which implies
[Equation]
Part ( i ) of Theorem 4 relaxes some of the conditions required in Theorem 1 for bounded gradients by imposing a Lipschitz condition on the immediate derivatives.
Part ( ii ) generalizes conditions satisfied, for instance, in orthogonal (unitary) RNNs (Arjovsky et al., 2016; Henaff et al., 2016) or fully regularized PLRNNs (Schmidt et al., 2021).
Proposition 3.
Let O z
= { z 1 , z 2 , . . .
z T , · · · } be an orbit generated by an RNN F θ ∈ R (parameterized by θ ), and ∥ J T ∥
̸ = 0 , T ≥ 2 .
If (cid:80) ∞ T =2 ln ∥ J T ∥ diverges to −∞ , then the Jacobian ∂ z T ∂ zvanishes as T tends to infinity.
Proof.
For ∥ J T ∥ ̸ = 0 , T ≥ 2 , we have
[Equation]
Hence if (cid:80) ∞ T =2 ln ∥ J T ∥ → −∞ , then
[Equation]
A. E MPIRICAL EVALUATION : D ATASETS
Lorenz attractor
The Lorenz system (Lorenz, 1963) is a simplified model for atmospheric convection, given by
[Equation]
The system is of particular interest for its chaotic regime and was studied here for σ = 16 , ρ = 45 .and
β
= 4 .
For these parameters the Lorenz system is known to have a maximal Lyapunov exponent λ max = 1 .
(Rosenstein et al., 1993).
To generate a time series, the ODEs were integrated with a step size ∆ t = 0 .
01 using scipy.integrate .
Accordingly, the prediction time is τ pred = ln(2) ∆ t λ max = 46 . 2 .
Duffing oscillator
The Duffing oscillator (Duffing, 1918) is an example of a periodically forced oscillator with nonlinear elasticity
[Equation]
Note that this system is non-autonomous, that is externally forced due to the r.h.s.
of eqn.
The following parameters were chosen to arrive at a chaotically forced oscillator: α = 1 . 0 , β
= − 1 . 0 , δ = 0 .
1 , γ = 0 .
35 , and ω = 1 . 4 .
For these parameters the Duffing oscillator has a maximum
Lyapunov exponent of λ max = 0 . 0995 .
The dataset used here was created with the code from (Gilpin, 2021) as a three dimensional embedding with step size ∆ t = 0 .
The prediction time is τ pred = 39 .
R¨ossler system Another prime textbook example for a chaotic system is the R¨ossler system (R¨ossler, 1976) given by:
[Equation]
For the parameters a = 0 . 15 , b = 0 .
2 and c = 10 , the maximal Lyapunov exponent is λ max = 0 .
(Rosenstein et al., 1993).
To arrive at a time series, a step size of ∆ t = 0 .
1 was chosen for integration.
This gives us a prediction time of τ pred = 77 . 0 for this system.
Mackey-Glass equation The Mackey-Glass equation (Glass & Mackey, 1979) is a nonlinear time delay differential equation
[Equation]
Here x ρ represents the value of the variable x at time t − ρ (note that strictly, mathematically, this makes the system infinite-dimensional).
Choosing the parameters to be β = 2 , γ = 1 . 0 , n = 9 . 65 , and ρ = 2 . 0 , leads to chaotic behavior with a maximum Lyapunov exponent of λ max = 0 .
The dataset was created as a 10-dimensional embedding with the code from (Gilpin, 2021) using ∆ t = 0 .
This yields a prediction time of τ pred = 82 .
Empirical temperature time series This time series was recorded at the Weather Station at the
Max Planck Institute for Biogeochemistry in Jena, Germany, spanning the time period betweenand 2016, and reassembled by Franc¸ois Chollet for the book Deep Learning with Python .
The data set can be accessed at https://www.kaggle.com/pankrzysiu/weather-archive-jena.
To expose the underlying chaotic dynamics of the time series, trends and yearly cycles were removed, and nonlinear noise-reduction was performed (using ghkss from TISEAN , see also Kantz et al.
Fig.
4 (a) shows a snippet of the temperature data in comparison with the denoised time-series.
High-frequency noise was further reduced through Gaussian kernel smoothing ( σ = 200 ), and the resulting time series was sub-sampled (every 5 th data point was retained).
Fig.(b) clearly reveals a fractional dimension of D eff = 2 . 8 for the de-noised and smoothed time-series.
This strongly suggests that the dynamics governing the time series are chaotic.
We created a time delay embedding (Kantz & Schreiber, 2003) with m = 5 (estimated by the false nearest neighbor technique, see Kennel et al.
and delay τ = 500 (obtained as the first minimum of the mutual information).
The first three embedding dimensions are shown in Fig. 4(c).
The maximal Lyapunov exponent of this time series was determined with lyap r from TISEAN (Hegger et al., 1999) to be λ max = 0 .
016 , see Fig. 3(a).
This value is in close agreement with the literature (Mill ´ an et al., 2010).
The predictability time of this system is estimated to be τ pred = 43 . 3 .
All datasets used were standardized (i.e., centered with unit variance) prior to training.
A. E MPIRICAL EVALUATION : MEASURES OF RECONSTRUCTION QUALITY
Attractor overlap To asses the geometrical similarity of the chaotic attractor produced by the
RNN to the one underlying the observations, we calculate the Kullback-Leibler divergence of the ground truth distribution p true ( x ) and the distribution p gen ( x | z ) generated by RNN simulation.
To do so in practice, we employ a binning approximation (see (Koppe et al., 2019))
[Equation]
where K is the total number of bins, and ˆ p ( k ) true ( x ) and ˆ p ( k ) gen ( x | z ) are estimates obtained as relative frequencies through sampling trajectories from the observed time-series and the trained RNN, respectively.
Power-spectral correlations Since in DS reconstruction we aim to capture invariant (timeindependent) properties of the underlying system, besides the geometrical agreement, we compare the similarity in true and RNN-reconstructed power spectra.
To do so, we generate a time series of length 100 , 000 from the RNN and calculate its power spectrum using the fast Fourier transform ( scipy.fft ).
To reduce the influence of noise we apply Gaussian kernel smoothing and cut off the long high-frequency tails of the spectra.
The dimension-wise correlation between observed and generated spectra are then averaged to give the PSC
[Figure]
A. F URTHER EMPIRICAL EVALUATIONS A.6. R ECONSTRUCTION : R ¨ OSSLER S YSTEM
[Figure]
A.6. R ECONSTRUCTION : H IGH - DIMENSIONAL M ACKEY -G LASS SYSTEM
[Figure]
A.6. R ECONSTRUCTION : P ARTIALLY OBSERVED L ORENZ S YSTEM
For this evaluation we trained models only on the variables { y, z } of the Lorenz system, eqn.
In order to compute the attractor overlap ( D stsp ) in the true state space, however, after training the observation matrix B was recomputed by linearly regressing the first 10 latent states onto the first 10 observations from all three Lorenz variables in eqn.
[Figure]
A.6. O THER INITIALIZATION PROCEDURES : S TANDARD BATCHING WITH ZERO RESETTING
A common procedure in training RNNs is partitioning the time series into chunks (as we did based on the Lyapunov spectrum), but then simply resetting the hidden states at the beginning of each chunk (window) to 0 .
Formally this would mean that we do not force the trajectory back on track as in our approach, but instead may kick it off the track.
To illustrate this, here we trained an LSTM on chunks (windows) with a length given by the optimal τ ( τ opt = 30 for the Lorenz system), but then initialized the hidden states to 0 at the beginning of each window.
The performance obtained this way is indicated by the green dashed line in Fig.
9 below.
[Figure]
A. S PARSELY FORCED BPTT
Loss truncation One implicit consequence of the teacher forcing, eqn.
(16), is the interruption of the hidden-to-hidden connections at these time points.
More specifically, if the system is forced at time t ∈ T , then there is no connection between z t and z t +1 , that is [Equation]
[Equation]
To see how these vanishing Jacobians truncate the loss gradients w.r.t to some parameter θ , let us focus on the loss gradients immediately after the forcing,
[Equation]
Eqn.
(95) shows that sparsely forced BPTT implicitly truncates the loss gradients because it interrupts the hidden-to-hidden connection from z t to z t +1 for t ∈ T .
More generally, defining (cid:101) t := max { t ′ ∈ T : t ′ ≤ t } , the overall loss gradients are truncated to
[Equation]