On the difficulty of learning chaotic dynamics with RNNs
Anonymous Author(s)
AffiliationAddress email
Abstract
Recurrent neural networks (RNNs) are wide-spread machine learning tools formodeling sequential and time series data.
They are notoriously hard to train because their loss gradients backpropagated in time tend to saturate or diverge duringtraining.
This is known as the exploding and vanishing gradient problem.
Previoussolutions to this issue either built on rather complicated, purpose-engineered architectures with gated memory buffers, or - more recently - imposed constraints thatensure convergence to a fixed point or restrict (the eigenspectrum of) the recurrencematrix.
Such constraints, however, convey severe limitations on the expressivity ofthe RNN.
Essential intrinsic dynamics such as multistability or chaos are disabled.
This is inherently at disaccord with the chaotic nature of many, if not most, timeseries encountered in nature and society.
It is particularly problematic in scientificapplications where one aims to reconstruct the underlying dynamical system.
Herewe offer a comprehensive theoretical treatment of this problem by relating theloss gradients during RNN training to the Lyapunov spectrum of RNN-generatedorbits.
We mathematically prove that RNNs producing stable equilibrium or cyclicbehavior have bounded gradients, whereas the gradients of RNNs with chaoticdynamics always diverge.
Based on these analyses and insights we suggest waysof how to optimize the training process on chaotic data according to the system’s
Lyapunov spectrum, regardless of the employed RNN architecture.
Introduction
Recurrent neural networks (RNNs) are widely used across various fields in engineering and sciencefor learning sequential tasks or modeling and predicting time series [53].
Yet, they struggle whenlong-term temporal dependencies, very slow or hugely varying time scales are involved [6, 33, 51,70, 77].
Time series or sequential data with such properties are, however, very common in fieldslike climate physics [81], neuroscience [21, 72], ecology [84] or language processing [10].
Training
RNNs on such data is hard because the loss gradients backpropagated in time easily saturate ordiverge in this process.
This is commonly referred to as the exploding and vanishing gradient problem(EVGP)
One solution to the EVGP is based on specifically designed RNN architectures with gating mechanisms, such as long short-term memory (LSTM) [34] or gated recurrent units (GRU) [11].
Thesearchitectures allow states at earlier time steps to more easily influence activity much later through akind of protected memory buffer, thus alleviating the EVGP by structural design.
In practice, suchmodels need to be backed up by further techniques like gradient clipping to keep the gradients incheck [65].
The relatively complex architectural design of these networks impedes their mathematical analysis and requires reverse engineering after training [55, 61, 62, 77].
Partly to foregothese complications, a variety of other solutions has been proposed recently, imposing restrictionson the recurrence matrix to bound the gradients [4, 9], or enforcing global stability by design or regularization [18, 48].
Often these procedures dramatically curtail the expressivity of the RNN [44,64, 77]; in particular, they rule out chaotic dynamics (for reasons discussed further below).
This is at odds with the plethora of chaotic phenomena in nature, engineering, and society.
Chaoticdynamics are commonplace, almost default in any complex physical or biological system.
Thisincludes scientific areas as diverse as neuroscience [16, 90], physiology [45], geophysics [78], climatesystems [85], astrophysics [50], ecology [13], chemical reactions [20], cell [63] or population [57]biology.
Chaotic phenomena are also crucial for the understanding of societal and epidemiologicalprocesses, such as the spread of diseases [56], or in economics [19].
They are further relevant inpurely technical contexts such as electrical engineering [40, 80] or laser optics [42].
They have evenbeen suggested to play an up to now largely neglected, but potentially very significant role in speechrecognition [74] and natural language processing [35].
Hence, in almost any practical setting, chaoticphenomena abound.
They cannot, in general, be ignored when devising RNN training algorithms.
Here we offer a comprehensive theoretical treatment of the relation between RNN dynamics andthe behavior of the loss gradients during training.
We find a close connection between an RNN’sloss gradients and the largest Lyapunov exponent of its freely generated orbits.
We mathematicallyprove that RNNs producing stable fixed point or cyclic behavior have bounded gradients.
Crucially,however, the loss gradients of RNNs producing chaotic dynamics always diverge.
Hence, the chaoticnature of many time series data induces a principle problem, and, despite significant efforts in the pastto solve the EVGP, training RNNs on such data remains an open issue.
We illustrate the implicationsof our theory for RNN training on several simulated and empirical chaotic time series, and adapt theold idea of sparsely forced BPTT as a simple yet effective remedy that enables to learn the underlyingdynamics despite exploding gradients.
Related works
Exploding and vanishing gradients.
While ‘classical’ remedies of the EVGP [6, 33, 65] rest onpurpose-tailored architectures with gating mechanisms, which safeguard information flow acrosslonger temporal distances [11, 34], the focus has recently shifted to simpler RNNs that address the
EVGP by restricting the recurrence matrix to be orthogonal [30, 31, 36], unitary [4], or antisymmetric[9], or by ensuring globally stable fixed point solutions [37, 39], for example through co-trained
Lyapunov functions [48].
However, all these approaches impose strong limitations on the dynamicalrepertoire of the RNN, enforcing global convergence to fixed points or simple cycles.
In doingso, they drastically reduce the expressiveness of these models [44, 64].
To address this problem,
Erichson et al.
[18] somewhat relaxed the constraints on the recurrence matrix by introducing a skewsymmetric decomposition combined with a Lipschitz condition on the activation function.
Anotherrecent approach discretizes oscillator ODEs to arrive at a stable system of coupled [70] or independent[71] oscillators which increase the RNN’s expressiveness while bounding its gradients.
By design(and as acknowledged by the authors), neither of these architectures is capable of producing chaoticdynamics, however, as the underlying ODEs do not allow for exponential divergence of close-bytrajectories (a prerequisite for chaos).
Given these often principle limitations of parametrically ordynamically strongly constrained models, a fruitful direction may be to modify the training processitself, e.g. through modified or auxiliary loss functions [77, 82], or special procedures for parameterupdating [38] or loss truncation [59, 92].
Our empirical evaluation will follow up on such ideas, butalso highlight that simple loss truncation, windowing, or architectural solutions like LSTMs are notsufficient.
Learning dynamical systems.
Surprisingly disconnected from the work on the EVGP and learninglong-term dependencies, a huge and long-standing literature deals with training RNNs on nonlineardynamical systems (DS) [67, 83, 87], including chaotic systems like the famous Lorenz equations[54] or chaotic turbulence in fluid dynamics [52].
Teacher forcing (TF; [12, 67, 92], see also [27]) isone of the earliest techniques introduced to keep RNN trajectories on track while training.
The ideabehind TF is to simply replace RNN states by observations when available, thereby also effectivelycutting off the gradients.
TF essentially derives from ideas in dynamical control theory, and adaptiveschemes that increasingly hand over control to the RNN throughout training have been devised[1, 2, 5].
A related technique from the control theory literature is “multiple shooting” [89]: Herethe whole observed time series is chopped into chunks, and for each chunk of trajectory a newinitial condition is estimated.
Explicit constraints ensure continuity between the separate trajectory bits during optimization.
State space models and the Expectation-Maximization algorithm becamepopular particularly in the 90es for uncovering the latent dynamics underlying a set of time seriesobservations [22], and remain an important tool until today [15, 49].
Most recently, approaches basedon variational inference and the reparameterization trick [47], like sequential variational autoencoders(SVAE), gained in popularity for DS approximation [8, 32]. “Deterministic” RNNs (i.e., with latentstates not treated as random variables), like conventional LSTMs [86], remain top choices for DSreconstruction, however.
Although connections between DS ideas and loss gradients have been drawn early on [6], so faronly particular scenarios (like fixed point attractors) have been considered.
Closest to our workis recent work by Schmidt et al.
[77], where non-divergence of loss gradients is established when
RNNs converge to fixed points or cycles.
However, this was done only for the particular classof piecewise-linear RNNs (PLRNNs), more restrictive conditions for cycles were imposed thanassumed here, and - most importantly - the chaotic case on which we focus here was not considered.
Recent studies [17, 88] also point out the general connections between Lyapunov exponents and lossgradients that we develop in sect.
3.1, but do not provide any in-depth theoretical treatment, proofs,or empirical evaluation of methods to alleviate exploding gradients in training, as we do here.
Thus, asystematic theoretical framework that relates RNN dynamics more generally, and across a range ofdifferent RNN architectures, to the behavior of its training gradients, is still lacking so far.
Theoretical analysis: Relation between RNN dynamics and loss gradients
In our analysis, we will cover all major types of system dynamics (fixed points, cycles, chaos, andquasi-periodicity), and mathematically investigate their implications for the loss gradients.
We willdo this for all major classes of RNNs, including standard RNNs with largely arbitrary activationfunction, LSTMs, GRUs, and PLRNNs.
The next section will first develop and illustrate the basicintuition behind the relations between RNN dynamics and loss gradients.
3.1 Preliminaries: RNN dynamics and loss gradients
Formally, all popular RNN architectures, including LSTMs, GRUs, or PLRNNs, are discrete time DS,defined by a (first-order-Markovian) recursive prescription for the temporal evolution of the latentstates z t ∈ R M of the form
[Equation]
where s t ∈ R N is the input at time t and θ are RNN parameters.
For instance, for standard RNNs wehave F θ ( z t − 1 , s t ) =
f (cid:0) W z t − 1 + Bs t + h (cid:1) , where f is an element-wise activation function liketanh or a rectified linear unit (ReLU).
Assuming we start at some initial value z 1 ∈ R M , and given a sequence of external inputs S = { s t } ,we can recursively rewrite eq.
(1) as
[Equation]
In DS theory, we characterize the long-term behavior of such sequences by its spectrum of Lyapunovexponents.
The Lyapunov exponents estimate the exponential growth rates in different local directionsof the system’s state space, and the largest Lyapunov exponent gives the dominant exponentialbehavior.
Let us denote the system’s Jacobian at time t by
[Equation]
Then, the maximal Lyapunov exponent along an RNN trajectory { z 1 , z 2 , · · · , z T , · · · } is defined as
[Equation]
where ∥ · ∥ denotes the spectral norm (or any subordinate norm) of a matrix.
If λ max < 0 nearby 130 trajectories will ultimately converge to a fixed point or cycle, while for λ max > 0 (a necessarycondition for chaos) initially nearby trajectories will exponentially separate, i.e. we will have divergence along one (or more) directions in state space.
This accounts for the sensitive dependenceon initial conditions in chaotic systems.
Now let L ( W , B , h ) be some loss function employed for RNN training that decomposes in timeas L
= (cid:80) Tt =
1 L t .
Suppose we fancy BPTT as our training algorithm (similar derivations could beperformed for RTRL), we recursively develop the loss gradients w.r.t.
some RNN parameter θ in time(i.e.
, across layers of the RNN unrolled in time) as
[Equation]
with
[Equation]
and
[Equation]
where ∂ + denotes the immediate derivative.
Now observe that the behavior of the loss gradientscrucially depends on the product series of Jacobians in eqn.
(6): If the maximum absolute eigenvaluesof the Jacobians J t will, in the geometric mean, be larger than 1 (i.e., (cid:13) (cid:13)(cid:13)(cid:81) T {+− 2 r =
+}
J T − r (cid:13)(cid:13)(cid:13) 1 /T > 1 ),gradients will explode as T → ∞ , while they will saturate if (cid:13)(cid:13)(cid:13)(cid:81)
[Equation]
[-1 /T > 1 ),gradients will explode as T → ∞ , while they will saturate if (cid:13)(cid:13)(cid:13)(cid:81) T − 2 r =
-]<
[Equation]
Thus, the keypoint to note is that the same terms that occur in the definition of the Lyapunov spectrum, eqn.
(4),resurface in the loss gradients, eqn.
This accounts for the tight links between systemdynamics and gradients.
3.2 Fixed points and cyclic dynamics
Let us start by considering the simplest types of dynamics that can occur in RNNs (or any discretetime DS): fixed points and cycles.
In fact, by far most of the literature on global stability in RNNsand on loss gradients focused on just fixed points [9, 18, 48], with only few authors who recentlystarted to also connect cyclic behavior to loss gradients [70, 77].
Recall that a fixed point of arecursive map z t = F ( z t − 1 ) is defined as a point z ∗ for which we have z ∗ =
F ( z ∗ ) .
Likewise,a k -cycle ( k > 1 ) is a set of temporally consecutive periodic points P k :=
{ z t 1 , z t 2 , . . .
, z t k } =
{ z t 1 , F ( z t 1 ) , . . .
, F k − 1 ( z t 1 ) } that we obtain from recursive application of the map such that each ofthe cyclic points z t r ∈
P k is a fixed point of the k times iterated map F k (with k being the smallestpositive integer for which this holds).
To simplify the subsequent treatment, we will collectively referto fixed points and cycles as k -cycles ( k ≥ 1 ).
Further recall that a fixed point or k -cycle is calledstable if the maximum absolute eigenvalue of the Jacobian evaluated at that point is smaller than 1 ,neutrally stable if exactly 1 , and unstable otherwise.
Although the results we develop in this and thefollowing sections will hold more widely, we will restrict our attention to recursive maps F θ from theclass of RNNs R = { standardRNN , LSTM , GRU , PLRNN } (see Appx.
A.1 for details).
Based on the observations made in the previous sections we can state the following theorem that links
RNN dynamics and loss gradients:
Theorem 1.
Consider an RNN F θ ∈ R parameterized by θ , and assume that it converges to a stablefixed point or k -cycle Γ k ( k ≥ 1) with B Γ k as its basin of attraction.
Then for every z 1 ∈ B Γ k (i)the Jacobian ∂ z T ∂ z 1 exponentially vanishes as T → ∞ ; (ii) for Γ k
the tangent vectors ∂ z T ∂θ and thusthe gradient of the loss function, ∂ L T ∂ θ , will be bounded from above, i.e. will not diverge for T → ∞ ;and (iii) for the PLRNN (27) both (cid:13)(cid:13)
∂
z T ∂θ (cid:13)(cid:13) and (cid:13)(cid:13) ∂
L T ∂ θ (cid:13)(cid:13) will remain bounded for every z 1 ∈ B Γ k as
T → ∞ .
Proof.
( i ) Assume that Γ k is a stable k -cycle ( k ≥ 1) denoted by
[Equation]
Then, the largest Lyapunov exponent of Γ k is given by
[Equation]
By assumption of stability of Γ k we have λ Γ k < 0 and also ρ (cid:0) (cid:81)
k − 1 s =
J t ∗ k − s (cid:1) < 1 , which implies
[Equation]
Now suppose that O z 1 is an orbit of (1) converging to Γ k , i.e. z 1 ∈ B Γ k .
Since O z 1 and Γ k have the 173 same largest Lyapunov exponent, we have
[Equation]
and hence for z 1 ∈ B Γ k
[Equation]
176 ( ii ) & ( iii ) See Appx.
A.2.1.
Remark 1.
The result of Theorem 1 part ( i ) will be generally true for any first-order-Markovian map (1), but the conclusions in part ( ii ) may hinge on its specific definition.
Remark 2.
None of the results above and throughout sect.
3 require the dynamics to be autonomous,the theory applies whether there is external input or not.
In fact, mathematically, non-autonomous(externally forced) systems can always be rewritten as autonomous dynamical systems [3, 68, 94],see Appx.
A.1.1 for details.
The results above ensure that loss gradients will not diverge (explode) as T → ∞ in RNNs that are“well-behaved” in the sense that they converge to a fixed point or cycle.
This is a generalization ofthe results given in Theorem 1 in Schmidt et al.
[77], where this was shown only a) for the specificclass of PLRNNs and b) for specific constraints imposed on the eigenvalue spectrum of the RNN’s
Jacobians which were relaxed in our theorem above.
While our treatment above is centered on the “exploding-gradients” case, various architecturalmodifications or regularization techniques can ensure that gradients do not vanish either, i.e. remainbounded from below as well.
This was established, for instance, in Schmidt et al.
[77] for PLRNNsusing ’manifold attractor regularization’.
In Appx.
A.2.1 we show that the results from Theorem 2from Schmidt et al.
[77] on doubly bounded (from below and above) loss gradients can indeed beextended to the more general case covered by Theorem 1 above.
3.3 Chaotic dynamics
We will now consider the all-important chaotic case.
Let F be a recursive map and O z 1 ={ z 1 , z 2 , z 3 , · · · } be an orbit of F .
The orbit is chaotic if (i) it is not asymptotically periodic and 197 (ii) has at least one positive Lyapunov exponent [25, 58].
If the system’s invariant set is bounded ,condition (ii) is considered a standard signature of chaos, as in this case two nearby orbits separateexponentially fast, but at the same time their mutual separation cannot go to infinity so that there arealso folds.
The following theorem states the sufficient condition for exploding gradients:
Theorem 2.
Suppose that an RNN F θ ∈ R (parameterized by θ ) has a chaotic attractor Γ ∗ with B Γ ∗ 202 as its basin of attraction.
Then, for almost every orbit with z∈ B Γ ∗ , (i) the Jacobians connectingtemporally distal states z T and z t ( T ≫ t ), ∂ z T ∂ z t , will exponentially explode for T → ∞ , and (ii) thetangent vector ∂
z T ∂θ
and so the gradients of the loss function, ∂ L T ∂θ , will diverge as T → ∞ .
Proof.
Let the RNN F θ ∈ R have a chaotic orbit denoted by Γ ∗ =
{ z ∗ 1 , z ∗ 2 , · · · , z ∗ T , · · · } .
Then, 206 denoting by J ∗
T the Jacobian of (1) at z ∗ T ∈ Γ ∗ , the largest Lyapunov exponent of Γ ∗ is given by
[Equation]
Since Γ ∗ is chaotic, so λ > 0 .
Hence, from (12), it is concluded that
[Equation]
Now, according to Oseledec’s multiplicative ergodic Theorem, almost all the points in the basin ofattraction of Γ ∗ have the same largest Lyapunov exponent λ .
Thus, (13) holds for almost everyz 1 ∈ B Γ ∗ .
( ii ) See Appx.
A.2.2.
Remark 3.
The first part of Theorem 2 holds for all first-order-Markovian recursive maps (1).
Notethat for LSTMs, ∂ z T ∂ z t ( z :=
( h , c ) T ) denotes the full Jacobian of both hidden and cell states.
We collect some further mathematical results and remarks related to Theorem 2 in Appx.
A.3.1.
Hence, the essential result is that for all popular RNNs R and activation functions, loss gradients willinevitably diverge if the RNN latent states converge to a chaotic attractor.
3.4 Quasi-periodicity
Quasi-periodicity is a long-term behavior which occurs on a torus and, superficially, bears somesimilarity to chaos in the sense that, strictly speaking, orbits are also aperiodic .
That is, as T → ∞ ,trajectories will never close up with themselves.
Moreover, every trajectory becomes arbitrarilyclose to any point on the torus, that is, it is dense.
One important difference between quasi-periodicand chaotic systems is, however, that in a quasi-periodic system, as time passes, two close initialconditions are linearly diverging, while in a chaotic system the divergence is exponential.
Theorem 3.
Assume that an RNN F θ ∈ R (parameterized by θ ) has a quasi-periodic attractor Γwith B Γ as its basin of attraction.
Then, for every z 1 ∈ B Γ
[Equation]
Proof.
See Appx.
A.2.3.
According to Theorem 3, for every orbit converging to a quasi-periodic attractor, the Jacobians ∂ z T ∂ z tmay diverge or vanish as T → ∞ , but this will not occur exponentially fast as T → ∞ .
Thus, even 229 for bounded non-chaotic RNNs we may sometimes stumble into the problem of diverging gradients.
Although this may be a less common scenario, we point out it may occur if we train RNNs on real datafrom oscillatory systems with incommensurate frequencies, as for instance encountered in electronicengineering.
In Appx.
A.3.2 we have collected further mathematical results on the connection between RNNdynamics and loss gradients that hold regardless of the RNN’s limiting behavior.
Empirical evaluation
Our theoretical results imply that chaotic time series pose a principle challenge for RNN training thatcannot easily be circumvented through specifically designed architectures, constraints, or regularization criteria.
If the underlying DS we aim to capture is chaotic, loss gradients propagated back intime will inevitably explode.
Hence we need to curtail gradients in an ideal way.
The issue arisesespecially in scientific ML where time series from chaotic systems are ubiquitous and the aim is toreconstruct the generating DS with its limiting behavior.
Thus, our exposition will focus on this area.
4.1 Training on systems with exploding gradients by sparse teacher forcing
To illustrate the connections between theory and RNN training, we revive the old idea of TF [92] asa mechanism for truncating error gradients while training.
However, we would like to do this suchthat important information about the system dynamics does not get lost, for which Lyapunov theoryoffers some guidance.
Specifically, we should not force the system back onto the true trajectoryall or most of the time (as in “classical TF”), but should effectively “re-calibrate” it only at certaintime points chosen wisely according to the system’s local divergence rates.
This procedure will bereferred to as sparsely forced BPTT in the following.
Assume we want to train an RNN with hiddenstates z t ∈ R M and linear (or affine) output layer on a time-series { x 1 , x 2 , · · · , x T } generated by a 251 chaotic system.
The linear output layer ˆ x t =
Bz t , B ∈ R N × M , maps the RNN hidden states intothe observation space.
This allows us to modify the original TF procedure by constructing a controlseries { ˜ z 1 , ˜ z 2 , · · · , ˜ z T } from the observations by “inverting” the linear output mapping
[Equation]
The idea is to supply this control signal only sparsely, separated by the learning interval τ betweenconsecutive forcings.
Hence, defining T = { nτ + 1 } n ∈ N 0 as the set of all time points at which we 256 force the RNN onto the ‘true’ values, the RNN updates can be written as
[Equation]
This forcing is applied after calculation of the loss, such that L t =
∥ x t − Bz t ∥ 22 irrespective of 258 whether t is in
T or not (and of course it is applied only during training, not at test time!).
Replacinghidden states z t with their teacher-forced signals ˜
z t simply breaks divergence between true andpredicted trajectories at time points t ∈ T , and also cuts off the Jacobians by breaking the temporalcontingency (for details see Appx.
A.7).
The learning interval τ hence controls how many time stepsare included in the gradient calculation and has to be chosen with care such as to balance the effectsof exploding gradients vs. those of losing relevant time scales and long-term dependencies.
Whileit is general wisdom that an optimal batch size will facilitate training, the point here is thus muchmore specific: Ideally τ should be chosen in accordance with the system’s
Lyapunov spectrum, forinstance based on the predictability time [7]
[Equation]
We emphasize that such a simple recipe for addressing the exploding gradient problem is based onmodifying the training routine, and is thus in principle applicable to any model architecture.
4.2 Example 1: Lorenz system and externally forced Duffing oscillator in chaotic regime
Let us illustrate these ideas on two classical textbook examples of chaotic DS, the chaotic Lorenzattractor as an autonomous system, and the chaotically forced Duffing oscillator as an examplewith explicit external input (see Appx.
A.4 for details).
Trajectories were repeatedly drawn fromthese systems, on which we trained a PLRNN, a vanilla RNN with tanh activation function, and a
LSTM by stochastic gradient descent (SGD) to minimize the MSE loss between predicted and actualobservations.
As optimizer we used Adam [46] from PyTorch [66] with a learning rate of 0 .
Forall models, training proceeded solely by sparsely forced BPTT and did not employ gradient clippingor any other technique that may interfere with optimal loss truncation.
In nonlinear DS reconstruction, we are mainly interested in reproducing invariant properties of theunderlying system such as the attractor geometry (or topology; [75, 79]) or the frequency composition(i.e.
, time-averaged properties), while measures like ahead-prediction errors are less meaningfulespecially on chaotic time series [49, 93].
Thus, in evaluating training performance, here we follow
Koppe et al.
[49] in using a Kullback-Leibler divergence D stsp to quantify the agreement betweenobserved and generated probability distributions across state-space to asses the overlap in attractorgeometry (Appx. A.5).
Moreover, we calculate the dimension-wise Hellinger distance D H between
[Figure]
power spectra to quantify the temporal agreement of the observed and generated time-series (Appx. A.5).
Fig.
1 shows the dependence of the reconstruction quality on the learning interval τ for all RNNarchitectures on (a) the Lorenz and (b) the externally forced Duffing system.
Fig.
2 provides particularexamples of reconstructions for τ chosen too small, too large, or about right.
For all models we find a system-dependent range for the optimal learning interval that agrees wellwith the predictability time defined in eqn.
(17), where estimates for the maximal Lyapunov exponentwere taken from the literature [23, 69].
As a reference, dashed lines represent the reconstructionperformance for all architectures when trained with classical BPTT and gradient clipping.
Thetraining procedure was the same as for sparsely forced BPTT, except that instead of supplying acontrol-signal, gradients were normalized to 1 prior to each parameter update.
As evidenced by themuch worse performance, gradient clipping does not effectively address the EVGP, even for LSTMs .
As further shown in Fig.
9 in Appx.
A.6.4, using the optimal window length τ but resetting the initialcondition to zero (instead of its control value ˜
z t ) for each chunk equally destroys performance.
Thissuggests that neither mere gradient normalization nor simple windowing are sufficient, but will wipeout essential information about the dynamics.
In Appx.
A.6 we collect further results on the chaotic Rössler attractor (Fig. 5), high-dimensional
Mackey-Glass equations (Fig. 7), and the Lorenz attractor with partial observations (Fig.
4.3 Example 2: Chaotic weather data
As for an empirical example, we trained all RNNs (vanilla RNN, PLRNN, LSTM) on a temperaturetime series recorded at the Weather Station at the Max Planck Institute for Biogeochemistry in Jena, [Figure]
Germany.
To expose the chaotic behavior and obtain a robust estimate of the maximal Lyapunovexponent, trends and yearly cycles were removed, and nonlinear noise-reduction was performed ([42];
Appx.
A.4).
The maximal Lyapunov exponent was determined with the TISEAN package [29], asshown in Figure 3 (a).
The value obtained is in close agreement with the literature [60].
Figure 3 shows that also for these empirical data the optimal training interval τ agrees well withthe predictability time, eqn.
(17), for all trained RNNs.
Furthermore, as was the case for the DSbenchmarks, gradient clipping was not able to satisfactorily tackle the EVGP, even when paired witharchitectures like LSTMs explicitly designed for alleviating this problem.
Similar results are reportedfor another real-world dataset, electroencephalogram (EEG) recordings, in Appx.
A.11.
Discussion and conclusions
In this paper we proved that RNN dynamics and loss gradients are intimately related for all majortypes of RNNs and activation functions.
If the RNN is “well behaved” in the sense that its dynamicsconverges to a fixed point or cycle, loss gradients will remain bounded, and established remedies[34, 77] can be used to refrain them from vanishing.
However, if the dynamics are chaotic, gradientswill always explode.
This constitutes a principle problem in RNN training that cannot easily bemastered through architectural design or gradient clipping.
It is furthermore a practically highlyrelevant one, as most time series we encounter in nature, and many from man-made systems as well,are inherently chaotic.
While we do not offer a full solution to this problem here, we suggest it mightbe tackled in training by taking a system’s local divergence rates as measured through the Lyapunovspectrum into account.
Hence, rather than conquering the EVGP by structural design or specificconstraints or regularization terms, we recommend to put the focus more on the training processitself.
We illustrated this point empirically using sparsely forced BPTT , a training technique that pullstrajectories back on track at times determined by the maximal Lyapunov exponent.
Doing so leads tooptimal reconstruction results for a variety of simulated and real-world benchmarks, regardless of thespecific RNN architecture employed in training.
We stress that our goal here above all was to provide a mathematically grounded perspective onthe problem, with the empirical section focused on elucidating the practical implications of thetheoretical results.
We believe that a more thorough theoretical understanding is important andneeded for guiding future research into more powerful training procedures that avoid explodinggradients without compromising expressiveness.
In our application examples, we developed the casefrom the perspective of scientific machine learning, which by now is a broad area in its own right withhuge societal relevance (e.g., climate or epidemiological time series), and where the reconstruction ofgeometrical or topological (invariant) properties is important, beyond mere prediction.
Nevertheless,we believe that our theoretical results will also have implications for other domains, like NLP [35].
While scientific time series problems traditionally have been extensively considered from a DSperspective (e.g., [41]), much more groundwork is needed, however, in areas like NLP, where, forinstance, it may not even be immediately clear how to best define a Lyapunov spectrum.