O N THE C URSE OF M EMORY IN R ECURRENT N EURAL N ETWORKS : A PPROXIMATION AND O PTIMIZATION A NALYSIS
A BSTRACT
We study the approximation properties and optimization dynamics of recurrent neural networks (RNNs) when applied to learn input-output relationships in temporal data.
We consider the simple but representative setting of using continuoustime linear RNNs to learn from data generated by linear relationships.
Mathematically, the latter can be understood as a sequence of linear functionals.
We prove a universal approximation theorem of such linear functionals and characterize the approximation rate.
Moreover, we perform a ﬁne-grained dynamical analysis of training linear RNNs by gradient methods.
A unifying theme uncovered is the non-trivial effect of memory, a notion that can be made precise in our framework, on both approximation and optimization: when there is long-term memory in the target, it takes a large number of neurons to approximate it.
Moreover, the training process will suffer from slow downs.
In particular, both of these effects become exponentially more pronounced with increasing memory - a phenomenon we call the “curse of memory”.
These analyses represent a basic step towards a concrete mathematical understanding of new phenomenons that may arise in learning temporal relationships using recurrent architectures.
I NTRODUCTION
Recurrent neural networks (RNNs) (Rumelhart et al., 1986) are among the most frequently employed methods to build machine learning models on temporal data.
Despite its ubiquitous application (Baldi et al., 1999; Graves & Schmidhuber, 2009; Graves, 2013; Graves et al., 2013; Graves &
Jaitly, 2014; Gregor et al., 2015), some fundamental theoretical questions remain to be answered.
These come in several ﬂavors.
First, one may pose the approximation problem, which asks what kind of temporal input-output relationships can RNNs model to an arbitrary precision.
Second, one may also consider the optimization problem, which concerns the dynamics of training (say, by gradient descent) the RNN.
While such questions can be posed for any machine learning model, the crux of the problem for RNNs is how the recurrent structure of the model and the dynamical nature of the data shape the answers to these problems.
For example, it is often observed that when there are long-term dependencies in the data (Bengio et al., 1994; Hochreiter et al., 2001), RNNs may encounter problems in learning, but such statements have rarely been put on precise footing.
In this paper, we make a step in this direction by studying the approximation and optimization properties of RNNs.
Compared with the static feed-forward setting, the key distinguishing feature here is the presence of temporal dynamics in terms of both the recurrent architectures in the model and the dynamical structures in the data.
Hence, to understand the inﬂuence of dynamics on learning is of fundamental importance.
As is often the case, the key effects of dynamics can already be revealed in the simplest linear setting.
For this reason, we will focus our analysis on linear RNNs, i.e. those with linear activations.
Further, we will employ a continuous-time analysis initially studied in the context of feed-forward architectures (E, 2017; Haber & Ruthotto, 2017; Li et al., 2017) and recently in recurrent settings (Ceni et al., 2019; Chang et al., 2019; Lim, 2020; Sherstinsky, 2018; Niu et al., 2019; Herrera et al., 2020; Rubanova et al., 2019) and idealize the RNN as a continuous-time dynamical system.
This allows us to phrase the problems under investigation in convenient analytical settings that accentuates the effect of dynamics.
In this case, the RNNs serve to approximate relationships represented by sequences of linear functionals.
On ﬁrst look the setting appears to be simple, but we show that it yields representative results that underlie key differences in the dynamical setting as opposed to static supervised learning problems.
In fact, we show that memory, which can be made precise by the decay rates of the target linear functionals, can affect both approximation rates and optimization dynamics in a non-trivial way.
Our main results are:
We give a systematic analysis of the approximation of linear functionals by continuoustime linear RNNs, including a precise characterization of the approximation rates in terms of regularity and memory of the target functional.
We give a ﬁne-grained analysis of the optimization dynamics when training linear RNNs, and show that the training efﬁciency is adversely affected by the presence of long-term memory.
These results together paint a comprehensive picture of the interaction of learning and dynamics, and makes concrete the heuristic observations that the presence of long-term memory affects RNN learning in a negative manner (Bengio et al., 1994; Hochreiter et al., 2001).
In particular, mirroring the classical curse of dimensionality (Bellman, 1957), we introduce the concept of the curse of memory that captures the new phenomena that arises from learning temporal relationships: when there is long-term memory in the data, one requires an exponentially large number of neurons for approximation, and the learning dynamics suffers from exponential slow downs.
These results form a basic step towards a mathematical understanding of the recurrent structure and its effects on learning from temporal data.
R ELATED W ORK
We will discuss related work on RNNs on three fronts concerning the central results in this paper, namely approximation theory, optimization analysis and the role of memory in learning.
A number of universal approximation results for RNNs have been obtained in discrete (Matthews, 1993;
Doya, 1993; Sch ¨ afer & Zimmermann, 2006; 2007) and continuous time (Funahashi & Nakamura, 1993; Chow & Xiao-Dong Li, 2000; Li et al., 2005; Maass et al., 2007; Nakamura & Nakagawa, 2009).
Most of these focus on the case where the target relationship is generated from a hidden dynamical system in the form of difference or differential equations.
The formulation of functional approximation here is more general, albeit our results are currently limited to the linear setting.
Nevertheless, this is already sufﬁcient to reveal new phenomena involving the interaction of learning and dynamics.
This will be especially apparent when we discuss approximation rates and optimization dynamics.
We also note that the functional/operator approximation using neural networks has been explored in Chen & Chen (1993); Tianping Chen & Hong Chen (1995);
Lu et al.
(2019) for nonrecurrent structures and reservoir systems for which approximation results similar to random feature models are derived (Gonon et al., 2020).
The main difference here is that we explicitly study the effect of memory in target functionals on learning using recurrent structures.
On the optimization side, there are a number of recent results concerning the training of RNNs using gradient methods, and they are mostly positive in the sense that trainability is proved under speciﬁc settings.
These include recovering linear dynamics (Hardt et al., 2018) or training in overparameterized settings (Allen-Zhu et al., 2019).
Here, our result concerns the general setting of learning linear functionals that need not come from some underlying differential/difference equations, and is also away from the over-parameterized regime.
In our case, we discover on the contrary that training can become very difﬁcult even in the linear case, and this can be understood in a quantitative way, in relation to long-term memory in the target functionals.
This points to the practical literature in relation to memory and learning.
The dynamical analysis here puts the ubiquitous but heuristic observations - that long-term memory negatively impacts training efﬁciency (Bengio et al., 1994; Hochreiter et al., 2001) - on concrete theoretical footing, at least in idealized settings.
This may serve to justify or improve current heuristic methods (Tseng et al., 2016; Dieng et al., 2017; Trinh et al., 2018) developed in applications to deal with the difﬁculty in training with long-term memory.
At the same time, we also complement general results on “vanishing and explosion of gradients” (Pascanu et al., 2013; Hanin & Rolnick, 2018; Hanin, 2018) that are typically restricted to initialization settings with more precise characterizations in the dynamical regime during training.
The long range dependency within temporal data has been studied for a long time in the time series literature, although its effect on learning input-output relationships is rarely covered.
For example, the Hurst exponent (Hurst, 1951) is often used as a measure of long-term memory in time series, e.g. fractional Brownian motion (Mandelbrot & Ness, 1968).
In contrast with the setting in this paper where memory involves the dependence of the output time series on the input, the Hurst exponent measures temporal variations and dependence within the input time series itself.
Much of the time series literature investigates statistical properties and estimation methods of data with long range dependence (Samorodnitsky, 2006; Taqqu et al., 1995; Beran, 1992; Doukhan et al., 2003).
One can also combine these classic statistical methodologies with the RNN-like architectures to design hybrid models with various applications (Loukas & Oke, 2007; Diaconescu, 2008; Mohan &
Gaitonde, 2018; Bukhari et al., 2020).
P ROBLEM F ORMULATION
The basic problem of supervised learning on time series data is to learn a mapping from an input temporal sequence to an output sequence.
Formally, one can think of the output at each time as being produced from the input via an unknown function that depends on the entire input sequence, or at least up to the time at which the prediction is made.
In the discrete-time case, one can write the data generation process
[Equation]
where x k , y k denote respectively the input data and output response, and { H k : k = 0 , 1 , . . .
} is a sequence of ground truth functions of increasing input dimension accounting for temporal evolution.
The goal of supervised learning is to learn an approximation of the sequence of functions { H k } given observation data.
Recurrent neural networks (RNN) (Rumelhart et al., 1986) gives a natural way to parameterize such a sequence of functions.
In the simplest case, the one-layer RNN is given by
[Equation]
Here, { h k } are the hidden/latent states and its evolution is governed by a recursive application of a feed-forward layer with activation σ , and ˆ y k is called the observation or readout.
We omit the bias term here and only consider a linear readout or output layer.
For each time step k , the mapping { x 0 , . . .
, x k − 1 } (cid:55)→ ˆ y k parameterizes a function ˆ H k ( · ) through adjustable parameters ( c, W, U ) .
Hence, for a particular choice of these parameters, a sequence of functions { ˆ H k } is constructed at the same time.
To understand the working principles of RNNs, we need to characterize how { ˆ H k } approximates { H k } .
The model (2) is not easy to analyze due to its discrete iterative nature.
Hence, here we employ a continuous-time idealization that replaces the time-step index k by a continuous time parameter t .
This allows us to employ a large variety of continuum analysis tools to gain insights to the learning problem.
Let us now introduce this framework.
Continuous-time formulation.
Consider a sequence of inputs indexed by a real-valued variable t ∈ R instead of a discrete variable k considered previously.
Concretely, we consider the input space
[Equation]
which is the linear space of continuous functions from R (time) to R d that vanishes at inﬁnity.
Here d is the dimension of each point in the time series.
We denote an element in X by x := { x t ∈ R d : t ∈ R } and equip X with the supremum norm (cid:107) x (cid:107) X :=
sup t ∈ R (cid:107) x t (cid:107) ∞ .
For the space of outputs we will take a scalar time series, i.e. the space of bounded continuous functions from R to R :
[Equation]
This is due to the fact that vector-valued outputs can be handled by considering each output separately.
In continuous time, the target relationship (ground truth) to be learned is
[Equation]
where for each t ∈ R , H t is a functional H t : X → R .
Correspondingly, we deﬁne a continuous version of (2) as a hypothesis space to model continuous-time functionals
[Equation]
whose Euler discretization corresponds to a discrete-time residual RNN.
The dynamics then naturally deﬁnes a sequences of functionals { ˆ H t ( x ) =
ˆ y t : t ∈ R } , which can be used to approximate the target functionals { H t } via adjusting ( c, W, U ) .
Linear RNNs in continuous time.
In this paper we mainly investigate the approximation and optimization property of linear RNNs, which already reveals the essential effect of dynamics.
The linear RNN obeys (6) with σ being the identity map.
Notice that in the theoretical setup, the initial time of the system goes back to −∞ with lim t →−∞ x t = 0 , ∀ x ∈ X , thus by linearity ( H t ( 0 ) = 0 ) we specify the initial condition of the hidden state h
−∞ = 0 for consistency.
In this case, (6) has the following solution
[Equation]
Since we will investigate uniform approximations over large time intervals, we will consider stable
RNNs, where W ∈ W m with
[Equation]
Owing to the representation of solutions in (7), the linear RNN deﬁnes a family of functionals
[Equation]
Here, m is the width of the network and controls the complexity of the hypothesis space.
Clearly, the family of functionals the RNN can represent is not arbitrary, and must possess some structure.
Let us now introduce some deﬁnitions of functionals that makes these structures precise.
Deﬁnition 3.1.
Let { H t : t ∈ R } be a sequence of functionals.
H t is causal if it does not depend on future values of the input: for every pair of x
, x (cid:48) ∈
X such that x s = x (cid:48) s for all s ≤ t , we have H t ( x ) =
H t ( x (cid:48) ) .
H t is linear and continuous if H t ( λ x + λ (cid:48)
x (cid:48) )
=
λH t ( x ) + λ (cid:48)
H t ( x (cid:48) ) for any x , x (cid:48) ∈ X and λ, λ (cid:48) ∈ R , and sup x ∈X , (cid:107) x (cid:107) X ≤ 1 | H t ( x ) | < ∞ , in which case the induced norm can be deﬁned as (cid:107) H t (cid:107) := sup x ∈X , (cid:107) x (cid:107) X ≤ 1 | H t ( x ) | .
H t is regular if for any sequence { x ( n ) ∈ X : n ∈ N } such that x ( n ) t → 0 for Lebesgue almost every t ∈ R , lim n →∞ H t ( x ( n ))
= 0 .
{ H t : t ∈ R } is time-homogeneous if H t ( x ) =
H t + τ ( x ( τ )) for any t, τ ∈ R , where x ( τ ) s = x
s − τ for all s ∈ R ,
i
e. x ( τ ) is x whose time index is shifted to the right by τ .
Linear, continuous and causal functionals are common deﬁnitions.
One can think of regular functionals as those that are not determined by values of the inputs on an arbitrarily small time interval, e.g. an inﬁnitely thin spike input.
Time-homogeneous functionals, on the other hand, are those where there is no special reference point in time: if the time index of both the input sequence and the functional are shifted in coordination, the output value remains the same.
Given these deﬁnitions, the following observation can be veriﬁed directly and its proof is immediate and hence omitted.
Proposition 3.1.
Let { ˆ H t : t ∈ R } be a sequence of functionals in the RNN hypothesis space ˆ H (see (9)).
Then for each t ∈ R , ˆ H t is a causal, continuous, linear and regular functional.
Moreover, the sequence of functionals { ˆ H t : t ∈ R } is time-homogeneous.
A PPROXIMATION T HEORY
The most basic approximation problem for RNN is as follows:
given some sequence of target functionals { H t : t ∈ R } satisfying appropriate conditions, does there always exist a sequence of RNN functionals { ˆ H t : t ∈ R } in ˆ H such that H t
≈ ˆ H t for all t ∈ R ?
We now make an important remark with respect to the current problem formulation that differs from previous investigations in the RNN approximation: we are not assuming that the target functionals { H t : t ∈ R } are themselves generated from an underlying dynamical system of the form
[Equation]
where
[Equation]
for any linear or nonlinear functions f, g .
This differs from previous work where it is assumed that the sequence of target functionals are indeed generated from such a system.
In that case, the approximation problem reduces to that of the functions f, g , and the obtained results often resemble those in feed-forward networks.
In our case, however, we consider general input-output relationships related by temporal sequences of functionals, with no necessary recourse to the mechanism from which these relationships are generated.
This is more ﬂexible and natural, since in applications it is often not clear how one can describe the data generation process.
Moreover, notice that in the linear case, if the target functionals { H t } are generated from a linear ODE system, then the approximation question is trivial: as long as the dimension of h t in the approximating RNN is greater than or equal to that which generates the data, we must have perfect approximation.
However, we will see that in the more general case here, this question becomes much more interesting, even in the linear regime.
In fact, we now prove precise approximation theories and characterize approximation rates that reveal intricate connections with memory effects, which may be otherwise obscured if one considers more limited settings.
Our ﬁrst main result is a converse of Proposition 3.1 in the form of an universal approximation theorem for certain classes of linear functionals.
The proof is found in Appendix A.
Theorem 4.1 (Universal approximation of linear functionals) .
Let { H t : t ∈ R } be a family of continuous, linear, causal, regular and time-homogeneous functionals on X .
Then, for any (cid:15) >there exists { ˆ H t : t ∈ R } ∈ ˆ H such that
[Equation]
The proof relies on the classical Riesz-Markov-Kakutani representation theorem, which says that each linear functional H t can be uniquely associated with a signed measure µ t such that H t ( x ) =
(cid:82)
R x
[Equation]
s dµ t ( s ) .
Owing to the assumptions of Theorem 4.1, we can further show that the sequence of representations { µ t } are related to an integrable function ρ : [0 , ∞ )
→ R d such that { H t } admits the common representation
[Equation]
Comparing this representation with the solution (7) of the continuous RNN, we ﬁnd that the approximation property of the linear RNNs is closely related to how well ρ ( t ) can be approximated by the exponential sums of the form ( c (cid:62) e Wt U ) (cid:62) .
Intuitively, (12) says that each output y t =
H t ( x ) is simply a convolution between the input signal and the kernel ρ .
Thus, the smoothness and decay of the input-output relationship is characterized by the convolution kernel ρ .
Due to this observation, we will hereafter refer to { H t } and ρ interchangeably.
Approximation rates.
While the previous result establishes the universal approximation property of linear RNNs for suitable classes of linear functionals, it does not reveal to us which functionals can be efﬁciently approximated.
In the practical literature, it is often observed that when there is some long-term memory in the inputs and outputs, the RNN becomes quite ill-behaved (Bengio et al., 1994; Hochreiter et al., 2001).
It is the purpose of this section to establish results which make these heuristics statements precise.
In particular, we will show that the rate at which linear functionals can be approximated by linear RNNs depends on the former’s smoothness and memory properties.
We note that this is a much less explored area in the approximation theory of RNNs.
To characterize smoothness and memory of linear functionals, we may pass to investigating the properties of their actions on constant input signals.
Concretely, let us denote by e i ( i = 1 , . . .
, d ) the standard basis vector in R d , and e i denotes a constant signal with e i,t =
e i 1 { t ≥ 0 } .
Then, smoothness and memory is characterized by the regularity and decay rate of the maps t (cid:55)→ H t ( e i ) , i = 1 , . . .
, d , respectively.
Our second main result shows that these two properties are intimately tied with the approximation rate.
The proof is found in Appendix B.
Theorem 4.2 (Approximation rates of linear RNN) .
Assume the same conditions as in Theorem 4.1.
Consider the output of constant signals y i ( t ) :
=
H t ( e i ) , i = 1 , . . .
, d. Suppose there exist constants α ∈ N + , β, γ > {+
+}
such that y i ( t ) ∈ C ( α +1)
( R ) , i = 1 , . . .
, d , and e βt y ( k ) i ( t ) =
o (1) as t → + ∞ , and
[Equation]
) where y ( k ) i ( t ) denotes the k th derivative of y i ( t ) .
Then, there exists a universal constant C ( α ) only depending on α , such that for any m ∈ N + , there exists a sequence of width- m RNN functionals { ˆ H t : t ∈ R } ∈
ˆ H m such that
[Equation]
The curse of memory in approximation.
For approximation of non-linear functions using linear combinations of basis functions, one often suffers from the “curse of dimensionality” (Bellman, 1957), in that the number of basis functions required to achieve a certain approximation accuracy increases exponentially when the dimension of input space d increases.
In the case of Theorem 4.2, the bound scales linearly with d .
This is because the target functional possesses a linear structure, and hence each dimension can be approximated independently of others, resulting in an additive error estimate.
Nevertheless, due to the presence of the temporal dimension, there enters another type of challenge, which we coin the curse of memory .
Let us now discuss this point in detail.
The key observation is that the rate result requires exponential decay of derivatives of H t ( e i ) , but the density result (Theorem 4.1) makes no such assumption.
The natural question is thus, what happens when no exponential decay is present?
We assume
d = 1 and consider an example in which the target functional’s representation satisﬁes ρ ( t ) ∈ C (1) ( R ) and ρ ( t ) ∼ t − (1+ ω ) as t → + ∞ .
Here ω > 0 indicates the decay rate of the memory effects in our target functional family.
The smaller its value, the slower the decay and the longer the system memory.
For any ω > 0 , the system’s memory vanishes more slowly than any exponential decay.
Notice that y (1) ( t ) = ρ ( t ) and in this case there exists no β > 0 making (13) true, and no rate estimate can be deduced from it.
A natural way to circumvent this obstacle is to introduce a truncation in time.
With T ( (cid:29) 1) we can deﬁne ˜ ρ ( t ) ∈ C (1) ( R ) such that ˜ ρ ( t ) ≡ ρ ( t ) for t ≤ T , ˜ ρ ( t ) ≡ 0 for t ≥ T + 1 , and ˜ ρ ( t ) is monotonically decreasing for T ≤ t ≤ T + 1 .
With the auxiliary linear functional ˜
H t ( x ) := (cid:82) t 0 x t − s ˜
ρ
( s ) ds, we can have an error estimate (with technical details provided in Appendix B)
[Equation]
[Equation] In order to achieve an error tolerance (cid:15) , according to the ﬁrst term above we require T ∼ (cid:15) − [Equation] ω , and then according to the second term we have
[Equation]
This estimate gives us a quantitative relationship between the degree of freedom needed and the decay speed.
With ρ ( t ) ∼ t − (1+ ω ) , the system has long memory when ω is small.
Denote the minimum number of terms needed to achieve an L 1 error (cid:15) as m ( ω, (cid:15) ) .
The above estimate shows an upper-bound of m ( ω, (cid:15) ) goes to inﬁnity exponentially fast as ω → 0 + with ﬁxed (cid:15) .
This is akin to the curse of dimensionality, but this time on memory, which manifests itself even in the simplest linear settings.
A stronger result would be that the lower bound of m ( ω, (cid:15) )
→ ∞ exponentially fast as ω → 0 + with ﬁxed (cid:15) , and this is a point of future work.
Note that this kind of estimates differ from the previous results in the literature (Kammler, 1979b; Braess & Hackbusch, 2005) regarding the order of m ( ω, (cid:15) )
∼ log(1 /(cid:15) ) as (cid:15) → 0 with ﬁxed ω
= 1 or 2 in the L ∞ or L 1 sense.
Note that the L 1 result has not been proved.
F INE - GRAINED A NALYSIS OF O PTIMIZATION D YNAMICS
According to Section 4, memory plays an important role in determining the approximation rates.
The result therein only depends on the model architecture, and does not concern the actual training dynamics.
In this section, we perform a ﬁne-grained analysis on the latter, which again reveals an interesting interaction between memory and learning.
The loss function (for training) is deﬁned as
[Equation]
Without loss of generality, here the input time series x is assumed to be ﬁnitely cut off at zero, i.e. x t = 0 for any t ≤ 0 almost surely.
Training the RNN amounts to optimizing E x J T with respect to the parameters ( c, W, U ) .
The most commonly applied method is gradient descent (GD) or its stochastic variants (say SGD), which updates the parameters in the steepest descent direction.
We ﬁrst show that the training dynamics of E x J T exhibits very different behaviors depending on the form of target functionals.
Take d = 1 and consider learning different target functionals with white noise x .
We ﬁrst investigate two choices for ρ : a simple decaying exponential sum and a scaled
Airy function.
The Airy function target is deﬁned as ρ ( t ) =
Ai( s 0 [ t − t 0 ]) , where Ai( t ) is the Airy function of the ﬁrst kind, given by the improper integral Ai( t ) =
1 π lim ξ →∞ (cid:82) ξ 0 cos (cid:16) u 3 3 + tu (cid:17) du .
Note that the effective rate of decay is controlled by the parameter t 0 : for t ≤ t 0 , the Airy function is oscillatory.
Hence for large t 0 , a large amount of memory is present in the target.
Observe from Figure 1 that the training proceeds efﬁciently for the exponential sum case.
However, in the second Airy function case, there are interesting “plateauing” behaviors in the training loss, where the loss decrement slows down signiﬁcantly after some time in training.
The plateau is sustained for a long time before an eventual reduction is observed.
As a further demonstration of that this behavior may be generic, we also consider a nonlinear forced dynamical system, the Lorenz 96 system (Lorenz, 1996), where the similar plateauing behavior is observed even for a non-linear RNN model trained with the Adam optimizer (Kingma & Ba, 2015).
All experimental details are found in Appendix C.3.1.
The results in Figure 1 hint at the fact that there are certainly some functionals that are much harder to learn than others, and it is the purpose of the remaining analyses to understand precisely when and why such difﬁculties occur.
In particular, we will again relate this to the memory effects in the target functional, which shows yet another facet of the curse of memory when it comes to optimization.
Dynamical analysis.
To make analysis amenable, we will make a series of simpliﬁcations to the loss function (17), by assuming that x is white noise, d = 1 , T → ∞ , and the recurrent kernel W is diagonal.
This allows us to write (see Appendix C.1 for details) the optimization problem as
[Equation]
We will subsequently see that these simpliﬁcations do not lose the key features of the training dynamics, such as the plateauing behavior.
We start with some informal discussion on a probable reason behind the plateauing.
A straightforward computation shows that, for k = 1 , 2 , . . .
, m ,
[Equation]
[Figure]
A similar expression holds for ∂J∂a k .
Write the (simpliﬁed) linear functional representation for linear
RNNs as ˆ ρ ( t ; a, w ) :
= (cid:80) mi =
a i e − w i t , which serves to learn the target ρ .
Observe that plateauing under the GD dynamics occurs if the gradient ∇ J is small but the loss J is large.
A sufﬁcient condition is that the residual ˆ ρ ( t ; a, w ) − ρ ( t ) is large only for large t (meaning the exponential multiplier to the residual is small).
That is, the learned functional differs from the target functional only at large times.
This again relates to the long-term memory in the target.
Based on this observation, we build this memory effect explicitly into the target functional by considering ρ of the parametrized form
[Equation]
(20) where ¯ ρ is the function which can be well-approximated by the model ˆ ρ , e.g. the exponential sum ¯ ρ ( t ) = (cid:80) m ∗ j =1 a ∗ j e − w ∗
j t (with w ∗ j > 0 , j = 1 , · · · , m ∗ ).
On the other hand, ρ 0 ,ω ( t ) :
= ρ 0 ( t − 1 /ω ) controls the target memory, with ρ 0 as any bounded template function in L 2 ( R ) ∩ C 2 ( R ) with subGaussian tails.
As ω → 0 + , the support of ρ 0 ,ω shifts towards large times, modelling the dominance of long-term memories.
In this case, if the initialization satisﬁes ˆ ρ ≈ ¯ ρ , the sufﬁcient condition informally discussed above is satisﬁed as ω → 0 + , which heuristically leads to the plateauing.
A simple example of (20) can be ρ ω ( t ) = a ∗ e
[Equation]
w [Equation] t + c 0 e − ( t − 1 /ω )2 2 σ, where a ∗ , c 0 , σ (cid:54) = 0 and w ∗ >are ﬁxed constants.
This corresponds to the simple case that m ∗ = 1 and ρ 0 is the Gaussian density.
Observe that as ω → 0 + , the memory of this sequence of functionals represented by ρ ω increases.
It can be numerically veriﬁed that this simple target functional gives rise to the plateauing behavior, which gets worse signiﬁcantly as ω → 0 + (see Figure 2 in Appendix C.3.2).
Our main result on training dynamics quantiﬁes the plateauing behavior theoretically for general functionals possessing the decomposition (20).
For rigorous statements and detailed proofs, see Theorem C.1 in Appendix C.2.
Theorem 5.1.
Deﬁne the loss function J ω as in (18) with the target ρ = ρ ω as deﬁned in (20).
Consider the gradient ﬂow training dynamics
[Equation]
21) where θ ω ( τ ) : = ( a ω ( τ ) , w ω ( τ ))
∈ R 2 m for any τ ≥ 0 , and θ 0 :
= ( a 0 , w 0 ) .
For any ω > 0 , m ∈ N + and θ 0 ∈ R m × R m + , 0 < δ (cid:28) 1 , deﬁne the hitting time
[Equation]
Assume that m > m ∗ , and the initialization is bounded and satisﬁes ˆ ρ ( t ; θ 0 )
≈ ¯ ρ ( t ) .
Then
[Equation]
23) for any ω > 0 sufﬁciently small, where c 0 and (cid:38) hide universal positive constants independent of ω , m and θ 0 .
Let us sketch the intuition behind Theorem 5.1.
Suppose that we currently have a good approximation ˆ ρ of the short-term memory part ¯ ρ , then we can show that the loss is large ( J = O (1) ) since the long-term memory part ρ 0 ,ω is not accounted for.
However, the gradient now is small ( ∇ J = o (1) ), since the gradient corresponding to the long-term memory part is concentrated at large t , and thus modulated by exponentially decayed multipliers (see (19)).
This implies slowdowns in the training dynamics in the region ˆ ρ ≈ ¯ ρ .
It remains to estimate a lower bound on the timescale of escaping from this region, which depends on the curvature of the loss function.
In particular, we show that ∇ 2 J is positive semi-deﬁnite when ω = 0 , but has O (1) positive eigenvalues and multiple o (1) (can be exponentially small) eigenvalues for any 0 < ω (cid:28) 1 .
Hence, a local linearization analysis implies an exponentially increasing escape timescale, as indicated in (23).
While the target form (20) may appear restrictive, we emphasize that some restrictions on the type of functionals is necessary, since plateauing does not always occur (see Figure 1).
In fact, a goal of the preceding analysis is to establish a family of functionals for which exponential slowdowns in training provably occurs, and this can be related to memories of target functionals in a precise way.
The curse of memory in optimization.
The timescale proved in Theorem 5.1 is veriﬁed numerically in Figure 3 in Appendix C.3.3, where we also show that the analytical setting here is representative of general cases, where plateauing occurs even for non-linear RNNs trained with accelerated optimizers, as long as the target functional has the memory structure imposed in (20).
Theorem 5.1 reveals another aspect of the curse of memory , this time in optimization.
When ω → 0 + , the inﬂuence of target functional H t does not decay, much like the case considered in the curse of memory in approximation.
However, different from the approximation case where an exponentially large number of hidden states is required to achieve approximation tolerance, here in optimization the adverse effect of memory comes with the exponentially pronounced slowdowns of the gradient ﬂow training dynamics.
While this is theoretically proved under sensible but restrictive settings, we show numerically in Appendix C.3.3 (Figure 4) that this is representative of general cases.
In the literature, a number of results have been obtained pertaining to the analysis of training dynamics of RNNs.
A positive result for training by GD is established in Hardt et al. , but this is in the setting of identifying hidden systems, i.e. the target functional comes from a linear dynamical system, hence it must possess good decay properties provided stablity.
On the other hand, convergence can also be ensured if the RNN is sufﬁciently over-parameterized (large m ; Allen-Zhu et al.
However, both of these settings may not be sufﬁcient in reality.
Here we provide an alternative analysis of a setting that is representative of the difﬁculties one may encounter in practice.
In particular, the curse of memory that we established here is consistent with the difﬁculty in RNN training often observed in applications, where heuristic attributions to memory are often alluded to Hu et al. ; Campos et al. ; Talathi & Vartak (2015); Li et al.
The analysis here makes the connection between memories and optimization difﬁculties precise, and may form a basis for future developments to overcome such difﬁculties in applications.
C ONCLUSION
In this paper, we analyzed the basic approximation and optimization aspects of using RNNs to learn input-output relationships involving temporal sequences in the linear, continuous-time setting.
In particular, we coined the concept curse of memory , and revealed two of its facets.
That is, when the target relationship has the long-term memory, both approximation and optimization become exceedingly difﬁcult.
These analyses make concrete heuristic observations of the adverse effect of memory on learning RNNs.
Moreover, it quantiﬁes the interaction between the structure of the model (RNN functionals) and the structure of the data (target functionals).
The latter is a much less studied topic.
Here, we adopt a continuous-time approach in order gain access to more quantitative tools, including classical results in approximation theory and stochastic analysis, which help us derive precise results in approximation rates and optimization dynamics.
The extension of these results to discrete time may be performed via numerical analysis in subsequent work.
More broadly, this approach may be a basic starting point for understanding learning from partially observed time series data in general, including gated RNN variants (Hochreiter & Schmidhuber, 1997;
Cho et al., 2014) and other methods such as transformers and convolution-based approaches (Vaswani et al., 2017; Oord et al., 2016).
These are certainly worthy of future exploration.
A U NIVERSAL A PPROXIMATION T HEOREM OF L INEAR F UNCTIONALS BY L INEAR RNN S
A key simpliﬁcation of considering linear functionals is due to the classical representation result below, which allows us to pass from the approximation of functionals to the approximation of functions.
In short, this theorem says that while for each measure µ , x (cid:55)→ (cid:82) R x (cid:62) s dµ ( s ) deﬁnes a linear functional, this is in fact the only way to deﬁne a linear functional: for any linear functional H there exists a unique measure µ such that H ( x ) =
(cid:82) R x (cid:62) s dµ ( s ) .
Theorem A.1 (Riesz-Markov-Kakutani representation theorem) .
Let H : X → R be a continuous linear functional.
Then, there exists a unique, vector-valued, regular, countably additive signed measure µ on R such that
[Equation]
Moreover, we have
[Equation]
Proof.
Well-known, see e.g. Bogachev (2007), CH 7.10.4.
We will use the representation theorem to prove Theorem 4.1.
First, we prove some lemmas.
The ﬁrst shows that there is in fact a common representation of a sequence of linear functionals satisfying the assumptions of Theorem 4.1.
Lemma A.1.
Let { H t } be a family of continuous, linear, regular, causal and time homogeneous functionals on X .
Then, there exists a measurable function ρ : [0 , ∞ )
→ R d that is integrable, i.e.
[Equation]
26) and
[Equation]
In particular, { H t } is uniformly bounded with sup t (cid:107) H t (cid:107) =
(cid:107) ρ
[Equation]
L 1 ([0 , ∞ )) and t (cid:55)→ H t ( x ) is continuous for all x ∈ X .
Proof.
By the Riesz-Markov-Kakutani representation theorem (Theorem A.1), for each t there is a unique regular signed Borel measure µ t such that
[Equation]
28) and (cid:80) i | µ t,i | ( R ) =
(cid:107) H t (cid:107) .
Since { H t } is causal, we must have (cid:82) ∞ t x (cid:62) s dµ t ( s ) = 0 for any x and thus
[Equation]
Now, by time homogeneity we have
[Equation]
Take τ = − t and set µ = − µ 0 to get
[Equation]
Note that we have (cid:107) µ (cid:107) 1 ([0 , ∞ ))
= (cid:107) µ 0 (cid:107) 1 ([0 , ∞ ))
= (cid:107) H 0 (cid:107) =
(cid:107) H t (cid:107) , and continuity follows from the fact that
[Equation]
which converges to 0 as δ → 0 by dominated convergence theorem.
Finally, we will show that each µ i is absolutely continuous with respect to λ (Lebesgue measure).
Take a measurable E ⊂ [0 , ∞ ) such that λ ( E ) = 0 and set E (cid:48)
=
[0 , ∞ ) \ E .
For each n ≥ 0 set K n ⊂ E, K (cid:48) n ⊂ E (cid:48) where K n , K (cid:48) n are closed and µ i ( E \ K n ) ≤ 1 /n , µ i ( E (cid:48) \ K (cid:48) n ) ≤ 1 /n .
For a ﬁxed i ∈ { 1 , . . .
, d } , deﬁne x ( n ) to be such that x ( n ) t − s,j = 0 for all j (cid:54)
=
i and all s .
For j = i , we set x ( n ) t − s,i = 1 if s ∈ K n and 0 if s ∈ K (cid:48) n , which can then be continuously extended to [0 , ∞ ) .
Observe that by construction, x ( n ) t − s → 0 for λ -a.e.
s , thus by dominated convergence theorem
[Equation]
This shows that µ i is absolutely continuous with respect to λ , and by the Radon-Nikodym theorem there exists a measurable function
ρ i : [0 , ∞ ) → R such that for any measurable A ⊂ R we have
[Equation]
34) for i = 1 , . . .
, d .
Hence, we have
[Equation]
35) with (cid:107) ρ (cid:107) L 1 ([0 , ∞ ))
= (cid:80) i (cid:82) ∞
| ρ i ( s ) | ds = (cid:107) µ (cid:107) 1 ([0 , ∞ )) < ∞ .
Lemma A.2.
Let ρ : [0 , ∞ )
→ R a Lebesgue integrable function, i.e. (cid:107) ρ (cid:107) L 1 ([0 , ∞ )) < ∞ .
Then, for any (cid:15) > 0 , there exists a polynomial p with p (0) =
such that
[Equation] [Equation]
Proof.
The approach here is similar to that of the approximation of functions using exponential sums (Kammler, 1976; Braess, 1986).
Alternatively, one may also appeal to the density of phase type distributions (He & Zhang, 2007; O’Cinneide, 1990) in the space of positive distributions, and generalizing them to signed measures.
Fix (cid:15) > 0 .
Deﬁne
[Equation] [Equation]
Then, we can check that
[Equation]
[Equation]
By density of continuous functions in L 1 there exists a continuous function ˜
R on [0 , 1] with ˜ R (0) =
such that
[Equation]
By M¨untz-Sz´asz theorem (M¨untz, 1914; Sz´asz, 1916), there exists a polynomial p with p (0) =
such that
[Equation]
40) and q ( u ) :
= p ( u ) /u is also a polynomial.
Therefore, we have
[Equation] [Equation]
We are now ready to present the proof of Theorem 4.1.
Proof of Theorem 4.1.
By (7), for each { ˆ H t } ∈ ˆ H we can write
[Equation]
By Lemma A.1, we can write
[Equation] [Equation]
where ρ is integrable.
Thus, we can apply Lemma A.2 to conclude that there exists polynomials p
i , i = 1 , . . .
, d with p i (0) =
such that
[Equation]
Notice that we can write each p i ( u )
=
(cid:80) m j =
α
ij u j for some m equaling the maximal order of { p i } .
Taking W = diag( − 1 , . .
. , − m ) , c = (1 , . . . , 1) and U ij = α ji , we have
[Equation]
, d
[Equation]
Consequently, we have for any x with (cid:107) x (cid:107) ∞ ≤ 1 ,
[Equation]
B A PPROXIMATION R ATES AND THE C URSE OF M EMORY IN A PPROXIMATION
We ﬁrst give the proof of Theorem 4.2.
Proof of Theorem 4.2.
We ﬁx i ∈ { 1 , . . .
, d } below until the last part of the proof.
By Lemma A.1, there exists ρ i ( t ) ∈
C α [0 , ∞ ) such that
[Equation]
By the assumption, ρ ( k ) i
[Equation]
t
[Equation]
o
[Equation] [Equation]
→ ∞ ,
[Equation]
Consider the transform
[Equation]
For k = 0 , . . .
, α , one can prove by induction that
[Equation]
where c ( j, k ) are some integer constants.
Together with the assumption, we have
[Equation]
where C ( α ) is a universal constant only depending on α .
Note that for j = 0 , . .
, α ,
[Equation]
hence q i ( s ) ∈ C α [0 , 1] with q i (0) =
q (1) i (0) = · · · = q ( α ) i (0)
= 0 .
By Jackson’s theorem (Jackson, 1930), for m = 1 , 2 , . . .
, there exists a polynomial Q i,m of degree m − 1 such that
[Equation]
Denote the polynomial Q i,m as
[Equation] [Equation]
and deﬁne
[Equation]
Then we have
[Equation]
56) where
[Equation] [Equation]
[Equation]
With a change of variable s = e − βα +1 t , we have the estimate
[Equation]
Finally we deﬁne U =
[ u 1 , . . . ,
u d ] ∈ R m × d and have
[Equation]
Recall the dynamical system (6)
[Equation]
(62) which is together determined by the parameters c, W, U .
Similar to the argument in the proof of
Theorem 4.1, for any x with (cid:107) x (cid:107) ∞ ≤ 1 and t , we have
[Equation]
The curse of memory in approximation.
Now we explain more technical details of why Theorem 4.2 implies the curse of memory in approximation, as pointed out in the main text.
We assume
d = 1 and consider an example
[Equation]
in which the density satisﬁes ρ ( t ) ∈ C (1) ( R ) and
[Equation]
Here ω > 0 indicates the decay rate of the memory effects in our target functional family { H t } .
The smaller its value, the slower the decay and the longer the system memory.
Notice that y (1) ( t ) =
ρ ( t ) , and in this case there exists no β > 0 making the following condition ((13) in the main text) true:
[Equation]
and
[Equation]
and no rate estimate can be deduced from it.
A natural way to circumvent this obstacle is to introduce a truncation in time.
With T ( (cid:29) 1) we can deﬁne ˜ ρ ( t ) ∈ C (1) ( R ) such that ˜ ρ ( t ) ≡ ρ ( t ) for t ≤ T , ˜ ρ ( t ) ≡ 0 for t ≥ T + 1 , and ˜ ρ ( t ) is monotonically decreasing for T ≤ t ≤ T + 1 .
Considering the linear functional ˜
H t ( x ) := (cid:82) t 0 x t − s ˜
ρ
( s ) ds, we have the truncation error estimate
[Equation]
Now the conclusion of Theorem 4.2 (i.e. (63)) is applicable to the truncated { ˜ H t } (with α = 1 ), and we have for ∀ β > 0 , there is a linear RNN ( U, W, c ) such that the associated functionals { ˆ H t } ∈
ˆ H m satisfy
[Equation]
It is straightforward to verify that when β
= 2 /T , the right-hand side of (68) achieves the minimum, which gives us sup
[Equation]
Combining (67) and (69) gives us
[Equation]
In order to achieve an error tolerance (cid:15) , according to the ﬁrst term above we require T ∼ (cid:15) − 1 ω , and then according to second term we have
[Equation]
This estimate gives us a quantitative relationship between the degree of freedom needed and the decay speed.
When ω is small, i.e., the system has long memory, the size of the RNN required grows exponentially.
C F INE - GRAINED A NALYSIS OF O PTIMIZATION D YNAMICS C. S IMPLIFICATIONS ON THE L OSS F UNCTION
Recall the loss function
[Equation]
The simpliﬁcations on E x J T are listed as follows.
Take the input data x to be the white noise, so that
[Equation]
where B t is the standard d -dimensional Wiener process.
As a consequence, simplifying (72) via Itˆo’s isometry gives
[Equation]
We focus on the temporal dimension and take the spatial dimension d = 1 in (74).
Moreover, to investigate the effect of long-term memory, it is necessary to consider the training on large time horizons. Hence, we take T → ∞ to get
[Equation]
where b is the sole column of U in (74) and ρ ( t ) becomes a scalar-valued target.
This corresponds to the so-called single-input-single-output (SISO) system.
Due to the difﬁculty of directly analyzing ∇ W e Wt and ∇ 2 W e Wt , we consider a further simpliﬁed ansatz.
Assume that W is a diagonal matrix with negative entries (to guarantee the stability of the model). That is, W = − diag( w ) with w ∈ R m + .
Then we can combine a = b ◦ c (where ◦ denotes the Hadamard product) and rewrite the model as
[Equation]
The optimization problem (75) becomes
[Equation]
Here we omit the subscript ∞ .4.
We apply a continuous-time idealization of the gradient descent dynamics by considering the gradient ﬂow with respect to J ( a, w ) .
That is,
[Equation]
with some initial value a (0) = a 0 ∈ R m , w (0) =
w 0 ∈ R m + .
As we will show later, applying the training dynamics (78) to optimize (77) is able to serve as a starting point in the ﬁne-grained dynamical analysis, since it still preserves the plateauing behavior observed in the optimization process (see Figure 1), provided additional structures related to memories as discussed next.
C. C ONCRETE D YNAMICAL A NALYSIS AND THE C URSE OF M EMORY IN O PTIMIZATION
We prove Theorem 5.1 in this section.
The basic insight is, by adding long-term memories in targets, one can increase the [-model of exponential sums ˆ ρ m ( t ; a, w ) = (cid:80) mi =
 a i e − w i t ,-]loss with little effect on the gradient and Hessian, which leads to a signiﬁcant slow down of the training dynamics near the short-term memory parts of the targets.
Therefore,
Theorem 5.1 is proved subsequently in the following procedure
We prove that J ω has a large value but small gradient when ˆ ρ ( t ; a, w ) ≡ ¯ ρ ( t ) .
in the following content if not speciﬁed.
We prove that when ˆ ρ ( t ; a, w ) ≡ ¯ ρ ( t ) , the Hessian ∇ 2 J ω is positive semi-deﬁnite for ω = 0 , but for ﬁnite, small ω > 0 , ∇ 2 J ω has O (1) positive eigenvalues and multiple o (1) eigenvalues.
Based on these results, we perform a local linearization analysis on the gradient ﬂow (21) initialized by ˆ ρ ( t ; a 0 , w 0 ) ≡ ¯ ρ ( t ) , from which and by continuity the timescale of plateauing is derived.
(1) Preliminary Results
As discussed around (20), we consider the target functional with a parametrized representation
[Equation]
Here, a ∗ j := a ∗ ( w ∗ j ) (cid:54) = 0 , w ∗ j > 0 and w ∗ i (cid:54) =
w ∗ j for any i (cid:54) =
j , i, j ∈ [ m ∗ ] , 3 and m ∗ < m . The former requirements are just non-degenerate conditions, and the last requirement ensures that the model can perfectly represent the well-approximated part of the target, ¯ ρ ( t ) .
The memory in the target is controlled by ρ 0 ,ω ( t )
=
ρ 0 ( t − 1 /ω ) , with ρ 0 as a ﬁxed template function which satisﬁes the following assumptions.
[Equation]
[Equation]
Remark C.1.
The above assumptions (i)(ii)(iii) are rather natural, and (iv) only restricts the single side tail of ρ 0 to be zero.
In the following analysis, we further focus on ρ 0 with light tails, e.g. the sub-Gaussian tail
[Equation]
for some ﬁxed positive constants c 0 , c 1 , t 0 .
Obviously, Gaussian densities and continuous functions with compact supports are sub-Gaussian functions.
We begin by the following preliminary estimate that is used throughout the subsequent analysis.
Lemma C.1.
For any n ∈ N , ω > 0 and w > 0 , let
[Equation]
Then • ∆ n,ω ( w ) is monotonically decreasing on (0 , ∞ ) ; • lim ω
→ 0 + ∆ n,ω ( w ) = 0 ; • In particular, if ρ 0 is sub-Gaussian, we further have ∆ n,ω ( w ) [Equation] ω
[Equation]
n [Equation] Here c 2 = e 14 c 1 > 1 , c 3 = e t 0 > 1 , and (cid:46) hides universal constants only depending on n and ρ 0 , t 0 , c 0 , c
Proof.
(i) Obviously ∆ n,ω ( w 1 ) ≤ ∆ n,ω ( w 2 ) for any w 1 > w 2 > 0 . (ii) By the assumptions on ρ 0 , we get lim
[Equation]
, and M 0 : = (cid:107) ρ 0 (cid:107) L ∞ ( R ) < ∞ , which gives t n e − wt | ρ 0 ,ω ( t ) | ≤ M 0 t n e − wt ∈ L 1 ([0 , ∞ )) for any n ∈ N , ω > 0 and w > 0 . By Lebesgue’s dominant convergence theorem, we have
[Equation]
n
[Equation]
(iii) Now we estimate ∆ n,ω ( w ) under the sub-Gaussian condition (79).
Suppose 0 < ω < 1 /t 0 , we have
[Equation]
{+Then we bound I 1 , I 
+}
and I 3 respectively:
[Equation]
where ω ∈ (0 , 1 / 2) , and (cid:46) hides universal constants only related to n and t 0 .
Let 1 /c 1 := 2 σ 2 , we have
[Equation]
where (cid:90) − t
[Equation]
holds for any ω ∈ (0 , 2 c 1 /w ) .
Here the last inequality is due to the Mellin Transform of absolute moments of the Gaussian density (see Proposition 1 in Li et al.
The argument is similar for I 3 , which gives the same bound as I 2 .
Combining all the estimates gives the desired conclusion.
The proof is completed.
The main idea to analyze plateauing behaviors is to investigate the local dynamics of the gradient ﬂow (21) when ˆ ρ =
¯ ρ , then extend the results to the setting ˆ ρ ≈ ¯ ρ by continuity. Recall that both of them are exponential sums, we can obtain the relation of parameters between ˆ ρ and ¯ ρ , according to the following lemma. Lemma C.2.
For any m ∈ N + , let λ = ( λ 1 , · · · , λ m ) with λ i (cid:54) = λ j for any i (cid:54) = j , i, j ∈ [ m ] .
Then the series of functions (cid:8) e λ i t (cid:9) m i =1 is linear independent on any interval I ⊂ R . Proof. The aim is to show
[Equation]
∈ [ m ] .
[Equation] (83) holds trivially for m = 1 . Assume that (83) holds for m − 1 , then [Equation]
[-Deﬁnition C.
-]By induction, we get c
{+i λ i − λ m ) = 
+}
for any i = 1 , · · · , m − 1 . Since λ 1 , · · · , λ m are distinct, we have c i = 0 for any i = 1 , · · · , m [-∗ j =
-]− 1 .
{+Together with (84), we get c m = 0 , which completes the proof. Deﬁnition C.1. Let m ≥ m ∗ . For any partition P : [ m ] = ∪ m ∗ j =
+}
I j with I j 1 ∩ I j 2 = ∅ for any j 1 (cid:54) =
j 2 , j 1 , j 2 ∈ { 0 } ∪ [ m ∗ ] , and I 0 = ∪ i 0 r =1 I 0 ,r with I 0 ,r 1 ∩ I 0 ,r 2 = ∅ for any r 1 (cid:54) = r 2 , r 1 , r 2 ∈ [ i 0 ] , where I j (cid:54) = ∅ for any j ∈ [ m ∗ ] and I 0 ,r (cid:54) = ∅ for any r ∈
[ i 0 ]
(if I 0 (cid:54) = ∅ ), deﬁne the afﬁne space (with respect to P ):
[Equation]
Denote the collection of all such afﬁne spaces by M ∗ := (cid:83) P M ∗P .
The following lemma characterizes the relation of parameters, by showing that M ∗ is exactly the set of equivalent points to ( a ∗ , w ∗ ) for the purpose of representation via exponential sums. Lemma C.3. For any ( a, w ) ∈ R m × R m + , ˆ ρ ( t ; a, w ) ≡ ¯ ρ ( t ) ⇔ ( a, w ) ∈ M ∗ .
Proof.
(i) ( ⇐ ) Since ( a, w ) ∈ M ∗ , there exists P such that ( a, w ) ∈ M ∗P . Then for any t ≥ 0 , ˆ ρ ( t ; a, w ) = m (cid:88) i = a i e − w i t
= m ∗ (cid:88)
j = (cid:88)
[Equation]
e − w
[Equation]
(cid:88)
[Equation]
(cid:88)
[Equation]
=
[Equation]
¯
[Equation]
( t ) .
(ii) ( ⇒ )
Let I
j = (cid:8) i ∈ [ m ] : w i =
w ∗ j (cid:9) for any j ∈ [ m ∗ ] , and I 0 = (cid:8) i ∈ [ m ] : w i (cid:54) =
w ∗ j for any j ∈ [ m ∗ ] (cid:9) .
Recall that ¯ ρ ( t ) =
(cid:80) m ∗ j =1 a ∗ j e − w ∗
j t is non-degenerate: a ∗ j (cid:54) = 0 , w ∗ j > 0 and w ∗ i (cid:54) =
w ∗ j for any i (cid:54) =
j , i, j ∈ [ m ∗ ] , we get [ m ] =
{+∪ m ∗ j =
+}
I j , I j 1 ∩ I j 2 = ∅ for any j 1 (cid:54) =
j 2 , j 1 , j 2 ∈ { 0 } ∪ [ m ∗ ] .
Combining Lemma C.2 and the non-degeneracy of ¯ ρ , I j (cid:54) = ∅ for any j ∈ [ m ∗ ] .
Assume that there are i 0 different components in ( w i ) i ∈I 0 , say v 1 , · · · , v i 0 , then v r (cid:54) =
w ∗ j for any r ∈
[ i 0 ] and j ∈ [ m ∗ ] .
Let I 0 ,r = (cid:8) i ∈ I 0 : w i = v r (cid:9) for any r ∈ [ i 0 ] , we get I 0 ,r (cid:54) = ∅ for any r ∈
[ i 0 ]
{+(if I 0 (cid:54) = ∅ ), and I 0 = ∪ i 0 r =
+}
I 0 ,r , and I 0 ,r 1 ∩ I 0 ,r 2 = ∅ for any r 1 (cid:54) =
r 2 , r 1 , r 2 ∈ [ i 0 ] .
{+Hence [ m ] = ∪ m ∗ j =
+}
{+I j with I 
+}
{+= ∪ i 0 r =
+}
I 0 ,r forms a P deﬁned in Deﬁnition C.1, and
[Equation]
r = (cid:32) (cid:88) i ∈I 0 ,r a i (cid:33)
e − v r t + m ∗ (cid:88)
j = (cid:32) (cid:88)
i ∈I
j a i − a ∗ j (cid:33)
e − w ∗ j t .
Again by Lemma C.2, we have (cid:80) i ∈I j a i = a ∗ j for any j ∈ [ m ∗ ] and (cid:80) i ∈I 0 ,r a i = 0 for any r ∈ [ i 0 ] , which gives ( a, w ) ∈ M ∗P .
The proof is completed.
Remark C.2.
Let I 0 = ∅ . It is straightforward to check that for any partition P , the dimension of M ∗P is (cid:80) m ∗ j =1 ( |I j | − 1)
= m − m ∗ .
In addition, it can be veriﬁed that the cardinality of M ∗ is m ∗ !
(cid:26) mm ∗ (cid:27) , where (cid:26) mm ∗ (cid:27) is the Stirling number of the second kind. (2) Loss and Gradient
Now we show that as ω → 0 + , the loss J ω remains lower bounded, while (cid:107)∇ J ω (cid:107) 2 converges to 0 .
This implies that the loss will saturate at a high value when learning long-term memories.
Proposition C.1.
For any ( a, w ) ∈ R m × R m + satisfying ˆ ρ ( t ; a, w ) ≡ ¯ ρ ( t ) , we have [Equation] where C ( ρ 0 ) , C (cid:48) ( ρ 0 ) > 0 are constants only depending on ρ 0 .
That is, the loss is lower bounded uniformly in ω .
Proof.
Recall the assumptions on ρ 0 , we have ρ 0 ( t ) (cid:54)≡ 0 and ρ 0 ∈ C ( R ) . Let t 1 ∈ R such that ρ 0 ( t 1 ) (cid:54) = 0 . By continuity, there exists δ 0 > 0 such that | ρ 0 ( t ) | ≥ | ρ 0 ( t 1 ) | / 2 for any t ∈ [ t 1 − δ 0 , t 1 + δ 0 ] .
Hence, for any ω > 0 sufﬁciently small such that − 1 /ω <
t 1 − δ 0 , we have (cid:107)
[Equation]
[Equation]
[Equation] , ∞ ) = (cid:90) ∞ [Equation]
Fix any (ˆ a, ˆ w ) ∈ R m × R m + such that ˆ ρ ( t ; ˆ a, ˆ w ) ≡ ¯ ρ ( t ) . Then
[Equation] which completes the proof. Proposition C.2. For any ( a, w ) ∈ R m × R m + satisfying ˆ ρ ( t ; a, w ) ≡ ¯ ρ ( t ) , we have
[Equation]
In particular, if ρ 0 has the sub-Gaussian tail (79), the estimate
[Equation]
holds for any ω ∈ (0 , min { 1 / 2 , 1 /t 0 , 2 c 1 /w min } ) .
Here w min := min i ∈ [ m ] w i > 0 , c 2 , c 3 > 1 are constants only related to c 1 , t 0 , and (cid:46) hides universal constants only depending on ρ 0 , t 0 , c 0 , c 1 .
Proof.
A straightforward computation shows, for k = 1 , 2 , · · · , m ,
[Equation]
Fix any (ˆ a, ˆ w ) ∈
R m × R m + satisfying ˆ ρ ( t ; ˆ a, ˆ w ) ≡ ¯ ρ ( t ) .
By Lemma C.3, we have (ˆ a, ˆ w ) ∈
M ∗ .
Recall Deﬁnition C.1, there exists a partition P : [ m ] = ∪ m {+∗ j =
 I j with I 
 = ∪+} i {+0 r =
 I 0 ,r , where I j (cid:54)+} = ∅ for any j ∈ [ m ∗ ]
and I 0 ,r (cid:54) = ∅ for any r ∈
[ i 0 ]
(if I 0 (cid:54) = ∅ ), such that (ˆ a, ˆ w ) ∈ M ∗P , which gives that (cid:80) i ∈I j ˆ a i = a ∗ j , ˆ w i =
w ∗ j for any i ∈ I j and j ∈ [ m ∗ ] ; (cid:80) i ∈I 0 ,r ˆ a i = 0 , ˆ w i = v r (cid:54) =
w ∗ j for any i ∈ I 0 ,r , r ∈
[ i 0 ] and j ∈ [ m ∗ ] .
Therefore, for any n ∈ N + , we have
[Equation]
This yields
[Equation]
and hence [Equation] By Lemma C.1, we get lim ω → 0 +
[Equation]
If ρ 0 has the sub-Gaussian tail (79), again by Lemma C.1, the estimate
[Equation]
holds for any n ∈ N , ω ∈ (0 , min { 1 / 2 , 1 /t 0 , 2 c 1 / ˆ w min } ) and k ∈ [ m ] . Here ˆ w min := min i ∈ [ m ] ˆ w i > 0 , c 2 , c 3 > 1 are constants only related to c 1 , t 0 , and (cid:46) hides universal constants only depending on n and ρ 0 , t 0 , c 0 , c 1 . Therefore [Equation]
The proof is completed. (3) Eigenvalues of Hessian Now we show that minimal eigenvalues of ∇ 2 J ω also converges to 0 as ω → 0 + .
Proposition C.3.
For any ( a, w ) ∈ R m × R m + satisfying ˆ ρ ( t ; a, w ) ≡ ¯ ρ ( t ) , denote the eigenvalues of ∇ 2 J ω ( a, w ) by λ 1 ( ω ) ≥ λ 2 ( ω ) ≥ · · · ≥ λ 2 m ( ω ) . If m > m ∗ , we have
[Equation]
for ω > 0 sufﬁciently small, where m (cid:48) ≤ 2 m ∗ + |I 0 | ≤ m + m ∗ .
In particular, if ρ 0 has the sub-Gaussian tail (79), the estimate
[Equation] holds for any ω ∈ (0 , min { 1 / 2 , 1 /t 0 , 2 c 1 /w min } ) .
Here w min := min i ∈ [ m ] w i > 0 , c 2 , c 3 > 1 are constants only related to c 1 , t 0 , and (cid:46) hides universal constants only depending on ρ 0 , t 0 , c 0 , c 1 .
Proof.
A straightforward computation shows, for k, j = 1 , 2 , · · · , m ,
[Equation]
Fix any (ˆ a, ˆ w ) ∈
R m × R m + satisfying ˆ ρ ( t ; ˆ a, ˆ w ) ≡ ¯ ρ ( t ) .
By (90), we have ∂ 2 J ω ∂a k ∂a [Equation] = [Equation]
∂a k
[Equation]
Let
[Equation]
L 2 [
[Equation]
∞ )
[Equation]
and
[Equation]
where ∆ n,ω ( w ) ( n = 1 , 2 ) is performed element-wisely.
One can verify that
[Equation]
Then we analyze ∇ 2 ¯ J (ˆ a, ˆ w ) and E ω
(ˆ a, ˆ w ) respectively.
(i) ∇ 2 ¯ J (ˆ a, ˆ w ) . Obviously (ˆ a, ˆ w ) is a global minimizer of ¯ J ( a, w ) due to ¯ J
(ˆ a, ˆ w ) = [-
-]0 .
Hence ∇ ¯ J (ˆ a, ˆ w ) = 0 and ∇ 2 ¯ J (ˆ a, ˆ w ) is positive semi-deﬁnite.
We further show that ∇ 2 ¯ J (ˆ a, ˆ w ) has multiple zero eigenvalues when m > m ∗ .
In fact, since
[Equation]
it is straightforward to verify that for any i, j ∈ I p , p ∈
[ m ∗ ] and any i, j ∈ I 0 ,r , r ∈ [ i 0 ] ,
[Equation]
where A i, : denotes the i -th row of matrix A . Notice that (cid:80) i ∈I 0 ,r ∇ 2 ¯ J (ˆ a, ˆ w ) m + i, : = 0 for any r ∈
[ i 0 ] , we conclude that the Hessian ∇ 2 ¯ J (ˆ a, ˆ w ) has at most 2 m ∗
{++ i 
 + i 2 ≤ 2 m ∗ + |I 0 | ≤ m + m ∗ different rows, 6 which yields rank(+} ∇ 2 ¯ J
(ˆ a, ˆ w )) ≤ 2 m ∗ + |I 0 | ≤ m + m ∗ . Therefore, r i
I ,r | When I
∅ , 2 m ∗ ; when I , since I for any r ∈ [ i 0 ] , let i the number of zero eigenvalues of ∇
[Equation]
J
[Equation]
ˆ
[Equation]
, ˆ w ) ≥ dim
[Equation]
− rank( ∇ [Equation] [Equation] ≥
[Equation]
|
[Equation] | ≥ m − m ∗ .
Since ∇ 2 ¯ J (ˆ a, ˆ w ) [-, µ ∈ (cid:83) mk =
-]is positive semi-deﬁnite, all the non-zero eigenvalues must be positive.
(ii)
[Equation]
w ) .
Let
[Equation]
By Gershgorin’s circle theorem, for any eigenvalue of E ω (ˆ a, ˆ w ) , say λ ( ω ) {+, we have λ ( ω ) ∈ (cid:83) mk =
+}
( G (1) k ∪ G (2) k ) .
Combining with Lemma C.1, we get
[Equation]
If ρ 0 has the sub-Gaussian tail (79), by (91), we further have
[Equation]
where ω ∈ (0 , min { 1 / 2 , 1 /t 0 , 2 c 1 / ˆ w min } ) , ˆ w min := min i ∈ [ m ] ˆ w i > 0 , c 2 , c 3 > 1 are constants only related to c 1 , t 0 , and (cid:46) hides universal constants only depending on ρ 0 , t 0 , c 0 , c 1 .
Combining (i), (ii) and applying Weyl’s theorem gives the desired result.
(4) Local Linearization Analysis
The previous analysis can now be tied directly to a quantitative dynamics via linearization arguments.
It is shown that under mild assumptions, the gradient ﬂow (21) can become trapped in plateaus with an exponentially large timescale.
That is, the curse of memory occurs, this time in optimization dynamics instead of approximation rates.
Theorem C.1 (Restatement of Theorem 5.1) .
For any ω > 0 , m ∈ N + and θ 0 = ( a 0 , w 0 )
∈
R m × R m + , 0 < δ (cid:28) 1 , deﬁne the hitting time
[Equation]
Assume that m > m ∗ , and the initialization satisﬁes ˆ ρ ( t ; θ 0 ) ≈ ¯ ρ ( t ) . Then we have
[Equation]
In particular, if ρ 0 has the sub-Gaussian tail (79), and the initialization is bounded as ( a 0 , w 0 ) ∈ [ a 0 l , a 0 r ] m × [ w 0 l , w 0 r ] m with constants a 0 l < a 0 r , 0 < w 0 l < w 0 r , we further have [Equation] [Equation] for any ω ∈ (0 , min { 1 / 2 , 1 /t 0 , 2 c 1 /w 0 r } ) sufﬁciently small, where (cid:38) hides universal constants only depending on ρ 0 , t 0 , c 0 , c 1 , w 0 r , a 0 l and a 0 r .
Proof.
Consider the asymptotic expansion with the form
[Equation]
for some δ ∈ (0 , 1) (with δ (cid:28) 1 ) and θ iω ( τ ) =
O (1) ( τ ≥ 0 , i = 0 , 1 , · · · ).
7 For consistency, we have θ 0 ω (0)
=
θ 0 and θ iω (0) =
for i = 1 , 2 , · · · .
By continuity, τ 0 > 0 and (cid:107) θ ω ( τ ) − θ 0 (cid:107) 2 ≤ δ for any τ ∈ [0 , τ 0 ] .
The aim is to quantify the scale of τ 0 .
Let g 0 :
= ∇ J ω ( θ 0 ) and H 0 : = ∇ 2 J ω ( θ 0 ) .
The local linearization on (21) shows
[Equation]
Combining with (108), we have
[Equation]
Therefore
[Equation]
which gives
[Equation]
To achieve a parameter separation gap δ 0 , i.e. (cid:107) θ ω
( τ ) − θ 0 (cid:107) 2 = δ 0 with δ[-
 = O ( δ ) and δ-] 0 = cδ , c ∈ (0 , 1] , we need to take τ such that
[Equation]
Let H
= P (cid:62) Λ P be the eigenvalue decomposition with P orthogonal and Λ = diag( λ 1 , · · · λ 2 m ) consisting of the eigenvalues of H 0 with λ 1 ≥ · · · ≥ λ 2 m .
Then
[Equation]
It is straightforward to verify that h ( τ ; λ ) :
= 1 | λ | | e − λτ − 1 | , τ ≥ 0 monotonically decreases on λ ∈ R for any τ ≥ 0 .
Hence
[Equation]
) and the right hand side monotonically increases on τ ≥ 0 . Combining (110), (111) gives [Equation]
We discuss for different cases: (i) (cid:107) g 0 (cid:107) 2 = 0 . Obviously the inequality (112) fails since (110) fails for any τ ≥ 0 , which gives τ 0 = + ∞ ; (ii) (cid:107) g 0 (cid:107) 2 (cid:54) = 0 and λ 2 m ≥ 0 . By (112), we get τ ≥
[Equation]
(cid:107) 2 ; (iii) (cid:107) g 0 (cid:107) 2 (cid:54) = 0 and λ 2 m < 0 . By (112), we get [Equation] If − λ 2 m ≤ 2 (cid:107) g 0 (cid:107) 2 , we have
[Equation]
; if − λ 2 m > 2 (cid:107) g 0 (cid:107) [-
-]2 , we have τ ≥ ln(1+ δ 0 ) − λ 2 m . Combining (i), (ii), (iii) gives
[Equation] Let the initialization satisfy ˆ ρ ( t ; θ 0 ) ≡ ¯ ρ ( t ) , and assume m > m ∗ . According to Propositions C. and C.3, we have
[Equation]
If ρ 0 has the sub-Gaussian tail (79), again by Propositions C.2 and C.3, we further have [Equation] for any ω ∈ (0 , min { 1 / 2 , 1 /t 0 , 2 c 1 /w 0 , min } ) , where w 0 , min :=
min i ∈ [ m ] w 0 ,i > 0 , c 2 , c 3 > 1 are constants only related to c 1 , t 0 , and (cid:38) hides universal constants only depending on ρ 0 , t 0 , c 0 , c 1 . Since the initialization is bounded as ( a 0 , w 0 ) ∈ [ a 0 l , a 0 r ] m × [ w 0 l , w 0 r ] m with a 0 l < a 0 r , 0 < w 0 l < w 0 r , let c 0 a = max {| a 0 l | , | a 0 r |} , we get
[Equation]
where (cid:38) hides universal constants only related to w 0 r , a 0 l and a 0 r .
The last task is to show the dynamics of the loss is much slower than the parameter separation when there is plateauing.
The argument is trivial since for any τ ∈ [0 , τ 0 ] ,
[Equation]
ω → 0 + .
By continuity, the proof is completed.
Remark C.3.
The estimate in Theorem C.1 shows a lower bound on the escape time, hence it does not appear to preclude the situation that the plateauing lasts forever.
However, in the proof above, if one supposes τ 0 = + ∞ in (104), i.e. the hypothetical situation where the parameters are trapped forever, and write ˜ g 0 : =
{+Pg 
+}
= (˜ g 0 , 1 , · · · , ˜ g 0 , 2 m ) , we have
[Equation]
τ
[Equation]
λ i ))
[Equation]
≥
[Equation]
˜ g [Equation] [Equation] h
[Equation]
τ [Equation] λ j ))for any j such that λ j < 0 . If ˜ g 0 ,j (cid:54) = 0 , (109) gives
[Equation]
which is a contradiction.
That is to say, the parameter separation has to achieve the gap δ within a ﬁnite time, even if it is exponentially large.
Remark C.4.
Recall Lemma C.3, ˆ ρ ( t ; a 0 , w 0 ) ≡ ¯ ρ ( t ) if and only if ( a 0 , w 0 ) ∈ M ∗ =
(cid:83) P M ∗P , where P is a partition over [ m ] as deﬁned in Deﬁnition C.1.
That is, as a union of afﬁne spaces, M ∗ is in fact an equivalent set for qualiﬁed initializations.
As discussed in Remark C.2, when there is no degeneracy, the cardinality of M ∗ is m ∗ ! (cid:26) mm ∗ (cid:27) (i.e. the number of P ), with each M ∗P an ( m − m ∗ ) -dimensional afﬁne space; when there is degeneracy in some M ∗P , it then becoms an uncountable set.
Certainly, initializations sufﬁciently near M ∗ are also qualiﬁed by continuity.
Remark C.5.
Motivated by the idea of weights degeneracy (see Deﬁnition C.1), we can further apply similar methods to a (global) landscape analysis on the loss function J ω .
The results there show that the plateaus are all over the landscape, even provided general targets (without memory structures).
See details in Appendix D.
C. N UMERICAL E XPERIMENTS C.3. M OTIVATING T ESTS
In this section we give details of Figure 1.
(1) Learning linear functionals using
linear RNNs
optimizer)
The target functional is H T ( x )
= (cid:82) T 0 ρ ( t ) x
T − t dt with white noise x , while the representation ρ is selected as the exponential sum or the scaled Airy function:
Exponential sum: ρ ( t ) =
[ c ∗ ] (cid:62) e W ∗ t b ∗ , where c ∗ , b ∗ are standard normal random vectors of m ∗ dimensions, and W = −
I − Z (cid:62) Z with Z ∈ R m ∗ × m ∗ is a Gaussian random matrix with i.i.d.
entries having variance 1 /m ∗ .
Airy function: ρ ( t ) =
Ai( s 0 [ t − t 0 ]) , where Ai( t ) is the Airy function of the ﬁrst kind, given by the improper integral
[Equation]
Note that in the ﬁrst example, the memory of target functional decays quickly.
However, for the second example, the effective rate of decay is controlled by the parameter t 0 .
For t ≤ t 0 , the Airy function is oscillatory.
Hence for large t 0 , a large amount of memory is present in the target.
In
Figure 1 we set m ∗ = 8 for exponential sums, and t 0 = 3 , s
= 2 .
25 for Airy functions.
In Figure 1 (a) and (b), we plot the gradient descent dynamics on training the linear RNNs (discretized using Euler method, hence equivalent to residual RNNs).
We observe an efﬁcient training process for the exponential sum case, while “plateauing” behaviors in the Airy function case.
This causes a severe slow down of training.
In addition, we also ﬁnd that the plateauing effect gets worse as t 0 (or s 0 ) is increased, which corresponds to more complex Airy functions in the sense of more memory effects.
That is, the long-term memory adversely affects the optimization process via gradient descent.
(2) Learning nonlinear functionals using nonlinear RNNs (with Adam optimizer)
To show that the plateauing behavior may be generic, we also consider a nonlinear forced dynamical system target, the Lorenz 96 system (Lorenz, 1996):
[Equation]
with cyclic indices z k + K = z k , and x is an external stochastic noise.
When the unresolved variables z k are unknown, the dynamics of the resolved variable y driven by x is a nonlinear dynamical system with memory effects (but not a linear functional).
We use a standard nonlinear RNN (with the tanh activation) to learn the sequence-to-sequence mapping x 0: T (cid:55)→ y 0: T with the Adam optimizer.
Figure 1 (c) shows that the training of the Lorenz 96 system with the presence of memory also exhibits the plateauing phenomenon. [Figure]
Here we take the width m = 2 in ˆ ρ .
The corresponding gradient ﬂow (21) is numerically solved by the Adams-Bashforth-Moulton method.
Observe that the plateauing time increases rapidly as the memory becomes longer ( ω decreases).
In all cases, the trained RNN model has a hidden dimension of 16 and the total length of the path is
T = 6 . 4 .
The continuous-time RNN is discretized using the Euler method with step size 0 .
C.3. L ONG - TERM M EMORY S IGNIFICANTLY C ONTRIBUTES TO P LATEAUS
Recall the simple example of target with memory
[Equation]
We aim to learn ρ with the exponential sum ˆ
[Equation]
[Equation]
a,
[Equation]
(cid:88)
[Equation]
a ∈
[Equation]
m , w ∈ [Equation] m + . That is, the simpliﬁed linear RNN model with a diagonal recurrent kernel (see (76) and Appendix C.1).
The optimization is performed by the gradient ﬂow (21), i.e. a continuous-time idealization of the gradient descent dynamics.
The ODE (21) is numerically solved by the AdamsBashforth-Moulton method.
The results are illustrated in Figure 2.
It is shown that the plateauing time increases rapidly as the memory 1 /ω becomes longer.
C.3. N UMERICAL V ERIFICATIONS
1) The timescale estimate
We ﬁrst numerically verify the timescale proved in Theorem 5.1 (or Theorem C.1).
That is, the time of plateauing (and also parameter separation (cid:107) θ
( τ ) − θ (0) (cid:107) 2 ) is exponentially large as the memory 1 /ω →
+ ∞ .
The results are shown in Figure 3, where we observe good agreement with the predicted scaling.
(2) General cases
To facilitate mathematical tractability, the analysis so far is done on the restrictive cases of the diagonal recurrent kernel W with negative entries, linear activations and the gradient ﬂow training dynamics.
However, we show here that the plateauing behavior - which we now understood as a generic feature of long-term memory of the target functional and its interaction with the optimization dynamics - is present even for general cases, and hence our simpliﬁed analytical setting is representative of the general situation.
In Figure 4, we still take the target functional as deﬁned in (119), but apply more general models to learn it, including using full (non-diagonal) recurrent kernels of the RNN with no restrictions on [Figure] entries, using non-linear ( tanh ) activations and using the Adam optimizer.
Furthermore, we do not take the It
ˆ o isometry simpliﬁcation
and instead use actual input sample paths of ﬁnite time horizons, just as one would do in training RNNs in practice.
We observe that the plateauing behavior is present in all cases.
Moreover, in the last case of Adam (which can be thought of as a momentumbased optimization method) [Figure] the plateauing behavior is somewhat alleviated [Figure]
although the separation of timescales is still present.
This is consistent with our supplemental analysis in Appendix E, where we show that the momentum-based methods will speed up training based on our dynamical (Appendix C.2) and landscape analysis (Appendix D) of plateauing [Figure] The learning rate is 1 . 0 for GD and 001 for Adam
10 initializations are and trained for each experiment. We consider two possible input distributions a) white noise inputs b) of the form x t = (cid:80) Jj =1 α j cos( λ j t ) λ j ∼ U [0 , 10] and α j ∼ N (0 , 1)
We D L ANDSCAPE A NALYSIS As mentioned in Remark C.5, we can perform a global landscape analysis on the loss function based on the idea of weights degeneracy, which arises from Deﬁnition C.1.
Recall that the loss function reads
[Equation]
The main results of the appendix are summarized as follows.
• In Theorem D.1, we prove that the loss function has inﬁnitely many critical points, which form a factorial number of afﬁne spaces; •
In Theorem D.2, we prove that such (critical) afﬁne spaces are much more than global minimizers provided the target being an exponential
sum;• In Theorem D.3, we prove that on such (critical) afﬁne spaces, the Hessian is singular in the sense of processing multiple zero eigenvalues; •
In Proposition D.1, we prove that the (critical) afﬁne spaces contain both saddles and degenerate stable points which are not global optimal.
Instead of a local dynamical analysis in Appendix C.2, we generalize similar methods to a global landscape analysis here, and the results hold for the loss function associated with general targets .
More speciﬁcally, these results complement our main results (see Theorem 5.1 or Theorem C.1) in the following aspects.
• It is shown that the weights degeneracy is quite common in the whole landscape of the loss function.
Unfortunately, the weights degeneracy often worsens the landscape to a large extent; • It is shown that the weights degeneracy leads to a large number of stable areas (i
e. critical afﬁne spaces), but most of them contribute to non-global minimizers; • It is shown that these stable areas can also be quite ﬂat, which often connect with local plateaus; • For the structure of these stable areas, there are both saddles and degenerate critical points (not global optimal).
In certain regimes, even saddles can be rather difﬁcult to escape from (see Theorem 5.1 or Theorem C.1).
As a consequence, the optimization problem of linear RNNs is globally and essentially difﬁcult to solve.
D. S YMMETRY A NALYSIS ON THE L ANDSCAPE
This subsection consists of two parts: in Appendix D.1.1, we give main results provided the existence of weights degeneracy; in Appendix D.1.2, we give sufﬁcient conditions to guarantee the existence.
Since the key observation to utilize weights degeneracy is to notice the permutation symmetry of coordinates of gradients, we also called it “symmetry analysis”.
D.1. G ENERIC T HEORY
We begin with the following deﬁnition, which describes the concept of weights degeneracy in a natural and rigorous way.
Deﬁnition D.1.
(coincided critical solutions and afﬁne spaces)
Let d ∈ N + and 1 ≤ d ≤ m .
We say ( a, w ) is a d -coincided critical solution of J m , if ∇ J m ( a, w ) = 0 , and w = ( w i ) ∈
R m + has d different components.
The coincided critical afﬁne spaces are deﬁned as coincided critical solutions that form afﬁne spaces.
To guarantee the existence of such solutions, it is necessary to have the following deﬁnition.
Deﬁnition D.2.
(ˆ a, ˆ w ) ∈ R m ⊗ R m + is called the non-degenerate global minimizer of J m , if and only if
[Equation]
and (ˆ a, ˆ w ) takes a non-degenerate form
[Equation]
i
[Equation]
For convenience, we also deﬁne an index set
[Equation]
degenerate global minimizers
[Equation]
any
[Equation]
n [Equation] [Equation] which is used frequently in the following analysis.
For any f ∈ L 2 [0 , ∞ ) , let L [ f ] be the Laplace transform of f , i.e. L [ f ]( s ) =
(cid:82) ∞
e − st f ( t ) dt, s > 0 .
We begin with the following lemma.
Lemma D.1.
Assume that ρ is smooth and √ w |L [ ρ ]( w ) | → 0 as w → 0 + and w → ∞ .
Then we have 1 ∈ N and thus N (cid:54) = ∅ .
Proof.
We aim to show that there exists ˆ a (cid:54) =
and ˆ w > 0 ,
such that
[Equation]
The basic idea is to limit the unbounded domain a ∈ R , w ∈ R + to a compact set without effecting the minimization of J 1 ( a, w ) .
We have
[Equation]
) , where a ( w ) :
=
2 w L [ ρ ]( w ) .
Write h ( w ) :
=
J 1 ( a ( w ) , w ) , then h (0 + ) = h ( ∞ )
=
(cid:107) ρ (cid:107) 2 L 2 [0 , ∞ ) .
Obviously h ( w ) <
(cid:107) ρ (cid:107)
L 2 [0 , ∞ ) for any w > 0 , hence min
[Equation]
which implies
[Equation]
That is to say, the minimization of J 1 ( a, w ) can be equivalently performed on a 2-dimensional smooth curve ( w, a ( w )) w ∈
[ w lb ,w ub ] , which is certainly a compact set.
By continuity, J 1 ( a, w ) has global minimizers, say (ˆ a, ˆ w ) .
Obviously ˆ w > 0 and ˆ a = a ( ˆ w ) (cid:54) =
(since ˆ a = 0 implies J 1 (ˆ a, w )
=
(cid:107) ρ (cid:107) 2 L 2 [0 , ∞ ) , certainly not a minimum), which completes the proof.
Remark D.1.
If the target is an exponential sum, i.e. ρ ( t ) = (cid:80) m ∗ j =1 a ∗ j e − w ∗
j t , we know ρ is smooth and √ w |L
[ ρ ]( w ) | → 0 as w → 0 + and w → + ∞ ; hence 1 ∈ N by Lemma D.1, and thus
N (cid:54) = ∅ .
In fact, L [ ρ ]( w ) = (cid:80) m ∗ j = a ∗ j w + w ∗ j implies that L [ ρ ]( w ) =
O (1) when w → 0 + , and
L [ ρ ]( w ) =
O (1 /w ) when w → + ∞ .
Theorem D.1.
Assume that N (cid:54) = ∅ with N deﬁned as (124).
Let M := sup N .
Then for any m, d ∈ N + , 1 ≤ d ≤ min { m, M } , there exists at least d ! (cid:26) m d (cid:27) d -coincided critical afﬁne spaces of J m , 10 where (cid:26) md (cid:27) ∈ N + is called the Stirling number of the second kind.
Proof.
(i) Existence.
The key observation is the permutation symmetry of ∇ J m : by (19), if a i = a j and w i = w j for some i (cid:54) =
j , then ∂J m ∂a
i = ∂J
m ∂a j and ∂J
m ∂w i =
∂J
m ∂w j .
For any m, d ∈ N + , 1 ≤ d ≤ m , suppose that w = ( w i ) ∈
R m + has d different components.
Then for any partition P : { 1 , · · · , m } = ∪ dj
=
I j with I j 1 ∩ I j 2 = ∅ for any j 1 (cid:54) =
j 2 , j 1 , j 2 = 1 , · · · , d , deﬁne the afﬁne space M
[Equation]
any
[Equation]
 [Equation] for some ( b, v ) ∈ R d ⊗ R d + , where v has exactly d different components.
Therefore, for any ( a, w ) ∈
M P , ( b,v ) , ( m,d ) , we have
J [Equation] [Equation] (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
d (cid:88) j
[Equation] (cid:88) i ∈I j a [Equation] e
[Equation] w [Equation] t − ρ ( t ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
L 2 [0 , ∞ )
= (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
d (cid:88)
j = b j e − v j t − ρ ( t ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
L 2 [0 , ∞ )
=
J d ( b, v ) , and similarly ∂J m ∂a k
( a, [Equation] [Equation] (cid:90)
[Equation]
e − v s t   d (cid:88)
j = b j e − v j t − ρ ( t ) 
 dt, k ∈ I s , s = 1 , 2 , · · · , d,
∂J
m ∂w k ( a, w ) =
[Equation] k (cid:90) ∞ 0 ( − t )
e − v s t   d (cid:88)
j = b j e − v j t − ρ ( t ) 
 dt, k ∈ I s , s = 1 , 2 , · · · , d.
Notice that ∂J [Equation] ∂b [Equation]
[Equation] (cid:90)
[Equation]
e − v s t   d (cid:88)
j = b j e − v j t − ρ ( t ) 
 dt, s = 1 , 2 , · · · , d, ∂J d ∂v s ( b, v ) =
2 b s (cid:90) ∞ 0 ( − t ) e − v s t   d (cid:88) j = b j e − v j t − ρ ( t )   dt, s = 1 , 2 , · · · , d, we have ∂J m
[Equation]
[Equation]
11 Since d ≤ min { m, M } , d ∈ N .
In fact, for any k ∈ N + , if k / ∈ N , there exists i ≤ k such that J ( i ) has no non-degenerate global minimizers, we have j / ∈ N for any j ≥ i , hence
M ≤ i − 1 ≤ k − 1 .
Hence M = ∞ implies N = N +
and M < ∞ implies M ∈ N , and both of them lead to d ∈ N .
Therefore, J d has non-degenerate global minimizers, i.e. there exists (ˆ b, ˆ v ) ∈ R d ⊗ R d + such that
[Equation]
and (ˆ b, ˆ v ) takes a non-degenerate form ˆ b i (cid:54) = 0 , ˆ v i (cid:54) =
ˆ v j for any i (cid:54) =
j, i
[Equation]
By (127), we get ∇ J d (ˆ b, ˆ v ) = 0 .
Combining with (126) and (128), we obtain ∇ J m (ˆ a, ˆ w ) =
for any (ˆ a, ˆ w ) ∈
M P , (ˆ b, ˆ v ) , ( m,d ) , i.e. (ˆ a, ˆ w ) belongs to a d -coincided critical afﬁne space.
Note that the afﬁne space the dimension dj =
( |I j | − 1) = m − d , since there are d linear equality constrains on the m -dimensional vector a .
(ii) Counting.
By the structure of afﬁne spaces discussed above, we can identify different afﬁne spaces with respect to the partition P .
For counting the number of different partitions P : { 1 , · · · , m } = ∪ dj =1 I j , it can be decomposed into the following two steps.
First, partitioning a set of m labelled objects into d nonempty unlabelled subsets.
By deﬁnition, the answer is the Stirling number of the second kind (cid:26) md (cid:27) .
Second, assign each partition to I 1 , · · · , I d accordingly.
There are d ! ways in total.
Therefore, the number of d -coincided critical afﬁne spaces is at least d ! (cid:26) md (cid:27) .
The proof is completed.
Combining Lemma D.1, Remark D.1 and Theorem D.1 gives the following theorem, which states that there are much more saddles and degenerate stable points which are not global optimal than global minimizers in the landscape (provided the target being an exponential sum).
Theorem D.2.
Fix any m ∈ N + relatively large.
Consider the loss J m associated with the target being a non-degenerate exponential sum, i.e. ρ ( t ) =
(cid:80) mj
=1 a ∗ j e − w ∗
j t , where a ∗ j (cid:54) = 0 and w ∗ i (cid:54) =
w ∗ j for any i (cid:54) =
j , i, j = 1 , · · · , m .
Assume that m ∈ N 12 with N deﬁned in (124).
Then in the landscape of J m , the number of coincided critical afﬁne spaces is at least Poly ( m ) times larger than the number of global minimizers.
Proof.
[-= 
-]
(i)
Global minimizers.
Since the target is an exponential sum, we have J m ( a, w ) ≥ 0 and J m (¯ a ∗ , ¯ w ∗ ) = 0 , where ¯ a ∗ =
Pa ∗ and ¯ w ∗ =
Pw ∗ with P ∈ R m × m to be some permutation matrix.
Next we show J m has no other global minimizers.
Suppose J m ( a, w ) = 0 , we have
[Equation]
It is easy to see that for any j = 1 , · · · , m , there exists i ( j ) such that w i ( j ) =
w ∗ j .
Otherwise, if w i (cid:54) =
w ∗ j , i = 1 , · · · , m , by (83) or Lemma C.2, we have a ∗ j = 0 , which is a contradiction.
Notice that w ∗ i (cid:54) =
w ∗ j for any i (cid:54) =
j , different w ∗ j ’s will correspond to different w i ’s, hence the correspondence is one-to-one.
Therefore, let w i =
w ∗ j ( i ) , (129) can be rewritten as
[Equation]
Again by Lemma C.2, we have a i = a ∗ j ( i ) .
That is to say, J m ( a, w ) = 0 implies a = Pa ∗ and w =
Pw ∗ with P ∈ R m × m to be some permutation matrix.
This gives m !
global minimizers.
(ii) Coincided critical afﬁne spaces.
Obviously N (cid:54) = ∅ , and M = sup N ≥ m .
According to
Theorem D.1, for any d ∈ N + , 1 ≤ d ≤ min { m, M } = m , we have at least d ! (cid:26) md (cid:27) d
-coincided critical afﬁne spaces of J m .
By (i), for any d ≤ m − 1 , there are no global minimizers in these afﬁne spaces.
Counting the total number
[Equation]
(iii) Comparison.
To give a bound between (130) and m ! , we need an elementary recurrence
[Equation]
• For d = m − 1 , let p m :=
[Equation]
, then
[Equation]
• For d = m − 2 , let q m :=
[Equation]
, then
[Equation]
Combining above gives
[Equation]
which is a quadratic polynominal on m .
The proof is completed.
Remark D.2.
We only take the last two terms of (130) for a lower bound, which is obviously rather loose.
In principle, a Poly( m ) bound with higher degrees can be similarly obtained.
That is to say, on one hand, there are inﬁnitely many critical points forming afﬁne spaces in the landscape of J m ; on the other hand, we see that even if only counting the number of afﬁne spaces, there are still much less global minimizers (provided the width m relatively large).
Remark D.3.
When the target ρ is not an exponential sum, it is straightforward to see Theorem D. still holds if there is a ﬁnite number (with the scale of no more than factorial) of global minimizers.
Now we get down to investigate ∇ 2 J m on the above coincided critical afﬁne spaces.
It is shown that ∇ 2 J m is singular and can have multiple zero eigenvalues.
Theorem D.3.
Fix any m, d ∈ N + , 1 ≤ d ≤ m .
On the d -coincided critical afﬁne spaces (induced by non-degenerate global minimizers of J d 13 ) of J m , ∇ 2 J m is with rank at most m + d , and hence has at least m − d zero eigenvalues.
Proof.
A straightforward computation shows that, for k, l = 1 , 2 , · · · , m ,
[Equation]
Let the induced d -coincided critical afﬁne space be M P , (ˆ b, ˆ v ) , ( m,d ) , as is derived in the proof of
Theorem D.1.
Since (ˆ b, ˆ v ) is the non-degenerate global minimizer of J d , we have (cid:90)
[Equation]
for any (ˆ a, ˆ w ) ∈
M P , (ˆ b, ˆ v ) , ( m,d ) .
This gives
[Equation]
[Equation]
Now we show that, for any i, j ∈ I s , i (cid:54) =
j , s = 1 , 2 , · · · , d , ∇ 2 J m (ˆ a, ˆ w ) i, : = ∇ 2 J m (ˆ a, ˆ w ) j, : .
In fact, for any k = 1 , · · · , m , let k ∈
I s (cid:48) , then by (131),
[Equation]
For k (cid:54) =
i and k (cid:54)
=
j , (132) gives
[-( ˆ w i + ˆ w k ) 
-]
[Equation]
By (134), for k = i (cid:54) =
j ,
[Equation]
and similarly for k = j (cid:54) = i ,
[Equation]
That is to say, there are at most m + d different rows in the symmetric matrix ∇ 2 J m (ˆ a, ˆ w ) ∈
R 2 m × 2 m , hence rank (cid:0) ∇ 2 J m (ˆ a, ˆ w ) (cid:1) ≤ m + d .
Therefore, the number of zero eigenvalues of ∇ 2 J m (ˆ a, ˆ w ) ≥ dim (cid:8) x ∈ R 2 m : ∇ 2 J m (ˆ a, ˆ w ) · x = 0 (cid:9) =
2 m − rank( ∇ 2 J m (ˆ a, ˆ w )) ≥ m − d .
The proof is completed.
Remark D.4.
The bound in Theorem D.3 is not sharp, since the estimate on rank (cid:0) ∇ 2 J m (cid:1) here is loose as only rows with the same elements are considered.
In practice (numerical tests), it is often observed that there are more zero eigenvalues of rank (cid:0) ∇ 2 J m (cid:1) on the coincided critical afﬁne space
M P , (ˆ b, ˆ v ) , ( m,d ) .
Remark D.5.
Theorem D.3 shows that, there are local plateaus around the d -coincided critical afﬁne spaces M P , (ˆ b, ˆ v ) , ( m,d ) for d ≤ m − 1 .
In addition, the 0 -eigenspace of J m is higherdimensional for smaller d , which may suggest that one can stuck on plateaus more easily.
D.1. S UFFICIENT C ONDITIONS
There is still a gap when connecting Theorem D.1 and Theorem D.2.
That is, it is necessary to guarantee sup N relatively large, i.e. J 1 , J 2 , · · · , J d all have non-degenerate global minimizers for d as large as possible.
Motivated by Kammler (1979a), we can give some sufﬁcient conditions by limiting the target ρ within a smaller function space, the so-called completely monotonic functions.
Deﬁnition D.3.
F ∈ C [0 , ∞ ]
∩ C ∞ (0 , ∞ ) is called completely monotonic, if and only if [Equation] and F ( ∞ ) = 0 .
Remark D.6.
Several examples of completely monotonic functions:
• ρ ( t ) = 1 / (1 + t ) α for any α > 0 ; • The non-degenerate exponential sum with positive coefﬁcients
[Equation]
· , m ∗ . Since the space of exponential sums is not close, we turn to consider the problem of ﬁnding a best approximation to a given ρ ∈ L 2 [0 , ∞ ) from the set V
[Equation] [Equation]
[Equation] [Equation]
[Equation]
D
[Equation]
d )]ˆ ρ = 0 for some w 1 , · · · w d ∈ R + (cid:9) (135) with respect to the common L 2 -norm, i.e. inf ˆ ρ ∈ V d ( R + ) (cid:107) ˆ ρ − ρ (cid:107) L 2 [0 , ∞ ) , where D denotes the common differential operator.
Obviously V d ( R + ) ⊂
L 2 [0 , ∞ ) and V d ( R + ) (cid:40)
V d +
( R + ) for any d ∈ N + .
Kammler (1979a) proves the following theorem.
Theorem D.4.
Assume ρ ∈ L 2 [0 , ∞ ) to be completely monotonic.
Then there exists a best approximation ˆ ρ 0 to ρ in V d ( R + ) , i.e.
[Equation]
When ρ / ∈ V d ( R + ) , any such best approximation admits a non-degenerate form
[Equation]
and satisﬁes the generalized Aigrain-Williams equations
[Equation]
Note that (136) and (137) are pretty similar to Deﬁnition D.2, except for a different choice of hypothesis function space.
Now we show a connection between these two problems.
Theorem D.5.
Assume ρ ∈ L 2 [0 , ∞ ) to be completely monotonic, and ρ / ∈ V d ( R + ) for some d ∈ N + .
Then J d has non-degenerate global minimizers (ˆ b, ˆ v ) ∈ R d ⊗ R d + .
Proof.
According to Theorem D.4, there exists a non-degenerate best approximation ˆ ρ 0 to ρ from
V d ( R + ) , i.e.
[Equation]
We aim to prove J d (ˆ b, ˆ v ) =
inf b ∈ R d ,v ∈ R d + J d ( b, v ) .
Deﬁne the following subsets of exponential sums
[Equation]
V
[Equation]
[Equation]
R + ) :
= (cid:26) ˆ
ρ ∈ V d ( R + ) : w = ( w i ) has k diﬀerent components (cid:27) , 1 ≤ k ≤ d, then we have inf b ∈ R d ,v
[Equation]
[Equation]
inf
[Equation]
[Equation]
ˆ ρ − ρ (cid:107) 2 L 2 [0 , ∞ ) .
It is straightforward to verify that
V d ( R + ) =
(cid:83) dk
=1 V d,k
( R + ) , and V d,k ( R + ) =
V k,k ( R + ) (cid:40)
V k
( R + ) for k = 1 , · · · , d .
By (140), we get
[Equation]
Since ˆ ρ 0 ∈ V d,d ( R + ) , we have
[Equation]
The last task is to show
[Equation]
+ ) (cid:107) ˆ ρ − ρ (cid:107) L 2 [0 , ∞ ) = inf ˆ ρ ∈V d,d ( R + ) (cid:107) ˆ ρ − ρ (cid:107) L 2 [0 , ∞ ) .
In fact, for any ˆ ρ ∈ V k,k , ˆ ρ ( t ) = (cid:80) ki =1 a i e − w i t , let ˜ a :
=
( a 1 , · · · , a k , 0) , ˜ w := ( w 1 , · · · , w k , 1 + max 1 ≤ i ≤ k w i ) , we get ˆ ρ ( t ) :
= (cid:80) k
+1 i =
˜ a
i e − ˜ w i t ∈ V k +1 ,k +1 , which implies V k,k ⊂ V k +1 ,k +1 .
Therefore,
[Equation]
which completes the proof.
Combining Theorem D.1 and Theorem D.5 immediately gives the following result.
Theorem D.6.
Assume ρ ∈ L 2 [0 , ∞ ) to be completely monotonic, and ρ / ∈ V 1 ( R + ) .
Let D :=
{ d ∈ N + : ρ / ∈ V d ( R + ) } , D 0 :=
sup D and write m (cid:48) :
= min { m, D 0 } .
Then the total number of coincided critical afﬁne spaces of J m is at least (cid:80) m (cid:48)
d =1 d !
[Equation]
md (cid:27) .
Proof.
We have 1 ∈ D and thus D (cid:54) = ∅ , D 0 ≥ 1 .
Since V d ( R + ) (cid:40) V d +1 ( R + ) for any d ∈ N + , we have D = { 1 , 2 , · · · , D 0 } if D 0 < ∞ , and D = N + if D 0 = ∞ .
14 Both of them gives { 1 , 2 , · · · , m (cid:48) } ⊂ D ,
i.e. ρ / ∈ V k
( R + ) for any k ≤ m (cid:48) .
By Theorem D.5, J ( k ) has non-degenerate global minimizers for any k ≤ m (cid:48) , i.e. m (
cid:48) ∈ N .
According to Theorem D.1, for any d ∈ N + , 1 ≤ d ≤ m (cid:48)
=
min { m, m (cid:48) } ≤ min { m, M } , there exists at least d ! (cid:26) md (cid:27) d -coincided critical afﬁne spaces of J m .
Sum over d gives the total number (cid:80) m (cid:48)
d =1 d ! (cid:26) md (cid:27) .
Remark D.7.
Examples:
• Suppose the target is ρ ( t )
= 1 / (1+ t )
α , α > 0 , then D = N + and D 0 = ∞ .
The total number of coincided critical afﬁne spaces of the corresponding J m is at least (cid:80) md
=1 d ! (cid:26) md (cid:27) .
• Suppose the target is an non-degenerate exponential sum with positive coefﬁcients: ρ ( t ) =
(cid:80) mj
=1 a ∗ j e − w ∗
j t , where a ∗ j > 0 and w ∗ i (cid:54) =
w ∗ j for any i (cid:54) =
j , i, j = 1 , · · · , m .
Then D = { 1 , 2 , · · · , m − 1 } and D
= m − 1 .
The total number of coincided critical afﬁne spaces of the corresponding J m is at least (cid:80) m − 1 d =1 d ! (cid:26) md (cid:27) , which is exactly (130).
An complement for Theorem D.2 is as follows.
Theorem D.7.
Fix any m ∈ N + relatively large.
Consider the loss J m associated with the target being a non-degenerate exponential sum with positive coefﬁcients, i.e. ρ ( t ) =
(cid:80) mj
=1 a ∗ j e − w ∗
j t , where a ∗ j > 0 and w ∗ i (cid:54) =
w ∗ j for any i (cid:54) =
j , i, j = 1 , · · · , m .
Then in the landscape of J m , the number of coincided critical afﬁne spaces is at least Poly ( m ) times larger than the number of global minimizers.
Proof.
By Theorem D.2, we only need to show is m ∈ N .
Since ρ ∈ L 2 [0 , ∞ ) is completely monotonic, and
ρ / ∈ V k
( R + ) for any k ≤ m − 1 , then by Theorem D.5, J ( k ) has non-degenerate global minimizers for any k ≤ m − 1 , i.e. m − 1 ∈ N .
The proof is completed by noticing that J m obviously has non-degenerate global minimizers, e.g. ( a ∗ , w ∗ ) .
D.1. A L OW - DIMENSIONAL E XAMPLE
To further understand the structure of coincided critical afﬁne spaces, we focus on a speciﬁc lowdimensional example here.
That is
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation] [Equation] , with the target to be a non-degenerate exponential sum ρ ( t ) = (cid:80) m ∗ j =1 a ∗ j e − w ∗
j t , where a ∗ j (cid:54) = 0 and w ∗ i (cid:54) =
w ∗ j for any i (cid:54) =
j , i, j = 1 , · · · , m ∗ .
As we will show later, the coincided critical afﬁne spaces of J 2 contain both saddles and degenerate stable points which are not global optimal .
By Lemma D.1, Remark D.1 and Theorem D.1, we know the 1 -coincided critical afﬁne space of Jexists, and it can be constructed by taking the non-degenerate global minimizer of J 1 , say (ˆ a, ˆ w ) with ˆ a (cid:54)
=
and ˆ w > 0 .
Then M (ˆ a, ˆ w ) , (2 , 1) :
= { ( a 1 , ˆ a − a 1 , ˆ w, ˆ w ) : a 1 ∈ R } ∈ R 4 is a line 15 , and ∇ J 2 ( a 1 , ˆ a − a 1 , ˆ w, ˆ w ) = 0 for any a 1 ∈ R .
Denote the Hessian of J 2 on the line M
(ˆ a, ˆ w ) , (2 , 1) by
A (ˆ a, ˆ w ) ( a 1 ) , i.e. A (ˆ a, ˆ w ) ( a 1 ) := ∇ 2 J 2 ( a 1 , ˆ a − a 1 , ˆ w, ˆ w ) .
We investigate the landscape of J 2 on the line M
(ˆ a, ˆ w ) , (2 , 1) by analyzing the eigenvalue distribution of A (ˆ a, ˆ w ) ( a 1 ) .
Proposition D.1.
Suppose m = m ∗ = 2 , and 0 < w ∗ 1 < w ∗ 2 .
Let I 1 :=
[0 , ˆ a ] and I 2 :=
( −∞ , 0) ∪ (ˆ a, + ∞ ) 16 .
Then
If a ∗ 1 a ∗ 2 < 0 , the minimal eigenvalue of A (ˆ a, ˆ w ) ( a 1 ) is 0 for any a 1 ∈ I 1 , and negative for any a 1 ∈ I 2 ; 2.
If a ∗ 1 a ∗ 2 > 0 and w ∗ 2 /w ∗ 1 < 2 + √ 3 , the minimal eigenvalue of A (ˆ a, ˆ w ) ( a 1 ) is negative for any a 1 ∈ I 1 , and 0 for any a 1 ∈ I 2 .
Proof.
Write c ( w ) :
= (cid:80) m ∗ j =1 a ∗ j (cid:104)
[Equation]
2 −
[Equation]
(cid:105) , and a 2 :
=
ˆ a − a 1 .
A straightforward computation shows that
[Equation]
{+a 1 2 ˆ w− a 
+}
[Table]
w 2ˆ w 1ˆ w − a
[Table]
{+2 ˆ w− a 
+}
[Table]
{+w− a 
+}
[Table]
{+w− a 
+}
[Table]
{+wa 
+}
[Table]
w 3 + 4 c ( ˆ w ) aa
[Table]
{+a 
+}
[Table]
{+w− a 
+}
[Table]
{+w− a 
+}
[Table]
wa 1 a 2 2 ˆ wa 22 2 ˆ w 3 + 4 c
( ˆ w ) a   . Considering the congruent transformation of A (ˆ a, ˆ w ) ( a 1 ) , which does not affect the index of inertia:
A (ˆ a, ˆ w ) ( a 1 ) =   1ˆ w 1ˆ w − a 1 2 ˆ w− a 2 2 ˆ w1ˆ w 1ˆ w − a
[Table]
2 ˆ w− a [Table] ˆ w
[Table] − a [Table] [Table]
w− a
[Table]
w
[Table]
a
[Table]
w
[Table]
+ 4 c ( ˆ w ) aa
[Table]
a
[Table]
w
[Table]
{+− a 
+}
[Table]
w− a
[Table]
w
[Table]
a
[Table]
a
[Table]
w
[Table]
a
[Table]
w
[Table]
+ 4 c ( ˆ w ) a
[Table]
{+  →   1ˆ w         a 
+}
[Table]
w
[Table]
+ 4 c ( ˆ w ) aa
[Table]
a
[Table]
w
[Table]
a
[Table]
a
[Table]
[Table] w [Table] a 22 4 ˆ w 3 + 4 c ( ˆ w ) a  , we see that A (ˆ a, ˆ w ) ( a 1 ) has one positive eigenvalue 1 / ˆ w and one eigenvalue 0.
What remains are the eigenvalues of A (cid:48)
(ˆ a, ˆ w ) ( a 1 ) :
= (cid:34) a 21 4 ˆ w 3 + 4 c ( ˆ w ) aa
[Table]
a
[Table]
ˆ wa 1 a 2 4 ˆ wa 22 4 ˆ w 3 + 4 c ( ˆ w ) a
[Equation]
To determine their signs, we compute
[Equation]
So we need to analyze the sign of ˆ ac ( ˆ w ) and ˆ a 2 + 16 ˆ w 3 ˆ ac ( ˆ w ) under different assumptions on ( a ∗ , w ∗ ) .
(i) a ∗ 1 a ∗ 2 < 0 .
By the optimality condition of (ˆ a, ˆ w ) for J 1 , we have
[Equation] [Equation]
and therefore
[Equation]
Write v j :=
w ∗ j / ˆ w , j = 1 , 2 , we get 0 < v 1 < v 2 , and
[Equation]
[Equation]
Therefore
[Equation]
(145) > 0 , which gives ˆ ac ( ˆ w ) > 0 and ˆ a
+ 16 ˆ w 3 ˆ ac ( ˆ w ) > 8 ˆ w 3 ˆ ac ( ˆ w ) > 0 .
(ii) a ∗ 1 a ∗ 2 > 0 , w ∗ 2 /w ∗ 1 < 2 + √ 3 .
By (145), ˆ ac ( ˆ w ) < 0 .
ˆ
[Equation]
ˆ
[Equation]
(cid:88)
[Equation]
a
[Equation]
Since 4 s 2 ( s 2 − 3 s + 1) 2 − 4 s 4 = {+4 s 2 ( s − 1) 
+}
[-[Equation] [Equation] Since 4 s 2 ( s 2 − 3 s + 1) 2 − 4 s 4 = 4 s 2 ( s − 1) 
-] [( s − 2) 2 − 3] , and 1 < s = u 2 /u 1 = ( ˆ w + w ∗ 2 ) / ( ˆ w + w ∗ 1 ) < w ∗ 2 /w ∗ 1 < 2 + √ 3 , we get ∆ c < 0 .
This implies c 2 − 2 s ( s 2 − 3 s + 1) c
+ s 4 > 0 and ˆ a 2 + 16 ˆ w 3 ˆ ac ( ˆ w ) > 0 .
In both (i) and (ii), ˆ a 2 + 16 ˆ w 3 ˆ ac ( ˆ w ) > 0 , which implies that there is at least one positive diagonal element of A (cid:48)
(ˆ a, ˆ w ) ( a 1 ) in a sufﬁciently small neighborhood of a 1 = 0 and a 1 = ˆ a .
By the
Rayleigh-Ritz Theorem and Weyl’s Theorem, A (cid:48)
(ˆ a, ˆ w ) ( a 1 ) has at least one positive eigenvalue in this neighborhood.
However, by (142), det( A (cid:48)
(ˆ a, ˆ w ) ( a 1 )) only changes the sign at a 1 = 0 and a 1 = ˆ a .
This implies another eigenvalue of A (cid:48)
(ˆ a, ˆ w ) ( a 1 ) changes the sign at a 1 = 0 and a 1 = ˆ a accordingly.
By different signs of ˆ ac ( ˆ w ) derived in (i) and (ii), and (142), the proof is completed.
Remark D.8.
From Proposition D.1, we see that there are both saddles and degenerate stable points of J 2 on the critical afﬁne spaces (line)
M (ˆ a, ˆ w ) , (2 , 1) , and each of them in fact forms afﬁne spaces (lines) respectively, but they are certainly not global minimizers.
Therefore, the gradientbased algorithms can get stuck around this afﬁne space, except that it meets saddles with negative eigenvalues of large magnitude.
E M OMENTUM H ELPS T RAINING : Q UADRATIC E XAMPLES
In practice, it is often the case that training is trapped in some very ﬂat regions (plateaus), where the loss function has rather small gradients and negative eigenvalues of Hessian.
Now we illustrate the escape dynamics (from plateaus) via a simple quadratic example.
Consider the loss function f ( x ) =
( x 21 − (cid:15)x 22 ) / 2 with 0 < (cid:15) (cid:28) 1 .
We check the escaping performance for continuous-in-time analogs of two optimization algorithms: gradient decent (GD) and momentum (heavy ball) method.
[Equation]
Consider the gradient ﬂow of f ( x ) with an initial value x 0 =
( δ, 1) (cid:62) , where 0 < δ (cid:28) 1 and δ
= O ( (cid:15) ) .
Thus (cid:107)∇ f ( x 0 ) (cid:107)
=
O ( (cid:15) ) , and (cid:26) x (cid:48)
[Equation]
[Equation]
[Equation]
It is easy to show that there are different timescales of (cid:96) 1 ( τ ) .
In fact, when τ = O (1 /(cid:15) ) , (cid:96) 1 ( τ ) =
O ( (cid:15) 2 ) e −| O (1 /(cid:15) ) | − (cid:15)e | O (1) |
= O ( (cid:15) ) .
However, when τ continuous to increase, say
[Equation]
(146) where δ 0 > 0 denotes the gap satisfying (cid:15)
=
o ( δ 0 ) , we get (cid:96) 1 ( τ ) ≤ (cid:96) 1 ( τ (cid:15) 1 )
=
O ( (cid:15) 2 ) − (cid:15)e 2 (cid:15) · 12 (cid:15) ln δ 0 (cid:15) /
2 =
O ( (cid:15) 2 ) − δ 0 / 2 < − δ 0 / 4 for any τ ≥ τ (cid:15) 1 .
(2) Momentum
The momentum algorithm has the update rule
[Equation]
[Equation]
where ρ ∈ R , η > 0 is the learning rate.
The continuous-in-time analog can be derived as (similar to the arguments in Su et al.
[Equation]
with x k := x ( k √ η ) and the step size √ η of the simple ﬁnite differences 17 .
Let x 1 = x 0 − η ∇ f ( x 0 ) , we also get x (cid:48) (0)
=
−√ η ∇ f
( x (0)) .
To facilitate a comparison to GD, we take η = 1 18 and ρ = 1 19 .
Plugging the expression of f , we can solve the ODE
[Equation]
Write (cid:96) 2 ( τ ) :
= f
( x ( τ )) .
It is not hard to show that there are still different timescales of (cid:96) 2 ( τ ) .
In fact, when τ = O (1 / √ (cid:15) ) , (cid:96) 2 ( τ ) =
O ( (cid:15) 2 ) | O (1) | − (cid:15) | O (1) | ( e | O (1) | + e −| O (1) | ) 2 =
O ( (cid:15) ) .
However, when τ continuous to increase, say
[Equation]
(148) we get (cid:96) 2 ( τ (cid:15) 2 ) =
O ( (cid:15) 2 ) − (cid:15) ( 1+ √ (cid:15) 2 ) 2 e 2 √ (cid:15)τ / 2 + O ( (cid:15) )
=
O ( (cid:15) ) − δ 0 < − δ 0 / 2 , hence (cid:96) 2 ( τ ) ≤
O ( (cid:15) ) − δ 0 < − δ 0 / 2 for any τ ≥ τ (cid:15) 2 .
Combining (1) and (2) , we have the following conclusions.
• For both training dynamics, there are different timescales in the loss function.
That is to say, relatively long time is needed to escape from the plateaus; • Comparing (146) and (148), we get different timescales of escaping: O (1 /(cid:15) · ln(1 /(cid:15) )) for GD and O (1 / √ (cid:15) · ln(1 /(cid:15) )) for momentum.
Just like the convex case, where momentum improves the convergence rate by weakening the dependence on condition number, we see momentum can also help to escape rather ﬂat saddles.