O N THE C URSE OF M EMORY IN R ECURRENT N EURAL N ETWORKS : A PPROXIMATION AND O PTIMIZATION A NALYSIS
Anonymous authors
Paper under double-blind review
A BSTRACT
We study the approximation properties and optimization dynamics of recurrent neural networks (RNNs) when applied to learn input-output relationships in temporal data.
We consider the simple but representative setting of using continuoustime linear RNNs to learn from data generated by linear relationships.
Mathematically, the latter can be understood as a sequence of linear functionals.
We prove a universal approximation theorem of such linear functionals and characterize the approximation rate.
Moreover, we perform a ﬁne-grained dynamical analysis of training linear RNNs by gradient methods.
A unifying theme uncovered is the non-trivial effect of memory, a notion that can be made precise in our framework, on both approximation and optimization: when there is long term memory in the target, it takes a large number of neurons to approximate it.
Moreover, the training process will suffer from slow downs.
In particular, both of these effects become exponentially more pronounced with increasing memory - a phenomenon we call the “curse of memory”.
These analyses represent a basic step towards a concrete mathematical understanding of new phenomenons that may arise in learning temporal relationships using recurrent architectures.
I NTRODUCTION
Recurrent neural networks (RNNs) (Rumelhart et al., 1986) are among the most frequently employed methods to build machine learning models on temporal data.
Despite its ubiquitous application (Baldi et al., 1999; Graves & Schmidhuber, 2009; Graves, 2013; Graves et al., 2013; Graves &
Jaitly, 2014; Gregor et al., 2015), some fundamental theoretical questions remain to be answered.
These come in several ﬂavors.
First, one may pose the approximation problem, which asks what kind of temporal input-output relationships can RNNs model to an arbitrary precision.
Second, one may also consider the optimization problem, which concerns the dynamics of training (say, by gradient descent) the RNN.
While such questions can be posed for any machine learning model, the crux of the problem for RNNs is how the recurrent structure of the model and the dynamical nature of the data shape the answers to these problems.
For example, it is often observed that when there are long term dependencies in the data (Bengio et al., 1994; Hochreiter et al., 2001), RNNs may encounter problems in learning, but such statements have rarely been put on precise footing.
In this paper, we make a step in this direction by studying the approximation and optimization properties of RNNs.
Compared with the static feed-forward setting, the key distinguishing feature here is the presence of temporal dynamics in terms of both the recurrent architectures in the model and the dynamical structures in the data.
Hence, to understand the inﬂuence of dynamics on learning is of fundamental importance.
As is often the case, the key effects of dynamics can already be revealed in the simplest linear setting.
For this reason, we will focus our analysis on linear RNNs, i.e. those with linear activations.
Further, we will employ a continuous-time analysis initially studied in the context of feed-forward architectures (E, 2017; Haber & Ruthotto, 2017; Li et al., 2017) and recently in recurrent settings (Ceni et al., 2019; Chang et al., 2019; Lim, 2020; Sherstinsky, 2018; Niu et al., 2019; Herrera et al., 2020; Rubanova et al., 2019) and idealize the RNN as a continuous-time dynamical system.
This allows us to phrase the problems under investigation in convenient analytical settings that accentuates the effect of dynamics.
In this case, the RNNs serve to approximate relationships represented by sequences of linear functionals.
On ﬁrst look the setting appears to be simple, but we show that it yields representative results that underlie key differences in the dynamical setting as opposed to static supervised learning problems.
In fact, we show that memory, which can be made precise by the decay rates of the target linear functionals, can affect both approximation rates and optimization dynamics in a non-trivial way.
Our main results are:
We give a systematic analysis of the approximation of linear functionals by continuoustime linear RNNs, including a precise characterization of the approximation rates in terms of regularity and memory of the target functional.
We give a ﬁne-grained analysis of the optimization dynamics when training linear RNNs, and show that the training efﬁciency is adversely affected by the presence of long term memory.
These results together paint a comprehensive picture of the interaction of learning and dynamics, and makes concrete the heuristic observations that the presence of long term memory affects RNN learning in a negative manner (Bengio et al., 1994; Hochreiter et al., 2001).
In particular, mirroring the classical curse of dimensionality (Bellman, 1957), we introduce the concept of the curse of memory that captures the new phenomena that arises from learning temporal relationships: when there is long term memory in the data, one requires an exponentially large number of neurons for approximation, and the learning dynamics suffers from exponential slow downs.
These results form a basic step towards a mathematical understanding of the recurrent structure and its effects on learning from temporal data.
R ELATED W ORK
We will discuss related work on RNNs on three fronts concerning the central results in this paper, namely approximation theory, optimization analysis and the role of memory in learning.
A number of universal approximation results for RNNs have been obtained in discrete (Sch ¨ afer &
Zimmermann, 2006; 2007; Matthews, 1993) and continuous time (Funahashi & Nakamura, 1993;
Nakamura & Nakagawa, 2009; Li et al., 2005; Chow & Xiao-Dong Li, 2000).
Most of these focus on the case where the target relationship is generated from a hidden dynamical system in the form of difference or differential equations.
The formulation of functional approximation here is more general, albeit our results are currently limited to the linear setting.
Nevertheless, this is already sufﬁcient to reveal new phenomena involving the interaction of learning and dynamics.
This will be especially apparent when we discuss approximation rates and optimization dynamics.
We also note that the functional/operator approximation using neural networks has been explored in (Chen & Chen, 1993; Tianping Chen & Hong Chen, 1995;
Lu et al., 2019) for non-recurrent structures and reservoir systems for which approximation results similar to random feature models are derived (Gonon et al., 2020).
The main difference here is that we explicitly study the effect of memory in target functionals on learning using recurrent structures.
On the optimization side, there are a number of recent results concerning the training of RNNs using gradient methods, and they are mostly positive in the sense that trainability is proved under speciﬁc settings.
These include recovering linear dynamics (Hardt et al., 2018) or training in overparameterized settings (Allen-Zhu et al., 2019).
Here, our result concerns the general setting of learning linear functionals that need not come from some underlying differential/difference equations, and is also away from the over-parameterized regime.
In our case, we discover on the contrary that training can become very difﬁcult even in the linear case, and this can be understood in a quantitative way, in relation to long term memory in the target functionals.
This points to the practical literature in relation to memory and learning.
The dynamical analysis here puts the ubiquitous but heuristic observations - that long term memory negatively impacts training efﬁciency (Bengio et al., 1994; Hochreiter et al., 2001) - on concrete theoretical footing, at least in idealized settings.
At the same time, we also complement general results on “vanishing and explosion of gradients” (Pascanu et al., 2013; Hanin & Rolnick, 2018; Hanin, 2018) that are typically restricted to initialization settings with more precise characterizations in the dynamical regime during training.
P ROBLEM F ORMULATION
The basic problem of supervised learning on time series data is to learn a mapping from an input temporal sequence to an output sequence.
Formally, one can think of the output at each time as being produced from the input via an unknown function that depends on the entire input sequence, or at least up to the time at which the prediction is made.
In the discrete-time case, one can write the data generation process
[Equation]
where { H k : k = 0 , 1 , . . . } is a sequence of ground truth functions of increasing input dimension accounting for temporal evolution.
The goal of supervised learning is to learn an approximation of the sequence of functions { H k } given observation data.
Recurrent neural networks (RNN) (Rumelhart et al., 1986) gives a natural way to parameterize such a sequence of functions.
In the simplest case, the one-layer RNN is given by
[Equation]
Here, { h k } are the hidden/latent states and its evolution is governed by a recursive application of a feed-forward layer with activation σ .
We omit the bias term here and only consider a linear readout or output layer.
For each time step k , the mapping { x 0 , . . .
, x k − 1 } (cid:55)→ ˆ y k parameterizes a function ˆ H k ( · ) through adjustable parameters ( c, W, U ) .
Hence, for a particular choice of these parameters, a sequence of functions { ˆ H k } is constructed at the same time.
To understand the working principles of RNNs, we need to characterize how { ˆ H k } approximates { H k } .
Equation (2) is not easy to analyze due to its discrete iterative nature.
Hence, here we employ a continuous-time idealization that replaces the time-step index k by a continuous time parameter t .
This allows us to employ a large variety of continuum analysis tools to gain insights to the learning problem.
Let us now introduce this framework.
Continuous-time formulation.
Consider a sequence of inputs indexed by a real-valued variable t ∈ R instead of a discrete variable k considered previously.
Concretely, we consider the input space
[Equation]
which is the linear space of continuous functions from R (time) to R d that vanishes at inﬁnity.
Here d is the dimension of each point in the time series.
We denote an element in X by x := { x t ∈ R d : t ∈ R } and equip X with the supremum norm (cid:107) x (cid:107) X :=
sup t ∈ R (cid:107) x t (cid:107) ∞ .
For the space of outputs we will take a scalar time series, i.e. the space of bounded continuous functions from R to R :
[Equation]
Vector-valued outputs can be handled by considering each output separately, and will not be explicitly treated subsequently.
In continuous time, the target relationship (ground truth) to be learned is
[Equation]
where for each t ∈ R , H t is a functional H t : X → R .
Correspondingly, we deﬁne a continuous version of (2) as a hypothesis space to model continuous-time functionals
[Equation]
whose Euler discretization corresponds to a discrete-time residual RNN.
The dynamics then naturally deﬁnes a sequences of functionals { ˆ H t ( x ) =
ˆ y t : t ∈ R } , which can be used to approximate the target functionals { H t } via adjusting ( c, W, U ) .
Linear RNNs in continuous time.
In this paper we mainly investigate the approximation and optimization property of linear RNN, which already reveals the essential effect of dynamics.
The linear RNN obeys (6) with σ being the identity map.
Notice that in the theoretical setup, the initial time of the system goes back to −∞ with lim t →−∞ x t = 0 , ∀ x ∈ X , thus by ( H t ( 0 ) = 0 ) we specify the initial condition of the hidden state h
−∞ = 0 for consistency.
In this case, (6) has the following solution
[Equation]
Since we will investigate uniform approximations over large time intervals, we will consider stable
RNNs, where W ∈ W m with
[Equation]
of W [Equation] negative real
[Equation]
Owing to the representation of solutions in (7), the linear RNN deﬁnes a family of functionals
[Equation]
Here, m is the width of the network and controls the complexity of the hypothesis space.
Clearly, the family of functionals the RNN can represent is not arbitrary, and must possess some structure.
Let us now introduce some deﬁnitions of functionals that makes these structures precise.
Deﬁnition 3.1. Let { H t : t ∈ R } be a sequence of functionals.
H t is causal if it does not depend on future values of the input: for every pair of x
, x (cid:48) ∈
X such that x s = x (cid:48) s for all s ≤ t , we have H t ( x ) =
H t ( x (cid:48) ) .
H t is continuous and linear if for any x , x (cid:48) ∈ X and λ, λ
(cid:48) ∈ R we have
[Equation]
and moreover that sup x ∈X , (cid:107) x (cid:107) X ≤ 1 H t ( x ) < ∞ , in which case we can deﬁne the induced norm as (cid:107) H t (cid:107) := sup x ∈X , (cid:107) x (cid:107) X ≤ 1 | H t ( x ) | . 3.
H t is regular if for any sequence { x ( n ) ∈ X , : n ≥ 0 } such that x ( n ) t → 0 for Lebesgue almost every t ∈ R , then lim n →∞ H t ( x ( n ))
= 0 .
{ H t : t ∈ R } is time-homogeneous if for every t, τ ∈ R , we have H t ( x ) =
H t + τ ( x ( τ )) , where x ( τ ) s = x
s − τ for all s ∈ R , i.e. x ( τ ) is x whose time index is shifted to the right by τ .
Linear, continuous and causal functionals are common deﬁnitions.
One can think of regular functionals as those that are not determined by values of the inputs on an arbitrarily small time interval, e.g. an inﬁnitely thin spike input.
Time-homogeneous functionals, on the other hand, are those where there is no special reference point in time: if the time index of both the input sequence and the functional are shifted in coordination, the output value remains the same.
Given these deﬁnitions, the following observation can be veriﬁed directly and its proof is immediate and hence omitted.
Proposition 3.1. Let { ˆ H t : t ∈ R } be a sequence of functionals in the RNN hypothesis space ˆ H (see (9)).
Then for each t ∈ R , ˆ H t is a causal, continuous, linear and regular functional.
Moreover, the sequence of functionals { ˆ H t : t ∈ R } is time-homogeneous.
A PPROXIMATION T HEORY
The most basic approximation problem for RNN is as follows:
given some sequence of target functionals { H t : t ∈ R } satisfying appropriate conditions, does there always exist a sequence of RNN functionals { ˆ H t : t ∈ R } in ˆ H such that H t
≈ ˆ H t for all t ∈ R
We now make an important remark with respect to the current problem formulation that differs from previous investigations in the RNN approximation: we are not assuming that the target functionals { H t : t ∈ R } are themselves generated from an underlying dynamical system of the form
[Equation]
where
[Equation]
for any linear or nonlinear functions f, g .
This differs from previous work where it is assumed that the sequence of target functionals are indeed generated from such a system.
In that case, the approximation problem reduces to that of the functions f, g , and the obtained results often resemble those in feed-forward networks.
In our case, however, we consider general input-output relationships related by temporal sequences of functionals, with no necessary recourse to the mechanism from which these relationships are generated.
This is more ﬂexible and natural, since in applications it is often not clear how one can describe the data generation process.
Moreover, notice that in the linear case, if the target functionals { H t } are generated from a linear ODE system, then the approximation question is trivial: as long as the dimension of h t in the approximating RNN is greater than or equal to that which generates the data, we must have perfect approximation.
However, we will see that in the more general case here, this question becomes much more interesting, even in the linear regime.
In fact, we now prove precise approximation theories and characterize approximation rates that reveal intricate connections with memory effects, which may be otherwise obscured if one considers more limited settings.
Our ﬁrst main result is a converse of Proposition 3.1 in the form of an universal approximation theorem for certain classes of linear functionals.
The proof is found in Appendix A.
Theorem 4.1 (Universal approximation of linear functionals) . Let { H t : t ∈ R } be a family of continuous, linear, causal, regular and time-homogeneous functionals on X .
Then, for any (cid:15) >there exists { ˆ H t : t ∈ R } ∈ ˆ H such that
[Equation]
The key ingredient in the proof is the classical Riesz-Markov-Kakutani representation theorem, which allows us to pass from the approximation of functionals to that of functions.
In particular, we can prove that if { H t } is a family of continuous, linear, regular, causal and time-homogeneous functionals then there exists a unique integrable function ρ : [0 , ∞ ) → R d such that
[Equation]
Comparing such a representation with the solution (7) of the continuous RNN, we ﬁnd that the approximation property of the linear RNNs is closely related to how well ρ can be approximated by the exponential sums, as detailed in Appendix A. Due to this representation, we will hereafter refer to H t and ρ ( t ) interchangeably.
Approximation rates.
While the previous result establishes the universal approximation property of linear RNNs for suitable classes of linear functionals, it does not reveal to us which functionals can be efﬁciently approximated.
In the practical literature, it is often observed that when there is some long-term memory in the inputs and outputs, the RNN becomes quite ill-behaved (Bengio et al., 1994; Hochreiter et al., 2001).
It is the purpose of this section to establish results which make these heuristics statements precise.
In particular, we will show that the rate at which linear functionals can be approximated by linear RNNs depends on the former’s smoothness and memory properties.
We note that this is a much less explored area in the approximation theory of RNNs.
To characterize smoothness and memory of linear functionals, we may pass to investigating the properties of their actions on constant input signals.
Concretely, let us denote by e i ( i = 1 , . . .
, d ) the standard basis vector in R d , and e i denotes a constant signal with e i,t =
e i 1 { t ≥ 0 } .
Then, smoothness and memory is characterized by the regularity and decay rate of the maps t (cid:55)→ H t ( e i ) , i = 1 , . . .
, d , respectively.
Our second main result shows that these two properties are intimately tied with the approximation rate.
The proof is found in Appendix B.
Theorem 4.2 (Approximation rates of linear RNN) .
Assuming the conditions as in Theorem 4.1.
Consider the output of constant signal y i ( t ) :
=
H t ( e i ) , i = 1 , . . .
, d. Suppose there exist constants α ∈ N + , β, γ > 0 such that for i = 1 , . . .
, d , y i ( t ) ∈ C ( α +1)
( R ) and for k = 1 , . . .
, α + 1 , e βt y ( k ) i ( t )
=
o (1) as t → + ∞ , and sup
[Equation]
Then, there exists a universal constant C ( α ) that depends only on α such that for any (cid:15) > 0 , there exists a sequence of width- m RNN functionals { ˆ H t : t ∈ R } ∈ ˆ H m such that
[Equation]
The curse of memory in approximation.
For approximation of non-linear functions using linear combinations of basis functions, one often suffers from the “curse of dimensionality” (Bellman, 1957), in that the number of basis functions required to achieve a certain approximation accuracy increases exponentially when the dimension of input space d increases.
In the case of Theorem 4.2, the bound scales linearly with d .
This is because the target functional possesses a linear structure, and hence each dimension can be approximated independently of others, resulting in an additive error estimate.
Nevertheless, due to the presence of the temporal dimension, there enters another type of challenge, which we coin the curse of memory .
Let us now discuss this point in detail.
The key observation is that the rate result requires exponential decay of derivatives of H t ( e i ) , but the density result (Theorem 4.1) makes no such assumption.
The natural question is thus, what happens when no exponential decay is present?
We assume
d = 1 and consider an example in which the target functional’s representation satisﬁes ρ ( t ) ∈ C (1) ( R ) and ρ ( t ) ∼ t − (1+ ω ) as t → + ∞ .
Here ω > 0 indicates the decay rate of the memory effects in our target functional family.
The smaller its value, the slower the decay and the longer the system memory.
Notice that y (1) ( t ) =
ρ ( t ) and in this case there exists no β > 0 making (14) true, and no rate estimate can be deduced from it.
A natural way to circumvent this obstacle is to introduce a truncation in time.
With T ( (cid:29) 1) we can deﬁne ˜ ρ ( t ) ∈ C (1) ( R ) such that ˜ ρ ( t ) ≡ ρ ( t ) for t ≤ T , ˜ ρ ( t ) ≡ 0 for t ≥ T + 1 , and ˜ ρ ( t ) is monotonically decreasing for T ≤ t ≤ T + 1 .
With the auxiliary linear functional ˜
H t ( x ) := (cid:82) t 0 x t − s ˜
ρ
( s ) ds, we can have an error estimate (with technical details provided in Appendix B)
[Equation] [Equation]
In order to achieve an error tolerance (cid:15) , according to the ﬁrst term above we require T ∼ (cid:15) − 1 ω , and then according to the second term we have
[Equation]
This estimate gives us a quantitative relationship between the degree of freedom needed and the decay speed.
When ω is small, i.e., the system has long memory, the size of the RNN required grows exponentially.
This is akin to the curse of dimensionality, but this time on memory, which manifests itself even in the simplest linear settings.
F INE - GRAINED A NALYSIS OF O PTIMIZATION D YNAMICS
According to Section 4, memory plays an important role in determining the approximation rates.
The result therein only depends on the model architecture, and does not concern the actual training dynamics.
In this section, we perform a ﬁne-grained analysis on the latter, which again reveals an interesting interaction between memory and learning.
The loss function (for training) is deﬁned as
[Equation]
where the input time series x is assumed to be ﬁnitely cut off, i.e. x t = 0 for any t ≤ t 0 almost surely. Without loss of generality we set t 0 = 0 .
Training the RNN amounts to optimizing E x J m with respect to the parameters ( c, W, U ) .
The most commonly applied method is (stochastic) gradient descent (GD), which updates them in the steepest descent direction.
We ﬁrst show that the training dynamics of E x
J m exhibits very interesting behaviors depending on the form of target functionals.
Take d = 1 and consider learning different target functionals
[Figure]
with x being the white noise. We ﬁrst investigate two choices for ρ : a simple decaying exponential sum and a scaled airy function.
Observe from Figure 1 that training proceeds efﬁciently for the exponential sum case.
However, in the second airy function case, there are interesting “plateauing” behaviors of the loss function, where the loss decrement slows down signiﬁcantly after some time in training.
The plateau is sustained for a long time before an eventual reduction is observed.
As a further demonstration of that this behavior may be generic, we also consider a nonlinear forced dynamical system, the Lorenz 96 system (Lorenz, 1996), where the similar plateauing behavior is observed even for a non-linear RNN model trained with the Adam optimizer (Kingma & Ba, 2015).
All experimental details are found in Appendix C.3.1.
The results in Figure 1 hint at the fact that there are certainly some functionals that are much harder to learn than others, and it is the purpose of the remaining analyses to understand precisely when and why such difﬁculties occur.
In particular, we will again relate this to the memory effects in the target functional, which shows yet another facet of the curse of memory when it comes to optimization.
Dynamical analysis.
To make analysis amenable, we will make a series of simpliﬁcations to the training loss (18), by assuming that x is the white noise, d = 1 , T → ∞ , and the recurrent kernel W is diagonal.
This allows us to write (see Appendix C.1 for details) the optimization problem as
[Equation]
We will subsequently see that these simpliﬁcations do not lose the key features of the training dynamics such as the plateauing behavior.
We start with some informal discussion on a probable reason behind the plateaus.
A straightforward computation shows, for k = 1 , 2 , . . .
, m ,
[Equation]
A similar expression holds for ∂J m ∂a k .
Write the (simpliﬁed) linear functional representation for linear
{+RNNs as ˆ ρ m ( t ; a, w ) : = (cid:80) mi =
 a i e − w i t , which serves to approximate ρ .+}
Observe that plateauing under the GD dynamics occurs if the gradient ∇ J m is small but the loss J m is large.
A sufﬁcient condition is that the residual ˆ ρ m ( t ; a, w ) − ρ ( t ) is large only for large t (meaning the exponential term multiplied to the residual is small).
That is, the learned functional differs from the target functional only at large times.
This again relates to the long term memory in the target.
Based on this observation, We build this memory effect explicitly into the target functional by considering ρ of the form
[Equation]
21) where ¯ ρ is a function which can be approximated well by ˆ ρ m , e.g. ¯ ρ ( t ) is the exponential sum (cid:80) m ∗ j =1 a ∗ j e − w ∗
j t with m ∗ <
m (where w ∗ j > 0 ).
On the other hand, ρ 0 ,ω ( t ) :
= ρ 0 ( t − 1 /ω ) controls the memory of ρ , with ρ 0 being any bounded template function in L 2 ( R ) ∩ C 2 ( R ) with sub-Gaussian tails.
As ω → 0 + , the support of ρ 0 ,ω shifts towards large times, modelling the dominance of long term memory in ρ .
In this case, if the initialization is such that ˆ ρ m ≈ ¯ ρ , the sufﬁcient condition informally discussed above is satisﬁed as ω → 0 + , which heuristically leads to the plateauing.
A simple example of such target functionals is ρ ( t ) = a [Equation] c
[Equation]
, where w ∗ > 0 and c 0 , σ are ﬁxed constants.
This corresponds to the simple case of m ∗ = 1 and ρ 0 being the Gaussian density.
Observe that as ω → 0 + , the memory of the sequence of functionals represented by ρ increases.
It can be numerically veriﬁed that this simple target functional gives rise to the plateauing behavior, which gets worse signiﬁcantly as memory increases (see Figure 2 in Appendix C.3.2).
Our main result on training dynamics quantiﬁes this behavior theoretically for general functionals possessing the decomposition (21).
We now state the informal version, and full technical details are given in Appendix C.2.
Theorem 5.1 (Informal) . Consider the gradient ﬂow
[Equation]
with the initial value ˆ ρ m ( t ; a (0) , w (0))
≈ ¯ ρ .
Then under technical conditions, the dynamical system (22) becomes trapped in a plateau with the timescale O (cid:0)
e c w ∗ /ω ω 2 ln(1 /ω ) (cid:1) as ω → 0 + .
Here O ( · ) hides universal constants independent of ω and c w ∗ > 0 is a global constant depending only on w ∗ .
The curse of memory in optimization.
The timescale proved in Theorem 5.1 is veriﬁed numerically in Appendix C.3.3, where we also show that the analytical setting here is representative of general cases, where plateauing occurs even in non-linear RNNs trained with accelerated optimizers, as long as the target functional has the memory structure imposed in (21).
Theorem 5.1 reveals another aspect of the curse of memory , this time in optimization.
When ω → 0 + , the inﬂuence of target functional H t does not decay, much like the case considered in the curse of memory in approximation.
However, different from the approximation case where an exponentially large number of hidden states is required to achieve approximation, here in optimization the adverse effect of the memory comes with the exponentially slow down of gradient ﬂow dynamics.
While this is theoretically proved under some sensible but restrictive conditions, we show numerically in
Appendix C.3.3 that this is representative of general cases.
In the literature, a number of results have been obtained pertaining to the analysis of training dynamics of RNNs.
A positive result for training by GD is established in (Hardt et al., 2018), but this is in the setting of identifying linear systems, i.e. the target functional comes from a linear dynamical system, hence it must possess good decay properties provided it is stable.
On the other hand, convergence can also be ensured if the RNN is sufﬁciently over-parameterized (large m ) (Allen-Zhu et al., 2019).
However, in reality both of these settings may not be sufﬁcient.
Here we provide an alternative analysis of a setting that is representative of the difﬁculties one may encounter in practical RNN training.
In particular, the curse of memory that we established is consistent with the difﬁculty in RNN training often observed in applications, where heuristic attributions to memory are often alluded to (Hu et al., 2018; Campos et al., 2017; Talathi & Vartak, 2015; Li et al., 2018).
The analysis here makes the connection between memory and optimization difﬁculty precise, and may form a basis for future developments to overcome such difﬁculties in applications.
C ONCLUSION
In this paper, we analyzed the basic approximation and optimization aspects of using RNNs to learn input-output relationships involving temporal sequences in the linear, continuous-time setting.
In particular, we coined the concept curse of memory , and revealed two of its facets.
That is, when the target relationship has the long term memory, both approximation and optimization become exceedingly difﬁcult.
These analyses make concrete heuristic observations of the adverse effect of memory on learning RNNs.
Moreover, it quantiﬁes the interaction between the structure of the model (RNN functionals) and the structure of the data (target functionals). The latter is a much less studied topic.
More broadly, this approach may be a basic starting point for understanding learning from partially observed time series data in general, including gated RNN variants (Hochreiter &
Schmidhuber, 1997; Cho et al., 2014) and other methods such as transformers and convolutionbased approaches (Vaswani et al., 2017; Oord et al., 2016).
These are certainly worthy of future exploration.
A U NIVERSAL A PPROXIMATION T HEOREM OF L INEAR F UNCTIONALS BY L INEAR RNN S
A key simpliﬁcation of considering linear functionals is due to the classical representation result below, which allows us to pass from the approximation of functionals to the approximation of functions.
Theorem A.1 (Riesz-Markov-Kakutani Representation Theorem) . Let H : X → R be a continuous linear functional.
Then, there exists a unique, vector-valued, regular, countably additive signed measure µ on R such that
[Equation]
Moreover, we have
[Equation]
Proof. Well-known, see e.g. Bogachev (2007), CH 7.10.4.
We will use the representation theorem to prove Theorem 4.1.
First, We prove some lemmas.
Lemma A.1. Let { H t } be a family of continuous, linear, regular, causal and time homogeneous functionals on X .
Then, there exists a measurable function ρ : [0 , ∞ )
→ R d that is integrable, i.e.
[Equation]
and
[Equation]
In particular, { H t } is uniformly bounded with sup t (cid:107) H t (cid:107) =
(cid:107) ρ (cid:107) L 1 ([0 , ∞ )) and t (cid:55)→ H t ( x ) is continuous for all x ∈ X .
Proof. By the Riesz-Markov-Kakutani representation theorem (Theorem A.1), for each t there is a unique regular signed Borel measure µ t such that
[Equation]
27) and (cid:80) i | µ t,i | ( R ) =
(cid:107) H t (cid:107) .
Since { H t } is causal, we must have (cid:82) ∞ t x (cid:62) s dµ t ( s ) = 0 for any x and thus
[Equation]
Now, by time homogeneity we have
[Equation]
Take τ = − t and set µ = − µ 0 to get
[Equation]
Note that we have (cid:107) µ (cid:107) 1 ([0 , ∞ ))
= (cid:107) µ 0 (cid:107) 1 ([0 , ∞ ))
= (cid:107) H 0 (cid:107) =
(cid:107) H t (cid:107) , and continuity follows from the fact that
[Equation]
which converges to 0 as δ → 0 by dominated convergence theorem.
Finally, we will show that each µ i is absolutely continuous with respect to λ (Lebesgue measure).
Take a measurable E ⊂ [0 , ∞ ) such that λ ( E ) = 0 and set E (cid:48)
=
[0 , ∞ ) \ E .
For each n ≥ 0 set K n ⊂ E, K (cid:48) n ⊂ E (cid:48) where K n , K (cid:48) n are closed and µ i ( E \ K n ) ≤ 1 /n , µ i ( E (cid:48) \ K (cid:48) n ) ≤ 1 /n .
For a ﬁxed i ∈ { 1 , . . .
, d } , deﬁne x ( n ) to be such that x ( n ) t − s,j = 0 for all j (cid:54)
=
i and all s .
For j = i , we set x ( n ) t − s,i = 1 if s ∈ K n and 0 if s ∈ K (cid:48) n , which can then be continuously extended to [0 , ∞ ) .
Observe that by construction, x ( n ) t − s → 0 for λ -a.e.
s , thus by dominated convergence theorem
[Equation]
This shows that µ i is absolutely continuous with respect to λ , and by the Radon-Nikodym theorem there exists a measurable function
ρ i : [0 , ∞ ) → R such that for any measurable A ⊂ R we have
[Equation]
33) for i = 1 , . . .
, d .
Hence, we have
[Equation]
with (cid:107) ρ
[Equation]
1 ([0 , ∞ )) < ∞ .
Lemma A.2. Let ρ : [0 , ∞ )
→ R a Lebesgue integrable function, i.e. (cid:107) ρ (cid:107) L 1 ([0 , ∞ )) < ∞ .
Then, for any (cid:15) > 0 , there exists a polynomial p with p (0) =
such that
[Equation]
[-Proof A.
-]Proof.
The approach here is similar to that of the approximation of functions using exponential sums (Kammler, 1976; Braess, 1986)
Fix (cid:15) > 0 .
Deﬁne
[Equation]
Then, we can check that
[Equation]
By density of continuous functions in L 1 there exists a continuous function ˜
R on [0 , 1] with ˜ R (0) = {+
+}
such that
[Equation]
By M¨untz-Sz´asz theorem (M¨untz, 1914; Sz´asz, 1916), there exists a polynomial p with p (0) =[-
-]
such that
[Equation]
39) and q ( u ) :
= p ( u ) /u is also a polynomial.
Therefore, we have
[Equation]
We are now ready to present the proof of Theorem 4.1.
[-Proof A.
-]
Proof of Theorem 4.1.
By (7), for each { ˆ H t } ∈ ˆ H we can write
[Equation]
[Equation]
By Lemma A.1, we can write
[Equation]
where ρ is integrable.
Thus, we can apply Lemma A.2 to conclude that there exists polynomials p
i , i = 1 , . . .
, d with p i (0) =
such that
[Equation]
Notice that we can write each p i ( u )
= (cid:80) mj
=1 α ij u j for some m equaling the maximal order of { p i } .
Taking W = diag( − 1 , . .
. , − m ) , c = (1 , . . . , 1) and U ij = α ji , we have
[Equation]
Consequently, we have for any x with (cid:107) x (cid:107) ∞ ≤ 1 ,
[Equation]
B A PPROXIMATION R ATES AND THE C URSE OF M EMORY IN A PPROXIMATION
We ﬁrst give the proof of Theorem 4.
[Equation] Proof of Theorem 4.2.
We ﬁx i ∈ { 1 , . . .
, d } below until the last part of the proof.
By Lemma A.1, there exists ρ i ( t ) ∈
C α [0 , ∞ ) such that
[Equation]
By the assumption,
[Equation]
Consider the transform
[Equation]
For k = [Equation]
[Equation]
α [Equation] one can prove by induction that
[Equation]
where c ( j, k ) are some integer constants.
Together with the assumption [Equation] we have
[Equation]
where C ( α ) is a universal constant only depending on α .
Note that for j = 0 , . .
, α ,
[Equation]
) hence q i ( s ) ∈ C α [0 , 1] with q i (0) =
q (1) i (0) = · · · = q ( α ) i (0)
= 0 .
By Jackson’s theorem (Jackson, 1930), for m = 1 , 2 , . . .
, there exists a polynomial Q i,m of degree m − 1 such that
[Equation]
Denote the polynomial Q i,m as
[Equation]
53) and deﬁne
[Equation]
Then we have
[Equation]
where
[Equation]
[Equation]
[Equation]
With a change of variable s = e − βα +1 t , we have the estimate
[Equation]
Finally we deﬁne U =
[ u 1 , . . . ,
u d ] ∈ R m × d and have
[Equation]
Recall the dynamical system (6)
[Equation]
[Equation]
which is together determined by the parameters c, W, U .
Similar to the argument in the proof of
Theorem 4.1, for any x with (cid:107) x (cid:107) ∞ ≤ 1 and t , we have
[Equation]
The curse of memory in approximation.
Now we explain more technical details of why Theorem 4.2 implies the curse of memory in approximation, as pointed out in the main text.
We assume
d = 1 and consider an example
[Equation]
in which the density satisﬁes ρ ( t ) ∈ C (1) ( R ) and
[Equation]
Here ω > 0 indicates the decay rate of the memory effects in our target functional family { H t } .
The smaller its value, the slower the decay and the longer the system memory.
Notice that y (1) ( t ) =
ρ ( t ) , and in this case there exists no β > 0 making the following condition ((14) in the main text) true:
[Equation]
and
[Equation]
and no rate estimate can be deduced from it.
A natural way to circumvent this obstacle is to introduce a truncation in time.
With T ( (cid:29) 1) we can deﬁne ˜ ρ ( t ) ∈ C (1) ( R ) such that ˜ ρ ( t ) ≡ ρ ( t ) for t ≤ T , ˜ ρ ( t ) ≡ 0 for t ≥ T + 1 , and ˜ ρ ( t ) is monotonically decreasing for T ≤ t ≤ T + 1 .
Considering the linear functional ˜
H t ( x ) := (cid:82) t 0 x t − s ˜
ρ
( s ) ds, we have the truncation error estimate
[Equation]
[Equation] Now the conclusion of Theorem 4.2 (i.e. (62)) is applicable to the truncated { ˜ H t } (with α = 1 ), and we have for ∀ β > 0 , there is a linear RNN ( U, W, c ) such that the associated functionals { ˆ H t } ∈
ˆ H m satisfy
[Equation]
It is straightforward to verify that when β
= 2 /T , the right-hand side of (67) achieves the minimum, which gives us
[Equation]
Combining (66) and (68) gives us
[Equation]
[Equation]
In order to achieve an error tolerance (cid:15) , according to the ﬁrst term above we require T ∼ (cid:15) − 1 ω , and then according to second term we have
[Equation]
This estimate gives us a quantitative relationship between the degree of freedom needed and the decay speed.
When ω is small, i.e., the system has long memory, the size of the RNN required grows exponentially.
C F INE - GRAINED A NALYSIS OF O PTIMIZATION D YNAMICS C. S IMPLIFICATIONS ON THE L OSS F UNCTION
Recall the loss function
[Equation]
[Equation]
The simpliﬁcations on E x J m are listed as follows.
Take the inputs x to be the white noise, so that
[Equation]
where B t is the standard d -dimensional Wiener process.
As a consequence, simplifying (71) via Itˆo’s isometry gives
[Equation]
Note that from the previous approximation results (Theorem 4.2 and its proof), it is clear that the spatial dimension plays little role in the overall learning process (since each spatial dimension can be handled separately).
Therefore, we focus on the temporal dimension and consider d = 1 in (73).
Moreover, to study the training dynamics in large time horizons, we may take T → ∞ .
Then we get
[Equation]
where ρ ( t ) becomes a scalar-valued target and b is the sole column of U before.
This corresponds to the so-called single-input-single-output (SISO) system.
Due to the difﬁculty of directly analyzing ∇ W e Wt and ∇ 2 W e Wt , a further simpliﬁed ansatz will be considered.
We suppose that W is a diagonal matrix with negative entries (to guarantee the stability of the model), i.e. W = − diag( w ) with w > 0 element-wisely.
Hence we can combine a = b ◦ c ( ◦ denotes the Hadamard product) and rewrite the model as
[Equation]
Then (74) becomes
[Equation]
where a ∈ R m , w ∈ R m + , and
the target ρ ∈ L 2 ([0 , ∞ )) ∩ C 2 ([0 , ∞ )) in principle.
As a further simpliﬁcation, we will use a continuous-time idealization of the gradient descent dynamics by considering the gradient ﬂow with respect to J m ( a, w ) .
That is, consider the dynamical system
[Equation]
with some initial values a (0) ∈ R m , w (0) ∈
R m + .
As we will show later, the loss function (76) is able to serve as the starting point in the ﬁne-grained dynamical analysis, since it still preserves the plateauing behavior observed in the optimization process.
C. C ONCRETE D YNAMICAL A NALYSIS AND THE C URSE OF M EMORY IN O PTIMIZATION
The procedure of our analysis is as follows:
We consider using the model of exponential sums ˆ ρ m ( t ; a, w )
= (cid:80) mi =
a i e − w i t , to learn the target with memory (21), i.e. ρ ( t ) =
¯ ρ ( t )+ ρ 0 ,ω
( t ) with ¯ ρ being the well-approximated part (e.g. another exponential sum) and ρ 0 ,ω ( t ) =
ρ 0 ( t − 1 /ω ) ( ω > 0 ) controlling the memory effect (e.g. the Gaussian density); 2.
We prove that J m has a large value but small gradient near the well-approximated part of the target (i
e. when ˆ ρ m ≈ ¯ ρ ); 3.
We prove that when ˆ ρ m ≈ ¯ ρ , the Hessian ∇ 2 J m is positive semi-deﬁnite as ω = 0 , but for 0 < ω (cid:28) 1 , ∇ 2 J m has O (1) positive eigenvalues and o (1) negative eigenvalues.
This is the most important part in our analysis;
Based on these results, we perform a local linearization analysis on the gradient ﬂow (77) with the initialization ˆ ρ m ≈ ¯ ρ , from which the exponential timescale of the plateauing behavior is derived.
That is the curse of memory in optimization.
(1) Target with memory
We consider the following form of the target with memory
[Equation]
Here, for the well-approximated part ¯ ρ , a ∗ j (cid:54) = 0 , w ∗ j > 0 and w ∗
i (cid:54) =
w ∗ j for any i (cid:54) =
j , i, j = 1 , · · · , m ∗ , and m ∗ < m (to ensure that the model can perfectly represent ¯ ρ ).
For the memory term ρ 0 ,ω , ω > 0 , ρ 0 is a ﬁxed template function which satisﬁes the following assumptions.
Assumptions : (i) ρ 0 ∈ L 2 ( R ) ∩ C 2 ( R ) ; (ii) (cid:107) ρ 0 (cid:107) L 2 [0 , ∞ ) > 0 ; (iii) ρ 0 is bounded on R , i.e. sup t ∈ R | ρ 0 ( t ) | < + ∞ ; (iv)
lim t →−∞ ρ 0 ( t ) = 0 .
Remark C.1. One can easily check a large set of functions satisfy the assumptions above, e.g. the sub-Gaussian function
[Equation]
) for some ﬁxed positive constants c 0 , c 1 , t 0 1 .
We begin by the following preliminary estimate that is used throughout the subsequent analysis.
Lemma [-C.
-]C.1.
For any w > 0 , n ∈ N , let
[Equation]
[Equation]
We have lim ω
→ 0 + ∆ n,ω ( w ) = 0 .
In particular, if ρ 0 is sub-Gaussian, we further have
[Equation]
where O ( · ) hides universal constants only related to n, w .
Proof.
Since ρ 0 is bounded on R , and t n e − wt ∈ L 1 ([0 , ∞ )) , by Lebesgue’s Dominant Convergence Theorem,
[Equation]
where the last equality is due to Assumption (iv)
[Equation]
That is, for any t ≥ 0 ,
[Equation]
One can then estimate ∆ n,ω ( w ) under the sub-Gaussian condition (79).
Suppose 0 < ω < 1 /t 0 , we have
[Equation]
[Table]
Let M 0 :
= sup t ∈ R | ρ 0 ( t ) | < + ∞ , then
[Equation]
where O ( · ) hides universal constants only related to n (with t 0 ﬁxed).
[Equation]
where (cid:90) − t
[Equation]
(cid:19) for any 0 < ω < 2 c 1 /w .
Here the last inequality utilizes the Mellin Transform of absolute moments of the Gaussian density (see Proposition 1 of Li et al.
The argument is similar for I 3 , which in fact gives the same bound with I 2 .
Therefore, for any w > 0 , n ∈ N , taking 0 < ω (cid:28) 1 gives | ∆ n,ω ( w ) | ≤ C n,w ω − n e − w/ω , where C n,w > 0 is some ﬁxed constant only depending on n, w (with σ > 0 ﬁxed).
The proof is completed.
(2) Loss and gradient
The main idea to show plateauing behaviors is to analyze the local dynamics of the gradient ﬂow (77) when ˆ ρ m ≈ ¯ ρ .
To do this, we need to analyze the derivatives of J m at ( a ∗ , w ∗ ) (with degeneracy since m > m ∗ ).
As we will show later, the loss J m remains lower bounded, but (cid:107)∇ J m (cid:107) 2 converges to 0 as ω → 0 + .
This shows that the loss will saturate at a high value.
Deﬁnition C.
(Weights degeneracy) . For any partition P : { 1 , · · · , m } = ∪ m ∗ j =
I
j , with I j 1 ∩I j 2 = ∅ for any j 1 (cid:54) =
j 2 , j 1 , j 2 = 1 , · · · , m ∗ , deﬁne the afﬁne space (with respect to P )
[Equation]
for any
[Equation]
Deﬁne the collection of all such afﬁne spaces by M ∗ := (cid:83) P M ∗P .
Intuitively, M ∗ is the set of equivalent points to ( a ∗ , w ∗ ) for the purpose of approximation via exponential sums.
Remark [-C.
-]C.2.
It is straightforward to check that the dimension of each M ∗ P is (cid:80) m ∗ j =1 ( |I j | − 1)
= m − m ∗ .
In addition, it can be veriﬁed that the cardinality of M ∗ is m ∗ !
(cid:26) mm ∗ (cid:27) , where (cid:26) mm ∗ (cid:27) is the Stirling number of the second kind (see details in the proof of Theorem D.1).
Proposition [-C.
-]C.1.
For any (ˆ a, ˆ w ) ∈ M ∗P , J m (ˆ a, ˆ w ) ≥ (cid:107) ρ 0 (cid:107) 2 L 2 [0 , ∞ ) > 0 .
That is, the loss value is lower bounded uniformly in ω .
Proof.
By Deﬁnition C.1 and Assumption (ii) on ρ 0 , we have
[Equation]
Proposition C.2. For any bounded subset M ∗ 0 ⊂ M ∗P and any (ˆ a, ˆ w ) ∈
M ∗ 0 , we have
[Equation]
In particular, if ρ 0 satisﬁes the sub-Gaussian condition (79), we further have
[Equation]
as [Equation] Here, o ( · ) and O ( · ) hide universal constants related only to m, w ∗ and the diameter of M ∗ 0 , and c w ∗ > 0 is a global constant only depending on w ∗ .
Proof.
A straightforward computation shows, for k = 1 , · · · , m , ∂J m
[Equation]
Notice that for any n = 1 , 2 , · · · , m (cid:88) i = ˆ
[Equation]
i
[Equation]
[Equation]
we get ∂J m
[Equation]
Combining with Lemma C.1 gives the desired conclusion.
[Equation]
The previous analysis justiﬁes the trapping in the plateau.
Now, we show that the dynamics will eventually escape this plateau by showing that, while the Hessian ∇ 2 J m (at ˆ ρ m ≈ ¯ ρ ) becomes positive semi-deﬁnite as ω → 0 + with O (1) positive eigenvalues, at sufﬁciently small but non-zero ω , ∇ 2 J m contains small negative eigenvalues, which results in the eventual escape.
Two propositions are given in this regard, which shows respectively the minimal eigenvalue of ∇ 2 J m is small and negative.
Proposition C.3. For any bounded subset M ∗ 0 ⊂ M ∗P , for any (ˆ a, ˆ w ) ∈
M ∗ 0 , denote the eigenvalues of ∇ 2 J m (ˆ a, ˆ w ) by λ 1 ≥ · · · ≥ λ 2 m , we have
[-+
 = o (1) , · · · , λ 2 m =-][Equation]
as ω → [Equation] with m (cid:48) ≤ 2 m ∗ .
In particular, if ρ 0 satisﬁes the sub-Gaussian condition (79), we can further quantify the scale o (1) by
[Equation]
as ω → 0 + .
Here, o ( · ) and O ( · ) hide universal constants only related to w ∗ and the diameter of M ∗ 0 , and c w ∗ > 0 is a global constant only depending on w ∗ .
Proof.
A straightforward computation shows, for k, l = 1 , · · · , m ,
[Equation]
By (86), we have
[Equation]
Let
[Equation]
and
E
[Equation]
= [Equation] with ∆ n,ω ( w ) ( n = 1 , 2 ) performed element-wisely and Diag( x ) denoting the diagonal matrix with the diagonal vector being x .
Then we have
[Equation]
We analyze ∇ 2 ˜ J m (ˆ a, ˆ w ) and E ω
(ˆ a, ˆ w ) respectively.
[Equation] w ) :
˜ J m (ˆ a, ˆ w ) = 0 implies ∇ ˜
J m (ˆ a, ˆ {+w ) = 
+}
[-w ) = 
-] and ∇ 2 ˜ J m (ˆ a, ˆ w ) is positive semi-deﬁnite.
We further show that ∇ 2 ˜ J m (ˆ a, ˆ w ) has multiple zero eigenvalues.
In fact, since
[Equation]
it is straightforward to verify that ∇ 2 ˜
J m (ˆ a, ˆ w ) i, : = ∇ 2 ˜ J m (ˆ a, ˆ w ) j, : and ˆ a j · ∇ 2 ˜
J m (ˆ a, ˆ w ) m
+ i, :
= ˆ a i · ∇ 2 ˜ J m (ˆ a, ˆ w ) m + j, : for any i, j ∈ I p , p = 1 , · · · , m ∗ , where A i, : denotes the i -th row of matrix A .
That is to say, there are at most 2 m ∗ different rows in
∇ 2 ˜
J m (ˆ a, ˆ w ) ∈ R 2 m × 2 m , hence rank( ∇ 2 ˜
J m (ˆ a, ˆ w )) ≤ 2 m ∗ .
Therefore, the number of zero eigenvalues of ∇ 2 ˜ J m (ˆ a, ˆ w ) ≥ dim (cid:110)
[Equation]
Since ∇ 2 ˜ J m (ˆ a, ˆ w ) is positive semi-deﬁnite, all the nonzero eigenvalues must be positive.
[Equation] : Let
[Equation]
By Gershgorin’s Circle Theorem, for any µ being the eigenvalue of E ω (ˆ a, ˆ w ) , µ ∈ (cid:83) mk =
( G (1) k ∪
G (2) k ) .
This gives | µ | ≤ | ˆ a k || ∆ 2 ,ω ( ˆ w k ) | + | ∆ 1 ,ω ( ˆ w k ) | , k = 1 , · · · , m .
Combining with Lemma C.1 and Weyl’s Theorem completes the proof.
Proposition C.4. Suppose m > 2 m ∗ , 2 and ∆ 1 ,ω ( w ∗ j ) (cid:54) = 0 for any ω > 0 , j = 1 , · · · , m ∗ .
Then for any (ˆ a, ˆ w ) ∈ M ∗P , ∇ 2 J m (ˆ a, ˆ w ) has negative eigenvalues.
Proof.
By (92), ∇ 2 J m (ˆ a, ˆ w ) = ∇ 2 ˜
J m (ˆ a, ˆ w ) + 2 E ω (ˆ a, ˆ w ) .
Denote the eigenvalues of ∇ 2 ˜ J m (ˆ a, ˆ w ) and E ω
(ˆ a, ˆ w ) by ˜ λ 1 ≥ · · · ≥ ˜ λ
2 m and µ 1 ≥ · · · ≥ µ 2 m respectively.
a) ∇ 2 ˜ J m (ˆ a, ˆ w
) : According to the proof of Proposition C.3, ∇ 2 ˜ J m (ˆ a, ˆ w ) has at least 2( m − m ∗ ) zero eigenvalues, i.e. ˜ λ 1 > 0 , · · · , ˜ λ m (cid:48) >
0 and ˜ λ m (cid:48)
+1 = · · · = ˜ λ 2 m = 0 with m (cid:48)
≤ 2 m ∗ .
b) E ω (ˆ a, ˆ w ) :
∆ ω ( w ∗ j ) (cid:54) = 0 implies that Diag(∆ 1 ,ω ( ˆ w )) is invertible.
By the congruent transformation in the form of
[Equation]
in which D 1 , D 2 are diagonal matrices and D 1 is invertible, we know
[Equation]
have the same index of inertia.
One can verify that the eigenvalues of ˜ E ω (ˆ a, ˆ w ) is ± ∆ 1 ,ω ( ˆ w i ) for i
= 1 , · · · , m .
This gives µ 1 > 0 , · · · , µ m > 0 and µ m +1 < 0 , · · · , µ 2 m < 0 .
By Weyl’s Theorem, the minimal eigenvalue of ∇ 2 J m (ˆ a, ˆ w ) ≤ ˜ λ j + µ k for j + k ≤ 2 m + 1 .
Let j = m > 2 m ∗ ≥ m (cid:48) and
k
= m + 1 , we have ˜ λ
j = 0 and µ k < 0 , which completes the proof.
Remark [-C.
-]C.3.
A simple case below shows that the assumption m > 2 m ∗ is not necessary.
Let m = 2 , m ∗ = 1 .
We can directly perform the congruent transformation on ∇ 2 J 2 (ˆ a, ˆ w ) : ∇ 2 J 2 (ˆ a, ˆ w )
=  
[Equation]
 →  
[Equation]
[Equation]
Since det
[Equation]

[Equation]
where o ( · ) hides polynomial dependence on ∆ i,ω ( w ∗ )
( i = 1 , 2 ) with the coefﬁcients being polynomials of ˆ a .
Then we obtain that ∇ 2 J 2 (ˆ a, ˆ w ) has negative eigenvalues for any (ˆ a, ˆ w ) ∈
M ∗ 0 and 0 < ω (cid:28) 1 .
Remark C.4. Under the assumption of Proposition C.4 and together with Proposition C.3, we obtain that ∇ 2 J m (ˆ a, ˆ w ) has small and negative eigenvalues for any (ˆ a, ˆ w ) ∈
M ∗ 0 , where M ∗ 0 is any bounded subset of M ∗P . [Equation]
) Local linearization analysis
The previous analysis can now be tied directly to a quantitative dynamics via linearization arguments.
It is shown that under mild assumptions, the training dynamics can be trapped in a plateau with an exponentially large timescale.
That is to say, the curse of memory does occur, and this time in optimization dynamics instead of approximation rates.
Theorem C.1 (Formal) . Let θ :=
( a, w ) and θ ( τ ) :
= ( a ( τ ) , w ( τ )) .
Consider the gradient ﬂow
[Equation]
Suppose ∆ n,ω ( w )
=
O (cid:0) ω − n e − w/ω (cid:1) as ω → 0 + for n = 0 , 1 , 2 , 3 and the assumptions of Proposition C.4 holds.
Then for any bounded subset M ∗ 0 ⊂ M ∗P and any (ˆ a, ˆ w ) ∈
M ∗ 0 , the dynamical system (93) becomes trapped in a plateau with the timescale O (cid:0)
e c w ∗ /ω ω 2 ln(1 /ω ) (cid:1) as ω → 0 + .
Here O ( · ) hides universal constants only related to m, w ∗ and the diameter of M ∗ 0 , and c w ∗ > 0 is a global constant only depending on w ∗ .
Proof. Consider the asymptotic expansion with the form
[Equation]
for some δ ∈ (0 , 1) (with δ (cid:28) 1 ) and θ i ( τ ) =
O (1) ( i = 0 , 1 , · · · ).
For consistency, we have θ 0 (0)
=
θ (0) and θ i (0)
=
for i = 1 , 2 , · · · .
Let τ 0 be the hitting time
[Equation]
Then by continuity and the deﬁnition of inﬁmum, τ 0 > 0 and (cid:107) θ
( τ ) − θ (0) (cid:107) 2 ≤ δ for any τ ∈ [0 , τ 0 ] .
It is our goal now to quantify the scale of τ 0 .
Let g 0 :
=
∇ J m ( θ (0)) and H 0 := ∇ 2 J m ( θ (0)) .
The local linearization on (93) shows
[Equation]
Combining with (94), we have
[Equation]
O
[Equation]
scale
[Equation]
at O ( δ ) scale ,
[Equation]
at O ( δ 2 ) scale .
Therefore
[Equation]
which gives
[Equation]
To achieve a parameter separation gap δ 0 , i [Equation] e. (cid:107) θ ( τ ) − θ (0) (cid:107) 2 ≤ δ 0 with δ
= O ( δ ) and δ 0 ≤ δ , we need to take
[Equation]
Let H
= P (cid:62) Λ P be the eigenvalue decomposition with P orthogonal and Λ = diag( λ 1 , · · · λ 2 m ) consisting of the eigenvalues of H 0 with λ 1 ≥ · · · ≥ λ 2 m .
Then
[Equation]
One can easily verify that
h τ ( λ ) :
= 1 | λ | | e − λτ − 1 | , τ ≥ 0 monotonically decreases on λ ∈ R for any τ ≥ 0 .
Hence
[Equation]
Let the right hand side of (98) equal to O ( δ ) , we can obtain a lower bound for the smallest τ such that (97) holds.
According to Proposition C.2, Proposition C.3 and Proposition C.4, we have [Equation] [Equation] e
[Equation]
w [Equation] /ω (cid:1) and 0 > λ 2 m =
O (cid:0) ω − 2 e − c w ∗ /ω (cid:1) as ω → 0 + , where O ( · ) hides universal constants only related to m, w ∗ and the diameter of M ∗ 0 .
For any 0 < ω, δ (cid:28) 1 with ω
= o ( δ ) , solving RHS of (98) =
O ( δ ) gives
[Equation]
The last task is to show the dynamics of plateauing of the loss is much slower than the parameter separation.
The argument is trivial since for any τ ∈ [0 , τ 0 ] ,
[Equation]
The proof is completed.
Remark [-C.
-]C.5.
The estimate above is a lower bound on the escape time, so it does not appear to preclude the situation that the plateau will last forever.
However, observe that in the proof above, suppose τ 0 = + ∞ in (95), i [Equation] e. the hypothetical situation where parameters are trapped forever
[Equation]
and write
[Equation]
we have
[Equation]
for any j such that λ j < 0 (say j = n ).
If there is some ˜ g 0 ,j (cid:54) = 0 , from (96) we get
[Equation]
which is a contradiction.
That is to say, the parameter separation has to achieve a gap δ (cid:48)
>
δ within a ﬁnite time.
This also implies a ﬁnite time of plateauing for any ω > 0 , even if it is exponentially large.
Remark [-C.
-]C.6.
As is shown, the exponential timescale estimate holds regardless of the speciﬁc partition P .
That is to say, the number of qualiﬁed afﬁne spaces for initialization (recall θ (0) =
(ˆ a, ˆ w ) ∈
M ∗ 0 ⊂ M ∗P ) is equal to that of partitions, i.e. m ∗ !
(cid:26) mm ∗ (cid:27) , the cardinality of M ∗ (see details in the proof of Theorem D.1).
In addition, initializations sufﬁciently near these afﬁne spaces are also qualiﬁed by continuity.
Remark C.7. The dynamical analysis above is generically performed locally.
However, motivated by the idea of weights degeneracy (see Deﬁnition C.1), we can further utilize similar methods to give a global landscape analysis on the loss function J m .
The results therein show that plateauing behaviors are likely to occur all over the landscape.
See details in Appendix D.
C. N UMERICAL E XPERIMENTS C.3. M OTIVATING T ESTS
In this section we give details of Figure 1.
(1) Learning linear functionals using linear RNNs (with GD optimizer)
The target functional is H T ( x )
= (cid:82) T 0 ρ ( t ) x
T − t dt with x being the white noise, while the representation ρ is selected as the exponential sum or the scaled airy function:
Exponential sum: ρ ( t ) =
[ c ∗ ] (cid:62) e W ∗ t b ∗ , where c ∗ , b ∗ are standard normal random vectors of m ∗ dimensions, and W = −
I − Z (cid:62) Z with Z ∈ R m ∗ × m ∗ is a Gaussian random matrix with i.i.d.
entries having variance 1 /m ∗ .
Airy function: ρ ( t ) =
Ai( s 0 [ t − t 0 ]) , where Ai( t ) is the Airy function of the ﬁrst kind, given by the improper integral Ai( t ) =
1 π lim ξ →∞ (cid:90)
[Equation]
Note that in the ﬁrst example, the memory of target functional decays quickly.
However, for the second example, the effective rate of decay is controlled by the parameter t 0 .
For t ≤ t 0 , the airy function is oscillatory.
Hence for large t 0 , a large amount of memory is present in the target.
In
Figure 1 we set m ∗ = 8 for exponential sums, and t 0 = 3 , s
= 2 .
25 for airy functions.
In Figure 1 (a) and (b), we plot the gradient descent dynamics on training the linear RNNs (discretized using Euler method, hence equivalent to residual RNNs).
We observe an efﬁcient training process for the exponential sum case, while “plateauing” behaviors in the airy function case.
This causes a severe slow down of training.
In addition, we also ﬁnd that the plateauing effect gets worse as t 0 (or s 0 ) is increased, which corresponds to more complex airy functions in the sense of more memory effects.
That is, the long term memory adversely affects the optimization process via gradient descent.
Learning nonlinear functionals using nonlinear RNNs (with Adam optimizer)
To show that the plateauing behavior may be generic, we also consider a nonlinear forced dynamical system target, the Lorenz 96 system (Lorenz, 1996):
[Equation]
with cyclic indices z k + K = z k , and x is an external stochastic noise.
When the unresolved variables z k are unknown, the dynamics of the resolved variable y driven by x is a nonlinear dynamical system with memory effects (but not a linear functional).
We use a standard nonlinear RNN (with the tanh activation) to learn the sequence-to-sequence mapping x 0: T (cid:55)→ y 0: T with the Adam optimizer.
Figure 1 (c) shows that the training of the Lorenz 96 system with the presence of memory also exhibits the plateauing phenomenon.
In all cases, the trained RNN model has a hidden dimension of 16 and the total length of the path is
T = 6 . 4 .
The continuous-time RNN is discretized using the Euler method with step size 0 . [Figure]
C.3. L ONG - TERM M EMORY S IGNIFICANTLY C ONTRIBUTES TO P LATEAUS
Recall the simple example of target with memory
[Equation]
We aim to learn ρ with the exponential sum
[Equation] [Equation]
That is, the simpliﬁed linear RNN model with a diagonal recurrent kernel (see (75) and Appendix C.1).
The optimization is performed by the gradient ﬂow (77), i.e. a continuous-time idealization of the gradient descent dynamics.
The ODE (77) is numerically solved by the AdamsBashforth-Moulton method.
The results are illustrated in Figure 2.
It is shown that the plateauing time increases rapidly as the memory 1 /ω becomes longer.
C.3. N UMERICAL V ERIFICATIONS
(1) The timescale estimate
We ﬁrst numerically verify the timescale proved in Theorem 5.1 (or Theorem C.1).
That is, the time of plateauing (and also parameter separation (cid:107) θ
( τ ) − θ (0) (cid:107) 2 ) is exponentially large as the memory 1 /ω →
+ ∞ .
The results are shown in Figure 3, where we observe good agreement with the predicted scaling.
General cases
To facilitate mathematical tractability, the analysis so far is done on the restrictive cases of the diagonal recurrent kernel W with negative entries, linear activations and the gradient ﬂow training dynamics.
However, we show here that the plateauing behavior - which we now understood as a generic feature of long-term memory of the target functional and its interaction with the optimization dynamics - is present even for general cases, and hence our simpliﬁed analytical setting is representative of the general situation.
In Figure 4, we still take the target functional as deﬁned in (101), but apply more general models to learn it, including using full (non-diagonal) recurrent kernels of the RNN with no restrictions on entries, using non-linear ( tanh ) activations and using the Adam optimizer.
Furthermore, we do not take the Itˆo isometry simpliﬁcation and instead use actual input sample paths of ﬁnite time horizons, just as one would do in training RNNs in practice.
We observe that the plateauing behavior is [Figure] [Figure] present in all cases.
Moreover, in the last case of Adam (which can be thought of as a momentumbased optimization method), the plateauing behavior is somewhat alleviated, although the separation of timescales is still present.
This is consistent with our supplemental analysis in Appendix E, where we show that the momentum-based methods will speed up training based on our dynamical (Appendix C.2) and landscape analysis (Appendix D) of plateauing [Figure]
D L ANDSCAPE A NALYSIS
As mentioned in Remark C. [Figure] we can perform a global landscape analysis on the loss function based on the idea of weights degeneracy, which arises from Deﬁnition C.1.
Recall that the loss function reads
[Equation]
The main results of the appendix are summarized as follows.
• In Theorem D.1, we prove that the loss function has inﬁnitely many critical points, which form a factorial number of afﬁne spaces; •
In Theorem D.2, we prove that such (critical) afﬁne spaces are much more than global minimizers provided the target being an exponential sum;
• In Theorem D.3, we prove that on such (critical) afﬁne spaces, the Hessian is singular in the sense of processing multiple zero eigenvalues; •
In Proposition D.1, we prove that the (critical) afﬁne spaces contain both saddles and degenerate stable points which are not global optimal.
Instead of a local dynamical analysis in Appendix C.2, we generalize similar methods to a global landscape analysis here, and the results hold for the loss function associated with general targets .
More speciﬁcally, these results complement our main results (see Theorem 5.1 or Theorem C.1) in the following aspects.
• It is shown that the weights degeneracy is quite common in the whole landscape of the loss function.
Unfortunately, the weights degeneracy often worsens the landscape to a large extent; • It is shown that the weights degeneracy leads to a large number of stable areas (i
e. critical afﬁne spaces), but most of them contribute to non-global minimizers; • It is shown that these stable areas can also be quite ﬂat, which often connect with local plateaus; • For the structure of these stable areas, there are both saddles and degenerate critical points (not global optimal).
In certain regimes, even saddles can be rather difﬁcult to escape from (see Theorem 5.1 or Theorem C.1).
As a consequence, the optimization problem of linear RNNs is globally and essentially difﬁcult to solve.
D. S YMMETRY A NALYSIS ON THE L ANDSCAPE
This subsection consists of two parts: in Appendix D.1.1, we give main results provided the existence of weights degeneracy; in Appendix D.1.2, we give sufﬁcient conditions to guarantee the existence.
Since the key observation to utilize weights degeneracy is to notice the permutation symmetry of coordinates of gradients, we also called it “symmetry analysis”.
D.1. G ENERIC T HEORY
We begin with the following deﬁnition, which describes the concept of weights degeneracy in a natural and rigorous way.
Deﬁnition D.1. (coincided critical solutions and afﬁne spaces)
Let d ∈ N + and 1 ≤ d ≤ m .
We say ( a, w ) is a d -coincided critical solution of J m , if ∇ J m ( a, w ) = 0 , and w = ( w i ) ∈
R m + has d different components.
The coincided critical afﬁne spaces are deﬁned as coincided critical solutions that form afﬁne spaces.
To guarantee the existence of such solutions, it is necessary to have the following deﬁnition.
Deﬁnition D.2.
(ˆ a, ˆ w ) ∈ R m ⊗ R m + is called the non-degenerate global minimizer of J m , if and only if
[Equation]
and (ˆ a, ˆ w ) takes a non-degenerate form
[Equation]
For convenience, we also deﬁne an index set
[Equation] degenerate global [Equation] which is used frequently in the following analysis.
For any f ∈ L 2 [0 , ∞ ) , let L [ f ] be the Laplace transform of f , i.e. L [ f ]( s ) =
(cid:82) ∞
e − st f ( t ) dt, s > 0 .
We begin with the following lemma.
exponential sum, the conclusion still ﬁnite
See D.3 in Appendix D.1.1 for details. Lemma D.1. Assume that ρ is smooth and √ w |L [ ρ ]( w ) | → 0 as w → 0 + and w → ∞ .
Then we have 1 ∈ N and thus N (cid:54) = ∅ .
Proof. We aim to show that there exists ˆ a (cid:54) =
and ˆ w > 0 ,
such that
[Equation]
The basic idea is to limit the unbounded domain a ∈ R , w ∈ R + to a compact set without effecting the minimization of J 1 ( a, w ) .
We have
[Equation]
where a ( w ) :
=
2 w L [ ρ ]( w ) .
Write h ( w ) :
=
J 1 ( a ( w ) , w ) , then h (0 + ) = h ( ∞ )
=
(cid:107) ρ (cid:107) 2 L 2 [0 , ∞ ) .
Obviously h ( w ) <
(cid:107) ρ (cid:107)
L 2 [0 , ∞ ) for any w > 0 , hence
[Equation]
] h ( w ) , 0 < w lb < w ub < ∞ , which implies min a,w>
[Equation]
[Equation]
That is to say, the minimization of J 1 ( a, w ) can be equivalently performed on a 2-dimensional smooth curve ( w, a ( w )) w ∈
[ w lb ,w ub ] , which is certainly a compact set.
By continuity, J 1 ( a, w ) has global minimizers, say (ˆ a, ˆ w ) .
Obviously ˆ w > 0 and ˆ a = a ( ˆ w ) (cid:54) =
(since ˆ a = 0 implies J 1 (ˆ a, w )
=
(cid:107) ρ (cid:107) 2 L 2 [0 , ∞ ) , certainly not a minimum), which completes the proof.
Remark [-D.
-]D.1.
If the target is an exponential sum, i.e. ρ ( t ) = (cid:80) m ∗ j =1 a ∗ j e − w ∗
j t , we know ρ is smooth and √ w |L
[ ρ ]( w ) | → 0 as w → 0 + and w → + ∞ ; hence 1 ∈ N by Lemma D.1, and thus
N (cid:54) = ∅ .
In fact, L [ ρ ]( w ) = (cid:80) m ∗ j = a ∗ j w + w ∗ j implies that L [ ρ ]( w ) =
O (1) when w → 0 + , and
L [ ρ ]( w ) =
O (1 /w ) when w → + ∞ .
Theorem D.1. Assume that N (cid:54) = ∅ with N deﬁned as (106).
Let M := sup N .
Then for any m, d ∈ N + , 1 ≤ d ≤ min { m, M } , there exists at least d ! (cid:26) m d (cid:27) d -coincided critical afﬁne spaces of {+J m , 
+}
where (cid:26) md (cid:27) ∈ N + is called the Stirling number of the second kind.
Proof. (i) Existence.
The key observation is the permutation symmetry of ∇ J m : by (20), if a i = a j and w i = w j for some i (cid:54) =
j , then ∂J m ∂a
i = ∂J
m ∂a j and ∂J
m ∂w i =
∂J
m ∂w j .
For any m, d ∈ N + , 1 ≤ d ≤ m , suppose that w = ( w i ) ∈
R m + has d different components.
Then for any partition P : { 1 , · · · , m } = ∪ dj
=
I j with I j 1 ∩ I j 2 = ∅ for any j 1 (cid:54) =
j 2 , j 1 , j 2 = 1 , · · · , d , deﬁne the afﬁne space
[Equation]
for some ( b, v ) ∈ R d ⊗ R d + , where v has exactly d different components.
Therefore, for any ( a, w ) ∈
M P , ( b,v ) , ( m,d ) , we have
J m ( a, w ) = (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) d (cid:88)
j = (cid:88)
i ∈I
j a i e − w i t − ρ ( t ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
L 2 [0 , ∞ )
= (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
d (cid:88)
j = b j e − v j t − ρ ( t ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
L 2 [0 , ∞ )
=
J d ( b, v ) , and similarly ∂J m ∂a k
( a, w ) =(cid:90) ∞
e − v s t   d (cid:88)
j = b j e − v j t − ρ ( t ) 
 dt, k ∈ I s , s = 1 , 2 , · · · , d,
∂J
m ∂w k ( a, w )
= 2 a k (cid:90) ∞ 0 ( − t )
e − v s t   d (cid:88)
j = b j e − v j t − ρ ( t ) 
 dt, k ∈ I s , s = 1 , 2 , · · · , d. Notice that ∂J d ∂b s ( b, v ) =(cid:90) ∞ e − v s t   d (cid:88) j = b j e − v j t
[Equation]
we have ∂J m
[Equation]
7 Since d ≤ min { m, M } , d ∈ N .
In fact, for any k ∈ N + , if k / ∈ N , there exists i ≤ k such that J ( i ) has no non-degenerate global minimizers, we have j / ∈ N for any j ≥ i , hence M ≤ i − 1 ≤ k − 1 .
Hence M = ∞ implies N = N +
and M < ∞ implies M ∈ N , and both of them lead to d ∈ N .
Therefore, J d has non-degenerate global minimizers, i.e. there exists (ˆ b, ˆ v ) ∈ R d ⊗ R d + such that
[Equation]
and (ˆ b, ˆ v ) takes a non-degenerate form ˆ b i (cid:54) = 0 ,
ˆ v i (cid:54) =
ˆ v j for any i (cid:54) =
j,
[Equation]
= [Equation] [Equation] [Equation]
By (109), we get ∇ J d (ˆ b, ˆ v ) = 0 .
Combining with (108) and (110), we obtain ∇ J m (ˆ a, ˆ w ) =
for any (ˆ a, ˆ w ) ∈
M P , (ˆ b, ˆ v ) , ( m,d ) , i e. (ˆ a, ˆ w ) belongs to a d -coincided critical afﬁne space.
Note that the afﬁne space is with the dimension (cid:80) dj
=1 ( |I j | − 1) = m − d , since there are d linear equality constrains on the m -dimensional vector a .
(ii) Counting.
By the structure of afﬁne spaces discussed above, we can identify different afﬁne spaces with respect to the partition P .
For counting the number of different partitions P : { 1 , · · · , m } = ∪ dj =1 I j , it can be decomposed into the following two steps.
First, partitioning a set of m labelled objects into d nonempty unlabelled subsets.
By deﬁnition, the answer is the Stirling number of the second kind (cid:26) md (cid:27) .
Second, assign each partition to I 1 , · · · , I d accordingly.
There are d ! ways in total.
Therefore, the number of d -coincided critical afﬁne spaces is at least d ! (cid:26) md (cid:27) .
The proof is completed.
Combining Lemma D.1, Remark D.1 and Theorem D.1 gives the following theorem, which states that there are much more saddles and degenerate stable points which are not global optimal than global minimizers in the landscape (provided the target being an exponential sum).
Theorem D.2. Fix any m ∈ N + relatively large.
Consider the loss J m associated with the target being a non-degenerate exponential sum, i.e. ρ ( t ) =
(cid:80) mj
=1 a ∗ j e − w ∗
j t , where a ∗ j (cid:54) = 0 and w ∗ i (cid:54) =
w ∗ j for any i (cid:54) =
j , i, j = 1 , · · · , m .
Assume that m ∈ N 8 with N deﬁned in (106).
Then in the landscape of J m , the number of coincided critical afﬁne spaces is at least Poly ( m ) times larger than the number of global minimizers.
The following simple lemma will be used in the proof of Theorem D.2.
Lemma D.2. For any m ∈ N + , λ = ( λ 1 , · · · , λ m ) , λ i (cid:54) =
λ j for any i (cid:54) =
j , i, j = 1 , · · · , m , the series of functions { e λ i t } mi =1 is linear independent on any interval I ⊂ R .
[-Proof D.
-]Proof.
The aim is to show m (cid:88) i =
[Equation] e [Equation]
t [Equation] [Equation] t
[Equation] i [Equation] [Equation]
= [Equation] [Equation] m.
(111) (111) holds trivially for m = 1 .
Assume that (111) holds for m − 1 , then for any interval I ⊂ R , m (cid:88)
i = c i e λ i t = 0 , t ∈ I ⇒ m − 1 (cid:88) i =
c i e ( λ i − λ m ) t + c m = 0 , t ∈ I (112) ⇒ m − 1 (cid:88) i = c i ( λ i − λ m ) e (
[Equation]
, t ∈ I.
By induction, we get c
i
λ i − λ m )
=
for i = 1 , · · · , m − 1 .
Since λ i ’s are distinct, we have c i
=for i
= 1 , · · · , m − 1 .
Combining with (112), we get c m = 0 , which completes the proof.
Proof of Theorem D.2.
(i)
Global minimizers.
Since the target is an exponential sum, we have J m ( a, w ) ≥ [-
-]0 and J m (¯ a ∗ , ¯ w ∗ ) = 0 , where ¯ a ∗ =
Pa ∗ and ¯ w ∗ =
Pw ∗ with P ∈ R m × m to be some permutation matrix.
Next we show J m has no other global minimizers.
Suppose J m ( a, w ) = 0 , we have
[Equation]
It is easy to see that for any j = 1 , · · · , m , there exists i ( j ) such that w i ( j ) =
w ∗ j .
Otherwise, if w i (cid:54) =
w ∗ j , i = 1 , · · · , m , by (111) or Lemma D.2, we have a ∗ j = 0 , which is a contradiction.
Notice that w ∗ i (cid:54) =
w ∗ j for any i (cid:54) =
j , different w ∗ j ’s will correspond to different w i ’s, hence the correspondence is one-to-one.
Therefore, let w i =
w ∗ j ( i ) , (113) can be rewritten as
[Equation]
Again by Lemma D.2, we have a i = a ∗ j ( i ) .
That is to say, J m ( a, w ) = 0 implies a = Pa ∗ and w =
Pw ∗ with P ∈ R m × m to be some permutation matrix.
This gives m !
global minimizers.
(ii) Coincided critical afﬁne spaces.
Obviously N (cid:54) = ∅ , and M = sup N ≥ m .
According to
Theorem D.1, for any d ∈ N + , 1 ≤ d ≤ min { m, M } = m , we have at least d ! (cid:26) md (cid:27) d
-coincided critical afﬁne spaces of J m .
By (i), for any d ≤ m − 1 , there are no global minimizers in these afﬁne spaces.
Counting the total number
[Equation]
(iii) Comparison.
To give a between and m , we need an elementary recurrence
[Equation]
• For d = m − 1 , let p m :=
[Equation]
, then
[Equation]
• For d = m − 2 , let q m
[Equation]
, then
[Equation]
Combining above gives
[Equation]
which is a quadratic polynominal on m .
The proof is completed
[Equation]
Remark [-D.
-]D.2.
We only take the last two terms of (114) for a lower bound, which is obviously rather loose.
In principle, a Poly( m ) bound with higher degrees can be similarly obtained.
That is to say, on one hand, there are inﬁnitely many critical points forming afﬁne spaces in the landscape of J m ; on the other hand, we see that even if only counting the number of afﬁne spaces, there are still much less global minimizers (provided the width m relatively large).
Remark [-D.
-]D.3.
When the target ρ is not an exponential sum, it is straightforward to see Theorem D. still holds if there is a ﬁnite number (with the scale of no more than factorial) of global minimizers.
Now we get down to investigate ∇ 2 J m on the above coincided critical afﬁne spaces.
It is shown that ∇ 2 J m is singular and can have multiple zero eigenvalues.
Theorem [-D.
-]D.3.
Fix any m, d ∈ N + , 1 ≤ d ≤ m .
On the d -coincided critical afﬁne spaces (induced by non-degenerate global minimizers of J d 9 ) of J m , ∇ 2 J m is with rank at most m + d , and hence has at least m − d zero eigenvalues.
[-Proof D.
-]Proof.
A straightforward computation shows that, for k, l = 1 , 2 , · · · , m ,
[Equation]
Let the induced d -coincided critical afﬁne space be M P , (ˆ b, ˆ v ) , ( m,d ) , as is derived in the proof of
Theorem D.1.
Since (ˆ b, ˆ v ) is the non-degenerate global minimizer of J d , we have
[Equation]
∞
[Equation]
for any (ˆ a, ˆ w ) ∈
M P , (ˆ b, ˆ v ) , ( m,d ) .
This gives
[Equation]
Now we show that, for any i, j ∈ I s , i (cid:54) =
j , s = 1 , 2 , · · · , d , ∇ 2 J m (ˆ a, ˆ w ) i, : = ∇ 2 J m (ˆ a, ˆ w ) j, : .
In fact, for any k = 1 , · · · , m , let k ∈
I s (cid:48) , then by (115), ∂
2 J m
[Equation]
For k (cid:54) =
i and k (cid:54)
=
j , (116) gives ∂ 2 J m
[Equation]
By (118), for k = i (cid:54) =
j ,
[Equation]
J m
[Equation]
[Equation]
v 2 s , and similarly for k = j (cid:54) = i ,
[Equation]
That is to say, there are at most m + d different rows in the symmetric matrix ∇ 2 J m (ˆ a, ˆ w ) ∈
R 2 m × 2 m , hence rank (cid:0) ∇ 2 J m (ˆ a, ˆ w ) (cid:1) ≤ m + d .
Therefore, the number of zero eigenvalues of ∇ 2 J m (ˆ a, ˆ w ) ≥ dim (cid:8) x ∈ R 2 m : ∇ 2 J [Equation] (ˆ [Equation]
= [Equation] m − rank( ∇ 2 J m (ˆ a, ˆ w )) ≥ m − d .
The proof is completed.
Remark [-D.
-]D.4.
The bound in Theorem D.3 is not sharp, since the estimate on rank (cid:0) ∇ 2 J m (cid:1) here is loose as only rows with the same elements are considered.
In practice (numerical tests), it is often observed that there are more zero eigenvalues of rank (cid:0) ∇ 2 J m (cid:1) on the coincided critical afﬁne space
M P , (ˆ b, ˆ v ) , ( m,d ) .
Remark [-D.
-]D.5.
Theorem D.3 shows that, there are local plateaus around the d -coincided critical afﬁne spaces M P , (ˆ b, ˆ v ) , ( m,d ) for d ≤ m − 1 .
In addition, the 0 -eigenspace of J m is higherdimensional for smaller d , which may suggest that one can stuck on plateaus more easily.
D.1. S UFFICIENT C ONDITIONS
There is still a gap when connecting Theorem D.1 and Theorem D.2.
That is, it is necessary to guarantee sup N relatively large, i.e. J 1 , J 2 , · · · , J d all have non-degenerate global minimizers for d as large as possible.
Motivated by Kammler (1979), we can give some sufﬁcient conditions by limiting the target ρ within a smaller function space, the so-called completely monotonic functions.
Deﬁnition D.3.
F ∈ C [0 , ∞ ] ∩ C ∞ (0 , ∞ ) is called completely monotonic, if and only if ( − 1) n F ( n ) ( t ) ≥ 0 , 0 < t < ∞ , n = 0 , 1 , · · · , and F ( ∞ ) = 0 .
Remark D.6. Several examples of completely monotonic functions:
• ρ ( t ) = 1 / (1 + t ) α for any α > 0 ; • The non-degenerate exponential sum with positive coefﬁcients
[Equation]
Since the space of exponential sums is not close, we turn to consider the problem of ﬁnding a best approximation to a given ρ ∈ L 2 [0 , ∞ ) from the set
[Equation]
for some
[Equation]
119) with respect to the common L 2 -norm, i.e. inf ˆ ρ ∈ V d ( R + ) (cid:107) ˆ ρ − ρ (cid:107) L 2 [0 , ∞ ) , where D denotes the common differential operator.
Obviously V d ( R + ) ⊂
L 2 [0 , ∞ ) and V d ( R + ) (cid:40)
V d +
( R + ) for any d ∈ N + .
Kammler (1979) proves the following theorem.
Theorem D.4. Assume ρ ∈ L 2 [0 , ∞ ) to be completely monotonic.
Then there exists a best approximation ˆ ρ 0 to ρ in V d ( R + ) , i.e.
[Equation]
When ρ / ∈ V d ( R + ) , any such best approximation admits a non-degenerate form
[Equation]
and satisﬁes the generalized Aigrain-Williams equations
[Equation]
Note that (120) and (121) are pretty similar to Deﬁnition D.2, except for a different choice of hypothesis function space.
Now we show a connection between these two problems.
Theorem D.5. Assume ρ ∈ L 2 [0 , ∞ ) to be completely monotonic, and ρ / ∈ V d ( R + ) for some d ∈ N + .
Then J d has non-degenerate global minimizers (ˆ b, ˆ v ) ∈ R d ⊗ R d + .
[-Proof D.
-]Proof.
According to Theorem D.4, there exists a non-degenerate best approximation ˆ ρ 0 to ρ from V d ( R + ) , i.e.
[Equation]
We aim to prove J d (ˆ b, ˆ v ) =
inf b ∈ R d ,v ∈ R d + J d
( b, v ) .
Deﬁne the following subsets of exponential sums
[Equation]
ˆ
[Equation]
R
[Equation]
[Equation]
w i ) has k diﬀerent components
[Equation]
k ≤ d, then we have
[Equation]
d + J d ( b, v ) = inf ˆ ρ ∈V d ( R + ) (cid:107) ˆ ρ − ρ (cid:107) 2 L 2 [0 , ∞ ) .
It is straightforward to verify that
V d ( R + ) =
(cid:83) dk
=1 V d,k
( R + ) , and V d,k ( R + ) =
V k,k ( R + ) (cid:40)
V k
( R + ) for k = 1 , · · · , d .
By (124), we get
[Equation]
Since ˆ ρ 0 ∈ V d,d ( R + ) , we have
[Equation]
The last task is to show inf ˆ ρ ∈V d ( R + ) (cid:107) ˆ
[Equation]
− ρ (cid:107)
[Equation]
[Equation]
= inf ˆ ρ ∈V d,d ( R + ) (cid:107) ˆ ρ − ρ (cid:107) L 2 [0 , ∞ ) .
In fact, for any ˆ ρ ∈ V k,k , ˆ ρ ( t ) = (cid:80) ki =1 a i e − w i t , let ˜ a :
=
( a 1 , · · · , a k , 0) , ˜ w
[Equation]
= (
[Equation]
[Equation]
we get ˆ ρ ( t ) :
= (cid:80) k
+1 i {+=
+} ˜ a
i e − ˜ w i t ∈ V k +1 ,k +1 , which implies V k,k ⊂ V k +1 ,k +1 .
Therefore,
[Equation]
which completes the proof.
Combining Theorem D.1 and Theorem D.5 immediately gives the following result.
Theorem D.6. Assume ρ ∈ L 2 [0 , ∞ ) to be completely monotonic, and ρ / ∈ V 1 ( R + ) .
Let D :=
{ d ∈ N + : ρ / ∈ V d ( R + ) } , D 0 :=
sup D and write m (cid:48) :
= min { m, D 0 } .
Then the total number of coincided critical afﬁne spaces of J m is at least (cid:80) m (cid:48)
d =1 d ! (cid:26) m d (cid:27) .
[-Proof D.
-]Proof.
We have 1 ∈ D and thus D (cid:54) = ∅ , D 0 ≥ 1 .
Since V d ( R + ) (cid:40) V d +1 ( R + ) for any d ∈ N + , we have D = { 1 , 2 , · · · , D 0 } if D 0 < ∞ , and D = N + if D 0 = ∞ .
10 Both of them gives { 1 , 2 , · · · , m (cid:48) } ⊂ D ,
i.e. ρ / ∈ V k
( R + ) for any k ≤ m (cid:48) .
By Theorem D.5, J ( k ) has non-degenerate global minimizers for any k ≤ m (cid:48) , i.e. m (
cid:48) ∈ N .
According to Theorem D.1, for any d ∈ N + , 1 ≤ d ≤ m (cid:48)
=
min { m, m (cid:48) } ≤ min { m, M } , there exists at least d ! (cid:26) md (cid:27) d
-coincided critical afﬁne spaces of J m .
Sum over d gives the total number (cid:80) m (cid:48)
d =1 d ! (cid:26) md (cid:27) .
Remark D.7. Examples:
• Suppose the target is ρ ( t )
= 1 / (1+ t )
α , α > 0 , then D = N + and D 0 = ∞ .
The total number of coincided critical afﬁne spaces of the corresponding J m is at least (cid:80) md
=1 d ! (cid:26) md (cid:27) .
• Suppose the target is an non-degenerate exponential sum with positive coefﬁcients: ρ ( t ) =
(cid:80) mj
= 1 a ∗ j e − w ∗ j t , where a ∗ j > 0 and w ∗ i (cid:54) =
w ∗ j for any i (cid:54) =
j , i, j = 1 , · · · , m .
Then D = { 1 , 2 , · · · , m − 1 } and D
= m − 1 .
The total number of coincided critical afﬁne spaces of the corresponding J m is at least (cid:80) m − 1 d =1 d ! (cid:26) md (cid:27) , which is exactly (114).
An complement for Theorem D.2 is as follows.
Theorem D.7. Fix any m ∈ N + relatively large.
Consider the loss J m associated with the target being a non-degenerate exponential sum with positive coefﬁcients, i.e. ρ ( t ) =
(cid:80) mj
=1 a ∗ j e − w ∗
j t , where a ∗ j > 0 and w ∗ i (cid:54) =
w ∗ j for any i (cid:54) =
j , i, j = 1 , · · · , m .
Then in the landscape of J m , the number of coincided critical afﬁne spaces is at least Poly ( m ) times larger than the number of global minimizers.
Proof. By Theorem D.2, we only need to show is m ∈ N .
Since ρ ∈ L 2 [0 , ∞ ) is completely monotonic, and
ρ / ∈ V k
( R + ) for any k ≤ m − 1 , then by Theorem D.5, J ( k ) has non-degenerate global minimizers for any k ≤ m − 1 , i.e. m − 1 ∈ N .
The proof is completed by noticing that J m obviously has non-degenerate global minimizers, e.g. ( a ∗ , w ∗ ) .
D.1. A L OW - DIMENSIONAL E XAMPLE
To further understand the structure of coincided critical afﬁne spaces, we focus on a speciﬁc lowdimensional example here.
That is
[Equation]
with the target to be a non-degenerate exponential sum ρ ( t ) = (cid:80) m ∗ j =1 a ∗ j e − w ∗
j t , where a ∗ j (cid:54) = 0 and w ∗ i (cid:54) =
w ∗ j for any i (cid:54) =
j , i, j = 1 , · · · , m ∗ .
As we will show later, the coincided critical afﬁne spaces of J 2 contain both saddles and degenerate stable points which are not global optimal .
By Lemma D.1, Remark D.1 and Theorem D.1, we know the 1 -coincided critical afﬁne space of Jexists, and it can be constructed by taking the non-degenerate global minimizer of J 1 , say (ˆ a, ˆ w ) with ˆ a (cid:54)
=
and ˆ w > 0 .
Then M (ˆ a, ˆ w ) , (2 , 1) :
= { ( a 1 , ˆ a − a 1 , ˆ w, ˆ w ) : a 1 ∈ R } ∈ R 4 is a line 11 , and ∇ J 2 ( a 1 , ˆ a − a 1 , ˆ w, ˆ w ) = 0 for any a 1 ∈ R .
Denote the Hessian of J 2 on the line M
(ˆ a, ˆ w ) , (2 , 1) by
A (ˆ a, ˆ w ) ( a 1 ) , i.e. A (ˆ a, ˆ w ) ( a 1 ) := ∇ 2 J 2 ( a 1 , ˆ a − a 1 , ˆ w, ˆ w ) .
We investigate the landscape of J 2 on the line M
(ˆ a, ˆ w ) , (2 , 1) by analyzing the eigenvalue distribution of A (ˆ a, ˆ w ) ( a 1 ) .
Proposition D.1. Suppose m = m ∗ = 2 , and 0 < w ∗ 1 < w ∗ 2 .
Let I 1 :=
[0 , ˆ a ] and I 2 :=
( −∞ , 0) ∪ (ˆ a, + ∞ ) 12 .
Then
If a ∗ 1 a ∗ 2 < 0 , the minimal eigenvalue of A (ˆ a, ˆ w ) ( a 1 ) is 0 for any a 1 ∈ I 1 , and negative for any a 1 ∈ I 2 ; 2.
If a ∗ 1 a ∗ 2 > 0 and w ∗ 2 /w ∗ 1 < 2 + √ 3 , the minimal eigenvalue of A (ˆ a, ˆ w ) ( a 1 ) is negative for any a 1 ∈ I 1 , and 0 for any a 1 ∈ I 2 .
Proof. Write c ( w ) :
= (cid:80) m ∗ j =1 a ∗ j (cid:104)
[Equation]
, and a 2 :
=
ˆ a − a 1 .
A straightforward computation shows that
[Equation]
Considering the congruent transformation of A (ˆ a, ˆ w ) ( a 1 ) , which does not affect the index of inertia:
[Equation]
 1ˆ w 1ˆ w − a 1 2 ˆ w− a 2 2 ˆ w1ˆ w 1ˆ w − a 1 2 ˆ w− a 2 2 ˆ w− a 1 2 ˆ w− a 1 2 ˆ wa 21 2 ˆ w 3 + 4 c ( ˆ w ) aa 1 a 2 2 ˆ w− a 2 2 ˆ w− a 2 2 ˆ wa 1 a 2 2 ˆ wa 22 2 ˆ w
[Equation]
  →  
[Equation]
[Equation]
[Equation]
  , we see that A (ˆ a, ˆ w ) ( a 1 ) has one positive eigenvalue 1 / ˆ w and one eigenvalue 0.
What remains are the eigenvalues of A (cid:48)
(ˆ a, ˆ w ) ( a 1 ) :=
[Equation]
To determine their signs, we compute
[Equation]
So we need to analyze the sign of ˆ ac ( ˆ w ) and ˆ a 2 + 16 ˆ w 3 ˆ ac ( ˆ w ) under different assumptions on ( a ∗ , w ∗ ) .
(i) a ∗ 1 a ∗ 2 < 0 .
By the optimality condition of (ˆ a, ˆ w ) for J 1 , we have
[Equation]
(128) and therefore
[Equation]
Write v j :=
w ∗ j / ˆ w , j = 1 , 2 , we get 0 < v 1 < v 2 , and ˆ a =
[Equation]
Therefore
[Equation] [Equation]
a
[Equation]
j (1 + v j )  −m ∗ (cid:88)
j = a ∗ j 1 + v j · m ∗ (cid:88) j = a ∗ j (1 + v j ) = − 16 a ∗ 1 a ∗ 2 ( v 1 − v 2 )(1 + v 1 ) 3 (1 + v 2 )(129) > 0 , which gives ˆ ac ( ˆ w ) > 0 and ˆ a
+ 16 ˆ w 3 ˆ ac ( ˆ w ) > 8 ˆ w 3 ˆ ac ( ˆ w ) > 0 .
(ii) a ∗ 1 a ∗ 2 > 0 , w ∗ 2 /w ∗ 1 < 2 + √ 3 .
By (129), ˆ ac ( ˆ w ) < 0 .
ˆ a 2 + 16 ˆ w 3 ˆ ac ( ˆ w ) =
3ˆ a 2 − 16ˆ a m ∗ (cid:88)
j = a ∗ j (1 + v j )=
    m ∗ (cid:88)
j = a ∗ j (1 + v j )
 −m ∗ (cid:88)
j
= a ∗ j 1 + v j · m ∗ (cid:88)
j = a ∗ j (1 + v j ) 
= a ∗u 41 u(cid:2) u
42 + c 2 u 41 + 6 cu 21 u 22 − 2 cu 31 u 2 − 2 cu 1 u 32 (cid:3) ( c = a ∗ 2 /a ∗ 1 , u j := 1 + v j > 1)
= a ∗u 4(cid:0)
s 4 + c 2 + 6 cs 2 − 2 cs − 2 cs 3 (cid:1) ( s =
u 2 /u 1 > 1)
= a ∗u(cid:2) c 2 − 2 s ( s 2 − 3 s + 1) c + s 4 (cid:3) .
Since 4 s 2 ( s 2 − 3 s + 1) 2 − 4 s 4 = 4 s 2 ( s − 1)
[( s − 2) 2 − 3] , and 1 < s = u 2 /u 1 = ( ˆ w + w ∗ 2 ) / ( ˆ w + w ∗ 1 ) < w ∗ 2 /w ∗ 1 < 2 + √ 3 , we get ∆ c < 0 .
This implies c 2 − 2 s ( s 2 − 3 s + 1) c
+ s 4 > 0 and ˆ a 2 + 16 ˆ w 3 ˆ ac ( ˆ w ) > 0 .
In both (i) and (ii), ˆ a 2 + 16 ˆ w 3 ˆ ac ( ˆ w ) > 0 , which implies that there is at least one positive diagonal element of A (cid:48)
(ˆ a, ˆ w ) ( a 1 ) in a sufﬁciently small neighborhood of a 1 = 0 and a 1 = ˆ a .
By the
Rayleigh-Ritz Theorem and Weyl’s Theorem, A (cid:48)
(ˆ a, ˆ w ) ( a 1 ) has at least one positive eigenvalue in this neighborhood.
However, by (126), det( A (cid:48)
(ˆ a, ˆ w ) ( a 1 )) only changes the sign at a 1 = 0 and a 1 = ˆ a .
This implies another eigenvalue of A (cid:48)
(ˆ a, ˆ w ) ( a 1 ) changes the sign at a 1 = 0 and a 1 = ˆ a accordingly.
By different signs of ˆ ac ( ˆ w ) derived in (i) and (ii), and (126), the proof is completed.
Remark D.8. From Proposition D.1, we see that there are both saddles and degenerate stable points of J 2 on the critical afﬁne spaces (line)
M (ˆ a, ˆ w ) , (2 , 1) , and each of them in fact forms afﬁne spaces (lines) respectively, but they are certainly not global minimizers.
Therefore, the gradientbased algorithms can get stuck around this afﬁne space, except that it meets saddles with negative eigenvalues of large magnitude.
E M OMENTUM H ELPS T RAINING : Q UADRATIC E XAMPLES
In practice, it is often the case that training is trapped in some very ﬂat regions (plateaus), where the loss function has rather small gradients and negative eigenvalues of Hessian.
Now we illustrate the escape dynamics (from plateaus) via a simple quadratic example.
Consider the loss function f ( x ) =
( x 21 − (cid:15)x 22 ) / 2 with 0 < (cid:15) (cid:28) 1 .
We check the escaping performance for continuous-in-time analogs of two optimization algorithms: gradient decent (GD) and momentum (heavy ball) method.
(1) Gradient decent
Consider the gradient ﬂow of f ( x ) with an initial value x 0 =
( δ, 1) (cid:62) , where 0 < δ (cid:28) 1 and δ
= O ( (cid:15) ) .
Thus (cid:107)∇ f ( x 0 ) (cid:107)
=
O ( (cid:15) ) , and (cid:26) x (cid:48)
[Equation]
[Equation]
It is easy to show that there are different timescales of (cid:96) 1 ( τ ) .
In fact, when τ = O (1 /(cid:15) ) , (cid:96) 1 ( τ ) =
O ( (cid:15) 2 ) e −| O (1 /(cid:15) ) | − (cid:15)e | O (1) |
= O ( (cid:15) ) .
However, when τ continuous to increase, say
[Equation]
(130) where δ 0 > 0 denotes the gap satisfying (cid:15)
=
o ( δ 0 ) , we get (cid:96) 1 ( τ ) ≤ (cid:96) 1 ( τ (cid:15) 1 )
=
O ( (cid:15) 2 ) − (cid:15)e 2 (cid:15) · 12 (cid:15) ln δ 0 (cid:15) /
2 =
O ( (cid:15) 2 ) − δ 0 / 2 < − δ 0 / 4 for any τ ≥ τ (cid:15) 1 .
(2) Momentum
The momentum algorithm has the update rule
[Equation]
[Equation]
where ρ ∈ R , η > 0 is the learning rate.
The continuous-in-time analog can be derived as (similar to the arguments in Su et al.
[Equation]
with x k := x ( k √ η ) and the step size √ η of the simple ﬁnite differences 13 .
Let x 1 = x 0 − η ∇ f ( x 0 ) , we also get x (cid:48) (0)
=
−√ η ∇ f
( x (0)) .
To facilitate a comparison to GD, we take η = 1 14 and ρ = 1 15 .
Plugging the expression of f , we can solve the ODE
[Equation]
Write (cid:96) 2 ( τ ) :
= f
( x ( τ )) .
It is not hard to show that there are still different timescales of (cid:96) 2 ( τ ) .
In fact, when τ = O (1 / √ (cid:15) ) , (cid:96) 2 ( τ ) =
O ( (cid:15) 2 ) | O (1) | − (cid:15) | O (1) | ( e | O (1) | + e −| O (1) | ) 2 =
O ( (cid:15) ) .
However, when τ continuous to increase, say
[Equation]
2 , (132) we get (cid:96) 2 ( τ (cid:15) 2 ) =
O ( (cid:15) 2 ) − (cid:15) ( 1+ √ (cid:15) 2 ) 2 e 2 √ (cid:15)τ / 2 + O ( (cid:15) )
=
O ( (cid:15) ) − δ 0 < − δ 0 / 2 , hence (cid:96) 2 ( τ ) ≤
O ( (cid:15) ) − δ 0 < − δ 0 / 2 for any τ ≥ τ (cid:15) 2 .
Combining (1) and (2) , we have the following conclusions
• For both training dynamics, there are different timescales in the loss function.
That is to say, relatively long time is needed to escape from the plateaus; • Comparing (130) and (132), we get different timescales of escaping: O (1 /(cid:15) · ln(1 /(cid:15) )) for GD and O (1 / √ (cid:15) · ln(1 /(cid:15) )) for momentum.
Just like the convex case, where momentum improves the convergence rate by weakening the dependence on condition number, we see momentum can also help to escape rather ﬂat saddles.