S TOCHASTIC G RADIENT D ESCENT L EARNS S TATE E QUATIONS WITH N ONLINEAR A CTIVATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
We study discrete time dynamical systems governed by the state equation h t
+1 =
φ ( Ah t + Bu t ) .
Here A , B are weight matrices, φ is an activation function and u t is the input data.
This relation is the backbone of recurrent neural networks (e g. LSTMs) which have broad applications in sequential learning tasks.
We utilize stochastic gradient descent to learn the weight matrices from a ﬁnite input/state trajectory { u t , h t } N t =0 .
We prove that SGD estimate linearly converges to the ground truth weights while using near-optimal sample size.
Our results apply to increasing activations whose derivatives are bounded away from zero.
The analysis is based on i) a novel SGD convergence result with nonlinear activations and ii) careful statistical characterization of the state vector.
Numerical experiments verify the fast convergence of SGD on ReLU and leaky ReLU in consistence with our theory.
I NTRODUCTION
A wide range of problems involve sequential data with a natural temporal ordering.
Examples include natural language processing, time series prediction, system identiﬁcation, and control design, among others.
State-of-the-art algorithms for sequential problems often stem from dynamical systems theory and are tailored to learn from temporally dependent data.
Linear models and algorithms; such as
Kalman ﬁlter, PID controller, and linear dynamical systems, have a long history and are utilized in control theory since 1960’s with great success (Brown et al.
; Ho & Kalman (1966); Åström & Hägglund (1995)).
More recently, nonlinear models such as recurrent neural networks (RNN) found applications in complex tasks such as machine translation and speech recognition (Bahdanau et al.
; Graves et al.
; Hochreiter & Schmidhuber (1997)).
Unlike feedforward neural networks, RNNs are dynamical systems that use their internal state to process inputs.
The goal of this work is to shed light on the inner workings of RNNs from a theoretical point of view.
In particular, we focus on the RNN state equation which is characterized by a nonlinear activation function φ , state weight matrix A , and input weight matrix B as follows
[Equation]
Here h t is the state vector and u t is the input data at timestamp t .
This equation is the source of dynamic behavior of RNNs and distinguishes RNN from feedforward networks.
The weight matrices
A and B govern the dynamics of the state equation and are inferred from data.
We will explore the statistical and computational efﬁciency of stochastic gradient descent (SGD) for learning these weight matrices.
Contributions: Suppose we are given a ﬁnite trajectory of input/state pairs ( u t , h t )
Nt =0 generated from the state equation (1.1).
We consider a least-squares regression obtained from N equations; with inputs ( u t , h t )
Nt =1 and outputs ( h t +1 )
Nt =1 .
For a class of activation functions including leaky
ReLU and for stable systems 1 , we show that SGD linearly converges to the ground truth weight matrices while requiring near-optimal trajectory length N .
In particular, the required sample size is
O ( n + p ) where n and p are the dimensions of the state and input vectors respectively.
The results are extended to unstable systems when the samples are collected from multiple independent RNN trajectories rather than a single trajectory.
Our theory applies to increasing activation functions whose derivatives are bounded away from zero, which includes leaky ReLU, and Gaussian input data.
Numerical experiments on ReLU and leaky ReLU corroborate our theory and demonstrate that SGD converges faster as the activation slope increases.
To obtain our results, we i) characterize the statistical properties of the state vector (e.g. well-conditioned covariance) and ii) derive a novel SGD convergence result with nonlinear activations; which may be of independent interest.
As a whole, this paper provides a step towards foundational understanding of RNN training via SGD.
1. R ELATED W ORK
Our work is related to the recent optimization and statistics literature on linear dynamical systems (LDS) and neural networks.
Linear dynamical systems: The state-equation (1.1) reduces to a LDS when φ is the linear activation ( φ ( x ) = x ).
Identifying the weight matrices is a core problem in linear system identiﬁcation and is related to the optimal control problem (e.g. linear quadratic regulator) with unknown system dynamics.
While these problems are studied since 1950’s (Ljung (1998; 1987); Åström & Eykhoff (1971)), our work is closer to the recent literature that provides data dependent bounds and characterize the non-asymptotic learning performance.
Recht and coauthors have a series of papers exploring optimal control problem (Simchowitz et al.
; Tu et al.
In particular, Hardt et al.
(2016) shows gradient descent learns single-input-single-output (SISO) LDS with polynomial guarantees.
Oymak & Ozay (2018) and Faradonbeh et al.
(2018) provide sample complexity bounds for learning
LDS.
Sanandaji et al.
(2011b;a); Pereira et al. (2010) study the identiﬁcation of sparse systems.
Neural networks: There is a growing literature on the theoretical aspects of deep learning and provable algorithms for training neural networks.
Most of the existing results are concerned with feedforward networks.
Ge et al.
; Li & Yuan (2017); Mei et al. (2018b); Soltanolkotabi (2017); Janzamin et al. (2015); Zhong et al. (2017b) consider learning fully-connected shallow networks with gradient descent.
Mei et al.
(2018a)
; Soltanolkotabi et al.
; Foster et al.
(2018) analyze empirical landscape of related nonlinear learning problems.
Brutzkus & Globerson (2017); Zhong et al.
(2017a)
; Du et al.
; Goel et al.
(2018) address convolutional neural networks; which is an efﬁcient weight-sharing architecture.
Brutzkus et al.
; Wang et al.
(2018) studies over-parameterized networks when data is linearly separable.
Janzamin et al.
; Oymak & Soltanolkotabi (2018) utilize tensor decomposition techniques for learning feedforward neural nets.
For recurrent networks, Sedghi & Anandkumar (2016) proposed tensor algorithms with polynomial guarantees and Khrulkov et al.
(2017) studied their expressive power.
More recently, Miller & Hardt (2018) showed that stable RNNs can be approximated by feed-forward networks.
P ROBLEM S ETUP
We ﬁrst introduce the notation.
(cid:107) · (cid:107) returns the spectral norm of a matrix and s min ( · ) returns the minimum singular value.
The activation φ : R → R applies entry-wise if its input is a vector.
Throughout, φ is assumed to be a 1 -Lipschitz function.
With proper scaling of its parameters, the system (1.1) with a Lipschitz activation can be transformed into a system with 1 -Lipschitz activation.
The functions Σ [ · ] and var [ · ] return the covariance of a random vector and variance of a random variable respectively.
I n is the identity matrix of size n × n .
Normal distribution with mean µ and covariance Σ is denoted by N ( µ , Σ ) .
Throughout, c, C, c 0 , c 1 , . . .
denote positive absolute constants.
Setup: We consider the dynamical system parametrized by an activation function φ ( · ) and weight matrices A ∈ R n × n , B ∈ R n × p as described in (1.1).
Here, h t is the n dimensional state-vector and u t is the p dimensional input to the system at time t .
As mentioned previously, (1.1) corresponds to the state equation of a recurrent neural network.
For most RNNs of interest, the state h t is hidden and we only get to interact with h t via an additional output equation.
For Elman networks Elman (1990), this equation is characterized by some output activation φ y and output weights C , D as follows
[Equation]
In this work, our attention is restricted to the state equation (1.1); which corresponds to setting y t = h t +1 in the output equation.
To analyze (1.1) in a non-asymptotic data-dependent setup, we assume a ﬁnite input/state trajectory of { u t , h t } N t =0 generated by some ground truth weight matrices I p (cid:21) ( A , B ) .
Our goal is learning the unknown weights A and B in a data and computationally efﬁcient way.
In essence, we will show that, if the trajectory length satisﬁes N (cid:38) n + p , SGD can quickly and provably accomplish this goal using a constant step size.
Appoach: Our approach is described in Algorithm 1.
It takes two hyperparameters; the scaling factor µ and learning rate η .
Using the RNN trajectory, we construct N triples of the form { u t , h t , h t +1 } Nt =1 .
We formulate a regression problem by deﬁning the output vector y t , input vector x t , and the target parameter C as follows
[Equation]
With this reparameterization, we ﬁnd the input/output identity y t = φ ( Cx t ) .
We will consider the least-squares regression given by
[Equation]
L t
[Equation]
Θ ) where L t ( Θ
[Equation]
For learning the ground truth parameter C , we utilize SGD on the loss function (2.3) with a constant learning rate η .
Starting from an initial point Θ 0 , after END SGD iterations, Algrorithm 1 returns an estimate ˆ C =
Θ END .
Estimates of A and B are decoded from the left and right submatrices of ˆ C respectively.
M AIN R ESULTS 3. P RELIMINARIES
The analysis of the state equation naturally depends on the choice of the activation function; which is the source of nonlinearity.
We ﬁrst deﬁne a class of Lipschitz and increasing activation functions.
Deﬁnition 3.1 ( β -increasing activation) .
Given 1 ≥ β ≥ 0 , the activation function φ satisﬁes φ (0) = 0 and 1 ≥ φ (cid:48)
( x ) ≥ β for all x ∈ R .
Our results will apply to strictly increasing activations where φ is β -increasing for some β > 0 .
Observe that, this excludes ReLU activation which has zero derivative for negative values.
However, it includes Leaky ReLU which is a generalization of ReLU.
Parameterized by 1 ≥ β ≥ 0 , Leaky
ReLU is a β -increasing function given by
[Equation]
In general, given an increasing and 1 -Lipschitz activation φ , a β -increasing function φ β can be obtained by blending φ with the linear activation, i.e. φ β ( x ) =
(1 − β ) φ ( x ) + βx .
A critical property that enables SGD is that the state-vector covariance Σ [ h t ] is well-conditioned under proper assumptions.
The lemma below provides upper and lower bounds on this covariance matrix in terms of problem variables.
Lemma 3.2 (State vector covariance) .
Consider the state equation (1.1) where h
=
and u t i.i.d.
∼
N (0 , I p ) .
Deﬁne the upper bound term B t as
[Equation]
• Suppose φ is 1 -Lipschitz and φ (0)
= 0 .
Then, for all t ≥ 0 , Σ [ h t ] (cid:22) B 2 t I n .
• Suppose φ is a β -increasing function and p ≥ n .
Then, Σ [ h t ] (cid:23) β 2 s min ( B ) 2 I n .
As a natural extension from linear dynamical systems, we will say the system is stable if (cid:107) A (cid:107) <and unstable otherwise.
For activations we consider, stability implies that if the input is set to 0 , state vector h t will exponentially converge to 0 i.e. the system forgets the past states quickly.
This is also the reason ( B t ) t ≥ 0 sequence converges for stable systems and diverges otherwise.
The condition number of the covariance will play a critical role in our analysis.
Using Lemma 3.2, this number can be upper bounded by ρ deﬁned as
[Equation]
Observe that, the condition number of B appears inside the ρ term.
3. L EARNING FROM S INGLE T RAJECTORY
Our main result applies to stable systems ( (cid:107) A (cid:107) < 1 ) and provides a non-asymptotic convergence guarantee for SGD in terms of the upper bound on the state vector covariance.
This result characterizes the sample complexity and the rate of convergence of SGD; and also provides insights into the role of activation function and the spectral norm of A .
Theorem 3.3 (Main result) .
Let { u t , h t +1 } Nt
=1 be a ﬁnite trajectory generated from the state equation (1.1) .
Suppose (cid:107) A (cid:107) < 1 , φ is β -increasing, h 0 = 0 , p ≥ n , and u t i.i.d.
∼ N (0 , I p ) .
Let ρ be same as (3.3) and c, C, c 0 be properly chosen absolute constants.
Pick the trajectory length N to satisfy
[Equation] , where L = 1 − log( cnρ ) log (cid:107) A (cid:107) .
Pick scaling µ = 1 /B ∞ , learning rate η = cβ 2 ρn ( n + p ) , and consider the loss function (2.3) .
With probability 1 − 4 N exp( − 100 n ) − 8 L exp( −O ( NLρ 2 )) , starting from an initial point Θ 0 , for all τ ≥ 0 , the SGD iterations described in Algorithm 1 satisﬁes
[Equation]
Here the expectation is over the randomness of the SGD updates.
Sample complexity: Theorem 3.3 essentially requires N (cid:38) ( n + p ) /β 4 samples for learning.
This can be seen by unpacking (3.3) and ignoring the logarithmic L term and the condition number of
B .
Observe that O ( n + p ) growth achieves near-optimal sample size for our problem.
Each state equation (1.1) consists of n sub-equations (one for each entry of h t +1 ).
We collect N state equations to obtain a system of Nn equations.
On the other hand, the total number of unknown parameters in A and B are n ( n + p ) .
This implies Theorem 3.3 is applicable as soon as the problem is mildly overdetermined i.e. Nn (cid:38) n ( n + p ) .
Computational complexity: Theorem 3.3 requires O ( n ( n + p ) log 1 ε ) iterations to reach εneighborhood of the ground truth.
Our analysis reveals that, this rate can be accelerated if the state vector is zero-mean.
This happens for odd activation functions satisfying φ ( − x ) = − φ ( x ) (e.g. linear activation).
The result below is a corollary and requires × n less iterations.
Theorem 3.4 (Faster learning for odd activations) .
Consider the same setup provided in Theorem 3.3.
Additionally, assume that φ is an odd function.
Pick scaling µ = 1 /B ∞ , learning rate η = cβ 2 ρ ( n + p ) , and consider the loss function (2.3) .
With probability 1 − 4 N exp( − 100 n ) − 8 L exp( −O ( NLρ 2 )) , starting from an initial point Θ 0 , for all τ ≥ 0 , the SGD iterations described in Algorithm 1 satisﬁes [Equation]
[Equation]
) where the expectation is over the randomness of the SGD updates.
Another aspect of the convergence rate is the dependence on β .
In terms of β , the SGD error (3.4) decays as (1 − O ( β 8 )) τ .
While it is not clear how optimal is the exponent 8 , numerical experiments in Section 6 demonstrate that larger β indeed results in drastically faster convergence.
M AIN I DEAS AND P ROOF S TRATEGY
We ﬁrst outline our high-level proof strategy for Theorem 3.3; which brings together ideas from statistics and optimization. We ﬁrst show that input data is well-behaved by proving that state-vector h t has a wellconditioned covariance as discussed in Lemma 3.2 and shown in Appendix B. The key idea is if φ is β -increasing, then the random input data u t provides sufﬁcient excitation for the output state h t +1 . Even if individual samples are well-behaved, analyzing (2.3) is still challenging due to temporal dependencies between the samples.
These dependencies prevent us from directly using statistical learning results that typically assume i
i
d. samples.
We show that the dependency between samples at time t and t + T decay exponentially fast in separation T (for stable systems).
This is outlined in Appendix C. 3.
This observation allows us to obtain nearly independent data by subsampling the original trajectory to get ( h iT , u iT ) i ≥ 0 .
Thanks to exponential decay, a logarithmically small T can be chosen to generate large subtrajectories of size N/T .
Appendix D uses additional perturbation arguments to establish the well-behavedness of the overall data matrix.
To conclude, we obtain a deterministic result which establishes fast convergence result for β -increasing activations and well-behaved dataset.
This is provided in Theorem 4.1 and proved in Appendix A.
The ﬁrst three steps are related to the statistical nature of the problem which can be decoupled from the last step.
Speciﬁcally, the last step derives a deterministic result that establishes the linear convergence of SGD for β -increasing functions.
For linear convergence proofs, a typical strategy is showing the strong convexity of the loss function i.e. showing that, for some α > 0 and all points v , u , the gradient satisﬁes
[Equation]
The core idea of our convergence result is that the strong convexity parameter of the loss function with β -increasing activations can be connected to the loss function with linear activations .
In particular, recalling (2.3), set y lin t =
Cx t and deﬁne the linear loss to be
[Equation]
Denoting the strong convexity parameter of the original loss by α φ and that of linear loss by α lin , we argue that α φ ≥ β 2 α lin ; which allows us to establish a convergence result as soon as α lin is strictly positive.
Next result is our SGD convergence theorem which follows from this discussion.
Theorem 4.1 (Deterministic convergence) .
Suppose a data set { x i , y i } Ni =1 is given; where output y i is related to input x i via y i =
φ ( (cid:104) x
i , θ (cid:105) ) for some θ ∈ R n .
Suppose β > 0 and φ is a β -increasing.
Let γ + ≥ γ − > 0 be scalars.
Assume that input samples satisfy the bounds
[Equation]
Let { r τ } ∞ τ =0 be a sequence of i.i.d.
integers uniformly distributed between 1 to N .
Then, starting from an arbitrary point θ 0 , setting learning rate η = β 2 γ − γ + B , for all τ ≥ 0 , the SGD iterations for quadratic loss
[Equation]
satisﬁes the error bound
[Equation]
where the expectation is over the random selection of the SGD iterations { r τ } ∞ τ =0 .
This theorem provides a clean convergence rate for SGD for β -increasing activations and naturally generalizes standard results on linear regression which corresponds to β = 1 . We remark that related results appear in the literature on generalized linear models. Kakade et al. ; Foster et al.
Mei et al.
(2018a) provide learning theoretic loss/gradient/hessian convergence results for isotonic regression, robust regression, and β -increasing activations.
Goel et al. (2018) establishes a similar result for leaky ReLU activations under the assumption of symmetric input distribution and inﬁnitely many samples (i.e. in population limit). Compared to these, we establish a deterministic linear convergence guarantee for SGD that works whenever the data matrix is full rank. We believe extensions to proximal gradient methods might be beneﬁcial for high-dimensional nonlinear problems (e.g. sparse/low-rank approximation, manifold constraints Cai et al.
; Beck & Teboulle (2009); Oymak et al.
Agarwal et al. (2010); Pereira et al. and is left as a future work.
To derive our main results in Section 3, we need to address the ﬁrst three steps outlined earlier and determine the conditions under which Theorem 4.1 is applicable to the data obtained from RNN state equation with high probability.
Below we provide desirable characteristics of the state vector; which enables our statistical results.
Assumption 1 (Well-behaved state vector) .
Let L > 1 be an integer.
There exists positive scalars γ + , γ − , θ and an absolute constant C > 0 such that θ ≤ 3 √ n and the following holds
• Lower bound: Σ [ h L − 1 ] (cid:23)
[Equation]
• Upper bound: for all t , the state vector satisﬁes
[Equation]
Here (cid:107)·(cid:107)
ψ 2 returns the subgaussian norm of a vector (see Def.
5.22 of Vershynin (2010)).
Assumption 1 ensures that covariance is well-conditioned, state vector is well-concentrated, and it has a reasonably small expectation.
Our next theorem establishes statistical guarantees for learning the RNN state equation based on this assumption.
Theorem 4.2 (General result) .
Let { u t , h t +1 } Nt
=1 be a length N trajectory of the state equation (1.1) .
Suppose (cid:107) A (cid:107) < 1 , φ is β -increasing, h 0 = 0 , and u t i.i.d.
∼ N (0 , I p ) .
Given scalars γ + ≥ γ − > 0 , set the condition number as ρ = γ + /γ − .
For absolute constants C, c, c 0 > 0 , choose trajectory length N to satisfy
[Equation]
Suppose Assumption 1 holds with L, γ + , γ − , θ .
Pick scaling to be µ = 1 / √ γ + and learning rate to be η = cβρ ( θ + √ 2) 2 ( n + p ) .
With probability 1 − 4 N exp( − 100 n ) − 8 L exp( −O ( NLρ 2 )) , starting from Θ 0 , for all τ ≥ 0 , the SGD iterations on loss (2.3) as described in Algorithm 1 satisﬁes
[Equation]
where the expectation is over the randomness of SGD updates.
The advantage of this theorem is that, it isolates the optimization problem from the statistical properties of state vector.
If one can prove tighter bounds on achievable ( γ + , γ − , θ ) , it will immediately imply improved performance for SGD.
In particular, Theorems 3.3 and 3.4 are simple corollaries of Theorem 4.2 with proper choices.
• Theorem 3.3 follows by setting γ + = B 2 ∞ , γ − =
β 2 s min ( B ) 2 , and θ = √ n .
• Theorem 3.4 follows by setting γ + = B 2 ∞ , γ − =
β 2 s min ( B ) 2 , and θ = 0 .
L EARNING U NSTABLE S YSTEMS
So far, we considered learning from a single RNN trajectory for stable systems ( (cid:107) A (cid:107) < 1 ).
For such systems, as the time goes on, the impact of the earlier states disappear.
In our analysis, this allows us to split a single trajectory into multiple nearly-independent trajectories.
This approach will not work for unstable systems ( A is arbitrary) where the impact of older states may be ampliﬁed over time.
To address this, we consider a model where the data is sampled from multiple independent trajectories.
Suppose N independent trajectories of the state-equation (1.1) are available.
Pick some integer
T 0 ≥ 1 .
Denoting the i th trajectory by the triple ( h ( i ) t +1 , h ( i ) t , u ( i ) t ) t ≥ 0 , we collect a single sample from each trajectory at time T 0 to obtain the triple ( h ( i ) T
+1 , h ( i ) T 0 , u
( i ) T 0 ) .
To utilize the existing optimization framework (2.3); for 1 ≤ i ≤ N , we set,
[Equation]
With this setup, we can again use the SGD Algorithm 1 to learn the weights A and B .
The crucial difference compared to Section 3 is that, the samples ( y i , h i , u i ) Ni =1 are now independent of each other; hence, the analysis is simpliﬁed.
As previously, having an upper bound on the condition number of the state-vector covariance is critical.
This upper bound can be shown to be ρ deﬁned as
[Equation]
The ¯ ρ term is similar to the earlier deﬁnition (3.3); however it involves B T 0 rather than B ∞ .
This modiﬁcation is indeed necessary since B ∞ = ∞ when (cid:107) A (cid:107) > 1 .
On the other hand, note that, B 2 Tgrows proportional to (cid:107) A (cid:107) 2 T 0 ; which results in exponentially bad condition number in T 0 .
Our ρ deﬁnition remedies this issue for single-output systems; where n = 1 and A is a scalar.
In particular, when β = 1 (e.g. φ is linear) ρ becomes equal to the correct value 1 2 .
The next theorem provides our result on unstable systems in terms of this condition number and other model parameters.
Theorem 5.1 (Unstable systems) .
Suppose we are given N independent trajectories ( h ( i ) t , u ( i ) t ) t ≥for 1 ≤ i ≤ N .
Each trajectory is sampled at time T 0 to obtain N samples ( y i , h i , u i )
Ni
=1 where the i th sample is given by (5.1) .
Suppose the sample size satisﬁes
[Equation] where ρ is given by (5.2) .
Assume the initial states are 0 , φ is β -increasing, p ≥ n , and u t i.i.d.
∼
N (0 , I p ) .
Set scaling µ = 1 / (cid:112) B T 0 , learning rate η = cβ 2 ρn ( n + p ) , and run SGD over the equations described in (2.2) and (2.3) .
Starting from Θ 0 , with probability 1 − 2 N exp( − 100( n + p ))
− 4 exp( −O ( Nρ 2 )) , all SGD iterations satisfy
[Equation]
where the expectation is over the randomness of the SGD updates.
N UMERICAL E XPERIMENTS
We conducted experiments on ReLU and Leaky ReLU activations.
Let us ﬁrst describe the experimental setup.
We pick the state dimension n = 50 and the input dimension p = 100 .
We choose the ground truth matrix A to be a scaled random unitary matrix; which ensures that all singular values of
A are equal.
B is generated with i.i.d.
N (0 , 1) entries.
Instead of using the theoretical scaling choice, we determine the scaling µ from empirical covariance matrices outlined in Algorithm 2.
Similar to our proof strategy, this algorithm equalizes the spectral norms of the input and state covariances to speed up convergence.
We also empirically determined the learning rate and used η = 1 / 100 in all experiments. [Figure]
Evaluation: We consider two performance measures in the experiments.
Let ˆ C be an estimate of the ground truth parameter C =
[ µ − 1 A B ] .
The ﬁrst measure is the normalized error deﬁned as (cid:107) ˆ C − C (cid:107) 2 F / (cid:107) C (cid:107) 2 F .
The second measure is the normalized loss deﬁned as
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation] [Figure] [Equation] In all experiments, we run Algorithm 1 for 50000 SGD iterations and plot these measures as a function of τ ; by using the estimate available at the end of the τ th SGD iteration for 0 ≤ τ ≤ 50000 .
Each curve is obtained by averaging the outcomes of 20 independent realizations.
Our ﬁrst experiments use N = 500 ; which is mildly larger than the total dimension n + p = 150 .
In Figure 1, we plot the Leaky ReLU errors with varying slopes as described in (3.1).
Here β = 0 corresponds to ReLU and β = 1 is the linear model.
In consistence with our theory, SGD achieves linear convergence and as β increases, the rate of convergence drastically improves 3 .
The improvement is more visible for less stable systems driven by A with a larger spectral norm.
In particular, while ReLU converges for small (cid:107) A (cid:107) , SGD gets stuck before reaching the ground truth when (cid:107) A (cid:107) = 0 .
To understand, how well SGD ﬁts the training data, in Figure 2a, we plotted the normalized loss for ReLU activation.
For more unstable system ( (cid:107) A (cid:107) = 0 . 9 ), training loss stagnates in a similar fashion to the parameter error.
We also veriﬁed that the norm of the overall gradient (cid:107)∇L (Θ τ ) (cid:107) F continues to decay (where Θ τ is the τ th SGD iterate); which implies that SGD converges before reaching a global minima.
As A becomes more stable, rate of convergence improves and linear rate is visible.
Finally, to better understand the population landscape of the quadratic loss with ReLU activations, Figure 2b repeats the same ReLU experiments while increasing the sample size ﬁve times to N = 2500 .
For this more overdetermined problem, SGD converges even for (cid:107) A (cid:107) = 0 .
9 ; indicating that
• population landscape of loss with ReLU activation is well-behaved, • however ReLU problem requires more data compared to the Leaky ReLU for ﬁnding global minima.
Overall, as predicted by our theory, experiments verify that SGD indeed quickly ﬁnds the optimal weight matrices of the state equation (1.1) and as the activation slope β increases, the convergence rate improves [Figure]
C ONCLUSIONS
This work showed that SGD can learn the nonlinear dynamical system (1.1); which is characterized by weight matrices and an activation function.
This problem is of interest for recurrent neural networks as well as nonlinear system identiﬁcation.
We showed that efﬁcient learning is possible with optimal sample complexity and good computational performance.
Our results apply to strictly increasing activations such as Leaky ReLU.
We empirically showed that Leaky ReLU converges faster than
ReLU and requires less samples; in consistence with our theory.
We list a few unanswered problems that would provide further insights into recurrent neural networks.
• Covariance of the state-vector: Our results depend on the covariance of the state-vector and requires it to be positive deﬁnite.
One might be able to improve the current bounds on the condition number and relax the assumptions on the activation function.
Deriving similar performance bounds for ReLU is particularly interesting.
• Hidden state: For RNNs, the state vector is hidden and is observed through an additional equation (2.1); which further complicates the optimization landscape.
Even for linear dynamical systems, learning the ( A , B , C , D ) system ((1.1), (2.1)) is a non-trivial task Ho & Kalman (1966);
Hardt et al.
What can be said when we add the nonlinear activations?
• Classiﬁcation task:
In this work, we used normally distributed input and least-squares regression for our theoretical guarantees.
More realistic input distributions might provide better insight into contemporary problems, such as natural language processing; where the goal is closer to classiﬁcation (e.g. ﬁnding the best translation from another language).
A D ETERMINISTIC C ONVERGENCE R ESULT FOR SGD
Proof of Theorem 4.1.
Given two distinct scalars a, b ; deﬁne φ (cid:48) ( a, b ) =
φ ( a ) − φ ( b ) a − b .
φ (cid:48) ( a, b ) ≥ β since φ is β -increasing.
Deﬁne w τ to be the residual w τ = θ τ − θ .
Observing
[Equation]
the SGD recursion obeys
[Equation]
where G r τ = x r τ φ (cid:48)
( x Tr τ θ τ ) φ
[Equation]
( x Tr τ θ τ , x
Tr τ θ ) x
Tr τ .
Since φ is 1 -Lipschitz and β -increasing, G r τ is a positive- semideﬁnite matrix satisfying
[Equation]
Consequently, we ﬁnd the following bounds in expectation
[Equation]
Observe that (A.1) essentially lower bounds the strong convexity parameter of the problem with β 2 γ − ; which is the strong convexity of the identical problem with the linear activation (i.e. β = 1 ).
However, we only consider strong convexity around the ground truth parameter θ i.e. we restricted our attention to ( θ , θ τ ) pairs.
With this, w τ +1 can be controlled as,
[Equation]
Setting η = β 2 γ − γ + B , we ﬁnd the advertised bound
[Equation]
Applying induction over the iterations τ , we ﬁnd the advertised bound (4.2)
[Equation]
Lemma A.1 (Merging L splits) .
Assume matrices X ( i
) ∈
R N i × q are given for 1 ≤ i ≤ L .
Suppose for all 1 ≤ i ≤ L , rows of X ( i ) has (cid:96) 2 norm at most √ B and each X ( i ) satisﬁes
[Equation]
γ − I n .
Set N = (cid:80) Li =
1 N i and form the concatenated matrix X =  
X (1)
X (
[Equation]
X ( L )   .
Denote i th row of X by x i .
Then, for each i , (cid:107) x i (cid:107) 2 (cid:96) 2 ≤ B and γ + I n (cid:23) X T X
[Equation]
=
[Equation]
N (cid:88) i =
[Equation]
I n .
Proof.
The bound on the rows (cid:107) x i (cid:107) (cid:96) 2 directly follows by assumption.
For the remaining result, ﬁrst observe that X T X
= (cid:80) Li
=1 X ( i ) T X
( i ) .
Next, we have
[Equation]
Combining these two yields the desired upper/lower bounds on X T X /N .
B P ROPERTIES OF THE NONLINEAR STATE EQUATIONS
This section characterizes the properties of the state vector h t when input sequence is normally distributed.
These bounds will be crucial for obtaining upper and lower bounds for the singular values of the data matrix
X =
[ x 1 . . .
x N ] T described in (2.2).
For probabilistic arguments, we will use the properties of subgaussian random variables.
Orlicz norm provides a general framework that subsumes subgaussianity.
Deﬁnition B.1 (Orlicz norms) .
For a scalar random variable Orlicz-
a norm is deﬁned as
[Equation]
Orlicz- a norm of a vector x ∈ R p is deﬁned as (cid:107)
[Equation]
(cid:107)
[Equation]
ψ a where B p is the unit (cid:96) 2 ball.
The subexponential norm is the Orlicz- 1 norm (cid:107)·(cid:107) ψ 1 and the subgaussian norm is the Orlicz- 2 norm (cid:107)·(cid:107) ψ 2 .
Lemma B.2 (Lipschitz properties of the state vector) .
Consider the state equation (1.1) .
Suppose activation φ is 1 -Lipschitz.
Observe that h t +1 is a deterministic function of the input sequence { u τ } tτ =0 .
Fixing all vectors { u i } i (cid:54) =
τ (i.e. all except u τ ), h t +1 is (cid:107) A (cid:107) t − τ (cid:107) B (cid:107) Lipschitz function of u τ for 0 ≤ τ ≤ t .
Proof.
Fixing { u i } i (cid:54)
= τ , denote h t +1 as a function of u τ by h t
+1 ( u τ ) .
Given a pair of vectors u τ , u (cid:48) τ using 1 -Lipschitzness of φ , for any t > τ , we have
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Proceeding with this recursion until t = τ , we ﬁnd
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
This bound implies h t +1 ( u τ ) is (cid:107) A (cid:107) t − τ (cid:107) B (cid:107) Lipschitz function of u τ .
Lemma B.3 (Upper bound) .
Consider the state equation governed by equation (1.1) .
Suppose u t i.i.d.
∼ N (0 , I p ) , φ is 1 -Lipschitz, φ (0)
=
and h 0 = 0 .
Recall the deﬁnition (3.2) of B t .
We have the following properties
• h t is a B t -Lipschitz function of the vector q t =
[ u T 0 . . .
u
Tt
[Equation]
1 ] T ∈ R tp .
• There exists an absolute constant c >
such that (cid:107) h t − E [ h t ] (cid:107) ψ 2 ≤ cB t and Σ [ h t ] (cid:22) B
2 t I n .
• h t satisﬁes
[Equation]
Also, there exists an absolute constant c > 0 such that for any m ≥ n , with probability 1 −
exp(
[Equation]
100 m
[Equation] [Equation]
h t
[Equation] [Equation]
Proof.
i)
Bounding Lipschitz constant: Observe that h t is a deterministic function of q t i.e. h t = f ( q t ) for some function f .
To bound Lipschitz constant of f , for all (deterministic) vector pairs q t and ˆ q t , we ﬁnd a scalar
L f satisfying,
[Equation]
B
[Equation]
Deﬁne the vectors, { a i } ti =0 , as follows a
[Equation]
Observing that a 0 = q t , a t = ˆ q t , we write the telescopic sum,
[Equation]
≤
[Equation]
a
[Equation]
f
[Equation]
a i ) (cid:107) [Equation]
Focusing on the individual terms f ( a i +1 ) − f ( a i ) , observe that the only difference is the u i , ˆ u i terms.
Viewing h t as a function of u i and applying Lemma B.2,
[Equation]
To bound the sum, we apply the Cauchy-Schwarz inequality; which yields
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
The ﬁnal line achieves the inequality (B.1) with L f =
B t hence h t is B t Lipschitz function of q t .
ii)
Bounding subgaussian norm: When u t i.i.d.
∼ N (0 , I p ) , the vector q t is distributed as N (0 , I tp ) .
Since h t a
B t Lipschitz function of q t , for any ﬁxed unit length vector v , α v := v T h t = v T f ( q t ) is still B t -Lipschitz function of q t .
Hence, using Gaussian concentration of Lipschitz functions, α v satisﬁes
P ( | α
[Equation]
This implies that for any v , α v − E
[ α v ] is O ( B t ) subgaussian Vershynin (2010).
This is true for all unit v , hence using Deﬁnition B.1, the vector h t satisﬁes (cid:107) h t − E
[ h t ] (cid:107) ψ 2 ≤ O ( B t ) as well.
Secondly, B t -Lipschitz function of a Gaussian vector obeys the variance inequality var [ α v ] ≤ B 2 t (page 49 of Ledoux (2001)), which implies the covariance bound
[Equation]
iii) Bounding (cid:96) 2 -norm: To obtain this result, we ﬁrst bound E [ (cid:107)
h
[Equation] [Equation]
Since φ is 1 -Lipschitz and φ (0) = 0 , we have the deterministic relation
[Equation]
Taking squares of both sides, expanding the right hand side, and using the independence of h t , u t and the covariance information of u t , we obtain
[Equation]
Now that the recursion is established, expanding h t on the right hand side until h 0 = 0 , we obtain
[Equation] [Equation]
≤ t
[Equation]
i =
[Equation]
Now using the fact that tr ( BB T ) ≤ rank ( B ) (cid:107) B (cid:107) 2 ≤ min { n, p }(cid:107) B (cid:107) 2 , we ﬁnd
[Equation]
Finally, using the fact that h t is B t -Lipschitz function and utilizing Gaussian concentration of q t ∼ N (0 , I tp ) ,
we ﬁnd
[Equation]
Setting t =
( c − 1) √ mB t for sufﬁciently large c > 0 ,
we ﬁnd P ( (cid:107) h t (cid:107) (cid:96) 2 ≥ √ nB t + ( c − 1) √ mB t ) ≤ exp( − 100 m ) .
Lemma B.4 (Odd activations) .
Suppose φ is strictly increasing and obeys φ ( x ) = − φ ( − x ) for all x and h 0 = 0 .
Consider the state equation (1.1) driven u t i.i.d.
∼ N (0 , I p ) .
We have that E [ h t ] = 0 .
Proof.
We will inductively show that { h t } t ≥ 0 has a symmetric distribution around 0 .
Suppose the vector h t satisﬁes this assumption.
Let S ⊂ R n be a set.
We will argue that P ( h t +1 ⊂ S ) =
P ( h t +1 ⊂ − S ) .
Since φ is strictly increasing, it is bijective on vectors, and we can deﬁne the unique inverse set S (cid:48)
=
φ − 1 ( S ) .
Also since φ is odd, φ ( − S (cid:48) )
= − S .
Since h t , u t are independent and symmetric, we reach the desired conclusion as follows
[Equation]
Theorem B.5 (State-vector lower bound) .
Consider the nonlinear state equation (1.1) with { u t } t ≥ 0 i.i.d.
∼
N (0 , I p ) .
Suppose φ is a β -increasing function for some constant β > 0 .
For any t ≥ 1 , the state vector obeys
[Equation]
Proof.
The proof is an application of Lemma B.7.
The main idea is to write h t as sum of two independent vectors, one of which has independent entries.
Consider a multivariate Gaussian vector g ∼ N (0 , Σ ) .
g is statistically identical to g 1 + g 2 where g 1 ∼ N (0 , s min ( Σ ) I d ) and g 2 ∼ N (0 ,
Σ − s min ( Σ ) I d ) are independent multivariate Gaussians.
Since Bu t ∼ N (0 , BB T ) , setting Σ = BB T and s min = s min ( Σ ) , we have that Bu t ∼ g 1 + g 2 where g 1 , g 2 are independent and g 1 ∼ N (0 , s min I n ) and g 2 ∼ N (0 ,
Σ − s min I n ) .
Consequently, we may write
[Equation]
For lower bound, the crucial component will be the g 1 term; which has i.i.d.
entries.
Applying Lemma B.7 by setting x = g 1 and y
= g 2 + Ah t , and using the fact that h t , g 1 , g 2 are all independent of each other, we ﬁnd the advertised bound, for all t ≥ 0 , via
[Equation]
The next theorem applies to multiple-input-single-output (MISO) systems where A is a scalar and B is a row vector.
The goal is reﬁning the lower bound of Theorem B.5.
Theorem B.6 (MISO lower bound) .
Consider the setup of Theorem B.5 with single output i.e. n = 1 .
For any t ≥ 1 , the state vector obeys
[Equation]
Proof.
For any random variable X , applying Lemma B.7, we have var [ φ ( X )] ≥ β 2 var [ X ] .
Recursively, this yields
[Equation]
Expanding these inequalities till h 0 , we obtain the desired bound
[Equation]
Lemma B.7 (Vector lower bound) .
Suppose φ is a β -increasing function.
Let x =
[ x 1 . . .
x n ] T be a vector with i.i.d.
entries distributed as x i ∼ X .
Let y be a random vector independent of x .
Then, Σ
[Equation]
Proof.
We ﬁrst apply law of total covariance (e.g. Lemma B.8) to simplify the problem using the following lower bound based on the independence of x and y ,
[Equation]
Now, focusing on the covariance Σ x
[ φ ( x + y )] , ﬁxing a realization of y , and using the fact that x has i.i.d.
entries; φ ( x + y ) has independent entries as φ applies entry-wise.
This implies that Σ x [ φ ( x + y )] is a diagonal matrix.
Consequently, its lowest eigenvalue is the minimum variance over all entries,
[Equation]
Fortunately, Lemma B.9 provides the lower bound var [ φ ( x i + y i )] ≥ β 2 var [ X ] .
Since this lower bound holds for any ﬁxed realization of y , it still holds after taking expectation over y ; which concludes the proof.
The next two lemmas are helper results for Lemma B.7 and are provided for the sake of completeness.
Lemma B.8 (Law of total covariance) .
Let x , y be two random vectors and assume y has ﬁnite covariance.
Then
[Equation]
Proof.
First, write Σ [ y ] =
E [ yy T ] − E
[ y ] E [ y T ] .
Then, applying the law of total expectation to each term,
[Equation]
Next, we can write the conditional expectation as E [ E [ yy T (cid:12)(cid:12) x ]]
=
E [ Σ [ y (cid:12)(cid:12) x ]] + E [ E [ y (cid:12)(cid:12) x ] E [ y (cid:12)(cid:12) x ]] T .
To conclude, we obtain the covariance of E [ y (cid:12)(cid:12) x ] via the difference,
[Equation]
which yields the desired bound.
Lemma B.9 (Scalar lower bound) .
Suppose φ is a β -increasing function with β > 0 as deﬁned in Deﬁnition 3.1.
Given a random variable X and a scalar y , we have
[Equation]
var [
[Equation]
Proof.
Since φ is β -increasing, it is invertible and φ − 1 is strictly increasing.
Additionally, φ − 1 is 1 /β Lipschitz since,
[Equation]
Using this observation and the fact that E [ X ] minimizes E ( X − α ) 2 over α , var [ φ ( X + y )] can be lower bounded as follows
[Equation]
Note that, the ﬁnal line is the desired conclusion.
C T RUNCATING S TABLE S YSTEMS
One of the challenges in analyzing dynamical systems is the fact that samples from the same trajectory have temporal dependence.
This section shows that, for stable systems, the impact of the past states decay exponentially fast and the system can be approximated by using the recent inputs only.
We ﬁrst deﬁne the truncation of the state vector.
Deﬁnition C.1 (Truncated state vector) .
Suppose φ (0) = 0 , initial condition h 0 = 0 , and consider the state equation (1.1) .
Given a timestamp t , L -truncation of the state vector h t is denoted by ¯ h t,L and is equal to q t where
[Equation]
is the state vector generated by the inputs u (cid:48)
τ satisfying
[Equation]
if
[Equation]
u τ else
[Equation]
In words, L truncated state vector ¯ h t,L is obtained by unrolling h t until time t − L and setting the contribution of the state vector h t − L to 0 .
This way, ¯ h t,L depends only on the variables { u τ } t − 1 τ = t − L .
The following lemma states that impact of truncation can be made fairly small for stable systems ( (cid:107) A (cid:107) < 1 ).
Lemma C.2 (Truncation impact – deterministic) .
Consider the state vector h t and its L -truncation ¯ h t,L from Deﬁnition C.1.
Suppose φ is 1 -Lipschitz.
We have that
[Equation]
else .
Proof.
When t ≤ L , Deﬁnition C.1 implies u (cid:48)
τ
= u τ hence h t = q t = ¯ h t,L .
When t > L , we again use Deﬁnition C.1 and recall that u (cid:48) τ
= 0 until time τ = t − L − 1 .
For all t − L < τ ≤ t , using 1 -Lipschitzness of φ , we have that
[Equation]
Applying this recursion between t − L < τ ≤ t and using the fact that q t − L = 0 implies the advertised result
[Equation]
C. N EAR INDEPENDENCE OF SUB - TRAJECTORIES
We will now argue that, for stable systems, a single trajectory can be split into multiple nearly independent trajectories.
First, we describe how the sub-trajectories are constructed.
Deﬁnition C.3 (Sub-trajectory) .
Let sampling rate L ≥ 1 and offset 1 ≤ ¯ τ ≤ L be two integers.
Let ¯ N = ¯
N ¯ τ be the largest integer obeying ( ¯ N − 1)
L + ¯ τ ≤ N .
We sample the trajectory { h t , u t } Nt =0 at the points ¯ τ, ¯ τ + L, . . .
, ¯ τ + ( ¯ N − 1) L + ¯ τ and deﬁne the ¯
τ th sub-trajectory as
[Equation]
Deﬁnition C.4 (Truncated sub-trajectory) .
Consider the state equation (1.1) and recall Deﬁnition C.1.
Given offset ¯ τ and sampling rate L , for 1 ≤ i ≤ ¯ N , the i th truncated sub-trajectory states are { ¯ h ( i ) }
¯ Ni =1 where the i th state is deﬁned as
[Equation]
The truncated samples are independent of each other as shown in the next lemma.
Lemma C.5.
Consider the truncated states of Deﬁnition C.4.
If (1.1) is generated by independent vectors { u t } t ≥ 0 , for any offset ¯ τ and sampling rate L , the vectors { ¯ h ( i ) } ¯ Ni =1 , { u ( i ) } ¯ Ni =1 are all independent of each other.
Proof.
By construction ¯ h ( i ) only depends on the vectors { u τ } L ( i − 1)+¯ τ − 1 τ
=
L ( i − 2)+¯ τ +1 .
Note that the dependence ranges [ L ( i − 2)
+ ¯ τ + 1 , L ( i − 1) + ¯ τ − 1] are disjoint intervals for different i ’s; hence (¯ h ( i ) )
¯ Ni =1 are independent of each other.
To show the independence of u ( i ) and ¯ h ( i ) ; observe that inputs u ( i ) =
u L ( i − 1)+¯ τ have timestamp ¯ τ modulo L ; which is not covered by the dependence range of (¯ h ( i ) )
¯ Ni =1 .
If the input is randomly generated, Lemma C.2 can be combined with a probabilistic bound on h t , to show that truncated states ¯ h ( i ) are fairly close to the actual states h ( i ) .
Lemma C.6 (Truncation impact – random) .
Given offset ¯ τ and sampling rate L , consider the state vectors of the sub-trajectory { h ( i ) }
¯ Ni =1 and L − 1 -truncations (¯ h ( i ) )
¯ Ni =1 .
Suppose { u t } t ≥ 0 i.i.d.
∼ N (0 , I p ) , (cid:107) A (cid:107) < 1 , h 0 = 0 , φ is 1 -Lipschitz, and φ (0)
= 0 .
Also suppose upper bound (4.3) of Assumption 1 holds for some θ ≤ √ n, γ + > 0 .
There exists an absolute constant c > 0 such that with probability at least 1 − 2 ¯ N exp( − 100 n ) , for all 1 ≤ i ≤ ¯ N , the following bound holds
[Equation]
In particular, we can always pick γ + = B 2 ∞ (via Lemma B.3).
Proof.
Using Assumption 1, we can apply Lemma F.3 on vectors { h ( i − 2)
L +¯ τ +1 } ¯ Ni
=1 .
Using a union bound, with desired probability, all vectors obey
[Equation]
2 ≤ ( c − 1) √ nγ + , for sufﬁciently large c .
Since θ ≤ √ n , triangle inequality implies (cid:107) h
( i − 2)
L +¯ τ +1 (cid:107) (cid:96) 2 ≤ c √ nγ + .
Now, applying
Lemma C.2, for all 1 ≤ i ≤ ¯ N , we ﬁnd
[Equation]
D P ROPERTIES OF THE DATA MATRIX
This section utilizes the probabilistic estimates from Section B to provide bounds on the condition number of data matrices obtained from the RNN trajectory (1.1).
Following (2.2), these matrices H , U and X are deﬁned as
H
[Equation] [Equation]
The challenge is that, the state matrix H has dependent rows; which will be addressed by carefully splitting the trajectory { u t , h t } Nt =0 into multiple sub-trajectories which are internally weakly dependent as discussed in Section
C.
We ﬁrst deﬁne the matrices obtained from these sub-trajectories.
Deﬁnition D.1.
Given sampling rate L and offset ¯ τ , consider the L -subsampled trajectory { h ( i ) , u ( i ) }
¯ Ni =1 as described in Deﬁnitions C.3 and C.4.
Deﬁne the matrices ¯ H =
¯ H (¯ τ ) ∈
R ¯ N × n , ˜ H =
˜ H (¯ τ ) ∈
R ¯ N × n , ˜ U =
˜
U (¯ τ ) ∈
R ¯ N × p , and ˜ X =
˜ X (¯ τ ) ∈
R ¯ N × ( n + p ) as
[Equation]
Lemma D.2 (Handling perturbation) .
Consider the nonlinear state equation (1.1) .
Given sampling rate L >and offset ¯ τ , consider the matrices ¯ H , ˜ H , ˜ X of Deﬁnition D.1 and let Q =
[ γ − 1 / 2 + ¯ H ˜ U
] ∈ R ¯ N × ( n + p ) .
Suppose Assumption 1 holds, φ is β -increasing, and u t i.i.d.
∼ N (0 , I p ) .
There exists an absolute constant
C > 0 such that if ¯ N ≥ C γ 2+ γ 2 − ( n + p ) , with probability 1 − 8 exp( − c γ 2 − γ 2+ ¯ N ) , for all matrices M obeying (cid:107) M − ¯ H (cid:107) ≤ √
[Equation]
10 , the perturbed Q matrices given by,
[Equation]
satisfy
[Equation]
Proof.
This result is a direct application of Theorem F.1 after determining minimum/maximum eigenvalues of population covariance.
The cross covariance obeys E
[ ¯ H T ˜ U ]
=
due to independence.
Also, for i > 1 , the truncated state vector ¯ h ( i ) is statistically identical to h L − 1 hence Σ [¯ h ( i ) ]
(cid:23) γ − I n .
Consequently, Σ [ u ( i ) ] =
I p , 1 γ + Σ [¯ h ( i ) ]
(cid:22) I n for all i and γ − γ + I n (cid:22) 1 γ + Σ
[¯ h ( i ) ] for all i > 1 .
Hence, setting q i = (cid:34) 1 √ γ + ¯ h ( i ) u
( i ) (cid:35) , for all
i >
[Equation] [Equation]
] (cid:22) I n .
Set the matrix ¯ Q =
[ q 2 . . .
q ¯ N ] T and note that Q =
[ q 1 ¯ Q T ] T .
Applying Theorem F.1 on ¯ Q and Corollary
F.2 on Q , we ﬁnd that, with the desired probability,
[Equation]
Setting E
=
M − ¯ H and observing ˜ Q = Q + [ γ − 1 / 2 +
E 0] , the impact of the perturbation E can be bounded naively via s min ( Q ) − γ − 1 / 2 + (cid:107) E (cid:107) ≤ s min ( ˜ Q ) ≤ (cid:107) ˜ Q (cid:107) ≤ (cid:107) Q (cid:107) + γ − 1 / 2 + (cid:107) E (cid:107) .
Using the assumed bound on (cid:107) E (cid:107) , this yields
[Equation]
This ﬁnal inequality is identical to the desired bound (D.3).
Theorem D.3 (Data matrix condition) .
Consider the nonlinear state-equation (1.1) .
Given γ + ≥ γ − > 0 , deﬁne the condition number ρ = γ + γ − .
For some absolute constants c, C > 0 , pick a trajectory length N where
[Equation]
and pick scaling µ = 1 √ γ + .
Suppose (cid:107) A (cid:107) < 1 , φ is β -increasing, u t i.i.d.
∼ N (0 , I p ) , and Assumption 1 holds with γ + , γ − , θ, L .
Matrix X
=
[ x 1 . . .
x N ] T of (D.1) satisﬁes the following with probability 1 − 4 N exp( − 100 n ) − 8 L exp( −O ( N 0 /
[Equation]
• Each row of X has (cid:96) 2 norm at most c 0 √ p + n where c 0 is an absolute constant.
• X T X obeys the bound
[Equation]
Proof.
The ﬁrst statement on (cid:96) 2 -norm bound can be concluded from Lemma D.4 and holds with probability 1 − 2 N exp( − 100( n + p )) .
To show the second statement, for a ﬁxed offset 1 ≤ ¯ τ ≤ L , consider Deﬁnition D. and the matrices ˜ H (¯ τ ) ,
˜ U (¯ τ ) , ˜ X (¯ τ ) .
Observe that X is obtained by merging multiple sub-trajectory matrices
{ ˜ X (¯ τ ) }
L ¯ τ =1 .
We will ﬁrst show the advertised bound for an individual ˜ X (¯ τ ) by applying Lemma D.2 and then apply Lemma A.1 to obtain the bound on the combined matrix X .
Recall that ¯ N ¯ τ is the length of the ¯ τ th sub-trajectory i.e. number of rows of ˜ X (¯ τ ) .
By construction 2 N 0 ≥ ¯
N ¯ τ ≥ N 0 for all 1 ≤ ¯ τ ≤ L .
Given 1 ≤ ¯ τ ≤ L and triple ¯ H (¯ τ ) , ˜ H (¯ τ ) , ˜ U (¯ τ ) , set Q = [ µ ¯ H (¯ τ )
˜
U (¯ τ ) ] .
Since
N 0 is chosen to be large enough, applying Theorem D.2 with µ = 1 / √ γ + choice, and noting ρ = γ + /γ − , we ﬁnd that, with probability 1 − 4 exp( − c 1 N 0 /ρ 2 ) , all matrices M satisfying (cid:107) M − ¯ H (¯ τ ) (cid:107) ≤ √ γ − N 0 / 10 and ˜
Q as in (D.2) obeys
[Equation]
Let us call this Event 1.
To proceed, we will argue that with high probability (cid:107) ˜ H (¯ τ ) − ¯ H (¯ τ ) (cid:107) is small so that the bound above is applicable with M =
˜ H (¯ τ ) choice; which sets ˜ Q = ˜ X (¯ τ ) in (D.5).
Applying Lemma C.6, we ﬁnd that, with probability 1 − 2 ¯ N ¯ τ exp( − 100 n ) ,
[Equation]
Let us call this Event 2.
We will show that our choice of L ensures right hand side is small enough and guarantees (cid:107) ¯ H (¯ τ ) − ˜ H (¯ τ ) (cid:107) ≤ √ γ − N 0 / 10 .
Set c = max { 200 c 2 0 , 1 } .
Desired claim follows by taking logarithms of upper/lower bounds and cancelling out √ N 0 terms as follows
[Equation]
Here we use the fact that log (cid:107) A (cid:107) < 0 since (cid:107) A (cid:107) < 1 and cnρ ≥ 0 .
Consequently, both Event 1 and Eventhold with probability 1 − 4 exp( − c 1 N 0 /ρ 2 ) − 2 ¯ N ¯ τ exp( − 100 n ) , implying (D.5) holds with ˜ Q = ˜
X (¯ τ ) .
Union bounding this over 1 ≤ ¯ τ ≤ L , (D.5) uniformly holds with ˜ Q = ˜
X (¯ τ ) and all rows of X are (cid:96) 2 -bounded with probability 1 − 4 N exp( − 100 n ) − 8 L exp( − c 1 N 0 /ρ 2 ) .
Applying Lemma A.1 on ( ˜ X (¯ τ ) )
L ¯
[Equation]
=1 , we conclude with the bound (D.4) on the merged matrix X .
Lemma D.4 ( (cid:96) 2 -bound on rows) .
Consider the setup of Theorem D.3.
With probability 1 − 2 N exp( − 100( n + p )) , each row of X has (cid:96) 2 -norm at most c √ p + n for some constant c > 0 .
Proof.
The t th row of X is equal to x t =
[ h Tt √ γ + u Tt ] T .
Since (cid:107) h t − E [ h t ] (cid:107) ψ 2 ≤ O ( √ γ + ) and (cid:107) u t (cid:107) ψ 2 ≤
O (1) , we have that (cid:107) x t − E
[ x t ] (cid:107) ψ 2 ≤ O (1) .
Now, applying Lemma F.3 on all rows { x t } Nt =1 , and using a union bound, with probability at least 1 − 2 N exp( − 100( n + p )) , we have that (cid:107) x t − E
[ x t ] (cid:107) (cid:96) 2 ≤ c √ n + p for all t .
To conclude, note that (cid:107) E [ x t ] (cid:107) (cid:96)
= (cid:107) E [ h t ] (cid:107) (cid:96) 2 / √ γ + ≤ θ ≤ 3 √ n via Assumption 1.
E P ROOFS OF M AIN R ESULTS E. P ROOF OF L EMMA 3.
The statement follows from upper bound Lemma B.3 and lower bound Lemma B.5.
E. P ROOF OF T HEOREM 4.
Proof.
To prove this theorem, we combine Theorem D.3 with deterministic SGD convergence result of Theorem 4.1.
Applying Theorem D.3, with the desired probability, inequality (D.4) holds and for all i , input data satisﬁes the bound (cid:107)
x
i (cid:107) (cid:96) 2 ≤ (cid:112) ( n + p ) / (2 c 0 ) for a sufﬁciently small constant c 0 > 0 .
As the next step, we will argue that these two events imply the convergence of SGD.
Let θ ( i ) , c ( i )
∈ R n + p denote the i th rows of Θ , C respectively.
Observe that the square-loss is separable along the rows of C via (cid:107) Θ − C (cid:107) 2 F = (cid:80) ni =1 (cid:107) θ ( i ) − c
( i ) (cid:107) 2 (cid:96) 2 .
Hence, SGD updates each row c ( i ) via its own state equation
[Equation]
where y t,i is the i th entry of y t .
Consequently, we can establish the convergence result for an individual row of C .
Convergence of all individual rows will imply the convergence of the overall matrix Θ τ to the ground truth C .
Pick a row index i ( 1 ≤ i ≤ n ), set c
= c ( i ) and denote i th row of Θ τ by θ τ .
Also denote the label corresponding to i th row by y t
= y t,i .
With this notation, SGD over (2.3) runs SGD over the i th row with equations y t =
φ ( (cid:104) c , x t (cid:105) ) and with loss functions
[Equation]
Substituting our high-probability bounds on x t (e.g. (D.4)) into Theorem 4.1, we can set B = ( n + p ) / (2 c 0 ) , γ + = ( θ + √ 2) 2 , and γ − = ρ − 1 / 2 .
Consequently, using the learning rate η = c
[Equation]
2 ( n + p ) , for all τ ≥ 0 , the τ th SGD iteration θ τ obeys
[Equation]
where the expectation is over the random selection of SGD updates.
This establishes the convergence for a particular row of C .
Summing up these inequalities (E.1) over all rows θ (1) τ , . .
, θ ( n ) τ (which converge to c (1) , . . .
, c ( n ) respectively) yields the targeted bound (4.4).
E. P ROOFS OF MAIN RESULTS ON STABLE SYSTEMS E.3. P ROOF OF T HEOREM 3.
Proof.
Applying Lemmas B.3 and 3.2, independent of L , Assumption 1 holds with parameters
[Equation]
B
[Equation]
This yields ( θ + √ 2) 2 = 6 n .
Hence, we can apply Theorem 4.2 with the learning rate η = c
[Equation]
6 ρn ( n + p ) where
[Equation]
E.2) and convergence rate 1 − β 2 η 2 ρ .
To conclude with the stated result, we use the change of variable c 0 / 6 → c 0 .
E.3. P ROOF OF T HEOREM 3.
Proof.
The proof is similar to that of Theorem 3.3.
Applying Lemmas B.3, B.4, and 3.2, independent of L , Assumption 1 holds with parameters
[Equation]
Hence, we again apply Theorem 4.2 with the learning rate η = cβ 2 2 ρ ( n + p ) where ρ is given by (E.2).
Use the change of variable c 0 / 2 → c 0 to conclude with the stated result.
E. L EARNING UNSTABLE SYSTEMS
In a similar fashion to Section 4, we provide a more general result on unstable systems that makes a parametric assumption on the statistical properties of the state vector.
Assumption 2 (Well-behaved state vector – single timestamp) .
Given timestamp T 0 > 0 , there exists positive scalars γ + , γ − , θ and an absolute constant C > 0 such that θ ≤ 3 √ n and the following holds
[Equation]
[Equation] [Equation]
The next theorem provides the parametrized result on unstable systems based on this assumption.
Theorem E.1 (Unstable system - general) .
Suppose we are given N independent trajectories ( h ( i ) t , u ( i ) t ) t ≥for 1 ≤ i ≤ N .
Sample each trajectory at time T 0 to obtain N samples ( y i , h i , u i )
Ni
=1 where i th sample is ( y i , h i , u i
[Equation]
=
[Equation]
h
[Equation]
[Equation]
[Equation]
Let C, c 0 > 0 be absolute constants.
Suppose Assumption 1 holds with T 0 and sample size satisﬁes N ≥
Cρ 2 ( n + p ) where ρ = γ + /γ − .
Assume φ is β -increasing, zero initial state conditions, and u t i.i.d.
∼ N (0 , I p ) .
Set scaling to be µ = 1 / √ γ + and learning rate to be η = cβ 2 ρ ( θ + √ 2) 2 ( n + p ) .
Starting from Θ 0 , we run SGD over the equations described in (2.2) and (2.3) .
With probability 1 − 2 N exp( − 100( n + p )) − 4 exp( −O ( Nρ 2 )) , all iterates satisfy
[Equation]
where the expectation is over the randomness of the SGD updates.
Proof.
Set x i =
[ γ − 1 / 2 + h
Ti u Ti ] T and X
=
[ x 1 . . .
x N ] T .
Since X has i.i.d.
rows, we can apply Theorem
F.1 and Lemma F.3 to ﬁnd with the desired probability that
• Rows of x i satisfy (cid:107) x
i − E
[ x i ] (cid:107) ψ 2 ≤ O (1) and E [ (cid:107) x
i (cid:107) (cid:96) 2 ] ≤ 3 √ n , hence all rows of X obeys (cid:107) x i (cid:107) (cid:96) 2 ≤ (cid:112) ( n + p ) / (2 c 0 ) , • X satisﬁes
[Equation]
To proceed, using γ − = ρ − 1 / 2 , B = ( n + p ) / (2 c 0 ) , and γ + = ( θ + √ 2) 2 , we apply Theorem 4.1 on the loss function (2.3); which yields the desired result.
E. P ROOF OF T HEOREM 5.
Proof.
The proof is a corollary of Theorem E.1.
We need to substitute the proper values in Assumption 2.
Applying Lemma B.3, we can substitute γ + = B 2 T 0 and θ = √ 6 n − √ 2 ≥ √ n .
Next, we need to ﬁnd a lower bound.
Applying Lemma 3.2 for n > 1 and Lemma B.6 for n = 1 , we can substitute γ − = γ
+ /ρ with the ρ deﬁnition of (5.2).
With these, the result follows as an immediate corollary of Theorem E.1.
F S UPPLEMENTARY S TATISTICAL R ESULTS
The following theorem bounds the empirical covariance of matrices with independent subgaussian rows.
Given a random vector x , deﬁne the de-biasing operation as zm ( x ) = x − E [ x ] .
Theorem F.1.
Let A ∈ R n ×
d be a matrix with independent subgaussian rows { a i } ni =1 satisfying (cid:107) zm
( a i ) (cid:107) ψ 2 ≤ O ( K ) and Σ
[ a i ] (cid:22) K
I d for some K > 0 and (cid:107) E
[ a i ] (cid:107) (cid:96) 2 ≤ θ .
Suppose Σ [ a i ] (cid:23) λ I d .
Sup- pose n ≥ O
( K 4 d/λ 2 ) .
Then, each of the following happens with probability at least 1 − 2 exp( − cK − 4 λ 2 n ) , • θ + (cid:112) 3 / 2 K ≥ 1 √ n (cid:107) A (cid:107) . • Suppose all rows of A have equal expectations. Then 1 √ n s min ( A ) ≥ (cid:112) 2 λ/ 3 .
Proof.
Let E = E
[ A ] , ¯ A =
A − E
[ A ] , ¯ a i = zm ( a i ) .
We will decompose A = ¯
A + E hence we will ﬁrst focus on bounding the upper and lower singular values of ¯ A by studying the random processes X v = (cid:107) ¯ Av (cid:107) 2 (cid:96)and Y v =
X v − E
[ X v ] over the unit sphere S d − 1 .
First, we provide a deviation bound for the quantity sup v ∈S d − 1 | Y v | .
To achieve this, we will utilize Talagrand’s mixed tail bound and show that increments of Y v are subexpoential.
Pick two unit vectors v , u ∈ R d .
Write x = u + v , y =
u − v .
We have that
X
[Equation]
= n (cid:88) i = ( ¯ a Ti x )( ¯ a Ti y ) .
Letting ˆ x = x / (cid:107) x (cid:107) (cid:96) 2 , ˆ y =
y / (cid:107) y (cid:107) (cid:96) 2 , observe that, multiplication of subgaussians x T ¯ a i , y T ¯ a i obey
[Equation]
Centering this subexponential variable around zero introduces a factor of 2 when bounding subexponential norm and yields (cid:107) ( x T ¯ a i )( y T ¯ a i ) − E
[( x T ¯ a i )( y T ¯ a i )]
(cid:107) ψ
1 ≤ O
( K 2 (cid:107) y (cid:107) (cid:96) 2 ) .
Now, using the fact that Y u − Y v is sum of n independent zero-mean subexponential random variables, we have the tail bound
[Equation]
Applying Talagrand’s chaining bound for mixed tail processes with distance metrics ρ 2 = K 2 (cid:107)·(cid:107) (cid:96) 2 √ n , ρ 1 =
K 2 (cid:107)·(cid:107) (cid:96) 2 n , (Theorem 3 . 5 of Dirksen (2013) or Theorem 2 . 2 .
23 of Talagrand (2014))
and using the fact that for unit sphere S d − 1 , Talagrand’s γ functionals (see Talagrand (2014)
[Equation]
with probability 1 − 2 exp( − min { t 2 , √ nt } ) .
Since n ≥ Cλ − 2 K 4 d for sufﬁciently large C > 0 , picking t = 116 c K − 2 λ √ n , with probability 1 − 2 exp( −O ( K − 4 λ 2 n )) , we ensure that right hand side of (F.1) is less than λ/ 8 .
This leads to the following inequalities
[Equation]
Upper bound on spectral norm: For spectral norm of A , we use the triangle inequality
[Equation]
Lower bound on minimum singular value: This part assumes that all row expectations are same.
Denote the size n all ones vector by 1 n and deﬁne the process Z v = 1 √ n 1 Tn ¯ Av .
Observe that ¯ A T 1 n = (cid:80) ni
=1 ¯ a i ∈ R d is a vector satisfying (cid:107)
¯
A T 1 n / √ n (cid:107)
ψ 2 ≤ O ( K ) .
Hence, again using n ≥ CK 4 λ − 2 d for sufﬁciently large C > 0 , applying Lemma F.3 with m =
c[-
-]
K − 4 λ 2 n > d by picking a sufﬁciently small constant c 0 > 1 /C , with probability at least 1 − 2 exp( − 100 c 0 K − 4 λ 2 n
[Equation]
Let P =
I n − 1 n 1 n 1 Tn be the projection onto the orthogonal complement of the all ones vector.
Note that PEv = 0 as the rows of E are equal.
With this observation, with desired probability, for any unit length v ,
[Equation]
which implies s min ( A ) / √ n ≥ (cid:112) 2 λ/ 3 .
The corollary below is obtained by slightly modifying the proof above by using 1 n (cid:107)
¯
A T [Equation] − E
[Equation] T [Equation] ] (cid:107) ≤ K 2in line (F.2) and only focusing on the spectral norm bound.
Corollary F.2.
Let A ∈ R n ×
d be a matrix with independent { a i } ni =1 subgaussian rows satisfying (cid:107) zm ( a i ) (cid:107) ψ 2 ≤ O ( K ) and Σ
[ a i ] (cid:22) K
I d for some K > 0 and (cid:107) E
[ a i ] (cid:107) (cid:96) 2 ≤ θ .
Suppose Σ [ a i ] (cid:23) λ I d .
Suppose n ≥ O ( K 2 d ) .
Then, with probability at least 1 − 4 exp( − cK − 2 n ) ,
[Equation]
The following lemma is fairly standard and is proved for the sake of completeness.
Lemma F.3 (Subgaussian vector length) .
Let a ∈ R n be a zero-mean subgaussian vector with (cid:107) a (cid:107) ψ 2 ≤ L .
Then, for any m ≥ n , there exists C > 0 such that
[Equation]
Proof.
We can pick a 1 / 2 cover C of the unit (cid:96) 2 -sphere with size log |C| ≤ 2 n .
For any v ∈ C , subgaussianity implies, P ( | v T a | ≥ t ) ≤ 2 exp( − ct 2 2 L 2 ) .
Setting t = CL √ m for sufﬁciently large constant C > 0 , and union bounding over all v ∈ C , we ﬁnd
[Equation]
To conclude, let v ( a ) ∈ C be a ’s neighbor satisfying (cid:107) v −
[Equation]
[Equation]
Hence, we have
[Equation]
To conclude, use the change of variable C →
C/ 2 .