Finite Sample Analysis Of Dynamic Regression Parameter Learning
Abstract
We consider the dynamic linear regression problem, where the predictor vector may vary with time.
This problem can be modeled as a linear dynamical system, with non-constant observation operator, where the parameters that need to be learned are the variance of both the process noise and the observation noise.
While variance estimation for dynamic regression is a natural problem, with a variety of applications, existing approaches to this problem either lack guarantees altogether, or only have asymptotic guarantees without explicit rates.
In particular, existing literature does not provide any clues to the following fundamental question:
In terms of data characteristics, what does the convergence rate depend on?
In this paper we study the global system operator – the operator that maps the noise vectors to the output.
We obtain estimates on its spectrum, and as a result derive the ﬁrst known variance estimators with ﬁnite sample complexity guarantees.
The proposed bounds depend on the shape of a certain spectrum related to the system operator, and thus provide the ﬁrst known explicit geometric parameter of the data that can be used to bound estimation errors.
In addition, the results hold for arbitrary sub Gaussian distributions of noise terms.
We evaluate the approach on synthetic and real-world benchmarks.
Introduction
A dynamic linear regression (West and Harrison, 1997, Chapter 3), or non-stationary regression, is a situation where we are given a sequence of scalar observations { Y t } t ≤ T ⊂ R , and observation vectors { u t } t ≤ T ⊂ R n such that Y t = ⟨ X t , u t ⟩ + z t where X t ∈ R n is a regressor vector and z t a random noise term.
In contrast to a standard linear regression, the vector X t may change with time.
One common objective for this problem is at time T , to estimate the trajectory of X t for t ≤ T , given the observation vectors and observations, { u t } t ≤ T , { Y t } t ≤ T , and possibly to forecast Y T +1 if u T +1 is also known.
In this paper we model the problem as follows:
[Equation]
where ⟨· , ·⟩ is the standard inner product on R n , z t , the observation noise , are zero-mean sub Gaussian random variables, with variance η 2 , and the process noise variables h t take values in R n , such that coordinates of h t are zero-mean sub Gaussian, independent, and have variance σ 2 .
All h t and z t variables are assumed to be mutually independent.
The vectors u t are an arbitrary sequence in R n , and the observed, known, quantities at time T are { Y t } t ≤ T and { u t } t ≤ T .
The system (1)-(2) is a special case of a Linear Dynamical System (LDS).
As is well known, when the parameters σ, η are given, the mean-squared loss optimal forecast for Y T +1 and estimate for
X T are obtained by the Kalman Filter (Anderson and Moore, 1979; Hamilton, 1994; Chui and
Chen, 2017).
In this paper we are concerned with estimators for σ, η , and ﬁnite sample complexity guarantees for these estimators.
Let us ﬁrst make a few remarks about the particular system (1)-(2).
First, as a natural model of time varying regression, this system is useful in a considerable variety of applications.
We refer to West and Harrison (1997), Chapter 3, for numerous examples.
In addition, an application to electricity consumption time series as a function of the temperature is provided in the experiments section of this paper.
Second, one may regard the problem of estimating σ, η in (1)-(2) as a pure case of ﬁnding the optimal learning rate for X t .
Indeed, the Kalman ﬁlter equations for (1)-(2), are given by (3)-(4) below, where (3) describes the ﬁltered covariance update and (4) the ﬁltered state update.
Here ¯ x t is the estimated state, given the observations Y 1 , . . .
, Y t , see West and Harrison (1997).
[Equation]
In particular, following (4), the role of σ and η may be interpreted as regulating how much the estimate of ¯ x t +1 is inﬂuenced, via the operator C t +1 η 2 , by the most recent observation and input
Y t +1 , u t +1 .
Roughly speaking, higher values of σ or lower values of η would imply that the past observations are given less weight, and result in an overﬁt of the forecast to the most recent observation.
On the other hand, very low σ or high η would make the problem similar to the standard linear regression, where all observations are given equal weight, and result in a lag of the forecast.
See
Figure 3 in Supplementary Material Section A for an illustration.
Finally, it is worth mentioning that the system (1)-(2) is closely related to the study of online gradient (OG) methods (Zinkevich, 2003; Hazan, 2016).
In this ﬁeld, assuming quadratic cost, one considers the update
[Equation]
where α is the learning rate, and studies the performance guarantees of the forecaster ⟨ ¯ x t , u t +1 ⟩ .
Compared to (4), the update (5) is simpler, and uses a scalar rate α instead of the input-dependent operator rate C t + (cid:14) η 2 of the Kalman ﬁlter.
However, due to the similarity, every domain of applicability of the OG methods is also a natural candidate for the model (1)-(2) and vice-versa.
As an illustration, we compare the OG to Kalman ﬁlter based methods with learned σ , η in the experiments section.
In this paper we introduce a new estimation algorithm for σ, η , termed STVE (Spectrum Thresholding Variance Estimator), and prove ﬁnite sample complexity bounds for it.
In particular, our bounds are an explicit function of the parameters T and { u t } Tt =1 for any ﬁnite T , and indicate that the estimation error decays roughly as T − 12 , with high probability.
To the best of our knowledge, these are the ﬁrst bounds of this kind.
As we discuss in detail in Section 2, most existing estimation methods for LDSs, such as subspace identiﬁcation (van Overschee and de Moor, 1996; Qin, 2006), or improper learning (Anava et al., 2013; Hazan et al., 2017; Kozdoba et al., 2019), do not apply to the system (1)-(2), due to non-stationarity.
On the other hand, the methods that do apply to (1)-(2) either lack guarantees, or have only asymptotic analysis which in addition relies strongly on Guassianity of the noises.
Moreover, our approach differs signiﬁcantly from the existing methods.
We show that the structure of equations (1)-(2) is closely related to, and inherits several important properties from, the classical discrete Laplacian operator on the line — leading to new arguments that have not been explored in the literature.
In particular, we use this connection to show that an appropriate inversion of the system produce estimators that are concentrated enough so that σ and η may be recovered.
The heart of the paper is the new deﬁnition of the estimators that exploits explicitly the shape of a certain data dependent operator, and the subsequent concentration analysis.
In particular, this approach yields the ﬁrst known geometric parameters of the data that can be used to bound convergence rates.
The rest of the paper is organized as follows: The related work is discussed in Section 2 and Section 3 contains the necessary deﬁnitions.
In Section 4 we describe in general lines the methods and the main results of this paper.
The technical estimates on certain operator spectra, that are critical to the analysis and may be of independent interest, are stated in Section 5.
In Section 6 we present experimental results on synthetic and real data.
Due to space constraints, while we outline the main arguments in the text, the full proofs are deferred to the Supplementary Material.
Literature
We refer to Chui and Chen (2017);
Hamilton (1994); Anderson and Moore (1979); Shumway and
Stoffer (2011) for a general background on LDSs, the Kalman Filter and maximum likelihood estimation.
Existing approaches to the variance estimation problem may be divided into three categories: (i)
General methods for parameter identiﬁcation in LDS, either via maximum likelihood estimation (MLE) (Hamilton, 1994), or via subspace identiﬁcation (van Overschee and de Moor, 1996; Qin, 2006).
In particular, ﬁnite sample bounds for system identiﬁcation were given in (Campi and Weyer, 2005; Vidyasagar and Karandikar, 2006) and in the recent work Tsiamis and Pappas (2019).
(ii)
Methods designed speciﬁcally to learn the noise parameters of the system, developed primarily in the control theory community, in particular via the innovation auto-correlation function, such as the classical Mehra (1970); Belanger (1974), or for instance more recent Wang et al.
; Dunik et al.
(iii) Improper Learning methods, such as Anava et al.
; Hazan et al.
; Kozdoba et al.
In these approaches, one does not learn the LDS directly, but instead learns a model from a certain auxiliary class and shows that this auxillary model produces forecasts that are as good as the forecasts of an LDS with “optimal” parameters.
Despite the apparent simplicity of the system (1)-(2), most of the above methods do not apply to this system.
This is due to the fact that most of the methods are designed for time invariant, asymptotically stationary systems, where the observation operator ( u t in our notation) is constant and the
Kalman gain (or, equivalently C t u t in eq.
(3)) converges with t .
In particular this limitation exists in all the system identiﬁcation results cited above, and is essential to the approaches taken there.
However, if the observation vector sequence u t changes with time – a necessary property for the dynamic regression problem – the system will no longer be asymptotically stationary.
In particular, due to this reason, neither the subspace identiﬁcation methods, nor any of the improper learning approaches above apply to system (1)-(2) .
Among the methods that do apply to (1)-(2) are the general MLE estimation, and some of the autocorrelation methods (Belanger, 1974; Dunik et al., 2018).
On one hand, both types of approaches may be applicable to systems apriori more general than (1)-(2).
On the other hand, the situation with consistency guarantees – the guarantee that one recovers true parameters given enough observations – for these methods is somewhat complicated.
Due to the non-convexity of the likelihood function, the MLE method is not guaranteed to ﬁnd the true maximum, and as a result the whole method has no guarantees.
The results in Belanger (1974); Dunik et al.
(2018) do have asymptotic consistency guarantees.
However, these rely on some explicit and implicit assumptions about the system, the sequence u t in our case, which can not be easily veriﬁed.
In particular, Belanger (1974); Dunik et al.
(2018) assume uniform observability of the system, which we do not assume, and in addition rely on certain implicit assumption about invertibility and condition number of the matrices related to the sequence u t .
Moreover, even if one assumes that the assumptions hold, the results are purely asymptotic, and for any ﬁnite T , do not provide a bound of the expected estimation error as a function of T and { u t } Tt =1 .
In addition, as mentioned earlier, MLE methods by deﬁnition must assume that the noises are Gaussian (or belong to some other predetermined parametric family) and autocorrelation based methods also strongly use the Gaussianity assumption.
Our approach, on the other hand, requires only sub
Gaussian noises with independent coordinates.
We note that there are straightforward extensions of our methods to certain cases with dependencies.
Indeed, the operator analysis part of this paper does not depend on the distribution of the noises.
Therefore, to achieve such an extension, one would only need to correspondingly extend the main probabilistic tool, the Hanson-Wright inequality (Hanson and Wright, 1971; Rudelson et al., 2013, see also Section 4 and Supplementary Material Section E).
One such extension, for vectors with the convex concentration property, was recently obtained in
Adamczak (2015).
Notation
We refer to Bhatia (1997) and Vershynin (2018) as general references on the notation introduced below, for operators and sub Gaussian variables, respectively.
Let A : R n → R m be an operator with a singular value decomposition
A =
U · Diag ( λ 1 , . .
λ s ) · W , where s ≤ min { m, n } and λ 1 ≥ λ 2 ≥ . .
≥ λ s > 0 .
Note that singular values are strictly positive by deﬁnition (that is, vectors corresponding to the kernel of A do not participate in the decomposition
A =
U · Diag ( λ 1 , . .
. , λ s ) · W ).
The Hilbert-Schmidt (Frobenius) norm is deﬁned as ∥ A ∥ HS = pP si =1 λ 2 i .
The nuclear and the operator norms are given by ∥ A ∥ nuc =
P si =
λ i and ∥
A ∥ op = λ 1 respectively.
A centered ( E X = 0 ) scalar random variable
X is sub-Gaussian with constant κ , denoted
X ∼ SG ( κ ) , if for all t > 0 it satisﬁes P ( | X | > t ) ≤ 2 exp (cid:16) − t 2 κ(cid:17) .
A random vector
X =
( X 1 , . . .
, X m ) is κ sub-Gaussian, denoted X ∼ SG m ( κ ) , if for every v ∈ R m with | v | = 1 the random variable ⟨ v, X ⟩ is κ sub-Gaussian.
A random vector X is σ -isotropic if for every v ∈ R m with | v | = 1 , E ⟨ v, X ⟩ = σ 2 .
Finally, a random vector X =
( X 1 , . . .
, X m ) is σ -isotropically κ sub-Gaussian with independent components, denoted X ∼ ISG m ( σ, κ ) if X i are independent, and for all i ≤ m , E X i = 0 , E
X 2 i =
σ 2 and X i ∼ SG ( κ ) .
Clearly, if X ∼ ISG m ( σ, κ ) then X is σ -isotropic.
Recall also that
X ∼ ISG m ( σ, κ ) implies X ∼ SG m ( κ ) (Vershynin, 2018).
The noise variables we discuss in this paper are ISG ( κ, σ ) .
Throughout the paper, absolute constants are denoted by c, c ′ , c ′′ , . . . .
etc.
Their values may change from line to line.
Overview of the approach
We begin by rewriting (1)-(2) in a vector form.
To this end, we ﬁrst encode sequences of T vectors in R n , { a t } t ≤ T ⊂ R n , as a vector a ∈ R Tn , constructed by concatenation of a t ’s. Next, we deﬁne the summation operator S ′ :
R T → R T which acts on any vector ( h 1 , h 2 , . .
. , h T ) ∈ R T by
S ′ ( h 1 , h 2 , . . . , h T ) = ( h 1 , h 1 + h 2 , . . . , X i ≤ T −h i ,
X i ≤ T h i ) .
Note that S ′ is an invertible operator.
Next, we similarly deﬁne the summation operator S :
R Tn
→ R Tn , an n -dimensional extension of S ′ , which sums n -dimensional vectors.
Formally, for ( h l ) Tnl =1 ∈ R Tn , and for 1 ≤ j ≤ n, 1 ≤ t ≤ T , ( Sh ) ( t − 1) · n + j = P i ≤ t h
( i − 1) · n + j .
Observe that if the sequence of process noise terms h 1 , . .
, h T ∈ R n is viewed as a vector h ∈ R Tn , then by deﬁnition
Sh is the R Tn encoding of the sequence X t .
Next, given a sequence of observation vectors u 1 , . . .
, u T ∈ R n , we deﬁne the observation operator
O u : R Tn
→ R T by ( O u x ) t = (cid:10) u t , (cid:0) x ( t − 1) · n +1 , . . .
, x ( t − 1) · n + n (cid:1)(cid:11) .
In words, coordinate t of O u x is the inner product between u t and t -th part of the vector x ∈ R Tn .
Deﬁne also Y
=
( Y 1 , ..., Y T ) ∈ R T to be the concatenation of Y 1 , ..., Y T .
With this notation, one may equivalently rewrite the system (1)-(2) as follows:
[Equation]
where h and z are independent zero-mean random vectors in R Tn and R T respectively, with independent sub Gaussian coordinates.
The variance of each coordinate of h is σ 2 and each coordinate of z has variance η 2 .
Up to now, we have reformulated our data model as a single vector equation.
Note that in that equation, the observations Y and both operators O u and S are known to us.
Our problem may now be reformulated as follows:
Given Y ∈ R T , assuming Y was generated by (7), provide estimates of σ, η .
As a motivation, we ﬁrst consider taking the expectation of the norm squared of eq.
For any operator A : R m → R m and zero-mean vector h with independent coordinates and coordinate variance σ 2 , we have E |
Ah | 2 = ∥
A ∥ 2 HS σ 2 , where
∥
A ∥ HS is the Hilbert-Schmidt (or Frobenius) norm of A .
Taking the norm and expectation of (7), and dividing by T 2 , we thus obtain
[Equation]
Next, note that ∥ O u S ∥ 2 HS is known, and an elementary computation shows that ∥ O u S ∥ 2 HS Tis of constant order (as a function of T ; see (25)), while the coefﬁcient of η 2 is 1 T .
Thus, if the quantity | Y | 2 Twere close enough to its expectation with high probability, we could take this quantity as a (slightly biased) estimator of σ 2 .
However, as it will become apparent later, the deviations of | Y | 2 T 2 around the expectation are also of constant order, and thus | Y | 2 T 2 can not be used as an estimator.
The reason for these high deviations of | Y | 2 T 2 is that the spectrum of O u S is extremely peaked.
The highest squared singular value of O u S is of order T 2 , the same order as sum of all of them, ∥ O u S ∥ 2 HS .
Contrast this with the case of identity operator, Id : R Tn → R Tn : We have E | Id ( h ) | 2 =
E | h | 2 =
Tnσ 2 , and one can also easily show that, for instance, V ar | Id ( h ) |
=
Tnσ 2 , and thus the deviations are of order √
Tnσ – a smaller order than E | Id ( h ) | 2 .
While for the identity operator the computation is elementary, for a general operator
A the situation is signiﬁcantly more involved, and the bounds on the deviations of | Y | 2 will be obtained from the Hanson-Wright inequality (Hanson and Wright, 1971, see also Rudelson et al.
combined with standard norm deviation bounds for isotropic sub Gaussian vectors.
With these observations in mind, we proceed to ﬂatten the spectrum of O u S by taking the pseudoinverse.
Let R : R T → R Tn be the pseudo-inverse, or Moore-Penrose inverse of O u S .
Speciﬁcally, let
[Equation]
be the singular value decomposition of O u S , where γ 1 ≥ γ 2 ≥ . . .
≥ γ T are the singular values.
For the rest of the paper, we will assume that all of the observation vectors u t are non-zero.
This assumption is made solely for notational simplicity and may easily be avoided, as discussed later in this section.
Under this assumption, since S is invertible and O u has rank T , we have λ t > 0 for all t ≤ T .
For i ≤ T , denote χ i = γ − 1 T +1 − i .
Then χ i are the singular values of R , arranged in a non-increasing order, and we have by deﬁnition
[Equation]
where W ∗ , U ∗ denote the transposed matrices of U, V , deﬁned in (9).
Similarly to Eq.
(8), we apply R to (7), and since ∥ RO u S ∥ HS = T , by taking the expectation of the squared norm we obtain
[Equation]
In this equation, the deviation term
[Equation]
is of order O ( 1 √ T ) with high probability (Theorem 1).
Moreover, the coefﬁcient of σ 2 is 1 , and the coefﬁcient of η 2 , which is ∥ R ∥ 2 HS T , is of order at least Ω
( 1 log 2 T ) (Theorem 3, see Section 5 for additional details) – much larger order than 1 √ T .
Since | RY | 2 and ∥ R ∥ 2 HS are known, it follows that we have obtained one equation satisﬁed by σ 2 and η 2 up to an error of 1 √ T , where both coefﬁcients are of order larger than the error.
the
[Equation]
where γ 1 ≥ γ 2 ≥ . . .
≥ γ T > 0 .
Denote χ i = γ − 1 T
+1 − i for 1 ≤ i ≤ T .
3: Construct the operators
[Equation]
and
[Equation]
4: Produce the estimates:
[Equation]
Next, we would like to obtain another linear relation between σ 2 , η 2 .
To this end, choose some p = αT , where 0 < α < 1 is of constant order.
The possible choices of p are discussed later in this section.
We deﬁne an operator R ′ :
R T → R Tn to be a version of R truncated to the ﬁrst p singular values.
If (10) is the SVD decomposition of R , then
[Equation]
Similarly to the case for R , we have
[Equation]
The deviations in (12) are also described by Theorem 1.
Note also that since ∥ R ′ ∥ 2 HS is the sum of p largest squared singular values of R , by deﬁnition it follows that ∥ R ′ ∥ 2 HS p
[Equation]
Now, given two equations in two unknowns, we can solve the system to obtain the estimates c σ 2 and b η 2 .
The full procedure is summarized in Algorithm 1, and the bounds implied by Theorem 1 on the estimators c σ 2 and b η 2 are given in Corollary 2.
We ﬁrst state these results, and then discuss in detail the various parameters appearing in the bounds.
Theorem 1.
Consider a random vector Y ∈ R T of the form Y
=
O u
Sh
+ z where h ∼ ISG Tn ( σ, κ ) and z ∼ ISG T ( η, κ ) .
Set | u min | = min t | u t | .
Then for any 0 < δ < 1 ,
[Equation]
where B is given by
[Equation]
The bounds on the estimators of Algorithm 1 are given in the following Corollary.
As discussed below, in addition to Theorem 1, the key to the derivation of this Corollary are the estimates of the spectrum of R , given in Theorem 3, Section 5.
800   p= T/
R 2 spectrum , χ(cid:107) R (cid:48)
( p ) (cid:107) 2 HS /p (cid:107) R (cid:107) 2 HS /T
.2 .p=
T/
R 2 spectrum , χ(cid:107) R (cid:48)
( p ) (cid:107) 2 HS /p (cid:107) R (cid:107) 2 HS /T
[Figure]
Corollary 2.
Let c σ 2 , b
η 2 be the estimators of σ 2 , η 2 obtained form Algorithm 1 with p ≥ 14 T .
Set | u max | = max t | u t | .
Then for any 0 < δ < 1 , with probability at least 1 − 8 δ ,
[Equation] with B given by (15) . We ﬁrst discuss the assumption | u min | > 0 .
This assumption is made solely for notational convenience, as detailed below.
To begin, note that some form of lower bound on the norms of the observation vectors u t must appear in the bounds.
This is simply because if one had u t = 0 for all T , then clearly no estimate of σ would have been possible.
On the other hand, our use of the smallest value | u min | may seem restrictive at ﬁrst.
We note however, that instead of considering the observation operator O u : R
Tn
→ R T , one may consider the operator
O ¯ u : R Tn
→ R ¯ T for any subsequence { ¯ u ¯ t } ¯ T ¯ t =1 .
The observation vector Y would be correspondingly restricted to the subsequence of indices.
This allows us to treat missing values and to exclude any outlier u t with small norms.
All the arguments in Theorems 1 and 3 hold for this modiﬁed O ¯ u without change.
The only price that will be paid is that T will be replaced by ¯ T in the bounds.
Moreover, we note that typically we have | u t | ≥ 1 by construction, see for instance Section 6.2.
Additional discussion of missing values may be found in Supplementary Material Section H.
Next, up to this point, we have obtained two equations, (11)-(12), in two unknowns, σ 2 , η 2 .
Note that in order to be able to obtain η 2 from these equations, at least one of the coefﬁcients of η 2 , either ∥ R ∥ 2 HS T or ∥ R ′ ∥
2 HS p must be of larger order than 1 √ T , the order of deviations.
Providing lower bounds on these quantities is one of the main technical contributions of this work.
This analysis uses the connection between the operator S and the Laplacian on the line, and resolves the issue of translating spectrum estimates for the Laplacian into the spectral estimates for R .
We note that there are no standard tools to study the spectrum of R , and our approach proceeds indirectly via the analysis of the nuclear norm of O u S .
These results are stated in Theorem 3.
In particular, we show that ∥ R ∥ 2 HS T is Ω( 1 log 2 T ) , which is the source of the log factor in (17).
Finally, in order to solve the equations (11)-(12), not only the equations must have large enough coefﬁcients, but the equations must be different .
This is reﬂected by the term
[Equation]
Equivalently, while ∥ R ′ ∥ 2 HS /p ∥
R ∥ 2 HS /T ≥ 1 by deﬁnition, we would like to have
[Equation]
for the bounds (16), (17) to be stable.
Note that since both ∥ R ′ ∥ 2 HS and ∥ R ∥ 2 HS are computed in
Algorithm 1, the condition (18) can simply be veriﬁed before the estimators c σ 2 , b
η 2 are returned.
It is worth emphasizing that for simple choices of p , say p = 14 T , the condition (18) does hold in practice.
Note that, for any p , we can have ∥ R ′ ∥ 2 HS /p
∥
R ∥ 2 HS /T
=
1 only if the spectrum of R is constant .
Thus (18) amounts to stating that the spectrum of R exhibits some decay.
As we show in experiments below, the spectrum (squared) of R , for u t derived from daily temperature features, or for random
Gaussian u t , indeed decays.
See Section 6, Figures 1a and 1b.
In particular, in both cases (18) holds with const > 1 .
Additional bounds on the quantity
[Equation]
under various assumptions on the sequence u t are given in Section G of the Supplementary Material.
Properties of O u S and R
As discussed in Section 4 (see the discussion following eq.
(11)), one of the crucial points enabling
Algorithm 1 and its analysis is the fact that the quantity ∥ R ∥ 2 HS T is bounded below by an expression that is of much higher order than the noise magnitude 1 √ T .
In this section we provide the formal statement of this and other associated results, and discuss the related arguments.
First, we obtain the following bound on the spectrum of O u S (Lemma 4,
Supplementary Material Section B).
Recall that the nuclear norm was deﬁned in Section 3, and that for a sequence u t we set | u max | = max t | u t | and | u min | = min t | u t | .
Then:
[Equation]
The proof of this bound exploits the connection between S and the Laplacian on the line.
In particular, we use the fact that the eigenvalues of the Laplacian are known precisely, satisfying λ l = 2 sin (cid:16) π ( T − l )
[Equation]
T (cid:17) .
Next, we state the lower (and upper) bounds for R .
Theorem 3.
Let R : R T → R Tn be the pseudoinverse of O u S .
Then
[Equation]
Due to the complicated structure of R as a pseudo-inverse of a composition of operators, there are no direct ways to control individual eigenvalues of R .
Thus the main technical issue resolved in
Theorem 3 is nevertheless obtaining lower bounds on ∥ R ∥ 2 HS .
Our approach is rather indirect, and we obtain these bounds from the nuclear norm bound (19) via a Markov type inequality on the eigenvalues.
Experiments 6.1 Synthetic Data
In this section the performance of STVE is evaluated on synthetic data.
The data was generated by the LDS (1)-(2), using Gaussian noises with σ 2 = 0 .
5 , η 2 = 2 .
The input dimension was n = 5 , and the input sequence u t sampled from the Gaussian N (0 , I n ) .
We run the STVE algorithm for different values of T , where for each T we sampled the datatimes.
Figure 2a shows the average (over 150 runs) estimation error for both process and observation noise variances for various values of T .
As expected from the bounds in Corollary 2, it may be observed in Figure 2a that the estimation errors decay roughly at the rate of const/ √
T .
A typical spectrum of R is shown in Figure 1b.
For larger T , the spectra also exhibits similar decay.
[Figure]
| σ 2 − (cid:98) σ 2 | | σ 2 − (cid:98) σ 2 | | η 2 − (cid:98) η 2 | T −0 .0 .0 .
[Figure]
[Figure]
[Figure]
[Figure] StationaryRegressionMLEKalmanSTVEKalmanOG [Figure] 6.2 Temperatures and Electricity Consumption
In this section we examine the relation between daily temperatures and electricity consumption in the data from Hong et al.
(2014) (see also Hong (2016)).
The following forecasting methods are compared: a stationary regression, an online gradient, and a Kalman ﬁlter for a dynamic regression, with parameters learned via MLE or STVE.
We ﬁnd that the Kalman ﬁlter methods provide the best performance, with no signiﬁcant difference between STVE and MLE derived systems.
The data consists of total daily electricity consumption (load) y t , and the average daily temperature, v t , for a certain region, for the period Jan- 2004 to Jun- 2008 .
Full details on the preprocessing of the data, as well as additional details on the experiments, are given in Supplementary Material Section I.
Here we note that the data contains missing load values, for 9 non-consecutive weeks (out of about 234 weeks total).
All methods discussed here, including STVE, can naturally incorporate missing values, as discussed in Supplementary Material Section H.
An elementary inspection of the data reveals that the load may be reasonably approximated as a quadratic function of the temperature,
y t
= x t, 1 · 1 + x t, 2 · v t + x t, 3 · v 2 t , where u t =
(1 , v t , v 2 t ) is the observation vector (features), and x t = ( x t, 1 , x t, 2 , x t, 3 ) is the possibly time varying regression vector.
This is shown in Figure 2b, where we ﬁt a stationary (time invariant) regression of the above form, using either only the ﬁrst or only the second half of the data.
We note that these regressions differ – the regression vector changes with time.
It is therefore of interest to track it via online regression.
We use the ﬁrst half of the data (train set) to learn the parameters σ, η of the online regression (1)-(2) via MLE optimization and using STVE.
We also use the train set to ﬁnd the optimal learning rate α for the OG forecaster described by the update equation (5).
This learning rate is chosen as the rate that yields smallest least squares forecast error on the train set.
In addition, we learn a time independent, stationary regression on the ﬁrst half of the data.
We then employ the learned parameters to make predictions of the load given the temperature, by all four methods.
The predictions for the system (1)-(2) are made with a Kalman ﬁlter (at time t , we use the ﬁltered state estimate ˜ x t , which depends only on y 1 , . . .
, y t and u 1 , . .
. , u t , and make the prediction ˜
y t +1 = ⟨ ˜ x t , u t +1 ⟩ ).
Daily squared prediction errors (that is, ( y t − ˜ y t ) 2 ) are shown in Figure 2c (smoothed with a moving average of 50 days).
We see that the adaptive models (MLE, STVE, OG) outperform the stationary regression already on the train set (ﬁrst half of the data), and that the difference in performance becomes dramatic on the second half (test).
It is also interesting to note that the performance of the
Kalman ﬁlter based methods (MLE, STVE) is practically identical, but both are somewhat better than the simpler OG approach.
We also note that by construction, we have | u min | ≥ 1 in this experiment, due to the constantcoordinate, and also | u max | ≤ 5 , due to the normalization.
Since these operations are typical for any regression problem, we conclude that the direct inﬂuence of | u min | and | u max | on the bounds in Theorem 1 and Corollary 2 will not usually be signiﬁcant.
Conclusion And Future Work
In this work we introduced the STVE algorithm for estimating the variance parameters of LDSs of type (1)-(2), and obtained the ﬁrst sample complexity guarantees for such estimators.
We have also shown how the shape of the spectrum of R can be exploited to obtain the estimators and the related bounds, thus providing the ﬁrst explicit geometric parameter of the data that affects the bounds.
As discussed in Section 1 and demonstrated in Section the system (1)-(2) is of independent interest in applications.
However, we also believe that the analysis presented here is an important ﬁrst step towards a ﬁnite time data-dependent quantitative understanding of general LDSs.
and perhaps even non-linear dynamical systems.
Acknowledgments and Disclosure of Funding
This research was supported by the ISRAEL SCIENCE FOUNDATION (grant No. 2199/20)
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A]
The main focus of this work is a theoretical analysis.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] Thus Supplementary Materail contains all the proofs.
Proofs outlines are given in the main body of the paper.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The data is publically available, see the references in Section 6.
The short code is not provided at the moment, but can be fully derived from Algorithm 1.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Sections 6 and I. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A]
The link to the data author’s documentation is provided.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]