Original: Robust Streaming PCA
Revised: Robust Streaming PCA

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: We consider streaming principal component analysis(PCA) when the stochasticdata-generating model is subject to perturbations.
Revised: We consider streaming principal component analysis when the stochastic datagenerating model is subject to perturbations.

Original: While existing models assume afixed covariance, we adopt a robust perspective where the covariance matrix belongsto a temporal uncertainty set.
Revised: While existing models assume a ﬁxed covariance, we adopt a robust perspective where the covariance matrix belongs to a temporal uncertainty set.

Original: Under this setting, we provide fundamental limitson any algorithm recovering principal components.
Revised: Under this setting, we provide fundamental limits on any algorithm recovering principal components.

Original: We analyze the convergenceof the noisy power method and Oja’s algorithm, both analyzed for the stationarydata generating model, and argue that the noisy power method is rate-optimal inour setting.
Revised: We analyze the convergence of the noisy power method and Oja’s algorithm, both studied for the stationary data generating model, and argue that the noisy power method is rate-optimal in our setting.

Original: Finally, we demonstrate the validity of our analysis through numericalexperiments.
Revised: Finally, we demonstrate the validity of our analysis through numerical experiments.

Original:  Introduction
Revised:  Introduction

Original: Principal component analysis (PCA) is one of the most extensively studied methods for obtaining thelow-dimensional representation of observed data [21].
Revised: Principal component analysis (PCA) is one of the most extensively studied methods for obtaining the low-dimensional representation of observed data [22].

Original: However, classical algorithms for PCA storeall the observations and use cubic-time complexity, thereby imposing prohibitively large computationtime and space requirements.
Revised: However, classical algorithms for PCA store all the observations and use cubic-time complexity, thereby imposing prohibitively large computationtime and space requirements.

Original: Recently, several works on PCA have focused on the design and analysis of single-pass (streaming)algorithms with near-optimal memory and storage complexity [40, 43, 24, 28].
Revised: Recently, several works on PCA have focused on the design and analysis of streaming algorithms with near-optimal memory and storage complexity [25, 29, 40, 43].

Original: These algorithmsassume that all the observations belong to the same low-dimensional space.
Revised: These algorithms assume that all the observations belong to the same low-dimensional space.

Original: However, this situationis unlikely when the unknown/unexplored alterations corrupt a system’s observations.
Revised: However, this situation is unlikely when the unknown/unexplored alterations corrupt a system’s observations.

Original: For instance,it is well known thatthe typical data attacks on power grids can significantly change the estimatedcovariance matrix of the data observed from sensors [8, 22, 9].
Revised: For instance, it is well known that typical data attacks on power grids can signiﬁcantly change the estimated covariance matrix of the data observed from sensors [9, 10, 23].

Original: Similarly, PCA can be used to explainstock returns in terms ofthe macroeconomic factors [26], product pricing taking into account crossproduct elasticity and demands [45], etc.
Revised: Similarly, PCA can be used to explain stock returns in terms of macroeconomic factors [27], and product pricing taking into account cross-product elasticity and demands [45].

Original: In all these scenarios the underlying data generating modelchanges at every time instant and the decisions are based on identifying the changed model.
Revised: In all these scenarios, the underlying data-generating model changes every instant, and the decisions are based on identifying the changed model.

Original: Recent work considers perturbations of the data lying in a fixed low-dimensional space [49].
Revised: Current work considers perturbations of the data lying in a ﬁxed low-dimensional space [49].

Original: Theydetermine the worst-case position of the adversarial data-point to incur the maximum error in thesubspace estimated through PCA and measure the distance between the two subspaces using thenotion of the principal angle between them.
Revised: They determine the worst-case position of the adversarial data point to incur the maximum error in the subspace estimated through PCA and measure the distance between the two subspaces using the notion of the principal angle between them.

Original: Another line of work considers PCA through the lens ofstochastic optimization [51, 52, 42, 4].
Revised: Another line of work considers PCA through the lens of stochastic optimization [5, 42, 51, 52].

Original: Our work differs from these approaches since we assume thatthe data generating model changes at every time instant.
Revised: Our work differs from these approaches since we assume that the data-generating model changes at every time instant.

Original: Further, we propose near-optimal algorithmsfor recovering principal components under this framework.
Revised: Further, we propose near-optimal algorithms for recovering principal components under this framework.

Original: We assume a system relies on the time-series of p -dimensional vectors sampled from the time-variantmodel.
Revised: We assume a system relies on the time-series of p -dimensional vectors sampled from the time-variant model.

Original: The available observations are the vectors { x t } Tt =1 .
Revised: The available observations are the vectors ( x t ) Tt =1 .

Original: We assume that T

Original: ≫ p ; justifying theuse of streaming (single-pass) algorithms.

Original: The noisy observation x t ∈ R p is a vector lying in thecolumn space of an unobserved full-rank matrix A t ∈ R p × k .
Revised: The noisy observation x t 2 R p is a vector lying in the column space of an unobserved full-rank matrix A t 2 R p ⇥ k .

Original: Precisely, from the standard spikedcovariance model [35], we consider the time-dependent environment:
Revised: Precisely, from the standard spiked covariance model [35], we consider the time-dependent environment:

Original: [Equation]
Revised: [Equation]

Original: where A t can vary with time.
Revised: where A t can vary with time.

Original: The parameter k ≪ p is the desired number of principal components.
Revised: The parameter k ⌧ p is the desired number of principal components.

Original: Assuming A t belongs to a temporal uncertainty set (defined in  (2)), our goal is to recover the top- kprincipal components of the terminal subspace A T .
Revised: Assuming A t belongs to a temporal uncertainty set (deﬁned in the equation (2)), our goal is to recover the top- k principal components of the terminal subspace A T .

Original: In financial applications, our model capturesthe market evolution, in terms of the changing A T with the ultimate goal of explaining the marketconditions for an appropriately chosen T .
Revised: In ﬁnancial applications, our model captures the market evolution in terms of the changing A T with the ultimate goal of explaining the market conditions for an appropriately chosen T .

Original: Previous works on the stationary environment assume A t =
Revised: Previous works on the stationary environment assume A t =

Original: A for all t and have focused oncomputing a basis for the column space of the matrix A using streaming algorithms [43, 28, 38,32, 20, 36].
Revised: A for all t and have focused on computing a basis for the column space of the matrix A using streaming algorithms [21, 29, 33, 36, 38, 43].

Original: The accurate reconstruction of the principal components for the standard streaming PCA problem depends on the magnitude of observation noise σ , the dimension of observations p ,the number of principal components k of the matrix A , and the spectral gap δ between k th andk +1 th singular value of A .
Revised: The accurate reconstruction of the principal components for the standard streaming PCA problem depends on the magnitude of observation noise  , the dimension of observations p , the number of principal components k of the matrix A , and the spectral gap  between k -th and k +1 -th spectrums. singular value of AA > .

Original: In a marked departure from previous work, we study the case when thecolumn space of A varies across time, possibly perturbed by Γ .
Revised: In a marked departure from previous work, we study the case when the column space of A varies across time, possibly perturbed by  .

Original: This paper explores these avenuesand proposes a tractable analysis framework for the streaming PCA problem, robust to perturbationsin the stochastic data-generating model.
Revised: This paper explores these avenues and proposes a tractable analysis framework for the streaming PCA problem, robust to perturbations in the stochastic data-generating model.

Original: Our contributions can be summarized as:(a)
Revised: Our contributions can be summarized as follows:

Original: (Lower Bound; Section 4)
Revised: (Lower Bound; Section 4)

Original: Our first contribution is establishing a fundamental lower boundfor estimating the principal components when the covariance matrix belongs to a temporaluncertainty set.
Revised: Our ﬁrst contribution is establishing a fundamental lower bound for estimating the principal components when the covariance matrix belongs to a temporal uncertainty set Tu (  ,  ) .

Original: In Theorem 1, we derive the minimax bounds of the expected error forrecovering the top singular vectors for any algorithm in the class Φ .
Revised: In Theorem 1, we derive the minimax bounds of the expected error for recovering the top singular vectors for any streaming algorithm.

Revised: Because the underlying distribution can vary

Original:  A key componentof this analysis involves carefully constructed instances to elicit the worst-case behaviorof any algorithm in Φ .
Revised: observations from the far past become less important and the estimation of principal components associated with A T should be determined by a subset of samples. Simultaneously, it becomes imperative to ﬁnd the block size B that can be used to recover the principal components This is in sharp contrast to the standard

Revised: spiked covariance

Original:  We show that for T
Revised: model We show that: • For T

Original: =
Revised: =

Original: O (Γ − 2 / 3 ) , the minimax estimation errordecreases as O ( p 1 / 2/δ √ T  ) .
Revised: O (  9 2 / 3 ) , the minimax estimation error decreases as O ( p 1 / 2 T 9 1 / 2 ) .

Original:  On the other hand, for T = Ω(Γ − 2 / 3 ) , the estimation errorstagnates to O ( p 1 / 3 Γ 1 / 3/δ ) and does not decrease upon collecting more observations 1 .
Revised: • On the other hand, for T = ⌦ (  9 2 / 3 ) , the error stagnates to O ( p 1 / 3  1 / 3 ) and does not decrease upon collecting more observations 1 .

Original: Since observations can belong to potentially different spaces, the estimation of principalcomponents associated with A T should be determined by a subset of samples. This is insharp contrast to the standard spiked covariance model (where data is generated from thefixed model), observations from the far past become less important when the covariancematrix is time-variant, and it becomes imperative to find the block size B that can be usedto recover the principal components.(b) (Algorithm Analysis; Section 5)
Revised: (Algorithm Analysis; Section 5)

Original: We then analyze two algorithms to recover the principalcomponents extensively used in the standard streaming PCA setting; the noisy powermethod [28] and Oja’s algorithm [48].
Revised: We then analyze two algorithms to recover the principal components extensively used in the standard streaming PCA setting; the noisy power method [29] and Oja’s algorithm [48].

Original: These algorithms represent two very different designprinciples for computing principal components from data; processing data in blocks visà-vis single observations (However, note that both algorithms perform computations in astreaming fashion, i.e., have O ( kp ) storage requirements).
Revised: These algorithms represent two very different design principles for computing principal components from data in streaming fashion, processing data in blocks vis-à-vis single observations.

Original: Wefirst determine the optimalchoice for critical parameters: the block size in the noisy power method on Lemma 1 andthe learning rate in Oja’s algorithm on Lemma 2.
Revised: We determine the optimal choice for critical parameters: the block size in the noisy power method on Lemma 1 and the learning rate in Oja’s algorithm on Lemma 2.

Original: Next, we leverage these results in orderto obtain an upper bound on the convergence error for these algorithms inTheorem 2 and
Revised: Next, we leverage these results to obtain an upper bound on the convergence error for these algorithms in

Original: Theorem 3 respectively.
Revised: Theorem 2 and 3, respectively.

Original: From these results, we have that: (i) the block size B for the noisypower method and inverse of the learning rate,  1/ζ for Oja’s algorithm play a similar roleeven in the non-stationary environment; (ii) In either case, the optimal parameters scale with∼
Revised: From these results, we have: • The block size B for the noisy power method and inverse of the learning rate, ⇣  1 for Oja’s algorithm, plays a similar role even in the non-stationary environment.

Original: Γ − 2 / 3  , where Γ isperturbation budget which will be defined later, and (iii) the derivedupper bound on the estimation error for  noisy power method matches the minimax error interms of p, δ, and Γ and becomes rate-optimal if it satisfies mild condition.
Revised: In either case, the optimal parameters scale with ⇥ (   2 / 3 ) , where  is the perturbation budget. • The derived upper bound on the estimation error for the noisy power method matches the minimax error in terms of p,  , and  and becomes rate-optimal if it satisﬁes the mild conditions.

Original: Notation We fix notation throughout the main body of the paper.
Revised: Notation. We ﬁx notation throughout the main body of the paper.

Original: Matrices are denoted by bolduppercase letters ( M ) and vectors are denoted by bold lowercase letters ( x ).
Revised: Matrices are denoted by bold uppercase letters (e.g. M ) and vectors are denoted by bold lowercase letters (e.g. x ).

Original: Let M ∈ R p × k .
Revised: For M 2 R p ⇥ k ,

Original: Then M i denote the i th column of M , M i : j denote the submatrix consists with i th∼ j th column of M and
Revised: M i denote the i th column of M , M i : j denote the submatrix consists with i th

Original:  M i,j be the ( i, j ) -element of M .
Revised: ⇠ j th column of M and M i,j be the ( i, j ) -element of M .

Original: ∥·∥ denote matrix 2-norm or equivalently operator norm for matricesand standard 2-norm for vectors.
Revised: k · k denote matrix 2-norm or equivalently operator norm for matrices and standard 2-norm for vectors.

Original: We use St k ( R p ) for the class of orthogonal matrices in R p × k .b
Revised: We use St k ( R p ) for the class of orthogonal matrices in R p ⇥ k .

Original:  ( M ) is the orthogonal matrix where the columns form basis for ran( M ) .
Revised: b ( M ) is the orthogonal matrix where the columns form basis for ran( M ) .

Original: λ i ( M ) represents the i thlargest eigenvalue of the matrix M and and s i ( M ) be the i th largest singular value.
Revised: s i ( M ) represents the i th largest singular value of the matrix.

Original: The singularvalue decomposition of M is defined as SVD( M ) =
Revised: The singular value decomposition of M is deﬁned as SVD( M ) =

Original: UDV ⊤ , where U ∈ St p ( R p ) ,
Revised: UDV > , where U 2 St p ( R p ) ,

Original: V ∈ St k
Revised: [Equation]

Original: ( R k ) , and D ∈ R p × k is a diagonal matrix whose i th diagonal element equals s i ( M ) .
Revised: ( R k ) , and D 2 R p ⇥ k is a diagonal matrix whose i th diagonal element equals s i ( M ) .

Original: Therefore, we assumewithout loss of generality that the singular values and respective singular vectors are ordered fromlargest to smallest.
Revised: Therefore, we assume without loss of generality that the singular values and respective singular vectors are ordered from largest to smallest.

Original: We denote by M ⊥ the orthogonal projection onto the orthogonal complement of the range of M  .
Revised: We denote by M ? the orthogonal projection onto the orthogonal complement of the ran( M ) .

Original: Therefore, if rk( M ) =
Revised: Therefore, if rk( M ) =

Original: k and SVD( M )
Revised: r and SVD( M )

Original: =
Revised: =

Original: UDV ⊤ , M ⊥ given by M ⊥ =
Revised: UDV > , M ? given by M ? =

Original: I − U 1: k U ⊤ 1: k .
Revised: I  U 1: r U > 1: r .

Original: Moreover, when rk( M ) = rk( ˜ M ) and SVD( ˜ M ) = ˜
Revised: Moreover, when rk( M ) = rk( ˜ M ) and SVD( ˜ M ) = ˜

Original: U
Revised: U

Original: ˜ D ˜ V ⊤ , the distance between ran( M ) and ran( ˜ M )is defined by: 
Revised: ˜ D ˜ V > , the distance between ran( M ) and ran( ˜ M ) is deﬁned by: d (ran( M ) , ran( ˜ M )) =

Original: [Equation]
Revised: k b ( M ) b ( M ) >  b ( ˜ M ) b ( ˜ M ) > k = k U 1: k U > 1: k  ˜U 1: k ˜U > 1: k k .

Original: We denote d (ran( M ) , ran( ˜ M )) by d ( M , ˜ M ) whenever clear from the context.
Revised: We denote d (ran( M ) , ran( ˜ M )) by d ( M , ˜ M ) whenever clear from the context.

Original: For
Revised: The letter A stands for abbreviation of sequence of matrices ( A t )

Original: A = ( A 1 , . . . 
Revised: Tt =1 . We write X

Revised: ⇠

Original: , A T ) , we denote the expectation of f ( x 1 , . . . , x T ) over x t ∼ N ( 0 , A t 
Revised: A when each element x t in sequence X = ( x t ) Tt =1 are drawn from N ( 0 , A t A > t +  2 I p ⇥ p ) .

Original:  A ⊤ t + σ 2 I )as E X∼A [ f ] .We denote ˜ O as the big-
Revised: We denote the expectation of f ( X ) over x t ⇠ N ( 0 , A t A > t +  2 I p ⇥ p ) as E X ⇠ A [ f ] .

Original:  O  notation with ignore the multiplicative dependency oflog( p ) + 2 log( T ) and 1 + Θ(log( T/ ∆) /k ) .
Revised: We denote ˜ O , ˜ ⇥ as the O , ⇥ notation with ignore the multiplicative dependency of log( pT 2 ) or smaller.

Original:  Related Work
Revised:  Related Work

Original: Principal component analysis  has been extensively studied across operations research, computerscience, and other communities.
Revised: Principal component analysis (PCA) has been extensively studied across operations research, computer science, and other communities.

Original: We highlight how our work differs from existing literature.
Revised: We highlight how our work differs from existing literature.

Original: Robust 
Revised: Robust PCA.

Original: PCA Robust PCA deals with the problem of retrieving the principal components robust to thepresence of outliers in the data.
Revised: Robust PCA deals with the problem of retrieving the principal components robust to the presence of outliers in the data.

Original: The cornerstone work in this direction is the Principal Component Pursuit framework wherein they assume that the matrix of observations M can be decomposed interms of a low-rank matrix L and a sparse matrix (with entries of arbitrarily large magnitude) S [14].
Revised: The cornerstone work in this direction is the principal component pursuit framework wherein they assume that the matrix of observations M can be decomposed in terms of a low-rank matrix L and a sparse matrix (with entries of arbitrarily large magnitude) S [15].

Original: Several works[23, 25, 16, 46] consider the robust PCA problem and propose algorithms in the offline,batch, and online setting.
Revised: Several works consider the robust PCA problem and propose algorithms in the ofﬂine, batch, and online settings [17, 24, 26, 46].

Original: Our work differs from this line of literature in two aspects: assumptionsabout the data generating model and convex optimization techniques.
Revised: Our work differs from this line of literature in two aspects: assumptions about the data-generating model and convex optimization techniques.

Original: First, our data generation modelis unrelated to those considered in the robust PCA literature.
Revised: First, our data generation model is unrelated to those considered in the robust PCA literature.

Original: Further, rich theories from convexoptimization can be used in the PCA framework to design efficient algorithms, but our problem is notamenable to such techniques.
Revised: Further, rich theories from convex optimization can be used in the PCA framework to design efﬁcient algorithms, but our problem is not amenable to such techniques.

Original: Streaming PCA Streaming algorithms for PCA have been proposed among other works on PCA [43,34, 28, 59, 2].
Revised: Streaming PCA. Streaming algorithms for PCA have been proposed, among other works on PCA [2, 29, 34, 43, 60].

Original: Algorithms analyzed in this work, such as the noisy power method [28] and Oja’salgorithm [48], are iterative methods for estimating the principal components.
Revised: Algorithms analyzed in this work, such as the noisy power method [29] and Oja’s algorithm [48], are iterative methods for estimating the principal components.

Original: These iterative schemesare instances of stochastic approximation-based solutions for the optimization version of the PCAproblem [4].
Revised: These iterative schemes are instances of stochastic approximation-based solutions for the optimization version of the PCA problem [5].

Original: The stochastic approximation is a root-finding framework extensively used for stochasticoptimization [37, 10].
Revised: The stochastic approximation is a root-ﬁnding framework extensively used for stochastic optimization [11, 37].

Original: Oja’s algorithm, originally proposed by [48], was the first such scheme.
Revised: Oja’s algorithm, originally proposed by [48], was the ﬁrst such scheme.

Original: This framework is also used to analyze gradient-type and proximal-type incremental methods akin toalgorithms for convex optimization.
Revised: This framework is also used to analyze gradient-type and proximal-type incremental methods akin to algorithms for convex optimization.

Original: Stochastic gradient descent-based algorithms for the streaming
Revised: Stochastic gradient descent-based algorithms for the streaming

Original: PCA problem, where a single observation is used at every point in time to update the principalcomponents’ estimate, have been extensively studied.
Revised: PCA problem, where a single observation is used at every point in time to update the principal components’ estimate, have been extensively studied.

Original: Along this line, [29] propose GRASTA, anincremental online gradient method for learning over different subspaces.
Revised: Along this line, [30] propose GRASTA, an incremental online gradient method for learning over different subspaces.

Original: Similarly, [5] proposes
Revised: Similarly, [6] proposes

Original: GROUSE, based on the idea of gradient updates over the Grassmannian manifold.
Revised: GROUSE, based on the idea of gradient updates over the Grassmannian manifold.

Original: In these and otherrelated works[53, 15, 60, 30, 30, 1, 58] consider Oja’s algorithm for the standard streaming PCAproblem.
Revised: These and other related works consider Oja’s algorithm for the standard streaming PCA problem [1, 16, 31, 31, 53, 59, 62].

Original: However, all these works are based on a completely different modeling assumption thanthis paper and do not provide theoretical guarantees for our setting.
Revised: However, all these works are based on a completely different modeling assumption than this paper and do not provide theoretical guarantees for our setting.

Original: Streaming and robust PCA algorithms are used in  presence of outliers or data with a lot of missingentries [17, 54].
Revised: Streaming and robust PCA algorithms are used in the presence of outliers or data with a lot of missing entries [18, 55].

Original: From this literature,  closest to our work is the work on robust subspace tracking [2,59, 29].
Revised: From this literature, the closest to our work is the work on robust subspace tracking [2, 30, 60].

Original: However, the robustness considered their is against erasures or sparse outliers.
Revised: However, the robustness considered there is against erasures or sparse outliers.

Original: While thesealgorithms provide theoretical guarantees in that setting, those guarantees cannot be extended to ourmodel.
Revised: While these algorithms provide theoretical guarantees in that setting, those guarantees cannot be extended to our model.

Original: Considerations of erasures and outliers under the model proposed in this paper is beyond thescope of this work and remains an interesting future direction.
Revised: Considerations of erasures and outliers under the model proposed in this paper are beyond the scope of this work and remain an interesting future direction.

Original:  Mathematical Framework
Revised:  Mathematical Framework

Original: Observations from the standard spiked covariance model belong to a fixed k -dimensional columnspace A ∈ R p .
Revised: Observations from the standard spiked covariance model belong to a ﬁxed k -dimensional column space A 2 R p .

Original: Previous work has focused on reconstructing this space from the observed timeseries.
Revised: Previous work has focused on reconstructing this space from the observed time series.

Original: Under our framework, we assume that the sequence of observations ( x t )
Revised: Under our framework, we assume that the sequence of observations ( x t )

Original: Tt =1 follows the timedependent spiked covariance model (1).
Revised: Tt =1 follows the time-dependent spiked covariance model (1).

Original: We consider the problem of computing top- k singular vectorsof A T .
Revised: We consider the problem of computing top- k singular vectors of A T .

Original: We formulate this model by addressinga minimax optimization as a robust optimizationproblem.
Revised: We formulate this model by addressing minimax optimization as a robust optimization problem.

Original: When the adversary is allowed to select a completely arbitrary sequence of matrices  A 1 , . .
Revised: When the adversary is allowed to select a completely arbitrary sequence of matrices ( A t )

Original: A T , itis impossible to accurately recover the column space of A T since only the single vector x T providesinformation about this space.
Revised: Tt =1 , it is impossible to accurately recover the column space of A T .

Original: Instead, we define temporal uncertainty sets to restrict the power of theadversary.
Revised: Instead, we deﬁne temporal uncertainty sets to restrict the power of the adversary.

Original: Definition 1.
Revised: Deﬁnition 1.

Original: Let Γ , δ ≥ 0 .
Revised: Let  ,   0 .

Original: We only allow the sequence of matrices 
Revised: We only allow the sequence of matrices ( A t )

Original: A 1 , . . . A T that lie in antemporal uncertainty set Tu ( δ, Γ) defined as:
Revised: Tt =1 that lie in an temporal uncertainty set Tu (  ,  ) deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: That is, we constrain the difference between any two consecutive covariance matrices of the underlying(noise-free) process.
Revised: We constrain the difference between any two consecutive covariance matrices of the underlying process by  .

Original: The assumption s k
Revised: The assumption s k

Original: ( A t A ⊤ t ) ≥ δ is crucial for establishing bounds of estimationerror.
Revised: ( A t A > t )   is crucial for establishing bounds of estimation error.

Original: It is also justified in any setting where the underlying phenomenology guarantees covariance ofrank k; the stated lower bound excludes pathological cases of near-rank k − 1 or smaller.
Revised: It is also justiﬁed in any setting where the underlying phenomenology guarantees covariance of rank k; the stated lower bound excludes pathological cases of near-rank k  1 or smaller.

Original: Having described the constraints on the perturbation power, wenow present the family of algorithmsand the performance metric of interest.
Revised: Having described the constraints on the perturbation power, we present the algorithms and the performance metric of interest.

Original: In particular, we consider algorithms that are (a) streaming,i.e., make a single pass through the time series in chronological order; only the previous samples arestored at any point in the past.
Revised: In particular, we consider (i) streaming algorithms, i.e., make a single pass through the time series in chronological order; only the previous samples are stored at any point in the past.

Original: These characteristics model the behavior of an algorithm that receivesdata in real-time and only stores samples in limited memory.
Revised: These characteristics model the behavior of an algorithm that receives data in real-time and only stores samples in limited memory.

Original: Among those streaming algorithms, weconsider the algorithms whose(b) output is a set of k orthonormal vectors in R p .
Revised: Among those streaming algorithms, we consider (ii) the algorithms whose output is a set of k orthonormal vectors in R p .

Original: The first requirementensures that the storage and computational memory requirements are independent of the number ofsamples T , and hence, the assumption T ≫ p is not prohibitive.

Original: Let us denote by Φ the family ofalgorithms just described.
Revised: Let us denote by  the family of algorithms just described.

Original: Let A
Revised: The output of algorithm  2  for a given observations X = ( x t ) Tt = sampled using model (2) is a set of k orthonormal vectors, which we denote by  X . We will also view  X as a matrix in R p ⇥ k or the subspace generated by  X . Following deﬁnition 2 illustrates how we treat the lower bound and when we call the algorithm optimal.

Original: = ( A 1 , . . . , A T ) ∈
Revised: We note that this formulation has been widely studied [14, 57, 58].

Original: Tu ( δ, Γ) be given.
Revised: Deﬁnition 2. Let A

Original: We formalize this definitionas follows: Definition 2. Σ( A ) is the time-dependent spiked covariance model represented by the A =( A 1 , . . .
Revised: =

Original: , A T ).
Revised: ( A t )

Original: Furthermore, X := ( x 1 , . . , x T ) ∼ Σ( A ) is a sequence of observation vectorsdrawn from Σ( A ) .{+Tt =1 
+}
Revised: {+Tt =1 
+}

Original: The output of algorithm ϕ for a given choice of the observations x t sampled using model (2) is a setof k orthonormal vectors which we denote by ϕ X ; abusing notation we will also view ϕ X as a matrixin R p × k or the subspace generated by ϕ X . Definition 3. Let ϕ ∈ Φ , A ∈ Tu( δ, Γ) and X ∼ Σ( A ) .
Revised: Tu

Original: (a) Given A and X , the estimation error of ϕ given X  is the distance between the space spannedby ϕ X and the column space of A T (= d ( ran ( A T ) , ϕ X ) ).
Revised: (  ,  ) , and  2  . The estimation error of  given X ⇠ A is the distance between the space spanned by  X and the column space of A T ( d ( ran ( A T ) ,  X ) ).

Original: The metric can be easily extended to the cumulative error.
Revised: The metric can be easily extended to the cumulative error.

Original: Please see the Appendix A for the discussion.
Revised: Refer Appendix A for the discussion.

Original: (b) R ϕ is the maximum expected estimation error of ϕ under A over all A ∈ Tu ( δ, Γ) .Precisely,
Revised: R  is the maximum expected estimation error of  under A over all A 2 Tu (  ,  ) .

Original: [Equation]
Revised: [Equation]

Original: (c) R ∗ is the minimax estimation error defined as the minimum of the largest expected estima- tion error incurred by ϕ ∈ Φ :
Revised: R ⇤ is the minimax estimation error deﬁned as the minimum of the largest expected estimation error incurred by  2  :

Original: [Equation]
Revised: [Equation]

Original: We note that minimax estimation problems have been widely studied [56, 13, 57].

Original: d

Original: An algorithm

Original: ϕ ∈ Φ is rate-optimal if R ϕ ≤ C · R ∗ , where constant C > 0 is independentof the problem parametersΓ , T , δ , p , k  .
Revised: An algorithm  2  is rate-optimal if R   C · R ⇤ , where constant C > 0 is independent of the problem parameters T ,  , p , k , and  .

Original: In this paper we establish the minimax estimation error R ∗ and propose rate-optimal sublinear-time, single-pass algorithms for robuststreaming PCA.
Revised: In this work, we establish the minimax estimation error R ⇤ and propose rate-optimal sublinear-time, single-pass algorithms for robust streaming PCA.

Original:  Minimax Lower Bound
Revised:  Minimax Lower Bound

Original: When A  t  belongs to the temporal uncertainty set Tu ( δ, Γ) (Definition 1) an algorithm designed torecover the principal components of A T from the observations cannot guarantee zero estimation error.
Revised: When A = ( A t ) Tt =1 belongs to the temporal uncertainty set Tu (  ,  ) (Deﬁnition 1), an algorithm designed to recover the principal components of A T from the observations cannot guarantee zero estimation error.

Original: Our first goal is to obtain a lower bound on the estimation error interms of the problem parameters δ, Γ , p  and the number of observations T . We now establish a lower bound on the expected error inrecovering the top singular vector of A T from observations X = ( x t ) Tt =1 .
Revised: Our ﬁrst goal is to obtain the minimax lower bound on the estimation error in the problem parameters T ,  , p , k , and  .

Original: In order to establish the lower bound, we leverage the fundamental limit of hypothesis tests [56].
Revised: In order to establish the lower bound, we leverage the fundamental limit of hypothesis tests [57].

Original: The crux of the proof lies in constructing the set of worst-case hypotheses and establishing a lowerbound on the probability of error in distinguishing between these hypotheses using observed data.
Revised: The crux of the proof lies in constructing the set of worst-case hypotheses and establishing a lower bound on the probability of error in distinguishing between these hypotheses using observed data.

Original: The complete proof is provided in Appendix C.
Revised: The complete proof is provided in Appendix D.

Original: Theorem 1 (Lower Bound) .
Revised: Theorem 1 (Lower Bound) .

Original: Assume  0≤ Γ < δ and T ≥ 1 .
Revised: Assume  >   0 and p > 2 k + 1 .

Original: For any algorithm ϕ ∈ Φ , there exists 
Revised: For any algorithm  2  , there exists a sequence

Original: asequence A ∈ Tu ( δ, Γ) and constant C > 0 which is independent with Γ , δ, T, p, k such that:
Revised: A 2 Tu (  ,  ) such that E X ⇠ A ⇥ d  [Equation] [Equation] X ⇤ has lower bound with order:

Original: [Equation]
Revised: [Equation]

Revised: By taking inf  2  , we get the same lower bound for R ⇤ .

Original: For the standard streaming PCA problem (Theorem 1 with the case of Γ = 0 ), the fundamental limitis Θ(( σ/δ )( p ( σ 2 + δ ) /T ) 1 / 2 )  which has the expected Θ(1 / √ T ) dependence [13, 57].
Revised: For the standard streaming PCA problem (Theorem 1 with the case of  = 0 ), the fundamental limit is ⇥ ((  /  )( p (  2 +  ) /T ) 1 / 2 ) , which has the expected ⇥ (1 / p T ) dependence [14, 58].

Original: On the otherhand, in  presence of perturbations ( Γ > 0 ), the lower bound exhibits a phase transition phenomenon,with the first term representing the effect of model ambiguity.
Revised: On the other hand, in the presence of perturbations (  > 0 ), the lower bound exhibits a phase transition phenomenon, with the ﬁrst term representing the effect of model ambiguity.

Original: To this end, define the critical time T cas
Revised: To this end, deﬁne the critical time T c as

Original: [Equation]
Revised: [Equation]

Original: For T
Revised: For T

Original: =
Revised: =

Original: O ( T c ) , the lower bound decreases with the rate of 1 / √ T .
Revised: O ( T c ) , the lower bound decreases with the rate of 1 / p T .

Original: However, when T = Ω( T c ) , thefirst term dominates the second term and R ∗ becomes independent to the number of observations T .
Revised: However, when T = ⌦ ( T c ) , the ﬁrst term dominates the second term, and R ⇤ becomes independent of the number of observations T .

Original: In this regime, the error stagnates to O ((Γ /δ ) 1 / 3 (cid:0) pσ 2 ( σ 2 + δ ) /δ
Revised: In this regime, the error stagnates to O ((  /  ) 1 / p

Original:  2 (cid:1) 1 / 3 ) .
Revised:  2 (  2 +  ) /  1 / 3 ) .

Original: This validates our intuitionthat in a dynamic environment past information quickly becoming stale.
Revised: Therefore, as our intuition suggests, the information quickly becomes stale in a dynamic environment.

Original: In Theorem 2 and 3, we will prove that the noisy power method and Oja’s algorithm attain a nearoptimal bound on the convergence guarantee.
Revised: Theorem 2 and 3 will prove that the noisy power method and Oja’s algorithm attain a near-optimal bound on the convergence guarantee.

Original: Theorem 2 guarantees that if s 1 ( A t A ⊤ t ) 
Revised: Theorem 2 guarantees that if s 1 ( A t A > t ) = ⇥

Original: = δ , theupper bound for estimation error on the noisy power method is of the following order:
Revised: (  ) , the upper bound for estimation error on the noisy power method is of the following order:

Original: [Equation]
Revised: [Equation]

Original: That is, if pσ 2 dominates kδ , the noisy power method becomes rate-optimal under the controlleduncertainty with Tu ( δ, Γ) .
Revised: That is, if p  2 dominates k  , the noisy power method becomes rate-optimal under the controlled uncertainty with Tu (  ,  ) .

Original: This regime is the case ofpractical noisy  situations, with σ 2 ̸≪ δ .
Revised: This regime is the case of noisy practical situations, with  2 6⌧  .

Original: Observealso that we have p > k from the setting.

Original:  Convergence Analysis
Revised:  Convergence Analysis

Original: In this section, we analyze two algorithms for the robust streaming PCA problem.
Revised: In this section, we analyze two algorithms for the robust streaming PCA problem.

Original: A generic templatefor algorithms ϕ ∈ Φ of interest to us is as follows: (i) ϕ is initialized with a random matrix withorthonormal columns ˆ U ∈ R p × k ; (ii) A running estimate of the principal components is maintainedas the columns of ˆ U ; (iii) Observations are projected onto the column space of ˆ U to update thisestimate.
Revised: A generic template for algorithms,  2  of interest to us is as follows: (i)  is initialized with a random matrix with orthonormal columns ˆ U 2 R p ⇥ k ; (ii) a running estimate of the principal components is maintained as the columns of ˆ U ; (iii) observations are projected onto the column space of ˆ U to update this estimate.

Original: We consider two algorithms:  robust version of the noisy power method (Algorithm 1) and Oja’salgorithm (Algorithm 2).
Revised: We consider two algorithms: a robust version of the noisy power method (Algorithm 1) and Oja’s algorithm (Algorithm 2).

Original: The key difference between the noisy power method and Oja’s algorithmis in the data used to estimate the principal components.
Revised: The critical difference between the noisy power method and Oja’s algorithm is the data used to estimate the principal components.

Original: In the noisy power method, the estimatesare updated after a batch of observations, whereas in Oja’s algorithm, the estimates are updatedafter scaling every observation with the learning rate.
Revised: In the noisy power method, the estimates are updated after a batch of observations, whereas in Oja’s algorithm, the estimates are updated after scaling every observation with the learning rate.

Original: Therefore, the parameters determining theperformance of these algorithms are the batch size B for the robust power method and the learningrate ζ for Oja’s algorithm.
Revised: Therefore, the parameters determining the performance of these algorithms are the batch size B for the robust power method and the learning rate ⇣ for Oja’s algorithm.

Original: The analysis of these algorithms cannot be readily established with existing techniques when thecovariance matrix belongs to a temporal uncertainty set since they rely on showing that the estimatesimprove every iteration.
Revised: The analysis of these algorithms cannot be readily established with existing techniques when the covariance matrix belongs to a temporal uncertainty set since they rely on showing that the estimates improve every iteration.

Original: Further, the application of many concentration results requires randommatrices to be bounded which is not the case when the observations are sampled from (1).
Revised: Further, applying many concentration results requires random matrices to be bounded, which is not the case when the observations are sampled from (1).

Original: Therefore,in order to simplify the analysis on both algorithms, we introduce Assumption 1, adapted from [34].
Revised: Therefore, in order to simplify the analysis of both algorithms, we introduce Assumption 1, adapted from [34].

Original: Assumption 1.
Revised: Assumption 1.

Original: Let ( A t )
Revised: Let ( A t )

Original: Tt ={+=1 
+}
Revised: Tt {+=1 
+}

Original: 1 ∈ Tu ( δ, Γ) .
Revised: Tu (  ,  ) .

Original: For ˜ δ ≥ δ and M , V > 0 , we consider the observations xtfor t ∈ [ T ] satisfy the following:
Revised: For ˜    and M , V > 0 , we consider the observations x t for t 2 [ T ] satisfy the following:

Original: (a) E [ x t x ⊤ t ] =
Revised: E [ x t x > t ] =

Original: A t A ⊤ t + [-σ 
-] I p × p while 
Revised: A t A > t + [-σ 
-] 2 I p ⇥ p while k A t A > t k  ˜  , 2.

Original: ∥ A t A ⊤ t ∥ ≤ ˜ δ , (b) ∥ x t x ⊤ t − ( A t A ⊤ t + [-σ 
-] I p × p ) ∥
Revised: k x t x > t  ( A t A > t + [-σ 
-] 2 I p ⇥ p ) k

Original: ≤ M almost surely, and 
Revised:  M a.s., and 3.

Original: (c) (cid:13)(cid:13) E (cid:2) ( [Equation]
Revised:  E ⇥ ( x t x > t  ( A t A > t +  2 I p ⇥ p ))

Original: (cid:3)(cid:13)(cid:13)
Revised: ⇤

Original: [Equation] V 
Revised:  V .

Original: [Equation] When the observations follow equation (1), we condition our analysis on  event E∆ .
Revised: When the observations follow model (1), we condition our analysis on the high-probability event E .

Original: Definition 4.
Revised: Deﬁnition 3.

Original: For z t ∼ N ( 0 k × 1 , I k × k ) and w t ∈ N ( 0p × 1 , I p × p ) , the event E∆ is defined as:
Revised: Let SVD( A t A > t +  2 I p ⇥ p )= U t D t U Tt and x t = U t D 1 / 2 t z t , where z t ⇠ N ( 0 , I p ⇥ p ) . We deﬁne the event E as:

Original: [Equation]
Revised: [Equation]

Original: The observations from model (1) satisfy Assumption 1 with M = [-
 (-] k ˜ δ + pσ 2 )(1+Θ(log( T/ ∆) /k )) ,
Revised: The observations from model (1) satisfy Assumption 1 with M = [-
 (-]2( k ˜  + p  2 )(1+ ⇥ (log( pT 2 ) /T )) ,

Original: V = 2 M (˜ δ + σ 2 ) with probability P [ E∆ ] ≥ 1 − ∆ .
Revised: V = 2 M (˜  +  2 ) with probability P [ E ]  1  1 /T .

Original: Although after conditioning E [ x t x ⊤ t | E∆ ] ̸ =
Revised: Although after conditioning E [ x t x > t | E ] 6 =

Original: A t A ⊤ t + σ 2 I p × p , we can use all the results in Section 5.1 and 5.2 with a sufficiently small ∆ .
Revised: A t A > t +  2 I p ⇥ p , we can use all the results in Section 5.1 and 5.2 with a multiplicative logarithmic factor.

Original: Pleaserefer to Appendix H for the details.
Revised: Please refer to Appendix B for the details.

Original: 5.1 Noisy Power Method
Revised: 5.1 Noisy Power Method

Revised: 1: Input: Stream of vectors: ( x t )

Revised: Tt =1 , block size: B , dimensions: p, k 2: Sample each element of ˆU (0) in N (0 , 1) 3: for ` = 1 : L = b T/B c do 4:

Revised: Y ( ` ) 0 2 R p ⇥ k 5: for t

Revised: =

Revised: ( `  1) B + 1 : ` B do 6:

Original:  The noisy power method is an iterative algorithm for computing the top- k principal components ofgiven matrix.
Revised: Y ( ` ) Y ( ` ) + 1 B x t x > t ˆU ( `  1) 7: end for 8: ˆU ( ` ) Gram-Schmidt ( Y ( ` )) 9: end for 10: Output: ˆU ( L ) The noisy power method is an iterative algorithm for computing the top- k principal components of a matrix.

Original: Starting from the random matrix ˆ U (0) in R p × k , the algorithm runs for L iterations,each processing B samples.
Revised: Starting from the random matrix ˆ U (0) in R p ⇥ k , the algorithm runs for L iterations, each processing B samples.

Original: By repeating this procedure, we expect the algorithm to reconstruct thecovariance matrix if the observations are derived from the fixed distribution.
Revised: By repeating this procedure, we expect the algorithm to reconstruct the covariance matrix if the observations are derived from the ﬁxed distribution.

Original: When observations are drawn fromthe model 1 under Assumption 1, the observations belong todifferent underlying subspace.
Revised: When observations are drawn from model (1) under Assumption 1, the later observation can be sampled from distributions with shifted covariance.

Original: Unlike the standard case, for any algorithm ϕ , the presence ofperturbation prevents the convergence of the columns of ˆ U ( ℓ ) to the singular vectors of the finalcovariance matrix U ( ℓ ) .
Revised: Unlike the standard case, for any algorithm  2  , the presence of perturbation prevents the convergence of the columns of ˆ U ( ` ) to the singular vectors of the ﬁnal covariance matrix U ( ` ) .

Original: The main difficulty here is that the columns of ˆ U ( ℓ ) do not convergetowards a fixed set of vectors but keep tracking the time-varying principal components.
Revised: The main difﬁculty here is that the columns of ˆ U ( ` ) do not converge towards a ﬁxed set of vectors but keep tracking the time-varying principal components.

Original: Recall thatour ultimate objective is to recover the principal components associated with the terminal observation.
Revised: Recall that our ultimate objective is to recover the principal components associated with the terminal observation.

Original: Hence, we decompose the covariance matrix in terms of the last observation and the remainingsamples.
Revised: Hence, we decompose the covariance matrix in terms of the last observation and the remaining samples.

Original: For ℓ -th block, we denote the covariance matrix for ℓB -th observation as M ( ℓ ) and have:
Revised: For ` -th block, we denote the covariance matrix for ` B -th observation as M ( ` ) and have:

Original: [Equation]
Revised: [Equation]

Original: Due topresence of perturbations in the robust model, E ( ℓ ) is a non-zero mean random variable.
Revised: Due to perturbations in the robust model, E ( ` ) is a non-zero mean random variable.

Original: Wefirst decompose E ( ℓ ) in terms of the contribution due to inherent noise and the perturbationsallowedby the robust model.
Revised: Therefore, we ﬁrst decompose E ( ` ) in terms of the contribution due to inherent noise and the perturbations the robust model allows.

Original: In Lemma 1, we decompose the error ∥E ( ℓ ) ∥ with respect to the block size Band the allowed perturbations  in the robust model.
Revised: Then, in Lemma 1, we decompose the error k E ( ` ) k with respect to the block size B and the allowed perturbations  in the robust model.

Original: The complete proofis provided in Appendix D.
Revised: We provided complete proof in Appendix E.

Original: Lemma 1 (Spectral norm of noise) .
Revised: Lemma 1 (Spectral norm of noise) .

Original: Assume that the observations ( x t )
Revised: Assume that the observations ( x t )

Original: Tt =1 generated according tothe Assumption 1.
Revised: Tt =1 generated according to the Assumption 1.

Original: For probability greater than 1 − 1 T , the matrices E ( ℓ ) in the equation (7) arebounded by:
Revised: With probability greater than 1  1 /T , the matrices E ( ` ) in the equation (6) are bounded by:

Original: [Equation]
Revised: [Equation]

Original: when the block size B is larger than M 2 log(2 pT 2 ) / V .
Revised: when the block size B is larger than M 2 log(2 pT 2 ) / V .

Original: Lemma 1 highlights the effect of allowing perturbations in the data generation model.
Revised: Lemma 1 highlights the effect of allowing perturbations in the data generation model.

Original: From classicalresults in statistics, our intuition tells  that the effect of noise washes out as the block size increases,i.e., the error decays with the ‘inverse square root’ of the block size.
Revised: From classical results in statistics, our intuition tells us that the effect of noise washes out as the block size increases, i.e., the error decays with the ‘inverse square root’ of the block size.

Original: Hence, barring memory anddata issues, a larger block size is better when Γ = 0 .
Revised: Hence, barring memory and data issues, a larger block size is better when  = 0 .

Original: In contrast, covariance perturbations add errorproportional to the block size.
Revised: In contrast, covariance perturbations add errors proportional to the block size.

Original: Consequently, we have a trade-off between two terms in this case, andthere exists an optimal block size  depending on the Γ .
Revised: Consequently, we have a trade-off between two terms in this case, and an optimal block size exists depending on the  .

Original: We establish the convergence guarantee of the robust power method in Theorem 2.
Revised: We establish the convergence guarantee of the robust power method in Theorem 2.

Original: In the proof of
Revised: In the proof of

Original: Theorem 2 (Appendix E), we bound the distance, d (cid:0) U ( L ) , ˆ U ( L ) (cid:1) between the output of Algorithm 1,ˆ U 1: k ( L ) and k -orthonormal vectors spanning the column space of A T , U 1: k
Revised: Theorem 2 (Appendix F), we bound the distance, d  U ( L ) , ˆ U ( L )  between the output of Algorithm 1, ˆ U 1: k ( L ) and k -orthonormal vectors spanning the column space of A T , U 1: k

Original: ( L ) .
Revised: ( L ) .

Original: We identify theoptimal block size B ,which is the unique parameter for  noisy power method and establish an upperbound on the estimation error of the noisy power method.
Revised: We identify the optimal block size B , the unique parameter for the noisy power method, and establish an upper bound on the estimation error of the noisy power method.

Original: Theorem 2 (Robust power method) .
Revised: Theorem 2 (Robust power method) .

Original: Assume that δ ≥ 0 .
Revised: Assume that   0 .

Original: 71 σ 2 and Γ =
Revised: 71  2 and  =

Original: O ( δ 3 / ( V log(2 pT 2 ))) .
Revised: O (  3 / ( V log(2 pT 2 ))) .

Original: Whenthe observations { x t } Tt =1 satisfies the assumption 1, for B = 
Revised: When the observations ( x t ) Tt =1 satisﬁes the assumption 1, for B = ⇥

Original: Θ( V 1 / 3 log(2 pT 2 ) 1 / 3 / Γ 2 / 3 ) we have: 
Revised: ( V 1 / 3 log(2 pT 2 ) 1 / 3 /  2 / 3 ) we have: d (ran( A T ) , NPM X

Original: [Equation]
Revised: [Equation]

Revised: = O

Original: [Equation] 
Revised: [Equation] [Equation]

Original:  with probability 1 − 1 /T − c Ω( p − k +1)
Revised: (8) with probability 1  2 /T  c ⌦ ( p  k +1)

Original: − exp( − Ω( p )) .
Revised:  e  ⌦ ( p ) .

Original: When Γ = Ω( δ 3 / ( V log(2 pT 2 )))
Revised: When  = ⌦ (  3 / ( V log(2 pT 2 )))

Original: we have that ( V Γlog(2 pT 2 )) 1 / 3 δ = Ω(1) .
Revised: we have that ( V  log(2 pT 2 )) 1 / 3 /  = ⌦ (1) .

Original: Therefore, the conditionon Γ in Theorem 2 is necessary to avoid a trivial upper bound ( ≤ 1 ) on R NPM .
Revised: Therefore, the condition on  in Theorem 2 is necessary to avoid a trivial upper bound ⇥ (1) .

Original: This condition encompasses several applications of interest alluded to earlier.
Revised: This condition encompasses several applications of interest alluded to earlier.

Original: For example, in the financial applicationsalluded to earlier, individual market changes of interest happen on  millisecond time-scale and it is ofsignificant interest to terminally detect cumulative market changes.
Revised: For example, in the ﬁnancial applications alluded to earlier, individual market changes of interest happen on a millisecond time scale. It is of signiﬁcant interest to terminally detect incremental market changes.

Original: Our results hold for sufficientlylarge value of cumulative changes (allowing for seconds time-scale) allowing us to study them.
Revised: Our results hold on to the large value of cumulative changes and allow us to study them.

Original: Theyfurther imply that the noisy power method is rate-optimal for non-trivial values of Γ .
Revised: They further imply that the noisy power method is rate-optimal for non-trivial values of  .

Original: When theobservations follow model (1), under the event E ∆ , Theorem 2 shows that the robust power methodcan achieve a estimation error of:
Revised: When the observations follow model (1), under the event E  , Theorem 2 shows that the robust power method can achieve an estimation error of:

Original: [Equation]
Revised: [Equation]

Original: with probability 1 − ∆ − 1 /T − c Ω( p − k +1)
Revised: if T = ⌦ (max( T c ,  ( p  2 )  1 )) ,  = ⌦ (( c ⌦ ( p  k +1)

Original: − exp( − Ω( p ))
Revised: + e  ⌦ ( p ) )

Original: if ˜ δ equals to δ . 
Revised:  2 ( p  2 )  1 ) , and ˜  = ⇥ (  ) . Then it becomes order-wise identical to the fundamental limit established in Theorem 1 when p dominates k  . The ﬁrst condition on T illustrates when past observations become less critical. The probability for the upper bound on the noisy power method [29] with random initialization should not be small to construct expectation bounds from the high probability bound.

Original: This is order-wise identicalto the fundamental limit established in Theorem 1 when pσ 2 dominates kδ .
Revised: We address this regime by condition on  , which is coarse due to exponential terms and ( p  2 )  1 .

Original: Establishing bounds on the estimation error when the underlying singular vectors change is intricatesince the subspace to which consecutive observations belong are potentially different.
Revised: Establishing bounds on the estimation error when the underlying singular vectors change is intricate since the subspace to which consecutive observations belong is potentially different.

Original: Conventionalproofs that analyze noisy power methods or Oja’s algorithm show that under a variety of assumptionsat every iteration ℓ , the distance between the estimated and true subspace, d ( U ( ℓ ) , ˆ U ( ℓ )) decreases.
Revised: Conventional proofs that analyze noisy power methods or Oja’s algorithm show that under a variety of assumptions at every iteration ` , the distance between the estimated and true subspace, d ( U ( ` ) , ˆ U ( ` )) decreases.

Original: For instance, the proof in [28] requires ∥E ( ℓ ) U ( ℓ ) ∥
Revised: For instance, the proof in [29] requires k E ( ` ) U ( ` ) k

Original: = 
Revised: = O

Original: O ( (cid:112) k/p )  which does not hold under ourmodel since ∥ M ( ℓ ) − M ( ℓ − 1) ∥ is in general greater than (cid:112) k/p .
Revised: ( p k/p ) , which does not hold under our model since k M ( ` )  M ( `  1) k is, in general, greater than p k/p .

Original: Similarly, the concentrationapproach in [32] can be used only when the covariance matrix is time-invariant.
Revised: Similarly, the concentration approach in [33] can be used only when the covariance matrix is time-invariant.

Original: We briefly describeour proof technique to establish Theorem 2, deferring details to Appendix E. Let 
Revised: We brieﬂy describe our proof technique to establish Theorem 2, deferring details to Appendix F. Let M ( L ) denotes the product

Original: M ( L ) denote theproduct (cid:81) Lℓ =1 (cid:0) M ( ℓ ) + E ( ℓ ) (cid:1) .
Revised: Q [Equation]  .

Original: Then, the output of the algorithm ˆ U 1: k
Revised: Then, the output of the algorithm ˆ U 1: k

Original: ( L ) is an orthonormal basisof M ( L ) ˆ U (0) , which estimates the first k principal components U 1: k of the M ( L ) .
Revised: ( L ) is an orthonormal basis of M ( L ) ˆ U (0) , which estimates the ﬁrst k principal components U 1: k of the M ( L ) .

Original: To addressthis, we construct sequences of k and ( p − k ) -dimensional subspaces of R p from observations{ ( x t ) ℓBt =( ℓ − 1) B } Lℓ =1 , denoted by { N ( ℓ ) } Lℓ =1 ∈
Revised: To address this, we construct sequences of k and ( p  k ) -dimensional subspaces of R p from observations { ( x t ) ` Bt =( `  1) B } L ` =1 , denoted by { N ( ` ) } L ` =1 2 St p  k ( R p ) and { W ( ` )

Original: St p − k ( R p ) and { W ( ℓ ) } Lℓ =1 ∈ St k
Revised: } L ` =1 2 St k

Original: ( R p ) respectively,such that for all iterations ℓ :
Revised: ( R p ) respectively, such that for all iterations ` :

Original: (a) ran (cid:0) ( M ( ℓ )+ E ( ℓ )) N ( ℓ ) (cid:1) ⊆ ran (cid:0) N ( ℓ +1) (cid:1) and ran (cid:0) ( M ( ℓ )+ E ( ℓ )) W ( ℓ ) (cid:1) ⊆
Revised: 1. ran  ( M ( ` )+ E ( ` )) N ( ` )  ✓ ran  N ( ` +1)  and ran  ( M ( ` )+ E ( ` )) W ( ` )  ✓ ran  W

Original: ran (cid:0) W ( ℓ +1) (cid:1) , 
Revised: ( ` +1)  , 2. k ( M ( ` ) + E ( ` ))

Original: (b) ∥ ( M ( ℓ ) + E ( ℓ )) N( ℓ ) ∥∥ (( M ( ℓ ) + E ( ℓ )) W ( ℓ ) ) − 1 ∥ < 1 ,
Revised: N

Original: (c) d ( U 1: k  ( ℓ ) , W ( ℓ +1) )
Revised: ( ` ) k · k (( M ( ` ) + E ( ` ))

Original: =
Revised: W ( ` ) )  1 k < 1 , 3.

Original: O ( ∥E ( ℓ ) ∥ ) and d ( U k
Revised: d ( U 1: k ( ` ) , W ( ` +1) ) , d ( U k

Original: +1: p ( ℓ ) , N ( ℓ +1) )
Revised: +1: p ( ` ) , N ( ` +1) )

Original: =
Revised: =

Original: O ( ∥E ( ℓ ) ∥ ) .
Revised: O ( k E ( ` ) k ) .

Original: The initial random matrix ˆU 1: k (0) consists of both N (1) and W (1) with high probability.
Revised: The initial random matrix ˆ U 1: k (0) consists of both N (1) and W (1) with high probability.

Original: From thefirst two properties, at every iteration ℓ , the projection of ˆU 1: k ( ℓ − 1) in W ( ℓ ) is amplified more thanthat on N ( ℓ ) and thus ˆU 1: k ( L ) becomes very close to W ( L ) after sufficiently large L .
Revised: From the ﬁrst two properties, at every iteration ` , the projection of ˆ U 1: k ( `  1) in W ( ` ) is ampliﬁed more than that on N ( ` ) and thus ˆU 1: k ( L ) becomes very close to W ( L ) after sufﬁciently large L .

Original: From the lastproperty, we can conclude that W ( L ) is close to U 1: k ( L ) , where the distance between W ( L ) and
Revised: From the last property, we can conclude that W ( L ) is close to U 1: k ( L ) , where the distance between W ( L ) and

Original: U 1: k ( L ) is proportional to ∥E ( L ) ∥ .
Revised: U 1: k ( L ) is proportional to k E ( L ) k .

Original: Combining these ideas with Lemma 1 establishes the results.
Revised: Combining these ideas with Lemma 1 establishes the results.

Original: 5.2 Oja’s Algorithm
Revised: 5.2 Oja’s Algorithm

Revised: Algorithm 2 Oja’s Algorithm with learning rate ⇣

Original:  1: Input: { x 1 } Tt =1 , Learning rate: ζ ,  dimensions: p, k& Output: ˆU ( T ) 2:  ˆU (0) ← p × k random matrix 3: for t = 1 : T do 4: ˆU ( t )← Gram-Schmidt (( I  + ζ x t x ⊤ t ) ˆU ( t − 1)) 5: end for 
Revised: [48] 1: Input: Stream of vectors: ( x t ) Tt =1 , learning rate: ⇣ , and dimensions: p, k 2: Sample each element of ˆU (0) in N (0 , 1) 3: for t = 1 : T do 4: ˆU ( t ) Gram-Schmidt (( I p ⇥ p + ⇣ x t x > t ) ˆU ( t  1)) 5: end for 6: Output: ˆU ( T )

Original: We now establish the convergence guarantee for Oja’s Algorithm (Algorithm 2) when observationsfollow (1).
Revised: We now establish the convergence guarantee for Oja’s Algorithm (Algorithm 2) when observations follow the equation (1).

Original: Unlike the noisy power method, Oja’s Algorithm is multiplicative in its construction of theestimated subspace.
Revised: Unlike the noisy power method, Oja’s Algorithm is multiplicative in its construction of the estimated subspace.

Original: We extend the existing analysis for Oja’s algorithm [34, 32, 3] by considering avirtual block with B = B ζ = ζ − 1 observations.
Revised: We extend the existing analysis for Oja’s algorithm [3, 33, 34] by considering a virtual block with B = B ⇣ = ⇣  1 observations.

Original: Building upon the analysis framework for the noisypower method and intuition from binomial approximation (1 + ζx ) 1 /ζ ≃ (1 + x ) we establish theconvergence guarantees for Oja’s algorithm.
Revised: Building upon the analysis framework for the noisy power method and intuition from binomial approximation (1 + ⇣ x ) 1 / ⇣ ' (1 + x ) we establish the convergence guarantees for Oja’s algorithm.

Original: Like the previous section, we decompose the block withtarget M ( ℓ ) and error matrix E ′{+
+} ( ℓ ) as:
Revised: Like the previous section, we decompose the block with target M ( ` ) and error matrix E {+
+} ( ` ) as:

Original: [Equation]
Revised: [Equation]

Original:  where E ′ is scaled error matrix.
Revised: (10) where E 0 is scaled error matrix.

Original: In Lemma 2, we provide bound for scaled error matrix with respect tothe learning parameter ζ , or the virtual block size B ζ =  1/ζ similar to Lemma 1.
Revised: In Lemma 2, we provide bound for scaled error matrix with respect to the learning parameter ⇣ , or the virtual block size B ⇣ = ⇣  1 similar to Lemma 1.

Original: The proof is providedin Appendix F.
Revised: The proof is provided in Appendix G.

Original: Lemma 2 (Spectral norm of noise, Oja’s algorithm case) .
Revised: Lemma 2 (Spectral norm of noise, Oja’s algorithm case) .

Original: Assume that the observations ( x t ) Tt =1generated according to the Assumption 1.
Revised: Assume that the observations ( x t ) Tt = generated according to the Assumption 1.

Original: For probability greater than 1 − 1 T , the matrices E ′{+
+} ( ℓ ) onthe equation (11) are bounded by:
Revised: With probability greater than 1  1 /T , the matrices E {+
+} ( ` ) on the equation (10) are bounded by:

Original: [Equation]
Revised: [Equation]

Original: when the virtual block size B ζ = ζ − 1 is larger than 2 M 2 log( pT 2 ) .
Revised: when the virtual block size B ⇣ = ⇣  1 is larger than 2 M 2 log( pT 2 ) .

Original: The difference in parameter M , rather than V  as in the case of Lemma 1 arises due to  use of adifferent concentration inequality.
Revised: The difference in parameter M , rather than V , as in the case of Lemma 1 arises due to the use of a different concentration inequality.

Original: The estimator of the noisy power method averages the outerproduct of vectors (equation (11)) making it straightforward to use Bernstein’s inequality.
Revised: The estimator of the noisy power method averages the outer product of vectors (equation (10)), making it straightforward to use Bernstein’s inequality.

Original: On theother hand, Oja’s algorithm averages the product of random matrices rather than the sum of randommatrices.
Revised: On the other hand, Oja’s algorithm averages the product of random matrices rather than the sum of random matrices.

Original: Therefore, we introduce the multiplicative concentration inequality [31], which requires the(probabilistic) norm bound for matrices.
Revised: Therefore, we introduce the multiplicative concentration inequality [32], which requires the (probabilistic) norm bound for matrices.

Original: Combining multiplicative concentration inequalities with Lemma 2 and the techniques developed for  noisy power method, we obtain a convergence guaranteesfor Oja’s algorithm in Theorem 3.
Revised: Combining multiplicative concentration inequalities with Lemma 2 and the techniques developed for the noisy power method, we obtain a convergence guarantee for Oja’s algorithm in Theorem 3.

Original: The complete proof is provided in Appendix G.
Revised: The complete proof is provided in Appendix H.

Original: Theorem 3 (Oja’s algorithm) .
Revised: Theorem 3 (Oja’s algorithm) .

Original: Assume that δ ≥ 0 .
Revised: Assume that   0 .

Original: 71 and Γ = O
Revised: 71 and  = O

Original: ( δ 3 / ( e 3(˜ δ − δ ) M 2 log( pT 2 ))) .
Revised: (  3 / ( e 3(˜    ) M 2 log( pT 2 ))) .

Original: Whenthe observations { x t } T t =1 satisfies the assumption 1, for  1/ζ = Θ( M 2 / 3 log( pT 2 ) 1 / 3 / Γ 2 / 3 ) wehave:
Revised: When the observations { x t } Tt =1 satisﬁes the assumption 1, for ⇣  1 = ⇥ ( M 2 / 3 log( pT 2 ) 1 / 3 /  2 / 3 ) we have:

Original: [Equation]
Revised: [Equation]

Revised: with probability 1  2 /T  c ⌦ ( p  k +1)

Original: [Equation] −
Revised:  e  ⌦ ( p ) .

Original: Unlike the noisy power method the upper bound in Theorem 3 is O ( p 2 / 3 ) rather than  optimaldependence of O ( p 1 / 3 ) from Theorem 1.
Revised: Unlike the noisy power method, the upper bound in Theorem 3 is O ( p 2 / 3 ) rather than the optimaldependence of O ( p 1 / 3 ) from Theorem 1.

Original: It is unclear whether the upper bound can be improved.
Revised: It is unclear whether the upper bound can be improved.

Original: Sharpening our analysis with a two-phase strategy [32, 39] (wherein the first phase identifies a goodinitial point and the second phase establishes convergence given an initial point) might be an excellentdirection for future investigation.
Revised: Sharpening our analysis with a two-phase strategy [33, 39] (wherein the ﬁrst phase identiﬁes a good initial point and the second phase establishes convergence given an initial point) might be an excellent direction for future investigation.

Original:  Numerical Results
Revised:  Numerical Results

Original: [Figure]
Revised: Key observations from Theorem 2 and 3 on each algorithm illustrate; (i) the existence of the optimal block size B and the learning rate ⇣ to obtain the minimum recovery error, and (ii)   3 / 2 dependencies of that optimal B and 1 / ⇣ .

Original: Lastly, in order to verify the established results for both algorithms, we examine the performanceofalgorithms for a variety of environments .
Revised: In order to verify the established results for both algorithms, this section provides the performance of algorithms for various environments.

Original: Theorem 2 and 3 show the (a) existence of the optimal blocksize B and the learning rate ζ to obtain the minimum recovery error, and (b) Γ − 3 / 2 dependencies ofthose optimal B and 1 /ζ .
Revised: We synthesized the A = ( A t ) Tt = and sample X = ( x t )

Original:  We demonstrate this phenomenon in Figure 1a and Figure 1b respectively.
Revised: Tt =1 from A . We generate ( A t ) Tt =1 2 R p ⇥ k as the product of three matrices, U t 2 St p ( R p ) , D t 2 R p ⇥ k (; diagonal), and V t 2 St k

Revised: ( R k ) .

Revised: To obtain the matrix of the next step, we rotate the ﬁrst matrix as U t =

Revised: U t  1 R t ( R t 2 SO( p ) ).

Original: Experimental details  are described in Appendix I.Our first observation on the Figure 1a is that
Revised: The vectors x t are sampled from the model (1). More details for experimental setup are described in Appendix I.

Original: [Figure]
Revised: [Figure]

Original:  the block size increases without any covariance perturbation ( Γ = 0 ).
Revised: Our ﬁrst observation in Figure 1a is that convergence error decreases as the block size increases without any covariance perturbation (  =0 ).

Original: This behavior is expected sincein the absence of rotation of underlying space, increasing the numberof past information results in better accuracy guarantees for the recovered space.
Revised: This behavior is expected since increasing the number of past information results in better accuracy guarantees for the recovered space in the absence of a covariance shift.

Original: However, if thecovariance perturbation exists ( Γ > 0 ),there exists the optimal learning parameter, and we have thesmaller optimal block size as the adversary stronger.
Revised: However, if the covariance perturbation exists (  > 0 ), the optimal learning parameter exists, and we have the smaller optimal block size with stronger perturbation.

Original: This phenomenon is alsothe expected behaviorsince an increase in the adversarial budget implies that the past information becomes less relevant.
Revised: This phenomenon is also expected since an increase in the adversarial budget implies that the past information becomes less relevant.

Original: Our observations also corroborate our theoretical results inthe Theorem 1.
Revised: Our observations also corroborate our theoretical results in Theorem 1.

Original: In Figure 1b, we focuson the optimal value of the block size B and  inverse of the learning rate 1 /ζ and its variation withthe perturbation budget Γ .
Revised: In Figure 1b, we focus on the optimal value of the block size B and the inverse of the learning rate 1 / ⇣ and its variation with the perturbation budget  .

Original: We plot the empirically optimal learning rate for the case of the noisypower method and Oja’s algorithm with Γ − 2 / 3 .
Revised: We plot the empirically optimal learning rate for the case of the noisy power method and Oja’s algorithm with   2 / 3 .

Original: We observe that the optimal block size and theinverse learning rate is proportional to Γ − 2 / 3 .
Revised: We observe that the optimal block size and the inverse learning rate are proportional to   2 / 3 .

Original: This experimental dependency of Γ − 2 / 3 verifies thetheoretically prescribed results in Theorem 2 andTheorem 3.
Revised: This experimental dependency of   2 / 3 veriﬁes the theoretically prescribed results in Theorem 2 and 3.

Revised: 6.1 Experiments on Stock Price Dataset

Revised: We provide the real-world benchmark using the S&P500 stock market dataset [44] in Kaggle to test our ﬁndings in the non-stationary environment.

Original: Conclusion
Revised: Refer the Appendix J for the non-stationarity of environments and further experimental details that do not appear in the main paper. The stream of vectors consisted of 133( = p ) companies’ normalized daily returns. Since each company in S&P500 has a different time horizon of available information, we considered 133 companies with the cost information from Mar.18, 1980, to Jul.22, 2022 ( T =10677 ). Then we calculated the ‘daily return,’ which is the difference of adjusted close cost between two days; normalized by the adjusted close cost of the day.

Revised: The stream of vectors from the environment can be seen as sampled from time-varying distributions with  ' 0 .

Revised: The objective is to predict the principal components of the covariance matrix of daily returns as in [4, 54, 61].

Revised: We tested the noisy power method and Oja’s algorithm on the preprocessed stream.

Revised: For the target space A T , we used the k -dimensional subspace consisting of top-k singular vectors of covariance estimator calculated with the ﬁnal 500 samples ( k =1 , . . .

Revised: The results in Figure 2 indicate that our ﬁndings are also valid in the real-world environment with covariance shifts.

Revised: First, each result shows optimal parameters regimes observed at the U-shaped curves on recovery errors.

Revised: Furthermore, the result on the noisy power method with varying block size B is akin to the recovery error on Oja’s algorithm, plotted with scaled inverse learning rate ⇣  1 .

Revised: These two observations support the main ﬁndings in Figure 1a.

Revised: We also note that different scaling for each k is natural since we have different spectral gaps  between k -th and k +1 -th spectrums.

Original: In this paper, we analyzed the fundamental limit of the minimax error and convergence of the noisypower method and Oja’s algorithm on the streaming PCA settings with time-varying covariance.
Revised: On the streaming PCA settings with time-varying covariance, we analyzed the fundamental lower bound of the minimax error and estimation errors for the noisy power method and Oja’s algorithm.

Original: Under this framework, whenthere exists no perturbation  ( Γ = 0 ), our theoretical result on a lowerlimit coincides with the order of the traditional outcome.
Revised: Under this framework, when no perturbation exists (  =0 ), our theoretical result on a lower limit coincides with the order of the traditional outcome.

Original: Furthermore, when perturbation exists( Γ > 0 ), we have a non-avoidable positive minimax recovery error, although the time horizonbecomes arbitrarily long.
Revised: Furthermore, when perturbation exists (  > 0 ), we have a non-avoidable positive minimax recovery error, although the time horizon becomes arbitrarily long.

Original: Next, we found that the noisy power method order-wisely achieves thisrecovery error and becomes rate-optimal if the kδ ≪ pσ 2  .
Revised: Next, we found that the noisy power method order-wisely achieves this recovery error and becomes rate-optimal if the  2 = ⌦ ( k  /p ) .

Original: Finally, we showed that optimal block sizeand inverse learning rate minimizing recovery error were proportional to Γ − 3 / 2  both theoreticallyand experimentally.
Revised: For the optimal learning parameters B or ⇣  1 , we showed that optimal block size and inverse learning rate minimizing recovery error are similar up to a multiplicative factor and proportional to   3 / 2 . Experimental results both on the synthetic data and real-world environments support the theoretical ﬁndings on the learning parameters.

Original: All our results are based on the spectral gap assumption. However, a recent lineof literature considers effective rank [11, 50] as a crucial metric.
Revised: Although our analysis requires the spectral gap assumption, a recent line of literature considers effective rank [12, 50] as a crucial metric.

Original: Establishing guarantees for robuststreaming PCA in terms of effective rank is an interesting direction for future work. 
Revised: Therefore, establishing guarantees for robust streaming PCA in terms of effective rank is a perfect direction for future work. Acknowledgement

Original: Checklist
Revised: This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2022-0-00311, Development of Goal-Oriented Reinforcement Learning Techniques for Contact-Rich Robotic Manipulation of Everyday Objects;

Revised: No.2019-0-

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] The datasets are public datasetsand the code is MIT license
Revised: [Yes] The datasets are public datasets and the code is MIT license

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] We explain our new framework and contributions in both abstract and introduction.
Revised: [Yes] We explain our new framework and contributions in both abstract and introduction.

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] There is a result which is partially suboptimal compare to the edge works.
Revised: [Yes] There is a result which is partially suboptimal compare to the edge works.

Original: We also describe and study the reason for that.
Revised: We also describe and study the reason for that.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?[Yes]
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original:  We provideour assumptions on the Section 3 and Assumption (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] We provide our assumptions on the Section 3 and Assumption(b) Did you include complete proofs of all theoretical results?

Original: [Yes] Detailed Proofs are inthe appendix.3.
Revised: [Yes] Detailed Proofs are in the appendix.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] Those areincluded in the supplemental material.
Revised: [Yes] Those are included in the supplemental material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] Those are included in the Appendix.
Revised: [Yes] Those are included in the Appendix.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] Yes, we repeated every experiment 10 times and reportederror bars or uncertainty markers.
Revised: [Yes] Yes, we repeated every experiment 10 times and reported error bars or uncertainty markers.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] Those are included in the Appendix.
Revised: [Yes] Those are included in the Appendix.

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [N/A] (b) Did you mention the license of the assets?
Revised: [N/A] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] We only used synthesized random matrix data.
Revised: [N/A] We only used synthesized random matrix data.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]