Original: Accelerating SGD for Highly Ill-Conditioned Huge-Scale Online Matrix Completion
Revised: Accelerating SGD for Highly Ill-Conditioned Huge-Scale Online Matrix Completion

Original: Abstract
Revised: Abstract

Original: The matrix completion problem seeks to recover a d × d ground truth matrix of low rank r ≪
Revised: The matrix completion problem seeks to recover a d × d ground truth matrix of low rank r ≪

Original: d from observations of its individual elements.
Revised: d from observations of its individual elements.

Original: Real-world matrix completion is often a huge-scale optimization problem, with d so large that even the simplest full-dimension vector operations with O ( d ) time complexity become prohibitively expensive.
Revised: Real-world matrix completion is often a huge-scale optimization problem, with d so large that even the simplest full-dimension vector operations with O ( d ) time complexity become prohibitively expensive.

Original: Stochastic gradient descent (SGD) is one of the few algorithms capable of solving matrix completion on a huge scale, and can also naturally handle streaming data over an evolving ground truth.
Revised: Stochastic gradient descent (SGD) is one of the few algorithms capable of solving matrix completion on a huge scale, and can also naturally handle streaming data over an evolving ground truth.

Original: Unfortunately,
Revised: Unfortunately,

Original: SGD experiences a dramatic slow-down when the underlying ground truth is illconditioned; it requires at least O ( κ log(1 /ϵ ))
Revised: SGD experiences a dramatic slow-down when the underlying ground truth is illconditioned; it requires at least O ( κ log(1 /ϵ ))

Original: iterations to get ϵ -close to ground truth matrix with condition number κ .
Revised: iterations to get ϵ -close to ground truth matrix with condition number κ .

Original: In this paper, we propose a preconditioned version of SGD that preserves all the favorable practical qualities of SGD for huge-scale online optimization while also making it agnostic to κ .
Revised: In this paper, we propose a preconditioned version of SGD that preserves all the favorable practical qualities of SGD for huge-scale online optimization while also making it agnostic to κ .

Original: For a symmetric ground truth and the Root Mean Square Error (RMSE) loss, we prove that the preconditioned SGD converges to ϵ -accuracy in O (log(1 /ϵ )) iterations, with a rapid linear convergence rate as if the ground truth were perfectly conditioned with κ = 1 .
Revised: For a symmetric ground truth and the Root Mean Square Error (RMSE) loss, we prove that the preconditioned SGD converges to ϵ -accuracy in O (log(1 /ϵ )) iterations, with a rapid linear convergence rate as if the ground truth were perfectly conditioned with κ = 1 .

Original: In our numerical experiments, we observe a similar acceleration for ill-conditioned matrix completion under the root mean square error (RMSE) loss,
Revised: In our numerical experiments, we observe a similar acceleration for ill-conditioned matrix completion under the root mean square error (RMSE) loss,

Original: Euclidean distance matrix (EDM) completion under pairwise square loss, and collaborative filtering under the Bayesian Personalized Ranking (BPR) loss.
Revised: Euclidean distance matrix (EDM) completion under pairwise square loss, and collaborative filtering under the Bayesian Personalized Ranking (BPR) loss.

Original:  Introduction
Revised:  Introduction

Original: The matrix completion problem seeks to recover an underlying d × d ground truth matrix M of low rank r ≪
Revised: The matrix completion problem seeks to recover an underlying d × d ground truth matrix M of low rank r ≪

Original: d from observations of its individual matrix elements M i,j .
Revised: d from observations of its individual matrix elements M i,j .

Original: The problem appears most prominently in the context of collaborative filtering and recommendation system, but also numerous other applications.
Revised: The problem appears most prominently in the context of collaborative filtering and recommendation system, but also numerous other applications.

Original: In this paper, we focus on the symmetric and positive semidefinite variant of the problem, in which the underlying matrix M can be factored as M = ZZ T where the factor matrix Z is d × r , though that our methods have natural extensions to the nonsymmetric case.
Revised: In this paper, we focus on the symmetric and positive semidefinite variant of the problem, in which the underlying matrix M can be factored as M = ZZ T where the factor matrix Z is d × r , though that our methods have natural extensions to the nonsymmetric case.

Original: We note that the symmetric positive semidefinite variant is actually far more common in collaborative filtering, due to the prevalence of item-item models, which enjoy better data (most platforms contain several orders of magnitude more users than items) and more stable recommendations (the similarity between items tends to change slowly over time) than user-user and user-item models.
Revised: We note that the symmetric positive semidefinite variant is actually far more common in collaborative filtering, due to the prevalence of item-item models, which enjoy better data (most platforms contain several orders of magnitude more users than items) and more stable recommendations (the similarity between items tends to change slowly over time) than user-user and user-item models.

Original: For the full-scale, online instances of matrix completion that arise in real-world collaborative filtering, stochastic gradient descent or SGD is the only viable algorithm for learning the underlying matrix M .
Revised: For the full-scale, online instances of matrix completion that arise in real-world collaborative filtering, stochastic gradient descent or SGD is the only viable algorithm for learning the underlying matrix M .

Original: The basic idea is to formulate a candidate matrix of the form XX T with respect to a learned factor matrix X ∈ R d × r , and to minimize a cost function of the form ϕ ( XX T − M ) .
Revised: The basic idea is to formulate a candidate matrix of the form XX T with respect to a learned factor matrix X ∈ R d × r , and to minimize a cost function of the form ϕ ( XX T − M ) .

Original: Earlier work used the root mean square error (RMSE) loss ∥ XX T − M ∥ 2 F = (cid:80) i,j ( XX T − M ) 2 i,j , though later work have focused on pairwise losses like the BPR [1] that optimize for ordering and therefore give better recommendations.
Revised: Earlier work used the root mean square error (RMSE) loss ∥ XX T − M ∥ 2 F = (cid:80) i,j ( XX T − M ) 2 i,j , though later work have focused on pairwise losses like the BPR [1] that optimize for ordering and therefore give better recommendations.

Original: For the RMSE loss, the corresponding SGD iterations with (rescaled) learning rate α > 0 reads
Revised: For the RMSE loss, the corresponding SGD iterations with (rescaled) learning rate α > 0 reads

Original: [Equation]
Revised: [Equation]

Original: where M ij is the sampled ( i, j ) -th element of the ground truth matrix M , and x i , x j and x i, + , x j, + denote the i -th and j -th rows of the current iterate X t and new iterate X t +1 .
Revised: where M ij is the sampled ( i, j ) -th element of the ground truth matrix M , and x i , x j and x i, + , x j, + denote the i -th and j -th rows of the current iterate X t and new iterate X t +1 .

Original: Pairwise losses like the BPR can be shown to have a similar update equation over three rows of X [1].
Revised: Pairwise losses like the BPR can be shown to have a similar update equation over three rows of X [1].

Original: Given that only two or three rows of X are accessed and updated at any time, SGD is readily accessible to massive parallelization and distributed computing.
Revised: Given that only two or three rows of X are accessed and updated at any time, SGD is readily accessible to massive parallelization and distributed computing.

Original: For very large values of d , the update equation (1) can be run by multiple workers in parallel without locks, with vanishing probability of collision [2].
Revised: For very large values of d , the update equation (1) can be run by multiple workers in parallel without locks, with vanishing probability of collision [2].

Original: The blocks of X that are more frequently accessed together can be stored on the same node in a distributed memory system.
Revised: The blocks of X that are more frequently accessed together can be stored on the same node in a distributed memory system.

Original: Unfortunately, the convergence rate of SGD can sometimes be extremely slow.
Revised: Unfortunately, the convergence rate of SGD can sometimes be extremely slow.

Original: One possible explanation, as many recent authors have pointed out [3–6], is that matrix factorization models are very sensitive to ill-conditioning of the ground truth matrix M .
Revised: One possible explanation, as many recent authors have pointed out [3–6], is that matrix factorization models are very sensitive to ill-conditioning of the ground truth matrix M .

Original: The number of SGD iterations grows at least linearly the condition number κ , which here is defined as the ratio between the largest and the r -th largest singular values of M .
Revised: The number of SGD iterations grows at least linearly the condition number κ , which here is defined as the ratio between the largest and the r -th largest singular values of M .

Original: Ill-conditioning causes particular concern because most realworld data are ill-conditioned.
Revised: Ill-conditioning causes particular concern because most realworld data are ill-conditioned.

Original: In one widely cited study [7], it was found that the dominant singular value accounts for only ≈ 80% prediction accuracy, with diversity of individual preferences making up the remainder ill-conditioned singular values.
Revised: In one widely cited study [7], it was found that the dominant singular value accounts for only ≈ 80% prediction accuracy, with diversity of individual preferences making up the remainder ill-conditioned singular values.

Original: Cloninger et al.
Revised: Cloninger et al.

Original: [8] notes that there are certain applications of matrix completion that have condition numbers as high as κ = 10 15 .
Revised: [8] notes that there are certain applications of matrix completion that have condition numbers as high as κ = 10 15 .

Original: This paper is inspired by a recent full-batch gradient method called ScaledGD [4, 9] and a closely related algorithm PrecGD [5] in which gradient descent is made immune to ill-conditioning in the ground truth by right-rescaling the full-batch gradient by the matrix ( X T X ) − 1 .
Revised: This paper is inspired by a recent full-batch gradient method called ScaledGD [4, 9] and a closely related algorithm PrecGD [5] in which gradient descent is made immune to ill-conditioning in the ground truth by right-rescaling the full-batch gradient by the matrix ( X T X ) − 1 .

Original: Applying this same strategy to the SGD update equation (1) yields the row-wise updates
Revised: Applying this same strategy to the SGD update equation (1) yields the row-wise updates

Original: [Equation]
Revised: [Equation]

Original: in which we precompute and cache the preconditioner P
Revised: in which we precompute and cache the preconditioner P

Original: =
Revised: =

Original: ( X T X ) − 1 ahead of time 1 , and update it after the iteration as
Revised: ( X T X ) − 1 ahead of time 1 , and update it after the iteration as

Original: [Equation]
Revised: [Equation]

Original: by making four calls to the Sherman–Morrison rank-1 update formula
Revised: by making four calls to the Sherman–Morrison rank-1 update formula

Original: [Equation]
Revised: [Equation]

Original: This way, the rescaled update equations use just O ( r 2 ) arithmetic operations, which for modest values of r is only marginally more than the O ( r ) cost of the unscaled update equations (1).
Revised: This way, the rescaled update equations use just O ( r 2 ) arithmetic operations, which for modest values of r is only marginally more than the O ( r ) cost of the unscaled update equations (1).

Original: Indeed, the nearest-neighbor algorithms inside most collaborative filters have exponential complexity with respect to the latent dimensionality r , and so are often implemented with r small enough for (1) and (2) to have essentially the same runtime.
Revised: Indeed, the nearest-neighbor algorithms inside most collaborative filters have exponential complexity with respect to the latent dimensionality r , and so are often implemented with r small enough for (1) and (2) to have essentially the same runtime.

Original: Here, we observe that the rescaled update equations (2) preserve essentially all of the practical advantages of SGD for huge-scale, online optimization: it can also be run by multiple workers in parallel without locks, and it can also be easily implemented over distributed memory.
Revised: Here, we observe that the rescaled update equations (2) preserve essentially all of the practical advantages of SGD for huge-scale, online optimization: it can also be run by multiple workers in parallel without locks, and it can also be easily implemented over distributed memory.

Original: The only minor difference is that separate copies of P should be maintained by each worker, and resynchronized once differences grow large.
Revised: The only minor difference is that separate copies of P should be maintained by each worker, and resynchronized once differences grow large.

Original: Contributions In this paper, we provide a rigorous proof that the rescaled update equations (1), which we name ScaledSGD, become immune to the effects of ill-conditioning in the underlying ground truth matrix.
Revised: Contributions In this paper, we provide a rigorous proof that the rescaled update equations (1), which we name ScaledSGD, become immune to the effects of ill-conditioning in the underlying ground truth matrix.

Original: For symmetric matrix completion under the root mean squared error (RMSE) loss function, regular SGD is known to have an iteration count of O ( κ 4 · dr log( d/ϵ )) within a local neighborhood of the ground truth [10].
Revised: For symmetric matrix completion under the root mean squared error (RMSE) loss function, regular SGD is known to have an iteration count of O ( κ 4 · dr log( d/ϵ )) within a local neighborhood of the ground truth [10].

Original: This figure is optimal in the dimension d , the rank r , and the final accuracy ϵ , but suboptimal by four exponents with respect to condition number κ .
Revised: This figure is optimal in the dimension d , the rank r , and the final accuracy ϵ , but suboptimal by four exponents with respect to condition number κ .

Original: In contrast, we prove for the same setting that ScaledSGD attains an optimal convergence rate, converging to ϵ -accuracy in O ( dr log( d/ϵ )) iterations for all values of the condition number κ .
Revised: In contrast, we prove for the same setting that ScaledSGD attains an optimal convergence rate, converging to ϵ -accuracy in O ( dr log( d/ϵ )) iterations for all values of the condition number κ .

Original: In fact, our theoretical result predicts that ScaledSGD converges as if the ground truth matrix is perfectly conditioned, with a condition number of κ = 1 .
Revised: In fact, our theoretical result predicts that ScaledSGD converges as if the ground truth matrix is perfectly conditioned, with a condition number of κ = 1 .

Original: At first sight, it appears quite natural that applying the ScaledGD preconditioner to SGD should result in accelerated convergence.
Revised: At first sight, it appears quite natural that applying the ScaledGD preconditioner to SGD should result in accelerated convergence.

Original: However, the core challenge of stochastic algorithms like SGD is that each iteration can have substantial variance that “drown out” the expected progress made in the iteration.
Revised: However, the core challenge of stochastic algorithms like SGD is that each iteration can have substantial variance that “drown out” the expected progress made in the iteration.

Original: In the case of ScaledSGD, a rough analysis would suggest that the highly ill-conditioned preconditioner should improve convergence in expectation, but at the cost of dramatically worsening the variance.
Revised: In the case of ScaledSGD, a rough analysis would suggest that the highly ill-conditioned preconditioner should improve convergence in expectation, but at the cost of dramatically worsening the variance.

Original: Surprisingly, we find in this paper that the specific scaling ( X T X ) − 1 used in ScaledSGD not only does not worsen the variance, but in fact improves it.
Revised: Surprisingly, we find in this paper that the specific scaling ( X T X ) − 1 used in ScaledSGD not only does not worsen the variance, but in fact improves it.

Original: Our key insight and main theoretical contribution is Lemma 4, which shows that the same mechanism that allows ScaledGD to converge faster (compared to regular GD) also allows ScaledSGD to enjoy reduced variance (compared to regular
Revised: Our key insight and main theoretical contribution is Lemma 4, which shows that the same mechanism that allows ScaledGD to converge faster (compared to regular GD) also allows ScaledSGD to enjoy reduced variance (compared to regular

Original: SGD).
Revised: SGD).

Original: In fact, it is this effect of variance reduction that is responsible for most ( κ 3 out of κ 4 ) of our improvement over the previous state-of-the-art.
Revised: In fact, it is this effect of variance reduction that is responsible for most ( κ 3 out of κ 4 ) of our improvement over the previous state-of-the-art.

Original: It turns out that a careful choice of preconditioner can be used as a mechanism for variance reduction , while at the same time also fulfilling its usual, classical purpose, which is to accelerate convergence in expectation.
Revised: It turns out that a careful choice of preconditioner can be used as a mechanism for variance reduction , while at the same time also fulfilling its usual, classical purpose, which is to accelerate convergence in expectation.

Original: Related work Earlier work on matrix completion analyzed a convex relaxation of the original problem, showing that nuclear norm minimization can recover the ground truth from a few incoherent measurements [11–15].
Revised: Related work Earlier work on matrix completion analyzed a convex relaxation of the original problem, showing that nuclear norm minimization can recover the ground truth from a few incoherent measurements [11–15].

Original: This approach enjoys a near optimal sample complexity but incurs an
Revised: This approach enjoys a near optimal sample complexity but incurs an

Original: O ( d 3 ) per-iteration computational cost, which is prohibitive for a even moderately large d .
Revised: O ( d 3 ) per-iteration computational cost, which is prohibitive for a even moderately large d .

Original: More recent work has focused more on a nonconvex formulation based on Burer and Monteiro [16], which factors the optimization variable as M = XX T where X ∈ R d × r and applies a local search method such as alternating-minimization [17–20], projected gradient descent [21, 22] and regular gradient descent [23–26].
Revised: More recent work has focused more on a nonconvex formulation based on Burer and Monteiro [16], which factors the optimization variable as M = XX T where X ∈ R d × r and applies a local search method such as alternating-minimization [17–20], projected gradient descent [21, 22] and regular gradient descent [23–26].

Original: A separate line of work [27–35] focused on global properties of nonconvex matrix recovery problems, showing that the problem has no spurious local minima if sampling operator satisfies certain regularity conditions such as incoherence or restricted isometry.
Revised: A separate line of work [27–35] focused on global properties of nonconvex matrix recovery problems, showing that the problem has no spurious local minima if sampling operator satisfies certain regularity conditions such as incoherence or restricted isometry.

Original: The convergence rate of SGD has been well-studied for general classes of functions [36–39].
Revised: The convergence rate of SGD has been well-studied for general classes of functions [36–39].

Original: For matrix completion in particular, Jin et al.
Revised: For matrix completion in particular, Jin et al.

Original: [10] proved that SGD converges towards an ϵ -accurate solution in O ( dκ 4 log(1 /ϵ ))
Revised: [10] proved that SGD converges towards an ϵ -accurate solution in O ( dκ 4 log(1 /ϵ ))

Original: iterations where κ is the condition number of M .
Revised: iterations where κ is the condition number of M .

Original: Unfortunately, this quartic dependence on κ makes SGD extremely slow and impractical for huge-scale applications.
Revised: Unfortunately, this quartic dependence on κ makes SGD extremely slow and impractical for huge-scale applications.

Original: This dramatic slow down of gradient descent and its variants caused by ill-conditioning has become well-known in recent years.
Revised: This dramatic slow down of gradient descent and its variants caused by ill-conditioning has become well-known in recent years.

Original: Several recent papers have proposed full-batch algorithms to overcome this issue [9, 40, 41], but these methods cannot be used in the huge-scale optimization setting where d is so large that even full-vector operations with O ( d ) time complexity are too expensive.
Revised: Several recent papers have proposed full-batch algorithms to overcome this issue [9, 40, 41], but these methods cannot be used in the huge-scale optimization setting where d is so large that even full-vector operations with O ( d ) time complexity are too expensive.

Original: As a deterministic full-batch method, ScaledGD [9] requires a projection onto the set of incoherent matrices at every iteration in order to maintain rapid convergence.
Revised: As a deterministic full-batch method, ScaledGD [9] requires a projection onto the set of incoherent matrices at every iteration in order to maintain rapid convergence.

Original: Instead our key finding here is that the stochasticity of SGD alone is enough to keep the iterates as incoherent as the ground truth, which allows for rapid progress to be made.
Revised: Instead our key finding here is that the stochasticity of SGD alone is enough to keep the iterates as incoherent as the ground truth, which allows for rapid progress to be made.

Original: The second-order method proposed in [41] costs at least O ( d ) per-iteration and has no straightforward stochastic analog.
Revised: The second-order method proposed in [41] costs at least O ( d ) per-iteration and has no straightforward stochastic analog.

Original: PrecGD [5] only applies to matrices that satisfies matrices satisfying the restricted isometry property, which does not hold for matrix completion.
Revised: PrecGD [5] only applies to matrices that satisfies matrices satisfying the restricted isometry property, which does not hold for matrix completion.

Original:  Background: Linear convergence of SGD
Revised:  Background: Linear convergence of SGD

Original: In our theoretical analysis, we restrict our attention to symmetric matrix completion under the root mean squared error (RMSE) loss function.
Revised: In our theoretical analysis, we restrict our attention to symmetric matrix completion under the root mean squared error (RMSE) loss function.

Original: Our goal is to solve the following nonconvex optimization
Revised: Our goal is to solve the following nonconvex optimization

Original: [Equation]
Revised: [Equation]

Original: in which we assume that the d × d ground truth ZZ T ⪰ 0 matrix is exactly rank- r , with a finite condition number
Revised: in which we assume that the d × d ground truth ZZ T ⪰ 0 matrix is exactly rank- r , with a finite condition number

Original: [Equation]
Revised: [Equation]

Original: In order to be able to reconstruct ZZ T from a small number of measurements, we will also need to assume that the ground truth has small coherence [42]
Revised: In order to be able to reconstruct ZZ T from a small number of measurements, we will also need to assume that the ground truth has small coherence [42]

Original: [Equation]
Revised: [Equation]

Original: Recall that µ takes on a value from 1 to d/r , with the smallest achieved by dense, orthonormal choices of Z whose rows all have magnitudes of 1 / √ d , and the largest achieved by a ground truth
Revised: Recall that µ takes on a value from 1 to d/r , with the smallest achieved by dense, orthonormal choices of Z whose rows all have magnitudes of 1 / √ d , and the largest achieved by a ground truth

Original: ZZ T containing a single nonzero element.
Revised: ZZ T containing a single nonzero element.

Original: Assuming incoherence µ =
Revised: Assuming incoherence µ =

Original: O (1) with respect to d , it is a well-known result that all d 2 matrix elements of ZZ T can be perfectly reconstructed from just
Revised: O (1) with respect to d , it is a well-known result that all d 2 matrix elements of ZZ T can be perfectly reconstructed from just

Original: O ( dr log d ) random samples of its matrix elements [12, 43].
Revised: O ( dr log d ) random samples of its matrix elements [12, 43].

Original: This paper considers solving (3) in the huge-scale, online optimization setting, in which individual matrix elements of the ground truth ( ZZ T ) i,j = z
Revised: This paper considers solving (3) in the huge-scale, online optimization setting, in which individual matrix elements of the ground truth ( ZZ T ) i,j = z

Original: Ti z j are revealed one-at-a-time, uniformly at random with replacement, and that a current iterate X is continuously updated to streaming data.
Revised: Ti z j are revealed one-at-a-time, uniformly at random with replacement, and that a current iterate X is continuously updated to streaming data.

Original: We note that this is a reasonably accurate model for how recommendation engines are tuned to user preferences in practice, although the uniformity of random sampling is admittedly an assumption made to ease theoretical analysis.
Revised: We note that this is a reasonably accurate model for how recommendation engines are tuned to user preferences in practice, although the uniformity of random sampling is admittedly an assumption made to ease theoretical analysis.

Original: Define the stochastic gradient operator as
Revised: Define the stochastic gradient operator as

Original: [Equation]
Revised: [Equation]

Original: where ( i, j ) ∼ Unif([ d ] × [ d ]) , where x i , x j ∈ R r are the i -th and j -th rows of X , and the scaling d 2 is chosen that, over the randomness of the sampled index ( i, j ) , we have exactly E [ SG ( X )]
Revised: where ( i, j ) ∼ Unif([ d ] × [ d ]) , where x i , x j ∈ R r are the i -th and j -th rows of X , and the scaling d 2 is chosen that, over the randomness of the sampled index ( i, j ) , we have exactly E [ SG ( X )]

Original: = ∇ f ( X ) .
Revised: = ∇ f ( X ) .

Original: Then, the classical online SGD algorithm can be written as
Revised: Then, the classical online SGD algorithm can be written as

Original: [Equation]
Revised: [Equation]

Original: where α > 0 .
Revised: where α > 0 .

Original: (SGD)
Revised: (SGD)

Original: Here, we observe that a single iteration of SGD coincides with full-batch gradient descent in expectation, as in E [ X t +1 | X t ] =
Revised: Here, we observe that a single iteration of SGD coincides with full-batch gradient descent in expectation, as in E [ X t +1 | X t ] =

Original: X t − α ∇ f ( X t ) .
Revised: X t − α ∇ f ( X t ) .

Original: Therefore, assuming that bounded deviations and bounded variances, it follows from standard arguments that the behavior of many iterations of SGD should concentrate about that of full-batch gradient descent X t +1 =
Revised: Therefore, assuming that bounded deviations and bounded variances, it follows from standard arguments that the behavior of many iterations of SGD should concentrate about that of full-batch gradient descent X t +1 =

Original: X t − α ∇ f ( X t ) .
Revised: X t − α ∇ f ( X t ) .

Original: Within a region sufficiently close to the ground truth, full-batch gradient descent is well-known to converge at a linear rate to the ground truth [23, 44].
Revised: Within a region sufficiently close to the ground truth, full-batch gradient descent is well-known to converge at a linear rate to the ground truth [23, 44].

Original: Within this same region, Jin et al.
Revised: Within this same region, Jin et al.

Original: [10] proved that SGD also converges linearly.
Revised: [10] proved that SGD also converges linearly.

Original: For an incoherent ground truth with µ = O (1) , they proved that SGD with an aggressive choice of step-size is able to recover the ground truth to ϵaccuracy O ( κ 4 dr log( d/ϵ )) iterations, with each iteration costing O ( r ) arithmetic operations and selecting 1 random sample.
Revised: For an incoherent ground truth with µ = O (1) , they proved that SGD with an aggressive choice of step-size is able to recover the ground truth to ϵaccuracy O ( κ 4 dr log( d/ϵ )) iterations, with each iteration costing O ( r ) arithmetic operations and selecting 1 random sample.

Original: This iteration count is optimal with respect to d, r, and 1 /ϵ, although its dependence on κ is a cubic factor (i.e. a factor of κ 3 ) worse than full-batch gradient descent’s figure of O ( κ log(1 /ϵ )) , which is itself already quite bad, given that κ in practice can readily take on values of 10 3 to 10 6 .
Revised: This iteration count is optimal with respect to d, r, and 1 /ϵ, although its dependence on κ is a cubic factor (i.e. a factor of κ 3 ) worse than full-batch gradient descent’s figure of O ( κ log(1 /ϵ )) , which is itself already quite bad, given that κ in practice can readily take on values of 10 3 to 10 6 .

Original: Theorem 1 (Jin, Kakade, and Netrapalli [10]) .
Revised: Theorem 1 (Jin, Kakade, and Netrapalli [10]) .

Original: For Z ∈ R d × r with σ max ( Z ) = 1 and f ( X ) =
Revised: For Z ∈ R d × r with σ max ( Z ) = 1 and f ( X ) =

Original: ∥
Revised: ∥

Original: XX T − ZZ T ∥
Revised: XX T − ZZ T ∥

Original: 2 F and h i ( X ) =
Revised: 2 F and h i ( X ) =

Original: ∥
Revised: ∥

Original: e Ti X ∥ 2 , define the following
Revised: e Ti X ∥ 2 , define the following

Original: [Equation]
Revised: [Equation]

Original: For an initial point X 0 ∈ R d × r that satifies f
Revised: For an initial point X 0 ∈ R d × r that satifies f

Original: ( X 0 ) ≤ 12 f max and max i
Revised: ( X 0 ) ≤ 12 f max and max i

Original: h
Revised: h

Original: i ( X 0 ) ≤ 12 h max , there exists some constant c such that for any learning rate
Revised: i ( X 0 ) ≤ 12 h max , there exists some constant c such that for any learning rate

Original: α < c · ( κ · h max · d 2 log d ) − 1 , with probability at least 1 − T/d 10 , we will have for all t ≤ T iterations of SGD that
Revised: α < c · ( κ · h max · d 2 log d ) − 1 , with probability at least 1 − T/d 10 , we will have for all t ≤ T iterations of SGD that

Original: [Equation]
Revised: [Equation]

Original: The reason for Theorem 1’s additional κ 3 dependence beyond full-batch gradient descent is due to its need to maintain incoherence in its iterates.
Revised: The reason for Theorem 1’s additional κ 3 dependence beyond full-batch gradient descent is due to its need to maintain incoherence in its iterates.

Original: Using standard techniques on martingale concentration, one can readily show that SGD replicates a single iteration of full-batch gradient descent over an epoch of d 2 iterations.
Revised: Using standard techniques on martingale concentration, one can readily show that SGD replicates a single iteration of full-batch gradient descent over an epoch of d 2 iterations.

Original: This results in an iteration count O ( κ · d 2 log(1 /ϵ )) with an optimal dependence on κ , but the entire matrix is already fully observed after collecting d 2 samples.
Revised: This results in an iteration count O ( κ · d 2 log(1 /ϵ )) with an optimal dependence on κ , but the entire matrix is already fully observed after collecting d 2 samples.

Original: Instead,
Revised: Instead,

Original: Jin et al.
Revised: Jin et al.

Original: [10] noted that the variance of SGD iterations is controlled by the step-size α times the maximum coherence µ X = dr · max i,t
Revised: [10] noted that the variance of SGD iterations is controlled by the step-size α times the maximum coherence µ X = dr · max i,t

Original: ∥ e Ti X t ∥ 2 over the iterates X t , X t − 1 , . .
Revised: ∥ e Ti X t ∥ 2 over the iterates X t , X t − 1 , . .

Original: , X 0 .
Revised: , X 0 .

Original: If the iterates can be kept incoherent with µ X = O (1) , then SGD with a more aggressive step-size will reproduce an iteration of full-batch gradient descent after an epoch of just O
Revised: If the iterates can be kept incoherent with µ X = O (1) , then SGD with a more aggressive step-size will reproduce an iteration of full-batch gradient descent after an epoch of just O

Original: ( dr log d ) iterations.
Revised: ( dr log d ) iterations.

Original: The main finding in Jin et al.
Revised: The main finding in Jin et al.

Original: [10]’s proof of Theorem 1 is that the stochasticity of SGD is enough to keep the iterates incoherent.
Revised: [10]’s proof of Theorem 1 is that the stochasticity of SGD is enough to keep the iterates incoherent.

Original: This contrasts with full-batch methods at the time, which required an added regularizer [20, 30, 45] or an explicit projection step [9].
Revised: This contrasts with full-batch methods at the time, which required an added regularizer [20, 30, 45] or an explicit projection step [9].

Original: (As pointed out by a reviewer, it was later shown by Ma et al.
Revised: (As pointed out by a reviewer, it was later shown by Ma et al.

Original: [46] that full-batch gradient descent is also able to maintain incoherence without a regularizer nor a projection.)
Revised: [46] that full-batch gradient descent is also able to maintain incoherence without a regularizer nor a projection.)

Original: Unfortunately, maintaining incoherence requires shrinking the step-size by a factor of κ , and the actual value of µ X that results is also a factor of κ 2 worse than the original coherence µ of the ground truth Z .
Revised: Unfortunately, maintaining incoherence requires shrinking the step-size by a factor of κ , and the actual value of µ X that results is also a factor of κ 2 worse than the original coherence µ of the ground truth Z .

Original: The resulting iteration count O ( κ 4 · dr log( d/ϵ )) is made optimal with respect to d, r, and 1 /ϵ , but only at the cost of worsening its the dependence on the condition number κ by another three exponents .
Revised: The resulting iteration count O ( κ 4 · dr log( d/ϵ )) is made optimal with respect to d, r, and 1 /ϵ , but only at the cost of worsening its the dependence on the condition number κ by another three exponents .

Original: Finally, the quality of the initial point X 0 also has a dependence on the condition number κ .
Revised: Finally, the quality of the initial point X 0 also has a dependence on the condition number κ .

Original: In order to guarantee linear convergence, Theorem 1 requires X 0 to lie in the neighborhood
Revised: In order to guarantee linear convergence, Theorem 1 requires X 0 to lie in the neighborhood

Original: ∥
Revised: ∥

Original: X 0 X T 0 − ZZ T ∥
Revised: X 0 X T 0 − ZZ T ∥

Original: F < λ min ( Z T Z ) =
Revised: F < λ min ( Z T Z ) =

Original: O ( κ − 1 ) .
Revised: O ( κ − 1 ) .

Original: This dependence on κ is optimal, because fullbatch gradient descent must lose its ability to converge linearly in the limit κ → ∞ [5, 6].
Revised: This dependence on κ is optimal, because fullbatch gradient descent must lose its ability to converge linearly in the limit κ → ∞ [5, 6].

Original: However, the leading constant can be very pessmistic, because the theorem must formally exclude spurious critical points X spur that have ∇ f ( X spur )
Revised: However, the leading constant can be very pessmistic, because the theorem must formally exclude spurious critical points X spur that have ∇ f ( X spur )

Original: = 

Revised: =

Original: but f ( X spur ) > 0 in order to be provably correct.
Revised: but f ( X spur ) > 0 in order to be provably correct.

Original: In practice, it is commonly observed that SGD converges globally, starting from an arbitrary, possibly random initialization [30], at a linear rate that is consistent with local convergence theorems like Theorem 1.
Revised: In practice, it is commonly observed that SGD converges globally, starting from an arbitrary, possibly random initialization [30], at a linear rate that is consistent with local convergence theorems like Theorem 1.

Original: It is now commonly argued that gradient methods can escape saddle points with high probability [47], and so their performance is primarily dictated by local convergence behavior [48, 49].
Revised: It is now commonly argued that gradient methods can escape saddle points with high probability [47], and so their performance is primarily dictated by local convergence behavior [48, 49].

Original:  Proposed algorithm and main result
Revised:  Proposed algorithm and main result

Original: Inspired by a recent full-batch gradient method called ScaledGD [4, 9] and a closely related algorithm PrecGD [5], we proposed the following algorithm
Revised: Inspired by a recent full-batch gradient method called ScaledGD [4, 9] and a closely related algorithm PrecGD [5], we proposed the following algorithm

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: As we mentioned in the introduction, the preconditioner P =
Revised: As we mentioned in the introduction, the preconditioner P =

Original: ( X T X ) − 1 can be precomputed and cached in a practical implementation, and afterwards efficiently updated using the Sherman–
Revised: ( X T X ) − 1 can be precomputed and cached in a practical implementation, and afterwards efficiently updated using the Sherman–

Original: Morrison formula.
Revised: Morrison formula.

Original: The per-iteration cost of ScaledSGD is O ( r 2 ) arithmetic operations and 1 random sample, which for modest values of r is only marginally more than the cost of SGD.
Revised: The per-iteration cost of ScaledSGD is O ( r 2 ) arithmetic operations and 1 random sample, which for modest values of r is only marginally more than the cost of SGD.

Original: Our main result in this paper is that, with a region sufficiently close to the ground truth, this simple rescaling allows ScaledSGD to converge linearly to ϵ -accuracy O ( dr log( d/ϵ )) iterations, with no further dependence on the condition number κ .
Revised: Our main result in this paper is that, with a region sufficiently close to the ground truth, this simple rescaling allows ScaledSGD to converge linearly to ϵ -accuracy O ( dr log( d/ϵ )) iterations, with no further dependence on the condition number κ .

Original: This iteration count is optimal with respect to d, r, 1 /ϵ, and κ , and in fact matches SGD with a perfectly conditioned ground truth κ = 1 .
Revised: This iteration count is optimal with respect to d, r, 1 /ϵ, and κ , and in fact matches SGD with a perfectly conditioned ground truth κ = 1 .

Original: In our numerical experiments, we observe that ScaledSGD converges globally from a random initialization at the same rate as SGD as if κ = 1 .
Revised: In our numerical experiments, we observe that ScaledSGD converges globally from a random initialization at the same rate as SGD as if κ = 1 .

Original: Theorem 2 (Main) .
Revised: Theorem 2 (Main) .

Original: For Z ∈ R d × r with σ max ( Z ) = 1 and f ( X ) =
Revised: For Z ∈ R d × r with σ max ( Z ) = 1 and f ( X ) =

Original: ∥
Revised: ∥

Original: XX T − ZZ T ∥
Revised: XX T − ZZ T ∥

Original: 2 F and g i ( X ) =
Revised: 2 F and g i ( X ) =

Original: e Ti X ( X T X ) − 1 X T e i , select a radius ρ < 1 / 2 and set
Revised: e Ti X ( X T X ) − 1 X T e i , select a radius ρ < 1 / 2 and set

Original: [Equation]
Revised: [Equation]

Original: For an initial point X 0 ∈ R d × r that satifies f
Revised: For an initial point X 0 ∈ R d × r that satifies f

Original: ( X 0 ) ≤ 12 f max and max
Revised: ( X 0 ) ≤ 12 f max and max

Original: i
Revised: i

Original: g
Revised: g

Original: i
Revised: i

Original: ( X 0 ) ≤ 12 g max , there exists some constant c such that for any learning rate α < c · [( g max + ρ ) · d 2 log d ] − 1 , with probability at least 1 − T/d 10 , we will have for all t ≤ T iterations of ScaledSGD that:
Revised: ( X 0 ) ≤ 12 g max , there exists some constant c such that for any learning rate α < c · [( g max + ρ ) · d 2 log d ] − 1 , with probability at least 1 − T/d 10 , we will have for all t ≤ T iterations of ScaledSGD that:

Original: [Equation]
Revised: [Equation]

Original: Theorem 2 eliminates all dependencies on the condition number κ in Theorem 1 except for the quality of the initial point, which we had already noted earlier as being optimal.
Revised: Theorem 2 eliminates all dependencies on the condition number κ in Theorem 1 except for the quality of the initial point, which we had already noted earlier as being optimal.

Original: Our main finding is that it is possible to maintain incoherence while making aggressive step-sizes towards a highly ill-conditioned ground truth ZZ T .
Revised: Our main finding is that it is possible to maintain incoherence while making aggressive step-sizes towards a highly ill-conditioned ground truth ZZ T .

Original: In fact, Theorem 2 says that, with high probability, the maximum coherence µ X over of any iterate X t will only be a mild constant factor of ≈ 16 times worse than the coherence µ of the ground truth ZZ T .
Revised: In fact, Theorem 2 says that, with high probability, the maximum coherence µ X over of any iterate X t will only be a mild constant factor of ≈ 16 times worse than the coherence µ of the ground truth ZZ T .

Original: This is particularly surprising in view of the fact that every iteration of ScaledSGD involves inverting a potentially highly ill-conditioned matrix ( X T X ) − 1 .
Revised: This is particularly surprising in view of the fact that every iteration of ScaledSGD involves inverting a potentially highly ill-conditioned matrix ( X T X ) − 1 .

Original: In contrast, even without inverting matrices, Theorem 1 says that SGD is only able to keep µ X within a factor of κ 2 of µ , and only by shrinking the step-size α by another factor of κ .
Revised: In contrast, even without inverting matrices, Theorem 1 says that SGD is only able to keep µ X within a factor of κ 2 of µ , and only by shrinking the step-size α by another factor of κ .

Original: However, the price we pay for maintaining incoherence is that the quality of the initial point X 0 now gains a dependence on dimension d , in addition to the condition number κ .
Revised: However, the price we pay for maintaining incoherence is that the quality of the initial point X 0 now gains a dependence on dimension d , in addition to the condition number κ .

Original: In order to guarantee fast linear convergence independent of κ , Theorem 2 requires X 0 to lie in the neighborhood ∥
Revised: In order to guarantee fast linear convergence independent of κ , Theorem 2 requires X 0 to lie in the neighborhood ∥

Original: X 0 X T 0 −
Revised: X 0 X T 0 −

Original: ZZ T ∥
Revised: ZZ T ∥

Original: F < µrλ min ( Z T Z ) /d =
Revised: F < µrλ min ( Z T Z ) /d =

Original: ( κd ) − 1 , so that ρ can be set to be the same order of magnitude as g max .
Revised: ( κd ) − 1 , so that ρ can be set to be the same order of magnitude as g max .

Original: In essence, the “effective” condition number of the ground truth has been worsened by another factor of d .
Revised: In essence, the “effective” condition number of the ground truth has been worsened by another factor of d .

Original: This shrinks the size of our local neighborhood by a factor of d , but has no impact on the convergence rate of the resulting iterations.
Revised: This shrinks the size of our local neighborhood by a factor of d , but has no impact on the convergence rate of the resulting iterations.

Original: In the limit that κ → ∞ and the search rank r becomes overparameterized with respect to the true rank
Revised: In the limit that κ → ∞ and the search rank r becomes overparameterized with respect to the true rank

Original: r ⋆ < r of ZZ T , both full-batch gradient descent and SGD slows down to a sublinear convergence rate, in theory and in practice [5, 6].
Revised: r ⋆ < r of ZZ T , both full-batch gradient descent and SGD slows down to a sublinear convergence rate, in theory and in practice [5, 6].

Original: While Theorem 2 is no longer applicable, we observe in our numerical experiments that ScaledSGD nevertheless maintains its fast linear convergence rate as if κ = 1 .
Revised: While Theorem 2 is no longer applicable, we observe in our numerical experiments that ScaledSGD nevertheless maintains its fast linear convergence rate as if κ = 1 .

Original: Following PrecGD [5], we believe that introducing a small identity perturbation to the scaling matrix of ScaledSGD, as in ( X T X + ηI ) − 1 for some η ≈ (cid:112) f
Revised: Following PrecGD [5], we believe that introducing a small identity perturbation to the scaling matrix of ScaledSGD, as in ( X T X + ηI ) − 1 for some η ≈ (cid:112) f

Original: ( X ) , should be enough to rigorously extend Theorem 2 to the overparameterized regime.
Revised: ( X ) , should be enough to rigorously extend Theorem 2 to the overparameterized regime.

Original: We leave this extension as future work.
Revised: We leave this extension as future work.

Original:  Key ideas for the proof
Revised:  Key ideas for the proof

Original: We begin by explaining the mechanism by which SGD slows down when converging towards an ill-conditioned ground truth.
Revised: We begin by explaining the mechanism by which SGD slows down when converging towards an ill-conditioned ground truth.

Original: Recall that
Revised: Recall that

Original: [Equation]
Revised: [Equation]

Original: As XX T converges towards an ill-conditioned ground truth
Revised: As XX T converges towards an ill-conditioned ground truth

Original: [Equation]
Revised: [Equation]

Original: the factor matrix X must become progressively ill-conditioned, with
Revised: the factor matrix X must become progressively ill-conditioned, with

Original: [Equation]
Revised: [Equation]

Original: Therefore, it is possible for components of the error vector XX T − ZZ T to become “invisible” by aligning within the ill-conditioned subspaces of X .
Revised: Therefore, it is possible for components of the error vector XX T − ZZ T to become “invisible” by aligning within the ill-conditioned subspaces of X .

Original: As SGD progresses towards the solution, these ill-conditioned subspaces of X become the slowest components of the error vector to converge to zero.
Revised: As SGD progresses towards the solution, these ill-conditioned subspaces of X become the slowest components of the error vector to converge to zero.

Original: On the other hand, the maximum step-size that can be taken is controlled by the most wellconditioned subspaces of X .
Revised: On the other hand, the maximum step-size that can be taken is controlled by the most wellconditioned subspaces of X .

Original: A simple idea, therefore, is to rescale the ill-conditioned components of the gradient ∇ f ( X ) in order to make the ill-conditioned subspaces of X more “visible”.
Revised: A simple idea, therefore, is to rescale the ill-conditioned components of the gradient ∇ f ( X ) in order to make the ill-conditioned subspaces of X more “visible”.

Original: More concretely, define the local norm of the gradient as ∥∇ f ( X ) ∥
Revised: More concretely, define the local norm of the gradient as ∥∇ f ( X ) ∥

Original: X = ∥∇ f
Revised: X = ∥∇ f

Original: ( X )( X T X ) 1 / 2 ∥ F and its corresponding dual norm as ∥∇ f ( X ) ∥ ∗ X = ∥∇ f
Revised: ( X )( X T X ) 1 / 2 ∥ F and its corresponding dual norm as ∥∇ f ( X ) ∥ ∗ X = ∥∇ f

Original: ( X )( X T X ) − 1 / 2 ∥ F .
Revised: ( X )( X T X ) − 1 / 2 ∥ F .

Original: It has long been known (see e.g. [23, 44]) that rescaling the gradient yields
Revised: It has long been known (see e.g. [23, 44]) that rescaling the gradient yields

Original: [Equation]
Revised: [Equation]

Original: where θ is the angle between the error vector XX T − ZZ T and the linear subspace { XY T + Y X T :
Revised: where θ is the angle between the error vector XX T − ZZ T and the linear subspace { XY T + Y X T :

Original: Y ∈ R d × r } .
Revised: Y ∈ R d × r } .

Original: This insight immediately suggests an iteration like X + =
Revised: This insight immediately suggests an iteration like X + =

Original: X − α ∇ f ( X )( X T X ) − 1 .
Revised: X − α ∇ f ( X )( X T X ) − 1 .

Original: In fact, the gradients of f have some Lipschitz constant L , so
Revised: In fact, the gradients of f have some Lipschitz constant L , so

Original: [Equation]
Revised: [Equation]

Original: for
Revised: for

Original: [Equation]
Revised: [Equation]

Original: However, a naive analysis finds that L X =
Revised: However, a naive analysis finds that L X =

Original: L/λ min ( X T X )
Revised: L/λ min ( X T X )

Original: ≈ L · κ , and this causes the step-size to shrink by a factor of κ .
Revised: ≈ L · κ , and this causes the step-size to shrink by a factor of κ .

Original: The main motivating insight behind ScaledGD [4, 9] and later PrecGD [5] is that, with a finer analysis, it is possible to prove Lipschitz continuity under a local change of norm.
Revised: The main motivating insight behind ScaledGD [4, 9] and later PrecGD [5] is that, with a finer analysis, it is possible to prove Lipschitz continuity under a local change of norm.

Original: Lemma 3 (Function descent) .
Revised: Lemma 3 (Function descent) .

Original: Let X, Z ∈ R n × r satisfy ∥ XX T − ZZ T ∥ F ≤ ρλ min ( Z T Z ) where ρ < 1 / 2 .
Revised: Let X, Z ∈ R n × r satisfy ∥ XX T − ZZ T ∥ F ≤ ρλ min ( Z T Z ) where ρ < 1 / 2 .

Original: Then, the function f ( X ) =
Revised: Then, the function f ( X ) =

Original: ∥
Revised: ∥

Original: XX T − ZZ T ∥
Revised: XX T − ZZ T ∥

Original: 2 F satisfies
Revised: 2 F satisfies

Original: [Equation]
Revised: [Equation]

Original: for all ∥ V ∥ X ≤ C · (cid:112) f
Revised: for all ∥ V ∥ X ≤ C · (cid:112) f

Original: ( X ) with L X = 6 + 8 C + 2 C 2 =
Revised: ( X ) with L X = 6 + 8 C + 2 C 2 =

Original: O (1 + C 2 ) .
Revised: O (1 + C 2 ) .

Original: This same idea can be “stochastified” in a straightforward manner.
Revised: This same idea can be “stochastified” in a straightforward manner.

Original: Conditioning on the current iterate X , then the new iterate X + = X − αSG ( X )( X T X ) − 1 has expectation
Revised: Conditioning on the current iterate X , then the new iterate X + = X − αSG ( X )( X T X ) − 1 has expectation

Original: [Equation]
Revised: [Equation]

Original: The linear term evaluates as E [ SG ( X )( X T X ) − 1 ] = ∇ f ( X )
Revised: The linear term evaluates as E [ SG ( X )( X T X ) − 1 ] = ∇ f ( X )

Original: ( X T X ) − 1 , while the quadratic term is
Revised: ( X T X ) − 1 , while the quadratic term is

Original: [Equation]
Revised: [Equation]

Original: where g i ( X ) =
Revised: where g i ( X ) =

Original: e Ti X ( X T X ) − 1 X T e i =
Revised: e Ti X ( X T X ) − 1 X T e i =

Original: ( ∥ e Ti X ∥ ∗ X ) 2 .
Revised: ( ∥ e Ti X ∥ ∗ X ) 2 .

Original: Combined, we obtain geometric convergence
Revised: Combined, we obtain geometric convergence

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We see that the step-size depends crucially on the incoherence
Revised: We see that the step-size depends crucially on the incoherence

Original: g i ( X ) ≤ g max of the current iterate.
Revised: g i ( X ) ≤ g max of the current iterate.

Original: If the current iterate X is incoherent with g max
Revised: If the current iterate X is incoherent with g max

Original: = O (1 /d ) , then a step-size of α
Revised: = O (1 /d ) , then a step-size of α

Original: = O (1 /d ) is possible, resulting in convergence in O ( dr log( d/ϵ )) iterations, which can be shown using standard martingale techniques [10].
Revised: = O (1 /d ) is possible, resulting in convergence in O ( dr log( d/ϵ )) iterations, which can be shown using standard martingale techniques [10].

Original: But if the current iterate is g max
Revised: But if the current iterate is g max

Original: = O (1) , then only a step-size of α
Revised: = O (1) , then only a step-size of α

Original: = O (1 /d 2 ) is possible, which forces us to compute d 2 iterations, thereby obviating the need to complete the matrix in the first place.
Revised: = O (1 /d 2 ) is possible, which forces us to compute d 2 iterations, thereby obviating the need to complete the matrix in the first place.

Original: Therefore, in order for prove rapid linear convergence, we need to additionally show that with high probability, the coherence g k ( X )=(
Revised: Therefore, in order for prove rapid linear convergence, we need to additionally show that with high probability, the coherence g k ( X )=(

Original: ∥
Revised: ∥

Original: e Tk X ∥ ∗ X ) 2 remains O (1) throughout ScaledGD iterations.
Revised: e Tk X ∥ ∗ X ) 2 remains O (1) throughout ScaledGD iterations.

Original: This is the most challenging part of our proof.
Revised: This is the most challenging part of our proof.

Original: Previous methods that applied a similar scaling to fullbatch GD [9] required an explicit projection onto the set of incoherent matrices at each iteration.
Revised: Previous methods that applied a similar scaling to fullbatch GD [9] required an explicit projection onto the set of incoherent matrices at each iteration.

Original: Applying a similar projection to ScaledSGD will take O ( d ) time, which destroys the scalability of our method.
Revised: Applying a similar projection to ScaledSGD will take O ( d ) time, which destroys the scalability of our method.

Original: On the other hand, Jin et al.
Revised: On the other hand, Jin et al.

Original: [10] showed that the randomness in SGD is enough to keep the coherence of the iterates within a factor of κ 2 times worse than the coherence of the ground truth, and only by a step-size of at most α
Revised: [10] showed that the randomness in SGD is enough to keep the coherence of the iterates within a factor of κ 2 times worse than the coherence of the ground truth, and only by a step-size of at most α

Original: = O ( κ − 1 ) .
Revised: = O ( κ − 1 ) .

Original: Surprisingly, here we show that the randomness in ScaledSGD is enough to keep the coherence of the iterates with a constant factor of the coherence the ground truth, using a step-size with no dependence on κ .
Revised: Surprisingly, here we show that the randomness in ScaledSGD is enough to keep the coherence of the iterates with a constant factor of the coherence the ground truth, using a step-size with no dependence on κ .

Original: The following key lemma is the crucial insight of our proof.
Revised: The following key lemma is the crucial insight of our proof.

Original: First, it says that function g k ( X ) satisfies a “descent lemma” with respect to the local norm ∥ · ∥ ∗ X .
Revised: First, it says that function g k ( X ) satisfies a “descent lemma” with respect to the local norm ∥ · ∥ ∗ X .

Original: Second, and much more importantly, it says that descending g k ( X ) along the scaled gradient direction ∇ f ( X )( X T X ) −incurs a linear decrement 1 − 2 ρ 1 − ρ g k ( X ) with no dependence of the condition number κ .
Revised: Second, and much more importantly, it says that descending g k ( X ) along the scaled gradient direction ∇ f ( X )( X T X ) −incurs a linear decrement 1 − 2 ρ 1 − ρ g k ( X ) with no dependence of the condition number κ .

Original: This is in direct analogy to the function value decrement in (6), which has no dependence on κ , and in direct contrast to the proof of Jin et al.
Revised: This is in direct analogy to the function value decrement in (6), which has no dependence on κ , and in direct contrast to the proof of Jin et al.

Original: [10], which is only able to achieve a decrement of (8 /κ )
Revised: [10], which is only able to achieve a decrement of (8 /κ )

Original: g k ( X ) due to the lack of rescaling by ( X T X ) − 1 .
Revised: g k ( X ) due to the lack of rescaling by ( X T X ) − 1 .

Original: Lemma 4 (Coherence descent) .
Revised: Lemma 4 (Coherence descent) .

Original: Let g k ( X ) =
Revised: Let g k ( X ) =

Original: e Tk X
Revised: e Tk X

Original: ( X T X ) − 1 X T e k .
Revised: ( X T X ) − 1 X T e k .

Original: Under the same conditions as
Revised: Under the same conditions as

Original: Lemma 3, we have
Revised: Lemma 3, we have

Original: [Equation]
Revised: [Equation]

Original: Conditioning on X , we have for the search direction V =
Revised: Conditioning on X , we have for the search direction V =

Original: SG ( X )( X T X ) − 1 and X + = X
Revised: SG ( X )( X T X ) − 1 and X + = X

Original: + V
Revised: + V

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: g k ( X
Revised: g k ( X

Original: [Equation]
Revised: [Equation]

Original: ] ≤
Revised: ] ≤

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: X
Revised: X

Original: [Equation]
Revised: [Equation]

Original: X
Revised: X

Original: [Equation]
Revised: [Equation]

Original: , E [ V
Revised: , E [ V

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ζα
Revised: ζα

Original: [Equation]
Revised: [Equation]

Original: g
Revised: g

Original: [Equation]
Revised: [Equation]

Original: X ) + α ·
Revised: X ) + α ·

Original: [Equation]
Revised: [Equation]

Original: 2 g max for α
Revised: 2 g max for α

Original: = O ( ρ − 1 d − 2 ) .
Revised: = O ( ρ − 1 d − 2 ) .

Original: [Equation]
Revised: [Equation]

Original: It then follows that g k ( X + ) converges geometrically towards 12 g max in expectation, with a convergence rate (1 − ζα ) that is independent of the condition number κ :
Revised: It then follows that g k ( X + ) converges geometrically towards 12 g max in expectation, with a convergence rate (1 − ζα ) that is independent of the condition number κ :

Original: [Equation]
Revised: [Equation]

Original: The proof of Theorem 2 then follows from standard techniques, by making the two decrement conditions (6) and (7) into supermartingales and applying a standard concentration inequality.
Revised: The proof of Theorem 2 then follows from standard techniques, by making the two decrement conditions (6) and (7) into supermartingales and applying a standard concentration inequality.

Original: We defer the rigorous proof to appendix E.
Revised: We defer the rigorous proof to appendix E.

Original:  Experimental validation
Revised:  Experimental validation

Original: In this section we compare the practical performance of ScaledSGD and SGD for the RMSE loss function in Theorem 2 and two real-world loss functions: the pairwise RMSE loss used to complete
Revised: In this section we compare the practical performance of ScaledSGD and SGD for the RMSE loss function in Theorem 2 and two real-world loss functions: the pairwise RMSE loss used to complete

Original: Euclidean Distance Matrices (EDM) in wireless communication networks; and the Bayesian Personalized Ranking (BRP) loss used to generate personalized item recommendation in collaborative filtering.
Revised: Euclidean Distance Matrices (EDM) in wireless communication networks; and the Bayesian Personalized Ranking (BRP) loss used to generate personalized item recommendation in collaborative filtering.

Original: In each case, ScaledSGD remains highly efficient since it only updates two or three rows at a time, and the preconditioner P can be computed through low-rank updates, for a per-iteration cost of O ( r 2 ) .
Revised: In each case, ScaledSGD remains highly efficient since it only updates two or three rows at a time, and the preconditioner P can be computed through low-rank updates, for a per-iteration cost of O ( r 2 ) .

Original: All of our experiments use random Gaussian initializations and an initial P = σ 2 I .
Revised: All of our experiments use random Gaussian initializations and an initial P = σ 2 I .

Original: To be able to accurately measure and report the effects of ill-conditioning on ScaledSGD and SGD, we focus on small-scale synthetic datasets in the first two experiments, for which the ground truth is explicitly known, and where the condition numbers can be finely controlled.
Revised: To be able to accurately measure and report the effects of ill-conditioning on ScaledSGD and SGD, we focus on small-scale synthetic datasets in the first two experiments, for which the ground truth is explicitly known, and where the condition numbers can be finely controlled.

Original: In addition, to gauge the scalability of ScaledSGD on large-scale real-world datasets, in the third experiment, we apply
Revised: In addition, to gauge the scalability of ScaledSGD on huge-scale real-world datasets, in the third experiment, we apply

Original: ScaledSGD to generate personalized item recommendation using MovieLens25M dataset [50], for which the underlying item-item matrix has more than 62,000 items and 100 million pairwise samples are used during training.
Revised: ScaledSGD to generate personalized item recommendation using MovieLens25M dataset [50], for which the underlying item-item matrix has more than 62,000 items and 100 million pairwise samples are used during training.

Original: (Due to space constraints, we defer the details on the experimental setup, mathematical formulations, and the actual update equations to Appendix A.)
Revised: (Due to space constraints, we defer the details on the experimental setup, mathematical formulations, and the actual update equations to Appendix A.)

Original: The code for all experiments are available at https://github.com/Hong-Ming/ScaledSGD .
Revised: The code for all experiments are available at https://github.com/Hong-Ming/ScaledSGD .

Original: Matrix completion with RMSE loss.
Revised: Matrix completion with RMSE loss.

Original: The problem formulation is discussed in Section 3.
Revised: The problem formulation is discussed in Section 3.

Original: Figureplots the error f ( X ) =
Revised: Figureplots the error f ( X ) =

Original: ∥
Revised: ∥

Original: XX T − M ∥ 2 F as the number of epochs increases.
Revised: XX T − M ∥ 2 F as the number of epochs increases.

Original: As expected, in the well-conditioned case, both ScaledSGD and SGD converges to machine error at roughly the same linear rate.
Revised: As expected, in the well-conditioned case, both ScaledSGD and SGD converges to machine error at roughly the same linear rate.

Original: However, in the ill-conditioned case, SGD slows down significantly while ScaledSGD converges at almost exactly the same rate as in the well-conditioned case.
Revised: However, in the ill-conditioned case, SGD slows down significantly while ScaledSGD converges at almost exactly the same rate as in the well-conditioned case.

Original: [Figure]
Revised: [Figure]

Revised: ScaledSGDSGD

Original: Euclidean distance matrix (EDM) completion.
Revised: Euclidean distance matrix (EDM) completion.

Original: The Euclidean distance matrix (EDM) is a matrix of pairwise distance between d points in Euclidean space [51].
Revised: The Euclidean distance matrix (EDM) is a matrix of pairwise distance between d points in Euclidean space [51].

Original: In applications such as wireless sensor networks, estimation of unknown distances, i.e., completing the EDM is often required.
Revised: In applications such as wireless sensor networks, estimation of unknown distances, i.e., completing the EDM is often required.

Original: We emphasize that this loss function is a pairwise loss, meaning that each measurement indexes multiple elements of the ground truth matrix.
Revised: We emphasize that this loss function is a pairwise loss, meaning that each measurement indexes multiple elements of the ground truth matrix.

Original: To demonstrate the efficacy of ScaledSGD, we conduct two experiments where D is wellconditioned and ill-conditioned respectively:
Revised: To demonstrate the efficacy of ScaledSGD, we conduct two experiments where D is wellconditioned and ill-conditioned respectively:

Original: Experiment 1.
Revised: Experiment 1.

Original: We uniformly sample 30 points in a cube center at origin with side length 2 , and use them to compute the ground truth EDM D .
Revised: We uniformly sample 30 points in a cube center at origin with side length 2 , and use them to compute the ground truth EDM D .

Original: In this case, each row x i ∈ R 3 corresponds to the coordinates of the i -th sample.
Revised: In this case, each row x i ∈ R 3 corresponds to the coordinates of the i -th sample.

Original: The corresponding matrix
Revised: The corresponding matrix

Original: X ∈ R 30 × 3 is well-conditioned because of the uniform sampling.
Revised: X ∈ R 30 × 3 is well-conditioned because of the uniform sampling.

Original: Experiment 2.
Revised: Experiment 2.

Original: The ground truth
Revised: The ground truth

Original: EDM is generated with 25 samples lie in the same cube in experiment 1, and 5 samples lie far away from the the cube.
Revised: EDM is generated with 25 samples lie in the same cube in experiment 1, and 5 samples lie far away from the the cube.

Original: These five outliers make the corresponding X become ill-conditioned.
Revised: These five outliers make the corresponding X become ill-conditioned.

Original: [Figure]
Revised: [Figure]

Original: Item-item collaborative filtering (CF).
Revised: Item-item collaborative filtering (CF).

Original: In the task of item-item collaborative filtering (CF), the ground truth M is an d × d matrix where d is the number of items we wish to rank and the i, j -th of
Revised: In the task of item-item collaborative filtering (CF), the ground truth M is an d × d matrix where d is the number of items we wish to rank and the i, j -th of

Original: M is a similarity measure between the items.
Revised: M is a similarity measure between the items.

Original: Our goal is to learn a low-rank matrix that preserves the ranking of similarity between the items.
Revised: Our goal is to learn a low-rank matrix that preserves the ranking of similarity between the items.

Original: For instance, given a pairwise sample ( i, j, k ) , if item i is more similar to item j than item k , then M ij > M ik .
Revised: For instance, given a pairwise sample ( i, j, k ) , if item i is more similar to item j than item k , then M ij > M ik .

Original: We want to learn a low-rank matrix that also has this property, i.e., the i, j -th entry is greater than the i, k -th entry.
Revised: We want to learn a low-rank matrix that also has this property, i.e., the i, j -th entry is greater than the i, k -th entry.

Original: To gauge the scalability of ScaledSGD on a large-scale real-world dataset, we perform simulation on item-item collaborative filtering using a 62 , 000 × 62 , 000 item-item matrix M obtained from
Revised: To gauge the scalability of ScaledSGD on a huge-scale real-world dataset, we perform simulation on item-item collaborative filtering using a 62 , 000 × 62 , 000 item-item matrix M obtained from

Original: MovieLens25M dataset.
Revised: MovieLens25M dataset.

Original: The CF model is trained using Bayesian Personalized Ranking (BRP) loss [1] on a training set, which consists of 100 million pairwise samples in M .
Revised: The CF model is trained using Bayesian Personalized Ranking (BRP) loss [1] on a training set, which consists of 100 million pairwise samples in M .

Original: The performance of CF model is evaluated using Area Under the  Curve (AUC) score [1] on a test set, which consists ofmillion pairwise samples in M .
Revised: The performance of CF model is evaluated using Area Under the ROC Curve (AUC) score [1] on a test set, which consists of 10 million pairwise samples in M .

Original: Figure 3 plots the training BPR loss and testing AUC score within the first epoch (filled with red) and the second epoch (filled with blue).
Revised: Figure 3 plots the training BPR loss and testing AUC score within the first epoch (filled with red) and the second epoch (filled with blue).

Original: In order to measure the efficacy of ScaledSGD, we compare its testing AUC score against a standard baseline NP-Maximum: the upper bound on the best achievable
Revised: In order to measure the efficacy of ScaledSGD, we compare its testing AUC score against a standard baseline NP-Maximum: the upper bound on the best achievable

Original: AUC score by non-personalized ranking methods, in which the same ranking is used for all items.
Revised: AUC score by non-personalized ranking methods, in which the same ranking is used for all items.

Original: Based on the results in Figure 3, we emphasize two important points in the figure.
Revised: Based on the results in Figure 3, we emphasize two important points in the figure.

Original: First, the percentage of training samples needed for ScaledSGD to achieve the same testing AUC scores as NP-Maximum is roughly 4 times smaller than SGD.
Revised: First, the percentage of training samples needed for ScaledSGD to achieve the same testing AUC scores as NP-Maximum is roughly 4 times smaller than SGD.

Original: Though both ScaledSGD and SGD are able to achieve higher AUC score than NP-Maximum before finishing the first epoch, ScaledSGD achieve the same AUC score as NP-Maximum after training on 11% of training samples while SGD requires 46% of them.
Revised: Though both ScaledSGD and SGD are able to achieve higher AUC score than NP-Maximum before finishing the first epoch, ScaledSGD achieve the same AUC score as NP-Maximum after training on 11% of training samples while SGD requires 46% of them.

Original: We note that in this experiment, the size of the training set is 100 million, this means that SGD would require 35 million more iterations than ScaledSGD before it can reach
Revised: We note that in this experiment, the size of the training set is 100 million, this means that SGD would require 35 million more iterations than ScaledSGD before it can reach

Original: NP-Maximum.
Revised: NP-Maximum.

Original: Second, the percentage of training samples needed for ScaledSGD to converge after the first epoch is also roughly 4 times smaller than SGD.
Revised: Second, the percentage of training samples needed for ScaledSGD to converge after the first epoch is also roughly 4 times smaller than SGD.

Original: Given that both ScaledSGD and SGD converge to AUC score at around 0 .
Revised: Given that both ScaledSGD and SGD converge to AUC score at around 0 .

Original: 9 within the second epoch (area filled with blue), we indicate the percentage of training samples when both algorithms reach 0 .
Revised: 9 within the second epoch (area filled with blue), we indicate the percentage of training samples when both algorithms reach 0 .

Original: 9 AUC score in Figure 3.
Revised: 9 AUC score in Figure 3.

Original: As expected, ScaledSGD is able to converge using fewer samples than SGD, with only 16% of training samples.
Revised: As expected, ScaledSGD is able to converge using fewer samples than SGD, with only 16% of training samples.

Original: SGD, on the other hand, requires 72% training samples.
Revised: SGD, on the other hand, requires 72% training samples.

Original:  Conclusions
Revised:  Conclusions

Original: We propose an algorithm called ScaledSGD for huge scale online matrix completion.
Revised: We propose an algorithm called ScaledSGD for huge scale online matrix completion.

Original: For the nonconvex approach to solving matrix completion, ill-conditioning in the ground truth causes SGD to slow down significantly.
Revised: For the nonconvex approach to solving matrix completion, ill-conditioning in the ground truth causes SGD to slow down significantly.

Original: ScaledSGD preserves all the favorable qualities of SGD while making it immune to ill-conditioning.
Revised: ScaledSGD preserves all the favorable qualities of SGD while making it immune to ill-conditioning.

Original: For the RMSE loss, we prove that with an initial point close to the ground truth, ScaledSGD converges to an ϵ -accurate solution in O (log(1 /ϵ )) iterations, independent of the condition number κ .
Revised: For the RMSE loss, we prove that with an initial point close to the ground truth, ScaledSGD converges to an ϵ -accurate solution in O (log(1 /ϵ )) iterations, independent of the condition number κ .

Original: We also run numerical experiments on a wide range of other loss functions commonly used in applications such as collaborative filtering, distance matrix recovery, etc.
Revised: We also run numerical experiments on a wide range of other loss functions commonly used in applications such as collaborative filtering, distance matrix recovery, etc.

Original: We find that ScaledSGD achieves similar acceleration on these losses, which means that it is widely applicable to many real problems.
Revised: We find that ScaledSGD achieves similar acceleration on these losses, which means that it is widely applicable to many real problems.

Original: It remains future work to provide rigorous justification for these observations.
Revised: It remains future work to provide rigorous justification for these observations.

Original: A Supplemental Details on Experiments in Main Paper A.1 Experimental setup and datasets used
Revised: A Supplemental Details on Experiments in Main Paper A.1 Experimental setup and datasets used

Original: Simulation environment We implement both ScaledSGD and SGD in MATLAB (version
Revised: Simulation environment. We implement both ScaledSGD and SGD in MATLAB (version

Original: R2021a).
Revised: R2021a).

Original: All simulations in this paper are performed on a computer with Apple silicon M1 pro chip with 10-core CPU, 16-core GPU, and 32GB of RAM.
Revised: All simulations in this paper are performed on a computer with Apple silicon M1 pro chip with 10-core CPU, 16-core GPU, and 32GB of RAM.

Original: Datasets The datasets we use for the experiments in the main paper are described below.
Revised: Datasets. The datasets we use for the experiments in the main paper are described below

Original: • Matrix completion with RMSE loss :
Revised: • Matrix completion with RMSE loss:

Original: In the simulationresult
Revised: In the simulation

Original:  shown in Figure 1,
Revised: result shown in Figure 1,

Original: we synthetically generate both the well-conditioned and ill-conditioned ground truth matrix M .
Revised: we synthetically generate both the well-conditioned and ill-conditioned ground truth matrix M .

Original: We fix both M to be a rank-3 matrix of size 30 × 30 .
Revised: We fix both M to be a rank-3 matrix of size 30 × 30 .

Original: To generate M , we sample a random orthonormal matrix U ∈ R 30 × 3 and set M = USU T .
Revised: To generate M , we sample a random orthonormal matrix U ∈ R 30 × 3 and set M = USU T .

Original: For well-conditioned case, we set S = diag(2 , 2 , 2) , thus M is perfectly conditioned with κ = 1 .
Revised: For well-conditioned case, we set S = diag(2 , 2 , 2) , thus M is perfectly conditioned with κ = 1 .

Original: For ill-conditioned case, we let S = diag(10 , 10 − 1 , 10 − 3 ) , so that M is ill-conditioned with κ = 10 4 .
Revised: For ill-conditioned case, we let S = diag(10 , 10 − 1 , 10 − 3 ) , so that M is ill-conditioned with κ = 10 4 .

Original: • Euclidean distance matrix completion : In this simulation shown in Figure 2, the ground truth Euclidean distance matrix D for experiments 1 and 2 are generated with respect to their sample matrix X as D ij =
Revised: • Euclidean distance matrix completion: In this simulation shown in Figure 2, the ground truth Euclidean distance matrix D for experiments 1 and 2 are generated with respect to their sample matrix X as D ij =

Original: ∥ x i − x j ∥ 2 2 .
Revised: ∥ x i − x j ∥ 22 .

Original: For the sample points in Experiment 1 , we randomly sample (without replacement) 30 points in 3-dimensional cube centered at origin with side length 2 , and the corresponding sample matrix X has conditioned number κ 
Revised: For the sample points in Experiment 1 , we randomly sample (without replacement) 30 points in 3-dimensional cube centered at origin with side length 2 , and the corresponding sample matrix X has conditioned number κ = 1 .

Original: = 1 .

Original: For the sample points in Experiment 2 , we take the first 5 sample points in experiment 1 and perturb its x-coordinate by 10 , and keep the rest of the 25 samples intact.
Revised: For the sample points in Experiment 2 , we take the first 5 sample points in experiment 1 and perturb its x-coordinate by 10 , and keep the rest of the 25 samples intact.

Original: The corresponding sample matrix X has conditioned number κ = 8 . 0828 .
Revised: The corresponding sample matrix X has conditioned number κ = 8 . 0828 .

Original: • Item-item collaborative filtering :
Revised: • Item-item collaborative filtering:

Original: In this simulation shown in Figure 3, we use the MovieLens25M dataset [50], which is a standard benchmark for algorithms for recommendation systems.
Revised: In this simulation shown in Figure 3, we use the MovieLens25M dataset [50], which is a standard benchmark for algorithms for recommendation systems.

Original: This dataset consists of 25 million ratings over 62,000 movies by 162,000 users, the ratings are stored in an user-item matrix G whose ( i, j ) -th entry is the rating that the i -th user gives the j -th movie.
Revised: This dataset consists of 25 million ratings over 62,000 movies by 162,000 users, the ratings are stored in an user-item matrix G whose ( i, j ) -th entry is the rating that the i -th user gives the j -th movie.

Original: The rating is from 1 to 5 , where a higher score indicates a stronger preference.
Revised: The rating is from 1 to 5 , where a higher score indicates a stronger preference.

Original: If the i, j -th entry is 0 , then no rating is given.
Revised: If the i, j -th entry is 0 , then no rating is given.

Original: For the simulation of item-item collaborative filtering, the ( i, j ) -th entry of the ground truth item-item matrix M contains the similarity score between the item i and j , and can be computed by measuring cosine similarity between the i -th and j -th column of G 
Revised: For the simulation of item-item collaborative filtering, the ( i, j ) -th entry of the ground truth item-item matrix M is the similarity score between the item i and j , which can be computed by measuring cosine similarity between the i -th and j -th column of G .

Original: The

Original: details are described in A.4.

Original: Hyperparameter and initialization
Revised: Hyperparameter and initialization.

Original: We start SGD and PrecSGD at the same initial point in each simulation.
Revised: We start ScaledSGD and SGD at the same initial point in each simulation.

Original: The initial points for each simulation are drawn from the standard Gaussian distribution.
Revised: The initial points for each simulation are drawn from the standard Gaussian distribution.

Original: • Matrix completion with RMSE loss : The step-size for both ScaledSGD and SGD are set to be α = 0 .
Revised: • Matrix completion with RMSE loss: The step-size for both ScaledSGD and SGD are set to be α = 0 .

Original: The search rank for both ScaledSGD and SGD are set to be r = 3 .
Revised: The search rank for both ScaledSGD and SGD are set to be r = 3 .

Original: • Euclidean distance matrix completion : Since SGD is only stable for small step-size in EDM completion problem, while ScaledSGD can tolerance larger step-sizes, we pick the largest possible step-size for ScaledSGD and SGD in both experiments.
Revised: • Euclidean distance matrix completion: Since SGD is only stable for small step-size in EDM completion problem, while ScaledSGD can tolerance larger step-sizes, we pick the largest possible step-size for ScaledSGD and SGD in both experiments.

Original: Experiment 1 : step-size for ScaledSGD α = 0 . 2 , step-size for SGD α = 0 .
Revised: Experiment 1 : step-size for ScaledSGD α = 0 . 2 , step-size for SGD α = 0 .

Original: Experiment 2 : stepsize for ScaledSGD α = 0 .
Revised: Experiment 2 : stepsize for ScaledSGD α = 0 .

Original: 2 , step-size for SGD α = 0 .
Revised: 2 , step-size for SGD α = 0 .

Original: The search rank for both ScaledSGD and SGD are set to be r = 3 .
Revised: The search rank for both ScaledSGD and SGD are set to be r = 3 .

Original: • Item-item collaborative filtering : Similar to EDM completion problem, SGD is only stable for small step-size, we  pick  the largest possible step-size for both ScaledSGD and SGD.
Revised: • Item-item collaborative filtering: The step-sizes for this experiment are set as follows: we first pick a small step-size and train the CF model over a sufficient number of epochs, this allows us to estimate the best achievable AUC score; we then set the step-sizes for both ScaledSGD and SGD to the largest possible step-size for which ScaledSGD and SGD is able to converge to the best achievable AUC score, respectively.

Original: The step-size for ScaledSGD α = 1 , 000 , step-size for SGD α = 0 .
Revised: The step-size for ScaledSGD α = 1 , 000 , step-size for SGD α = 0 .

Original: The search rank for both ScaledSGD and SGD are set to be r = 3 .
Revised: The search rank for both ScaledSGD and SGD are set to be r = 3 .

Original: A.2 Matrix completion with RMSE loss
Revised: A.2 Matrix completion with RMSE loss

Original: We now turn to the practical aspects of implementing ScaledSGD for RMSE loss function.
Revised: We now turn to the practical aspects of implementing ScaledSGD for RMSE loss function.

Original: In practical setting, suppose we are given a set Ω
Revised: In practical setting, suppose we are given a set Ω

Original: = { ( i, j ) } that contains indices for which we know the value of M ij , our goal is to recover the missing elements in M by solving the following nonconvex 
Revised: = { ( i, j ) } that contains indices for which we know the value of M ij , our goal is to recover the missing elements in M by solving the following nonconvex optimization

Original: [Equation]
Revised: [Equation]

Original: The gradient of f ( X ) is
Revised: The gradient of f ( X ) is

Original: [Equation]
Revised: [Equation]

Original: ScaledGD update equations for RMSE loss Each iteration of ScaledSGD samples one element ( i,
Revised: ScaledSGD update equations for RMSE loss. Each iteration of ScaledSGD samples one element ( i,

Original: j ) ∈
Revised: j ) ∈

Original: Ω uniformly.
Revised: Ω uniformly.

Original: The resulting iteration updates only two rows of X
Revised: The resulting iteration updates only two rows of X

Original: [Equation]
Revised: [Equation]

Original: The update on P is low-rank
Revised: The update on P is low-rank

Original: [Equation]
Revised: [Equation]

Original: and can be computed by calling four times of rank-1 Sherman–Morrison–Woodbury (SMW) update formula in O ( r 2 ) time
Revised: and can be computed by calling four times of rank-1 Sherman–Morrison–Woodbury (SMW) update formula in O ( r 2 ) time

Original: [Equation]
Revised: [Equation]

Original: In practice, this low-rank update can be “pushed” onto a centralized storage of the preconditioner
Revised: In practice, this low-rank update can be “pushed” onto a centralized storage of the preconditioner

Original: P .
Revised: P .

Original: Heuristically, independent copies of P can be maintained by separate, distributed workers, and a centralized dispatcher can later merge the updates to P by simply adding the cumulative low-rank updates onto the existing centralized copy.
Revised: Heuristically, independent copies of P can be maintained by separate, distributed workers, and a centralized dispatcher can later merge the updates to P by simply adding the cumulative low-rank updates onto the existing centralized copy.

Original: A.3 Euclidean distance matrix (EDM) completion
Revised: A.3 Euclidean distance matrix (EDM) completion

Original: Suppose that we have d points x 1 , . . .
Revised: Suppose that we have d points x 1 , . . .

Original: , x d ∈ R r in r dimensional space, the Euclidean distance matrix D ∈ R d × d is a matrix of pairwise squared distance between d points in Euclidean space, namely, D ij =
Revised: , x d ∈ R r in r dimensional space, the Euclidean distance matrix D ∈ R d × d is a matrix of pairwise squared distance between d points in Euclidean space, namely, D ij =

Original: ∥ x i − x j ∥ 2 .
Revised: ∥ x i − x j ∥ 2 .

Original: Many applications, such as wireless sensor networks, communication and machine learning, require Euclidean distance matrix to provide necessary services.
Revised: Many applications, such as wireless sensor networks, communication and machine learning, require Euclidean distance matrix to provide necessary services.

Original: However, in practical scenario, entries in D that correspond to points far apart are often missing due to high uncertainty or equipment limitations in distance measurement.
Revised: However, in practical scenario, entries in D that correspond to points far apart are often missing due to high uncertainty or equipment limitations in distance measurement.

Original: The task of Euclidean distance matrix completion is to recover the missing entries in D from a set of available measurement, and this problem can be formulated as a rank r matrix completion problem with respect to pairwise square loss function.
Revised: The task of Euclidean distance matrix completion is to recover the missing entries in D from a set of available measurement, and this problem can be formulated as a rank r matrix completion problem with respect to pairwise square loss function.

Original: Specifically, let X ∈ R
Revised: Specifically, let X ∈ R

Original: d
Revised: d

Original: × r be a matrix containing x 1 , . . .
Revised: × r be a matrix containing x 1 , . . .

Original: , x d in its row and let
Revised: , x d in its row and let

Original: M =
Revised: M =

Original: XX T be the Grammian of X .
Revised: XX T be the Grammian of X .

Original: Each entry of D can be written in terms of three entries in M
Revised: Each entry of D can be written in terms of three entries in M

Original: [Equation]
Revised: [Equation]

Original: Hence, given a set of sample Ω = { ( i, j ) } in D , the pairwise square loss function for EDM completion reads
Revised: Hence, given a set of sample Ω = { ( i, j ) } in D , the pairwise square loss function for EDM completion reads

Original: [Equation]
Revised: [Equation]

Original: The gradient of f ( X ) is∇
Revised: The gradient of f ( X ) is

Original: [Equation]
Revised: [Equation]

Revised: X.

Original: ScaledGD update equations for EDM completion Each iteration of ScaledSGD samples one element ( i,
Revised: ScaledSGD update equations for EDM completion. Each iteration of ScaledSGD samples one element ( i,

Original: j ) ∈
Revised: j ) ∈

Original: Ω uniformly.
Revised: Ω uniformly.

Original: The resulting iteration updates only two rows of X
Revised: The resulting iteration updates only two rows of X

Original: [Equation]
Revised: [Equation]

Original: Similarly, the update on P is low-rank and can be computed by calling four times of equation (8).
Revised: Similarly, the update on P is low-rank and can be computed by calling four times of equation (8).

Original: A.4 Item-item collaborative filtering
Revised: A.4 Item-item collaborative filtering

Original: In the task of item-item collaborative filtering, the ground truth M is an d × d matrix where d is the number of items we wish to rank and the i, j -th of M is a similarity measure between the items.
Revised: In the task of item-item collaborative filtering, the ground truth M is an d × d matrix where d is the number of items we wish to rank and the i, j -th of M is a similarity measure between the items.

Original: Our goal is to learn a low-rank matrix that preserves the ranking of similarity between the items.
Revised: Our goal is to learn a low-rank matrix that preserves the ranking of similarity between the items.

Original: For instance, suppose that item i is more similar to item j than item k , then M ij > M ik , we want to learn a low-rank matrix XX T that also has this property, i.e., x Ti x j ≥ x Ti x k where x i is the i -th row of X .
Revised: For instance, suppose that item i is more similar to item j than item k , then M ij > M ik , we want to learn a low-rank matrix XX T that also has this property, i.e., x Ti x j ≥ x Ti x k where x i is the i -th row of X .

Original: Similarity score An important building block of item-item recommendation systems is the socalled item-item similarity matrix [52–54], which we denote by M .
Revised: Similarity score. An important building block of item-item recommendation systems is the socalled item-item similarity matrix [52–54], which we denote by M .

Original: The i, j -th entry of this matrix is the pairwise similarity scores of items i and j .
Revised: The i, j -th entry of this matrix is the pairwise similarity scores of items i and j .

Original: There are various measures of similarity.
Revised: There are various measures of similarity.

Original: In our experiments we adopt a common similarity measure known as cosine similarity [53].
Revised: In our experiments we adopt a common similarity measure known as cosine similarity [53].

Original: As a result, the item-item matrix can be computed from the user-item matrix.
Revised: As a result, the item-item matrix can be computed from the user-item matrix.

Original: In particular, let g i , g j denote the i -th and j -th columns of the user-item matrix G , corresponding to the ratings given by all users to the i -th and j -th items.
Revised: In particular, let g i , g j denote the i -th and j -th columns of the user-item matrix G , corresponding to the ratings given by all users to the i -th and j -th items.

Original: Then the ( i, j ) -th element of the item-item matrix M is set to
Revised: Then the ( i, j ) -th element of the item-item matrix M is set to

Original: M ij = g Ti g j / ( ∥ g i ∥∥ g j ∥ ) .
Revised: M ij = g Ti g j / ( ∥ g i ∥∥ g j ∥ ) .

Original: In general, the item-item matrix computed this way will be very sparse and not capable of generating good recommendations.
Revised: In general, the item-item matrix computed this way will be very sparse and not capable of generating good recommendations.

Original: Our goal is to complete the missing entries of this matrix, assuming that that M is low-rank.
Revised: Our goal is to complete the missing entries of this matrix, assuming that that M is low-rank.

Original: As we will see, we can formulate this completion problem as an optimization problem over the set of rank - r matrices.
Revised: As we will see, we can formulate this completion problem as an optimization problem over the set of rank - r matrices.

Original: Pairwise entropy loss (BPR loss)
Revised: Pairwise entropy loss (BPR loss).

Original: The Bayesian Personalized Ranking (BRP) loss [1] is a widely used loss function in the context of collaborative filtering.
Revised: The Bayesian Personalized Ranking (BRP) loss [1] is a widely used loss function in the context of collaborative filtering.

Original: For the task of predicting a personalized ranking of a set of items (videos, products, etc.), BRP loss often outperforms RMSE loss because it is directly optimized for ranking; most collaborative filtering models that use RMSE loss are essentially scoring each individual item based on user implicit feedbacks, in applications that only positive feedbacks are available, the models will not be able to learn to distinguish between negative feedbacks and missing entries.
Revised: For the task of predicting a personalized ranking of a set of items (videos, products, etc.), BRP loss often outperforms RMSE loss because it is directly optimized for ranking; most collaborative filtering models that use RMSE loss are essentially scoring each individual item based on user implicit feedbacks, in applications that only positive feedbacks are available, the models will not be able to learn to distinguish between negative feedbacks and missing entries.

Original: The BPR loss in this context can be defined as follows.
Revised: The BPR loss in this context can be defined as follows.

Original: Let Ω =
Revised: Let Ω =

Original: { ( i, j, k ) } denote a set of indices for which we observe the ranking of similarity between items i, j, k .
Revised: { ( i, j, k ) } denote a set of indices for which we observe the ranking of similarity between items i, j, k .

Original: Our observations are of the form Y ijk = 1 if M ij > M ik and Y ijk = 0 otherwise.
Revised: Our observations are of the form Y ijk = 1 if M ij > M ik and Y ijk = 0 otherwise.

Original: In other words, Y ijk = 1 if item i is more similar to item j than to item k .
Revised: In other words, Y ijk = 1 if item i is more similar to item j than to item k .

Original: We form a candidate matrix of the form XX T , where X ∈ R d × r .
Revised: We form a candidate matrix of the form XX T , where X ∈ R d × r .

Original: Our hope is that XX T preserves the ranking between the items.
Revised: Our hope is that XX T preserves the ranking between the items.

Original: The BPR loss function is designed to enforce this property.
Revised: The BPR loss function is designed to enforce this property.

Original: Let x i denote the i -th row of X and set z ijk = ( XX T ) ij − ( XX T )
Revised: Let x i denote the i -th row of X and set z ijk = ( XX T ) ij − ( XX T )

Original: ik = x
Revised: ik = x

Original: Ti ( x j − x k ) .
Revised: Ti ( x j − x k ) .

Original: The BPR loss attempts to preserve the ranking of samples in each row of M by minimizing the logistic loss with respect to ( Y ijk , σ ( z ijk )) , where σ ( · ) is the sigmoid function:
Revised: The BPR loss attempts to preserve the ranking of samples in each row of M by minimizing the logistic loss with respect to ( Y ijk , σ ( z ijk )) , where σ ( · ) is the sigmoid function:

Original: [Equation]
Revised: [Equation]

Original: Then the gradient of f ( X ) is
Revised: Then the gradient of f ( X ) is

Original: [Equation]
Revised: [Equation]

Revised: ScaledSGD update equations for BPR loss.

Original: Similarly to the previous section, each iteration of ScaledSGD samples one element ( i, j, k ) ∈
Revised: Similarly to the previous section, each iteration of ScaledSGD samples one element ( i, j , k ) ∈

Original: Ω uniformly.
Revised: Ω uniformly.

Original: The resulting iteration updates only three rows of X , as in
Revised: The resulting iteration updates only three rows of X , as in

Original: [Equation]
Revised: [Equation]

Original: Similar to before, the preconditioner P can be updated via six call of equation (8) in O ( r 2 ) time.
Revised: Similar to before, the preconditioner P can be updated via six call of equation (8) in O ( r 2 ) time.

Original: The AUC score The AUC score [1] is a popular evaluation metric for recommendation system.
Revised: The AUC score. The AUC score [1] is a popular evaluation metric for recommendation system.

Original: Roughly speaking, the AUC score of a candidate matrix XX T is the percentage of ranking of the entries of M that is preserved by XX T .
Revised: Roughly speaking, the AUC score of a candidate matrix XX T is the percentage of ranking of the entries of M that is preserved by XX T .

Original: Specifically, for each sample ( i, j, k ) ∈ Ω , we define a indicator variable δ ijk as
Revised: Specifically, for each sample ( i, j, k ) ∈ Ω , we define a indicator variable δ ijk as

Original: [Equation]
Revised: [Equation]

Original: and Y
Revised: ijk

Original: [Equation]
Revised: [Equation]

Original: if
Revised: and Y ijk

Original: [Equation]
Revised: [Equation]

Original: ijk ≤ 0 andY ijk =
Revised: and

Original: [Equation]
Revised: [Equation]

Original: otherwise , where we recall that Y ijk is our observation and z ijk = ( XX T ) ij − ( XX T ) ik .
Revised: where we recall that Y ijk is our observation and z ijk = ( XX T ) ij − ( XX T ) ik .

Original: In other words, δ ijk = 1 only if the ranking between M ij and M ik is preserved by ( XX T ) ij and ( XX T ) ik .
Revised: In other words, δ ijk = 1 only if the ranking between M ij and M ik is preserved by ( XX T ) ij and ( XX T ) ik .

Original: The
Revised: The

Original: AUC score is then defined as the ratio
Revised: AUC score is then defined as the ratio

Original: [Equation]
Revised: [Equation]

Original: Thus, a higher AUC score indicates that the candidate matrix XX T perserves a larger percentage of the pairwise comparisons in | Ω | .
Revised: Thus, a higher AUC score indicates that the candidate matrix XX T perserves a larger percentage of the pairwise comparisons in | Ω | .

Original:  CF modeltraining for Figure 3 We precompute a dataset  of  item-item pairwise comparisons M from the MovieLens25M datasets of user-item ratings, and then run ScaledSGD and SGD overepochs on this dataset.
Revised: Training a CF model for Figure 3. We precompute a dataset Ω of 110 million item-item pairwise comparisons using the user-item ratings from the MovieLens25M dataset, and then run ScaledSGD and SGD over 2 epochs on this dataset.

Original: Let d = 62 , 000 denote the number of items and let Ω ⊆
Revised: Let d = 62 , 000 denote the number of items and let Ω ⊆

Original: [ d ] 3 denote the set of observations, i.e., the set of entries ( i, j, k ) where we observe Y ijk = 1 if M ij > M ik and Y ijk = 0 if M ij < M ik .
Revised: [ d ] 3 denote the set of observations, i.e., the set of entries ( i, j, k ) where we observe Y ijk =if M ij > M ik and Y ijk = 0 if M ij < M ik .

Original: We construct Ω by sampling 110 million pairwise measurements  uniformly at random without replacement from ( i, j, k ) ∼ [ d ] 3that have either M ij > M ik or M ij < M ik .
Revised: We construct Ω by sampling 110 million pairwise measurements that have either M ij > M ik or M ij < M ik uniformly at random without replacement from ( i, j, k ) ∼ [ d ] 3 .

Original: We do this because the item-item matrix M remains highly sparse, and there are many pairs of ( i, j ) and ( i, k ) for which M ij = M ik = 0 .
Revised: We do this because the item-item matrix M remains highly sparse, and there are many pairs of ( i, j ) and ( i, k ) for which M ij = M ik = 0 .

Original: To ensure independence between training and testing, we divide the set Ω into two disjoint sets Ω train and Ω test .
Revised: To ensure independence between training and testing, we divide the set Ω into two disjoint sets Ω train and Ω test .

Original: The first set Ω train consists of 100 million of all observations, which we use to fit our model.
Revised: The first set Ω train consists of 100 million of all observations, which we use to fit our model.

Original: The second set Ω test consists of 10 million samples for which we use to calculate the AUC score of our model on new data.
Revised: The second set Ω test consists of 10 million samples for which we use to calculate the AUC score of our model on new data.

Revised: Upper bounds on the non-personalized ranking AUC score (NP-Maximum).

Original: B Additional Experiments on pointwise cross-entropy loss 
Revised: As oppose to personalized ranking method, non-personalized ranking method generate the same ranking for every pair of item j and k , independent of item i . In the context of item-item collaborative filtering, the non-personalized ranking method can be defined as follows. Given a set of pairwise comparisons Ω = ( { i, j, k } ) and observations Y ijk , we optimized the ranking between item j and k on a candidate vector x , where x ∈ R d . Let x i denote the i -th entry of x , the non-personalized ranking method attempts to preserve the ranking between the x j and x k by minimizing the logistic loss with respect to ( Y ijk , σ ( x j − x k )) where σ ( · ) is the sigmoid function:

Revised: [Equation]

Revised: The gradient of f ( x ) and the SGD update equations are

Revised: [Equation]

Revised: Notice that non-personalized ranking method is not a matrix completion problem, the regular SGD is used to minimized f ( x ) .

Revised: To find the upper bound on the non-personalized ranking AUC score, we directly optimized the non-personalized ranking on the test set Ω test , and evaluate the corresponding

Revised: AUC score on Ω test .

Revised: Since we perform both training and evaluation on Ω test , this corresponding AUC score is the upper bound on the best achievable AUC score on Ω test .

Revised: B Additional Experiments on pointwise cross-entropy loss

Original: This problem is also known as 1-bit matrix completion [55].
Revised: This problem is also known as 1-bit matrix completion [55].

Original: Here our goal is to recover a rank- r matrix M through binary measurements.
Revised: Here our goal is to recover a rank- r matrix M through binary measurements.

Original: Specifically, we are allowed to take independent measurements on every entry M ij , which we denote by Y ij .
Revised: Specifically, we are allowed to take independent measurements on every entry M ij , which we denote by Y ij .

Original: Let σ ( · ) denote the sigmoid function, then
Revised: Let σ ( · ) denote the sigmoid function, then

Original: Y ij = 1 with probability σ ( M ij ) and Y ij = 0 otherwise.
Revised: Y ij = 1 with probability σ ( M ij ) and Y ij = 0 otherwise.

Original: After a number of measurements are taken on each entry in the set Ω , let y ij denote the percentage of measurements on the i, j -th entry that is equal to 1 .
Revised: After a number of measurements are taken on each entry in the set Ω , let y ij denote the percentage of measurements on the i, j -th entry that is equal to 1 .

Original: The plan is to find the maximum likelihood estimator for M by minimizing a cross-entropy loss defined as follow
Revised: The plan is to find the maximum likelihood estimator for M by minimizing a cross-entropy loss defined as follow

Original: [Equation]
Revised: [Equation]

Original: We assume an ideal case where the number of measurements is large enough so that y ij = σ ( M ij ) and the entries are fully observed.
Revised: We assume an ideal case where the number of measurements is large enough so that y ij = σ ( M ij ) and the entries are fully observed.

Original: The gradient of f ( X ) is
Revised: The gradient of f ( X ) is

Original: [Equation]
Revised: [Equation]

Original: ScaledGD update equations for pointwise cross-entropy loss.
Revised: ScaledSGD update equations for pointwise cross-entropy loss.

Original: Each iteration of ScaledSGD samples one element ( i,
Revised: Each iteration of ScaledSGD samples one element ( i,

Original: j ) ∈
Revised: j ) ∈

Original: Ω uniformly.
Revised: Ω uniformly.

Original: The resulting iteration updates only two rows of X
Revised: The resulting iteration updates only two rows of X

Original: [Equation]
Revised: [Equation]

Original: The preconditioner P can be updated by calling four times of equation (8) as in RMSE loss.
Revised: The preconditioner P can be updated by calling four times of equation (8) as in RMSE loss.

Original: Matrix completion with pointwise cross-entropy loss.
Revised: [Table] with pointwise cross-entropy loss.

Original: We apply ScaledSGD to perform matrix completion through minimizing pointwise cross-entropy loss.
Revised: We apply ScaledSGD to perform matrix completion through minimizing pointwise cross-entropy loss.

Original: In this experiment, the wellconditioned and ill-conditioned ground truth matrix M is the same as those in Figure 1, and the process of data generation are described in A.1.
Revised: In this experiment, the wellconditioned and ill-conditioned ground truth matrix M is the same as those in Figure 1, and the process of data generation are described in A.1.

Original: The learning rate for both ScaledSGD and SGD are set to be α = 1 .
Revised: The learning rate for both ScaledSGD and SGD are set to be α = 1 .

Original: The search rank for both ScaledSGD and SGD are set to be r = 3 .
Revised: The search rank for both ScaledSGD and SGD are set to be r = 3 .

Original: Figure 4 plots the error f ( X ) =
Revised: Figure 4 plots the error f ( X ) =

Original: ∥
Revised: ∥

Original: XX T − M ∥ 2 F against the number of epochs.
Revised: XX T − M ∥ 2 F against the number of epochs.

Original: Observe that the results shown in Figure 4 are almost identical to that of the RMSE loss shown in Figure 1.
Revised: Observe that the results shown in Figure 4 are almost identical to that of the RMSE loss shown in Figure 1.

Original: Illconditioning causes SGD to slow down significantly while ScaledSGD is unaffected.
Revised: Illconditioning causes SGD to slow down significantly while ScaledSGD is unaffected.

Original: [Figure]
Revised: [Figure]

Original: C Additional Experiments with Noise
Revised: C Additional Experiments with Noise

Original: To mimic the real-world datasets, we corrupt each entry of ground truth matrix M by white Gaussian noise.
Revised: To mimic the real-world datasets, we corrupt each entry of ground truth matrix M by white Gaussian noise.

Original: We first generate a noiseless well-conditioned and ill-conditioned matrix ˜ M following same procedure as the one described in A.1.
Revised: We first generate a noiseless well-conditioned and ill-conditioned matrix ˜ M following same procedure as the one described in A.1.

Original: For well-conditioned case, we set the singular value as
Revised: For well-conditioned case, we set the singular value as

Original: S = diag(10 , 10 , 10) .
Revised: S = diag(10 , 10 , 10) .

Original: For the ill-conditioned case, we set S = diag(10 , 10 − 1 , 10 − 3 ) .
Revised: For the ill-conditioned case, we set S = diag(10 , 10 − 1 , 10 − 3 ) .

Original: To obtain a noisy ground truth, we generate a matrix of white Gaussian noise W corresponding to a fixed signal to noise ratio (SNR), which is defined as SNR = 20 log 10 ( ∥ ˜ M
Revised: To obtain a noisy ground truth, we generate a matrix of white Gaussian noise W corresponding to a fixed signal to noise ratio (SNR), which is defined as SNR = 20 log 10 ( ∥ ˜ M

Original: ∥ F / ∥
Revised: ∥ F / ∥

Original: W ∥ F ) .
Revised: W ∥ F ) .

Original: Finally, we set
Revised: Finally, we set

Original: M = ˜
Revised: M = ˜

Original: M + W .
Revised: M + W .

Original: For the experiments in this section, we set SNR = 15 dB.
Revised: For the experiments in this section, we set SNR = 15 dB.

Original: For the case of wellconditioned ˜ M , the resulting M = ˜
Revised: For the case of wellconditioned ˜ M , the resulting M = ˜

Original: M + W is full-rank with condition number κ = 310 .
Revised: M + W is full-rank with condition number κ = 310 .

Original: For the case of ill-conditioned ˜ M , the resulting M is full-rank with condition number κ = 423 . 5022 .
Revised: For the case of ill-conditioned ˜ M , the resulting M is full-rank with condition number κ = 423 . 5022 .

Original: Matrix completion with RMSE loss on noisy datasets We plot the convergence rate of
Revised: Matrix completion with RMSE loss on noisy datasets. We plot the convergence rate of

Original: ScaledSGD and SGD under the noisy setting in Figure 5.
Revised: ScaledSGD and SGD under the noisy setting in Figure 5.

Original: In this experiment, we pick a larger search rank r = 5 to accommodate the noisy ground truth.
Revised: In this experiment, we pick a larger search rank r = 5 to accommodate the noisy ground truth.

Original: Observe that SGD slows down in both the well-conditioned and ill-conditioned case due to the addition of white Gaussian noise and the larger search rank r , while ScaledSGD converge linearly toward the noise floor.
Revised: Observe that SGD slows down in both the well-conditioned and ill-conditioned case due to the addition of white Gaussian noise and the larger search rank r , while ScaledSGD converge linearly toward the noise floor.

Original: We also plot the noise floor, which can be computed as follows.
Revised: We also plot the noise floor, which can be computed as follows.

Original: First we take the eigendecomposition of M = Q Λ Q T , where Q is an orthonormal matrix and Λ is a diagonal matrix containing the eigenvalues of M sorted in descending order in its diagonal entries.
Revised: First we take the eigendecomposition of M = Q Λ Q T , where Q is an orthonormal matrix and Λ is a diagonal matrix containing the eigenvalues of M sorted in descending order in its diagonal entries.

Original: Let Λ ′ be a diagonal matrix such that Λ ′ ii =
Revised: Let Λ ′ be a diagonal matrix such that Λ ′ ii =

Original: Λ ii if i ≤ r , and Λ ′ ii = 

Revised: Λ ii if i ≤ r , and Λ ′ ii =

Original: otherwise, then the noise floor is defined as the RMSE between
Revised: otherwise, then the noise floor is defined as the RMSE between

Original: M and its best rank-
Revised: M and its best rank-

Original: r approximation M ′ =
Revised: r approximation M ′ =

Original: Q Λ ′ Q T , which is equal to 12 | Ω | (cid:80) ( i,j ) ∈ Ω (cid:0) M ′ ij − M ij (cid:1) 2 .
Revised: Q Λ ′ Q T , which is equal to 12 | Ω | (cid:80) ( i,j ) ∈ Ω (cid:0) M ′ ij − M ij (cid:1) 2 .

Original: The step-sizes in the simulation are set to be the largest possible step-sizes for which ScaledSGD and SGD can converge to the noise floor.
Revised: The step-sizes in the simulation are set to be the largest possible step-sizes for which ScaledSGD and SGD can converge to the noise floor.

Original: For ScaleSGD, the step-size is set to be α = 0 .
Revised: For ScaleSGD, the step-size is set to be α = 0 .

Original: For
Revised: For

Original: SGD, the step-size is set to be α = 0 .
Revised: SGD, the step-size is set to be α = 0 .

Original: SNR = 20 log 10 ( ∥ ˜ M ∥ F / ∥ W ∥ F ) = 15 dB. 
Revised: SNR = 20 log 10 ( ∥ ˜ M ∥ F / ∥ W ∥ F ) = 15 dB. [Figure]

Original:  Matrix completion with pointwise cross-entropy loss on noisy datasets We plot the convergence rate of ScaledSGD and SGD under the noisy setting in Figure 6.
Revised: ScaledSGDSGDNoise Floor Matrix completion with pointwise cross-entropy loss on noisy datasets. We plot the convergence rate of ScaledSGD and SGD under the noisy setting in Figure 6.

Original: Similar to RMSE loss in noisy setting, SGD show down in both well-conditioned and ill-conditioned case, while ScaledSGD converge linearly toward the noise floor.
Revised: Similar to RMSE loss in noisy setting, SGD show down in both well-conditioned and ill-conditioned case, while ScaledSGD converge linearly toward the noise floor.

Original: In this simulation, the search rank is set to be r = 5 .
Revised: In this simulation, the search rank is set to be r = 5 .

Original: The step-size are set to be the largest possible step-sizes for which ScaledSGD and SGD can converge[Figure] to the noise floor.
Revised: The step-size are set to be the largest possible step-sizes for which ScaledSGD and SGD can converge to the noise floor.

Original: For ScaleSGD, the step-size is set to be α = 0 .
Revised: For ScaleSGD, the step-size is set to be α = 0 .

Original: For SGD, the step-size is set to be α = 0 .
Revised: For SGD, the step-size is set to be α = 0 .

Original: SNR = 15 dB.
Revised: SNR = 15 dB.

Original: [Figure]
Revised: [Figure]

Original: D Additional simulation on item-item collaborative filtering
Revised: D Additional simulation on item-item collaborative filtering

Original: Finally, we perform  additional experiments on item-item collaborative filtering in order to compare the ability of ScaledSGD and SGD to generate good recommendations using matrix factorization.
Revised: Finally, we perform three additional experiments on item-item collaborative filtering in order to compare the ability of ScaledSGD and SGD to generate good recommendations using matrix factorization.

Original: Dataset For additional simulations on item-time collaborative filtering, we use the Small-scale and Full-scale MovieLens Latest datasets [50] in order to gauge the performance of our algorithm on different scales.
Revised: Dataset. For additional simulations on item-time collaborative filtering, we use the MovieLensLatest-Small and MovieLens-Latest-Full datasets [50] in order to gauge the performance of our algorithm on different scales.

Original: First, we run experiments on the Small-scale MovieLens Latest dataset that has 100,000 ratings over 9,000 movies by 600 users.
Revised: First, we run a small-scale experiment on the MovieLens-LatestSmall dataset that has 100,000 ratings over 9,000 movies by 600 users.

Original: Second, wealso run  experiments on the Full-scale MovieLens Latest dataset with 27 million total ratings over 58,000 movies by 280,000 users.
Revised: Second, we run two medium to large-scale experiments on the MovieLens-Latest-Full dataset with 27 million total ratings over 58,000 movies by 280,000 users.

Original: Experimental Setup
Revised: Experimental Setup.

Original: The process of training a collaborative filtering model is described in A.
Revised: The process of training a collaborative filtering model is described in A.

Original: The hyperparameters and initializations for  experiments in this section are described below
Revised: The hyperparameters and initializations for the three experiments in this section are described below.

Original: • Small-scale dataset We sample m =  | Ω  | = 500,000 pairwise observations.
Revised: • MovieLens-Latest-Small dataset: We sample | Ω train | = 1 million and | Ω test | = 100, pairwise observations for training and testing, respectively.

Original: We set our search rank to be r = 10 , so the optimization variable X is of size 4180 × 10 .
Revised: We set our search rank to be r = 3 , so the optimization variable X is of size 9000 × 3 .

Original: Both ScaledGD and SGD are initialized using a random Gaussian initial point.
Revised: Both ScaledSGD and SGD are initialized using a random Gaussian initial point.

Original: For ScaledSGD the step-size is 10 2 and for SGD the step-size is  10 − 1 .
Revised: For ScaledSGD the step-size is 10 3 and for SGD the step-size is 5 × 10 − 2 .

Original: • Full-scale dataset We sample m =  | Ω  | =2 .
Revised: • MovieLens-Latest-Full dataset: In the first experiment, we sample | Ω train | = 10 million and | Ω test | =

Original: 7 million pairwise observations in a first experiment, and m =  | Ω  | = 10 million pairwise observations in a second experiment.
Revised: 1 million pairwise observations for training and testing, respectively. In the second experiment, we sample | Ω train | = 30 million and | Ω test | = 3 million pairwise observations for training and testing, respectively.

Original: In both cases, we set our search rank to be r = 4 , so the optimization variable X is of size 59047 × 4 and 4321441 × 4 respectively.
Revised: In both cases, we set our search rank to be r = 3 , so the optimization variable X is of size 58000 × 3 .

Original: For ScaledSGD the step-size is  10 2 and for SGD the step-size is  10 − 1 .
Revised: For ScaledSGD the step-size is 5 × 10 3 and for SGD the step-size is 5 × 10 − 2 .

Original: Difference in learning rates between SGD and ScaledSGD In our experiments, the step-sizes for both ScaledSGD and SGD are set using a hyperparameter search.

Original: We observe that under identical conditions, ScaledSGD is able to tolerate a much larger step-size than SGD.

Original: Thus, we set the stepsize to be the largest magnitude possible for both methods.

Original: Results
Revised: Results.

Original: The results of our experiments forSGD and ScaledSGD  are plotted in Figures 7, 8, and 9.
Revised: The results of our experiments for ScaledSGD and SGD are plotted in Figures 7, 8, and 9.

Original: In all three cases, we see both algorithms converging towards a good fit before apparently overfitting.
Revised: In all three cases, ScaledSGD reachs the AUC scores that are greater than NP-Maximum’s within the first epoch, while SGD requires more than one epoch to achieve the same AUC score as NPMaximum’s in the small-scale (Figure 7) and medium-scale (Figure 8) setting.

Original: This is reflected in the fact that both algorithms decrease the training loss towards some minimum value before increasing the training loss again. Both algorithms have a comparable rate ofconvergence, although SGD is actually a little bit faster in the smaller examples. Crucially, we observe in all three cases that ScaledSGD attains a better asymptote  than SGD, which we suspect is due to its ability tobetter minimize the training loss.
Revised: In addition, of all three cases, ScaledSGD is able to converge to the asymptote of AUC score within the second epoch, while SGD needs more than 2 epochs to converge to the asymptote in the small-scale (Figure 7) and medium-scale (Figure 8) setting. These results demonstrates that ScaledSGD remain highly efficient across small-scale (Figure 7), medium-scale (Figure 8), large-scale(Figure 9) and hugescale (Figure 3) settings.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ScaledSGDSGD     50 0. ScaledSGDSGD     50 0. ScaledSGDSGD
Revised: [Figure]

Original: E Proof of the theoretical results
Revised: E Proof of the theoretical results

Original: In this section, we show that, in expectation, the search direction V = SG ( X )( X T X − 1 ) makes a geometric decrement to both the function value f and the incoherence g .
Revised: In this section, we show that, in expectation, the search direction V = SG ( X )( X T X − 1 ) makes a geometric decrement to both the function value f and the incoherence g .

Original: A key idea is to show that the size of the decrement in f is controlled by the coherence g max ≥ g k ( X ) of the current iterate, and this motivates the need to decrement g k in order to keep the iterates incoherent.
Revised: A key idea is to show that the size of the decrement in f is controlled by the coherence g max ≥ g k ( X ) of the current iterate, and this motivates the need to decrement g k in order to keep the iterates incoherent.

Original: Our key result is that both decrements are independent of the condition number κ .
Revised: Our key result is that both decrements are independent of the condition number κ .

Original: E.1 Preliminaries
Revised: E.1 Preliminaries

Original: We define the inner product between two matrices as ⟨ X, Y ⟩ def = tr( X T Y ) , which induces the
Revised: We define the inner product between two matrices as ⟨ X, Y ⟩ def = tr( X T Y ) , which induces the

Original: Frobenius norm as ∥ X ∥ F = (cid:112) ⟨ X, X ⟩ .
Revised: Frobenius norm as ∥ X ∥ F = (cid:112) ⟨ X, X ⟩ .

Original: The vectorization vec( X ) is the column-stacking operation that turns an m × n matrix into a length- mn vector; it preserves the matrix inner product ⟨ X, Y ⟩ = vec( X ) T vec( Y ) and the Frobenius norm ∥ vec( X ) ∥ =
Revised: The vectorization vec( X ) is the column-stacking operation that turns an m × n matrix into a length- mn vector; it preserves the matrix inner product ⟨ X, Y ⟩ = vec( X ) T vec( Y ) and the Frobenius norm ∥ vec( X ) ∥ =

Original: ∥ X ∥ F .
Revised: ∥ X ∥ F .

Original: We denote λ i ( M ) and σ i ( M ) as the i -th eigenvalue and singular value of a symmetric matrix
Revised: We denote λ i ( M ) and σ i ( M ) as the i -th eigenvalue and singular value of a symmetric matrix

Original: M = M T , ordered from the most positive to the most negative.
Revised: M = M T , ordered from the most positive to the most negative.

Original: We will often write λ max ( M ) and λ min ( M ) to index the most positive and most negative eigenvalues, and σ max ( M ) and σ min ( M ) for the largest and smallest singular values.
Revised: We will often write λ max ( M ) and λ min ( M ) to index the most positive and most negative eigenvalues, and σ max ( M ) and σ min ( M ) for the largest and smallest singular values.

Original: Recall for any matrix V ∈ R d × r , we define its local norm with respect to X ∈ R d × r as
Revised: Recall for any matrix V ∈ R d × r , we define its local norm with respect to X ∈ R d × r as

Original: [Equation]
Revised: [Equation]

Original: Also recall that we have defined the stochastic gradient operator
Revised: Also recall that we have defined the stochastic gradient operator

Original: [Equation]
Revised: [Equation]

Original: where ( i, j ) ∼ [ d ] 2 is selected uniformly at random.
Revised: where ( i, j ) ∼ [ d ] 2 is selected uniformly at random.

Original: This way SGD is written X + = X − αSG
Revised: This way SGD is written X + = X − αSG

Original: ( X ) and ScaledSGD is written X + = X − αSG
Revised: ( X ) and ScaledSGD is written X + = X − αSG

Original: ( X )( X T X ) − 1 for step-size α > 0 .
Revised: ( X )( X T X ) − 1 for step-size α > 0 .

Original: E.2 Function value convergence
Revised: E.2 Function value convergence

Original: Recall that Theorem 1, due to Jin et al.
Revised: Recall that Theorem 1, due to Jin et al.

Original: [10], says that SGD converges to ϵ accuracy in
Revised: [10], says that SGD converges to ϵ accuracy in

Original: O ( κ 4 log(1 /ϵ )) iterations, with a four-orders-of-magnitude dependence on the condition number κ .
Revised: O ( κ 4 log(1 /ϵ )) iterations, with a four-orders-of-magnitude dependence on the condition number κ .

Original: By comparison, our main result Theorem 2 says that ScaledSGD converges to ϵ accuracy in
Revised: By comparison, our main result Theorem 2 says that ScaledSGD converges to ϵ accuracy in

Original: O (log(1 /ϵ )) iterations, completely independence on the condition number κ .
Revised: O (log(1 /ϵ )) iterations, completely independence on the condition number κ .

Original: In this section, we explain how the first two factors of κ are eliminated, by considering the full-batch counterparts of these two algorithms.
Revised: In this section, we explain how the first two factors of κ are eliminated, by considering the full-batch counterparts of these two algorithms.

Original: First, consider full-batch gradient descent on the function f ( X ) =
Revised: First, consider full-batch gradient descent on the function f ( X ) =

Original: ∥
Revised: ∥

Original: XX T − ZZ T ∥ 2 F .
Revised: XX T − ZZ T ∥ 2 F .

Original: It follows from the local Lipschitz continuity of f that
Revised: It follows from the local Lipschitz continuity of f that

Original: [Equation]
Revised: [Equation]

Original: where X t +1 =
Revised: where X t +1 =

Original: X t − α ∇ f ( X t ) .
Revised: X t − α ∇ f ( X t ) .

Original: Here, the linear progress term determines the amount of progress that can proportionally be made with a sufficiently small step-size α , whereas the inverse step-size term basically controls how large the step-size can be.
Revised: Here, the linear progress term determines the amount of progress that can proportionally be made with a sufficiently small step-size α , whereas the inverse step-size term basically controls how large the step-size can be.

Original: In the case of full-batch gradient descent, it is long known that an X t that is sufficiently close to Z will satisfy the following
Revised: In the case of full-batch gradient descent, it is long known that an X t that is sufficiently close to Z will satisfy the following

Original: [Equation]
Revised: [Equation]

Original: and therefore, taking λ max ( Z T Z ) =1 and λ min ( Z T Z )
Revised: and therefore, taking λ max ( Z T Z ) =

Original:  = κ − 1 where κ is the condition number, we have linear convergence
Revised: [Equation] and λ min ( Z T Z ) = κ − 1 where κ is the condition number, we have linear convergence

Original: [Equation]
Revised: [Equation]

Original: for step-sizes of α ≤ 2 / ( κL ) .
Revised: for step-sizes of α ≤ 2 / ( κL ) .

Original: Therefore, it follows from this analysis that full-batch gradient descent takes T
Revised: Therefore, it follows from this analysis that full-batch gradient descent takes T

Original: = O ( κ 2 log(1 /ϵ )) iterations to converge to ϵ -accuracy.
Revised: = O ( κ 2 log(1 /ϵ )) iterations to converge to ϵ -accuracy.

Original: In this iteration count, one factor of κ arises from the linear progress term, which shrinks as O ( κ − 1 ) as κ grows large.
Revised: In this iteration count, one factor of κ arises from the linear progress term, which shrinks as O ( κ − 1 ) as κ grows large.

Original: The second factor of κ arises because the inverse step-size term is a factor of κ larger than the linear progress term, which restricts the maximum step-size to be no more than O ( κ − 1 ) .
Revised: The second factor of κ arises because the inverse step-size term is a factor of κ larger than the linear progress term, which restricts the maximum step-size to be no more than O ( κ − 1 ) .

Original: The following lemma, restated from the main text, shows that an analogous analysis for full-batch
Revised: The following lemma, restated from the main text, shows that an analogous analysis for full-batch

Original: ScaledSGD proves an iteration count of T
Revised: ScaledGD proves an iteration count of T

Original: = O (log(1 /ϵ ))
Revised: = O (log(1 /ϵ ))

Original: with no dependence on the condition number κ .
Revised: with no dependence on the condition number κ .

Original: In fact, it proves that full-batch ScaledSGD converges like full-batch gradient descent with a perfect condition number κ = 1 .
Revised: In fact, it proves that full-batch ScaledGD converges like full-batch gradient descent with a perfect condition number κ = 1 .

Original: Lemma 5 (Function descent, Lemma 3 restated) .
Revised: Lemma 5 (Function descent, Lemma 3 restated) .

Original: Let X, Z ∈ R n × r satisfy ∥ XX T − ZZ T ∥ F ≤ ρλ min ( Z T Z ) where ρ < 1 / 2 .
Revised: Let X, Z ∈ R n × r satisfy ∥ XX T − ZZ T ∥ F ≤ ρλ min ( Z T Z ) where ρ < 1 / 2 .

Original: Then, the function f ( X ) =
Revised: Then, the function f ( X ) =

Original: ∥
Revised: ∥

Original: XX T − ZZ T ∥
Revised: XX T − ZZ T ∥

Original: 2 F satisfies
Revised: 2 F satisfies

Original: [Equation]
Revised: [Equation]

Original: for all ∥ V ∥ X ≤ C · (cid:112) f
Revised: for all ∥ V ∥ X ≤ C · (cid:112) f

Original: ( X ) with L X = 6 + 8 C + 2 C 2 =
Revised: ( X ) with L X = 6 + 8 C + 2 C 2 =

Original: O (1 + C 2 ) .
Revised: O (1 + C 2 ) .

Original: It follows that the iteration X t +1 =
Revised: It follows that the iteration X t +1 =

Original: X t − α ∇ f ( X t )( X Tt X t ) − 1 yields
Revised: X t − α ∇ f ( X t )( X Tt X t ) − 1 yields

Original: [Equation]
Revised: [Equation]

Original: for step-sizes of α ≤ 2 /L X , where L X = 6 + 8(4) + 2(4) 2 .
Revised: for step-sizes of α ≤ 2 /L X , where L X = 6 + 8(4) + 2(4) 2 .

Original: Therefore, we conclude that full-batch
Revised: Therefore, we conclude that full-batch

Original: ScaledGD takes T
Revised: ScaledGD takes T

Original: = O (log(1 /ϵ )) iterations to converge to ϵ -accuracy, as if the condition number were perfectly κ = 1 .
Revised: = O (log(1 /ϵ )) iterations to converge to ϵ -accuracy, as if the condition number were perfectly κ = 1 .

Original: Note that Lemma 5 has been proved in both Tong et al.
Revised: Note that Lemma 5 has been proved in both Tong et al.

Original: [9] and Zhang et al.
Revised: [9] and Zhang et al.

Original: For completeness, we give a proof inspired by Zhang et al.
Revised: For completeness, we give a proof inspired by Zhang et al.

Original: Proof of Lemma 5.
Revised: Proof of Lemma 5.

Original: We prove (11) via a direct expansion of the quadratic
Revised: We prove (11) via a direct expansion of the quadratic

Original: [Equation]
Revised: [Equation]

Original: and it then follows by simple counting that
Revised: and it then follows by simple counting that

Original: [Equation]
Revised: [Equation]

Original: Now, from Weyl’s inequality that
Revised: Now, from Weyl’s inequality that

Original: [Equation]
Revised: [Equation]

Original: and therefore ∥ XX T − ZZ T ∥ [Equation] /λ min ( X T X ) ≤ ρ/ (1 − ρ ) ≤ 1 because ρ < 1 / 2 .
Revised: and therefore ∥ XX T − ZZ T ∥ F /λ min ( X T X ) ≤ ρ/ (1 − ρ ) ≤ 1 because ρ < 1 / 2 .

Original: It follows that
Revised: It follows that

Original: L 2 ≤ 6 .
Revised: L 2 ≤ 6 .

Original: If ∥ V ∥ X ≤ C · (cid:112) f
Revised: If ∥ V ∥ X ≤ C · (cid:112) f

Original: ( X ) , then L 3 

Revised: ( X ) , then L 3

Original: ∥ V ∥ X ≤ 8 C and L 4 

Revised: ∥ V ∥ X ≤ 8 C and L 4

Original: ∥ V ∥ 2 X ≤ 2 C 2 .
Revised: ∥ V ∥ 2 X ≤ 2 C 2 .

Original: For the upper-bound in (12), we have simply
Revised: For the upper-bound in (12), we have simply

Original: [Equation]
Revised: [Equation]

Original: For the lower-bound in (12), we evoke Zhang et al.
Revised: For the lower-bound in (12), we evoke Zhang et al.

Original: [5, Lemma 12] with RIP constant δ = 0 and regularization parameter η = 0 to yield
Revised: [5, Lemma 12] with RIP constant δ = 0 and regularization parameter η = 0 to yield

Original: [Equation]
Revised: [Equation]

Original: in which cos θ is defined between XX T − ZZ T and the set { XY T + Y X [Equation] Y ∈ R d × r } , as in
Revised: in which cos θ is defined between XX T − ZZ T and the set { XY T + Y X T : Y ∈ R d × r } , as in

Original: [Equation]
Revised: [Equation]

Original: It follows from Zhang et al.
Revised: It follows from Zhang et al.

Original: [5, Lemma 13] that
Revised: [5, Lemma 13] that

Original: [Equation]
Revised: [Equation]

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: for ρ < 1 /
Revised: for ρ < 1 /

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: E.3 Coherence convergence
Revised: E.3 Coherence convergence

Original: We now explain that ScaledSGD eliminates the last two factors of κ from SGD because it is able to keep its iterates a factor of κ 2 more incoherent.
Revised: We now explain that ScaledSGD eliminates the last two factors of κ from SGD because it is able to keep its iterates a factor of κ 2 more incoherent.

Original: First, consider regular SGD on the function f ( X ) =
Revised: First, consider regular SGD on the function f ( X ) =

Original: ∥
Revised: ∥

Original: XX T − ZZ T ∥ 2 F .
Revised: XX T − ZZ T ∥ 2 F .

Original: Conditioning on the current iterate, we have via the local Lipschitz continuity of f :
Revised: Conditioning on the current iterate, we have via the local Lipschitz continuity of f :

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where X + = X − αSG ( X ) .
Revised: where X + = X − αSG ( X ) .

Original: In expectation [Equation] the linear progress term of SGD coincides with that of full-batch gradient descent in (10).
Revised: In expectation, the linear progress term of SGD coincides with that of full-batch gradient descent in (10).

Original: The inverse step-size term, however [Equation] is up to a factor of d 2 times larger.
Revised: The inverse step-size term, however, is up to a factor of d 2 times larger.

Original: To see this, observe that
Revised: To see this, observe that

Original: [Equation]
Revised: [Equation]

Original: In a coarse analysis, we can simply bound max i ∥
Revised: In a coarse analysis, we can simply bound max i ∥

Original: e Ti X ∥ 2 F ≤ λ max ( X T X ) =
Revised: e Ti X ∥ 2 F ≤ λ max ( X T X ) =

Original: O (1) to yield 
Revised: O (1) to yield f

Original: [Equation]
Revised: [Equation]

Original: for step-sizes of α ≤ 4 / ( κd 2 ) .
Revised: for step-sizes of α ≤ 4 / ( κd 2 ) .

Original: Hence, we conclude that it takes T
Revised: Hence, we conclude that it takes T

Original: = O ( κ 2 d 2 log(1 /ϵ )
Revised: = O ( κ 2 d 2 log(1 /ϵ )

Original: ) iterations to converge to ϵ -accuracy, with an epoch of d 2 iterations of SGD essentially recreating a single iteration of full-batch gradient descent.
Revised: ) iterations to converge to ϵ -accuracy, with an epoch of d 2 iterations of SGD essentially recreating a single iteration of full-batch gradient descent.

Original: Unfortunately, the matrix is already fully observed after d 2 iterations, and so this result is essentially vacuous.
Revised: Unfortunately, the matrix is already fully observed after d 2 iterations, and so this result is essentially vacuous.

Original: Here, Jin et al.
Revised: Here, Jin et al.

Original: [10] pointed out that the term h max = max i ∥ e Ti X ∥ 2 F measures the coherence of the d × r iterate X , and can be as small as O (1 /d ) for small values of rank r = O (1) .
Revised: [10] pointed out that the term h max = max i ∥ e Ti X ∥ 2 F measures the coherence of the d × r iterate X , and can be as small as O (1 /d ) for small values of rank r = O (1) .

Original: Conditioned on the current iterate X , they observed that the function
Revised: Conditioned on the current iterate X , they observed that the function

Original: h
Revised: h

Original: i ( X ) =
Revised: i ( X ) =

Original: ∥
Revised: ∥

Original: e Ti X ∥ 2 F converges towards a finite value in expectation
Revised: e Ti X ∥ 2 F converges towards a finite value in expectation

Original: [Equation]
Revised: [Equation]

Original: Let us define γ as the ratio between the coherences of the ground truth Z and the iterate X :
Revised: Let us define γ as the ratio between the coherences of the ground truth Z and the iterate X :

Original: [Equation]
Revised: [Equation]

Original: Crucially, we require γ = κ 2 in order for h i ( X ) to converge towards 12 h max in expectation:
Revised: Crucially, we require γ = κ 2 in order for h i ( X ) to converge towards 12 h max in expectation:

Original: [Equation]
Revised: [Equation]

Original: As a consequence, we conclude that, while SGD is able to keep its iterates X incoherent, their actual coherence h max = max i ∥ e Ti X ∥ 2 F is up to a factor of κ 2 worse than the coherence max j ∥ e Tj Z ∥ 2 F of the ground truth Z .
Revised: As a consequence, we conclude that, while SGD is able to keep its iterates X incoherent, their actual coherence h max = max i ∥ e Ti X ∥ 2 F is up to a factor of κ 2 worse than the coherence max j ∥ e Tj Z ∥ 2 F of the ground truth Z .

Original: Using a standard supermartingale argument, Jin et al.
Revised: Using a standard supermartingale argument, Jin et al.

Original: [10] extended the analysis above to prove that if the ground truth Z has coherence max j ∥ e Tj Z ∥ 2 F = O (1 /d ) , then the SGD generates iterates X that have coherence max i ∥
Revised: [10] extended the analysis above to prove that if the ground truth Z has coherence max j ∥ e Tj Z ∥ 2 F = O (1 /d ) , then the SGD generates iterates X that have coherence max i ∥

Original: e Ti X ∥ 2 F ≤ h max
Revised: e Ti X ∥ 2 F ≤ h max

Original: = O
Revised: = O

Original: (( κ 2 /d ) log d ) , which is two factors worse in κ as expected.
Revised: (( κ 2 /d ) log d ) , which is two factors worse in κ as expected.

Original: Combined, this proves that SGD converges to ϵ accuracy in T
Revised: Combined, this proves that SGD converges to ϵ accuracy in T

Original: =
Revised: =

Original: O ( κ 4 dr log( d/ϵ )) iterations with the step-size of α
Revised: O ( κ 4 dr log( d/ϵ )) iterations with the step-size of α

Original: = O ( κ − 1 d − 1 h − 1max )
Revised: = O ( κ − 1 d − 1 h − 1max )

Original: and iterate coherence h max
Revised: and iterate coherence h max

Original: = O (( κ 2 /d )
Revised: = O (( κ 2 /d )

Original: log d ) , which is another two factors of κ worse than full-batch gradient descent.
Revised: log d ) , which is another two factors of κ worse than full-batch gradient descent.

Original: The following lemma, restated from the main text, shows that an analogous analysis for ScaledSGD proves that the algorithm maintains iterates X whose coherences have no dependence on κ .
Revised: The following lemma, restated from the main text, shows that an analogous analysis for ScaledSGD proves that the algorithm maintains iterates X whose coherences have no dependence on κ .

Original: Here, we need to define a different incoherence functiong
Revised: Here, we need to define a different incoherence function

Original: [Equation]
Revised: [Equation]

Original: in order to “stochastify” our previous analysis for full-batch ScaledGD.
Revised: )in order to “stochastify” our previous analysis for full-batch ScaledGD.

Original: Surprisingly, the factors of ( X T X ) − 1 in both the new definition of g i ( X ) and the search direction SG ( X )( X T X ) − 1 do not hurt incoherence, but in fact improves it.
Revised: Surprisingly, the factors of ( X T X ) − 1 in both the new definition of g i ( X ) and the search direction SG ( X )( X T X ) − 1 do not hurt incoherence, but in fact improves it.

Original: Lemma 6 (Coherence descent, Lemma 4 restated) .
Revised: Lemma 6 (Coherence descent, Lemma 4 restated) .

Original: Let X, Z ∈ R n × r satisfy ∥ XX T − ZZ T ∥ F ≤ ρλ min ( Z T Z ) where ρ < 1 / 2 .
Revised: Let X, Z ∈ R n × r satisfy ∥ XX T − ZZ T ∥ F ≤ ρλ min ( Z T Z ) where ρ < 1 / 2 .

Original: Then, the functions f ( X ) =
Revised: Then, the functions f ( X ) =

Original: ∥
Revised: ∥

Original: XX T − ZZ T ∥
Revised: XX T − ZZ T ∥

Original: 2 F and g k ( X ) =
Revised: 2 F and g k ( X ) =

Original: e Tk X
Revised: e Tk X

Original: ( X T X ) − 1 X T
Revised: ( X T X ) − 1 X T

Original: e
Revised: e

Original: k satisfy
Revised: k satisfy

Original: [Equation]
Revised: [Equation]

Original: Conditioning on X , we have for the search direction V =
Revised: Conditioning on X , we have for the search direction V =

Original: SG ( X )( X T X ) − 1 and X + = X
Revised: SG ( X )( X T X ) − 1 and X + = X

Original: + V
Revised: + V

Original: [Equation]
Revised: [Equation]

Original: max for α = O (ρ − 1 d
Revised: for α = O (

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where ζ = 1 − 2 ρ 1 − ρ .
Revised: (16) where ζ = 1 − 2 ρ 1 − ρ .

Original: It then follows that g k ( X + ) converges geometrically towards 12 g max in expectation, with a convergence rate (1 − ζα ) that is independent of the condition number κ :
Revised: It then follows that g k ( X + ) converges geometrically towards 12 g max in expectation, with a convergence rate (1 − ζα ) that is independent of the condition number κ :

Original: [Equation]
Revised: [Equation]

Original: Before we prove Lemma 6, we first need to prove a simple claim.
Revised: Before we prove Lemma 6, we first need to prove a simple claim.

Original: Lemma 7 (Change of norm) .
Revised: Lemma 7 (Change of norm) .

Original: The local norm ∥ V ∥ ∗ X = ∥
Revised: The local norm ∥ V ∥ ∗ X = ∥

Original: V ( X T X ) − 1 / 2 ∥ satisfies
Revised: V ( X T X ) − 1 / 2 ∥ satisfies

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The upper-bound follows because
Revised: The upper-bound follows because

Original: [Equation]
Revised: [Equation]

Original: where P Y =
Revised: where P Y =

Original: ( Y T Y ) − 1 and P X =
Revised: ( Y T Y ) − 1 and P X =

Original: ( X T X ) − 1 and therefore
Revised: ( X T X ) − 1 and therefore

Original: P 1 / 2 X P−
Revised: P 1 / 2 X P

Original: [Equation]
Revised: [Equation]

Original: Y P1 /
Revised: P

Original: [Equation]
Revised: [Equation]

Original: X ⪰ I + P 1 / 2 X

Original: [ X T
Revised: X ⪰

Original: [Equation]
Revised: [Equation]

Original: Y − X ) +
Revised: P

Original: [Equation]
Revised: [Equation]

Original: Y − X ) T X
Revised: 2 X [ X

Original: ] P1 /
Revised: [Equation] P

Original: [Equation]
Revised: [Equation]

Original: X and σ max [ P 1 / 2 X [ X T ( Y − X ) P 1 / 2 X ] ≤ ∥ Y − X ∥ X because XP 1 / 2 X is orthonormal.
Revised: X and σ max [ P 1 / 2 X [ X T ( Y − X ) P 1 / 2 X ] ≤ ∥ Y − X ∥ X because XP 1 / 2 X is orthonormal.

Original: The lower-bound follows similarly.
Revised: The lower-bound follows similarly.

Original: We are ready to prove Lemma 6.
Revised: We are ready to prove Lemma 6.

Original: Proof of Lemma 6.
Revised: Proof of Lemma 6.

Original: It follows from the intermediate value version of Taylor’s theorem that there exists some ˜ X
Revised: It follows from the intermediate value version of Taylor’s theorem that there exists some ˜ X

Original: = X + tV with t ∈ [0 , 1] that
Revised: = X + tV with t ∈ [0 , 1] that

Original: [Equation]
Revised: [Equation]

Original: Let P =
Revised: Let P =

Original: ( X T X ) − 1 and U =
Revised: ( X T X ) − 1 and U =

Original: e i e Ti
Revised: e i e Ti

Original: ( I − XPX T ) V and G =
Revised: ( I − XPX T ) V and G =

Original: V PX T e i e Ti .
Revised: V PX T e i e Ti .

Original: By direct computation, we have
Revised: By direct computation, we have

Original: [Equation]
Revised: [Equation]

Original: by differentiating XP and XPX T respectively.
Revised: by differentiating XP and XPX T respectively.

Original: A coarse count yields 12 ⟨∇ 2 g i ( X )
Revised: A coarse count yields 12 ⟨∇ 2 g i ( X )

Original: [ V ] , V ⟩ ≤ 5( ∥ V ∥ ∗ X ) 2 and therefore
Revised: [ V ] , V ⟩ ≤ 5( ∥ V ∥ ∗ X ) 2 and therefore

Original: [Equation]
Revised: [Equation]

Original: which is the first claim.
Revised: which is the first claim.

Original: Now, observe that the two functions have gradient {+∇ g i ( X ) = 
+}
Revised: Now [Equation] observe that the two functions have gradient {+∇ g i ( X ) = 
+}

Revised: [ I − X

Revised: [Equation]

Revised: X T X )

Original: [Equation]
Revised: [Equation]

Original: Directly substituting yields
Revised: Directly substituting yields

Original: [Equation]
Revised: [Equation]

Original: where the second line follows from the fact that
Revised: where the second line follows from the fact that

Original: [Equation]
Revised: [Equation]

Original: The second claim follows from the following three identities
Revised: The second claim follows from the following three identities

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: We have (17) via Weyl’s inequality:
Revised: We have (17) via Weyl’s inequality:

Original: [Equation]
Revised: [Equation]

Original: We have (18) by rewriting
Revised: We have (18) by rewriting

Original: [Equation]
Revised: [Equation]

Original: and rewriting ZZ T
Revised: and rewriting ZZ T

Original: =
Revised: =

Original: XX T − E where E =
Revised: XX T − E where E =

Original: XX T − ZZ T and evoking (17) as in ∥ 
Revised: XX T − ZZ T and evoking (17) as in ∥ ZZ T X

Original: [Equation]
Revised: ( X T X ) − 2 X T ∥ ≤ ∥ XX T X ( X T X ) − 2 X T ∥ (cid:124) (cid:123)(cid:122) (cid:125) =

Original:  and noting that 1 + ρ 1 − ρ = 11 − ρ .
Revised: + ∥ E ∥ · ∥ X ( X T X ) − 2 X T ∥ (cid:124) (cid:123)(cid:122) (cid:125) ≤ ρ/ (1 − ρ ) and noting that 1 + ρ 1 − ρ = 11 − ρ .

Original: We have (19) again by substituting ZZ T =
Revised: We have (19) again by substituting ZZ T =

Original: XX T − E 
Revised: XX T − E e T i X

Revised: ( X T X ) − 2 X T ZZ T X

Revised: ( X T X ) − 1 X T e i

Original: [Equation]
Revised: = e T i X ( X T X ) − 1 X T e i − e T

Original:  and then noting that1 − ρ 1 − ρ = 1 − 2 ρ 1 − ρ .
Revised: i X ( X T X ) − 2 X T EX ( X T X ) − 1 X T e i [Equation] ) and then noting that

Original: E.4 Proof of the main result
Revised: E.4 Proof of the main result

Original: In the previous two subsections, we showed that when conditioned on the current iterate X t , a single step of ScaledSGD X t +1 =
Revised: In the previous two subsections, we showed that when conditioned on the current iterate X t , a single step of ScaledSGD X t +1 =

Original: X t − αSG ( X t )( X Tt X t ) − 1 is expected to geometrically converge both the loss function f and each of the incoherence functions
Revised: X t − αSG ( X t )( X Tt X t ) − 1 is expected to geometrically converge both the loss function f and each of the incoherence functions

Original: g i , as in
Revised: g i , as in

Original: E [f ( X t +1 )] ≤ ( [Equation] − α
Revised: E [

Original: [Equation]
Revised: [Equation]

Original: In this section, we will extend this geometric convergence to T iterations of ScaledSGD.
Revised: In this section, we will extend this geometric convergence to T iterations of ScaledSGD.

Original: Our key challenge is to verify that the variances and maximum deviations of the sequences f ( X 0 ) , f ( X 1 ) , . . .
Revised: Our key challenge is to verify that the variances and maximum deviations of the sequences f ( X 0 ) , f ( X 1 ) , . . .

Original: , f ( X T ) and g i ( X 0 ) , g i ( X 1 ) , . .
Revised: , f ( X T ) and g i ( X 0 ) , g i ( X 1 ) , . .

Original: , g i ( X T ) have the right dependence on the dimension d , the radius ρ , the condition number κ , the maximum coherence g max , and the iteration count t , so that T iterations of ScaledSGD with a step-size of α ≤ c/ [( g max + ρ )
Revised: , g i ( X T ) have the right dependence on the dimension d , the radius ρ , the condition number κ , the maximum coherence g max , and the iteration count t , so that T iterations of ScaledSGD with a step-size of α ≤ c/ [( g max + ρ )

Original: d 2 log d ] results in no more than a multiplicative factor of 2 deviation from expectation.
Revised: d 2 log d ] results in no more than a multiplicative factor of 2 deviation from expectation.

Original: Crucially, we must check that the cumulated deviation over T iterations does not grow with the iteration count T , and that the convergence rate is independent of the condition number κ .
Revised: Crucially, we must check that the cumulated deviation over T iterations does not grow with the iteration count T , and that the convergence rate is independent of the condition number κ .

Original: We emphasize that the actual approach of our proof via the Azuma–Bernstein inequality is textbook; to facilitate a direct comparison with
Revised: We emphasize that the actual approach of our proof via the Azuma–Bernstein inequality is textbook; to facilitate a direct comparison with

Original: SGD, we organize this section to closely mirror Jin et al.
Revised: SGD, we organize this section to closely mirror Jin et al.

Original: [10]’s proof of Theorem 1.
Revised: [10]’s proof of Theorem 1.

Original: Let f max = ρ 2 · λ 2 min ( Z T Z ) and g max = 16 ( [Equation]
Revised: Let f max = ρ 2 · λ 2min ( Z T Z ) and g max = 16 (1 − 2 ρ ) 2 max

Original: [Equation]

Original: ) 2 max i g i ( Z ) .
Revised: i g i ( Z ) .

Original: Our goal is to show that the following event happens with probability 1 − T/d 10 :
Revised: Our goal is to show that the following event happens with probability 1 − T/d 10 :

Original: [Equation]
Revised: [Equation]

Original: Equivalently, conditioned on event E t , we want to prove that the probability of failure at time t +is δ ≡ 1 /d 10 .
Revised: Equivalently, conditioned on event E t , we want to prove that the probability of failure at time t +is δ ≡ 1 /d 10 .

Original: We split this failure event into a probability of δ 2 that the function value clause fails to hold, as in f ( X t +1 ) >
Revised: We split this failure event into a probability of δ 2 that the function value clause fails to hold, as in f ( X t +1 ) >

Original: (1 − α/ 2) t · f max , and a probability of δ 2 d that any one of the d incoherence caluses fails to hold, as in g i ( X t +1 ) > g max .
Revised: (1 − α/ 2) t · f max , and a probability of δ 2 d that any one of the d incoherence caluses fails to hold, as in g i ( X t +1 ) > g max .

Original: Then, cumulated over T steps, the total probability of failure would be T · δ = T/d 10 as desired.
Revised: Then, cumulated over T steps, the total probability of failure would be T · δ = T/d 10 as desired.

Original: We begin by setting up a supermartingale on the loss function f .
Revised: We begin by setting up a supermartingale on the loss function f .

Original: Our goal is to show that the variance and the maximum deviation of this supermartingale have the right dependence on α, d, ρ, κ, g max , so that a step-size of α ≤ c/ ( g max d 2 log d ) with a sufficiently small c > 0 will keep the cumulative deviations over T iterations within a factor of 2.
Revised: Our goal is to show that the variance and the maximum deviation of this supermartingale have the right dependence on α, d, ρ, κ, g max , so that a step-size of α ≤ c/ ( g max d 2 log d ) with a sufficiently small c > 0 will keep the cumulative deviations over T iterations within a factor of 2.

Original: Note that, by our careful choice of the coherence function g i , the following statement for ScaledSGD match the equivalent statements for SGD with a perfect condition number κ = 1 ; see Jin et al.
Revised: Note that, by our careful choice of the coherence function g i , the following statement for ScaledSGD match the equivalent statements for SGD with a perfect condition number κ = 1 ; see Jin et al.

Original: [10, Section B.2].
Revised: [10, Section B.2].

Original: Lemma 8 (Function value supermartingale) .
Revised: Lemma 8 (Function value supermartingale) .

Original: Let f ( X ) =
Revised: Let f ( X ) =

Original: ∥
Revised: ∥

Original: XX T − ZZ T ∥ 2 F .
Revised: XX T − ZZ T ∥ 2 F .

Original: Define f max = ρ 2 · λ 2min ( Z T Z ) and g max = 16 (1 − 2 ρ ) 2 max i g i ( Z ) .
Revised: Define f max = ρ 2 · λ 2min ( Z T Z ) and g max = 16 (1 − 2 ρ ) 2 max i g i ( Z ) .

Original: For a sufficiently small c > 0 , the following with learning rate α ≤ c/ ( g max d 2 log d ) is a supermartingale
Revised: For a sufficiently small c > 0 , the following with learning rate α ≤ c/ ( g max d 2 log d ) is a supermartingale

Original: [Equation]
Revised: [Equation]

Original: meaning that E [ F t +1 | X t , . .
Revised: meaning that E [ F t +1 | X t , . .

Original: , X 0 ] ≤ F t holds for all t ∈ { 0 , 1 , 2 , . . . } .
Revised: , X 0 ] ≤ F t holds for all t ∈ { 0 , 1 , 2 , . . . } .

Original: Moreover, there exist sufficiently large constants C dev , C var > 0 such that the following holds with probability one:
Revised: Moreover, there exist sufficiently large constants C dev , C var > 0 such that the following holds with probability one:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof is technical but straightforward; it is deferred to Section E.5.
Revised: The proof is technical but straightforward; it is deferred to Section E.5.

Original: Lemma 9 (Function value concentration) .
Revised: Lemma 9 (Function value concentration) .

Original: Let the initial point satisfy f
Revised: Let the initial point satisfy f

Original: ( X 0 ) ≤ 12 f max .
Revised: ( X 0 ) ≤ 1 2 f max .

Original: Then, there exists a sufficiently small constant c > 0 such that for all learning rates α <
Revised: Then, there exists a sufficiently small constant c > 0 such that for all learning rates α <

Original: c/ ( g max d 2 log d ), we have
Revised: c/ ( g max d 2 log d )

Original: [Equation]
Revised: [Equation]

Original: Proof.

Original: Let σ
Revised: we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: =
Revised: d

Original: [Equation]
Revised: [Equation]

Revised: Proof.

Revised: Let σ 2 = (cid:80) tτ

Revised: =1 Var [ F τ | X τ − 1 , . . .

Original: , X 0 ] and let R satisfy E
Revised: , X 0 ] and let R satisfy E

Original: [ F τ | X τ − 1 , . . .
Revised: [ F τ | X τ − 1 , . . .

Original: , X 0 ]
Revised: , X 0 ]

Original: ≤ X τ + R almost surely for all τ ∈ { 1 , 2 , . . .
Revised: ≤ X τ + R almost surely for all τ ∈ { 1 , 2 , . . .

Original: , t } .
Revised: , t } .

Original: Recall via the standard Azuma–Bernstein concetration inequality for supermartingales that Pr ( F t ≥ F 0 + s ) ≤ exp (cid:16) − s 2 / 2 σ 2 + Rs/ 3 (cid:17) .
Revised: Recall via the standard Azuma–Bernstein concetration inequality for supermartingales that Pr ( F t ≥ F 0 + s ) ≤ exp (cid:16) − s 2 / 2 σ 2 + Rs/ 3 (cid:17) .

Original: Equivalently, there exists a large enough constant C > 0 in s = C · (1 − α ) t (cid:104)(cid:112) σ 2 log d + R log d (cid:105) such that the following is true
Revised: Equivalently, there exists a large enough constant C > 0 in s = C · (1 − α ) t (cid:104)(cid:112) σ 2 log d + R log d (cid:105) such that the following is true

Original: [Equation]
Revised: [Equation]

Original: Given that f ( X 0 ) ≤ 12 f max and therefore (1 − α ) t f
Revised: Given that f ( X 0 ) ≤ 12 f max and therefore (1 − α ) t f

Original: ( X 0 ) ≤ 12 (cid:0) 1 − α 2 (cid:1) t · f max holds by hypothesis
Revised: ( X 0 ) ≤ 12 (cid:0) 1 − α 2 (cid:1) t · f max holds by hypothesis

Original: , the desired claim is true if we can show that s ≤ 12 (cid:0) 1 − α 2 (cid:1) t · f max .
Revised: , the desired claim is true if we can show that s ≤ 12 (cid:0) 1 − α 2 (cid:1) t · f max .

Original: Crucially, we observe that the variance term in s does not blow-up with time t
Revised: Crucially, we observe that the variance term in s does not blow-up with time t

Original: [Equation]
Revised: [Equation]

Original: due to the geometric series expansion (cid:80) 
Revised: due to the geometric series expansion (cid:80) tτ

Original: [Equation]{+=
 β t − τ = (1 − β t +1 ) / (1 − β ) .+}
Revised: {+=
 β t − τ = (1 − β t +1 ) / (1 − β ) .+}

Original: Substituting the deviations term, choosing a step-size α ≤ c/ ( ρd 2 log d ) for sufficiently small c yields
Revised: Substituting the deviations term, choosing a step-size α ≤ c/ ( ρd 2 log d ) for sufficiently small c yields

Original: [Equation]
Revised: [Equation]

Original: We now set up a supermartingale on each of the incoherence functions g i .
Revised: We now set up a supermartingale on each of the incoherence functions g i .

Original: Again, our goal is to show that the variance and the maximum deviation of this supermartingale have the right dependence on α, d, ρ, κ, g max , so that a step-size of α ≤ c/ ( ρd 2 log d ) with a sufficiently small c > 0 will keep the cumulative deviations over T iterations within a factor of 2.
Revised: Again, our goal is to show that the variance and the maximum deviation of this supermartingale have the right dependence on α, d, ρ, κ, g max , so that a step-size of α ≤ c/ ( ρd 2 log d ) with a sufficiently small c > 0 will keep the cumulative deviations over T iterations within a factor of 2.

Original: Note that Jin et al.
Revised: Note that Jin et al.

Original: [10, Section B.2]’s proof tracks a different function
Revised: [10, Section B.2]’s proof tracks a different function

Original: h i ( X ) =
Revised: h i ( X ) =

Original: e Ti XX T e i that is substantially simpler, but pays a penalty of two to three factors of the condition number κ .
Revised: e Ti XX T e i that is substantially simpler, but pays a penalty of two to three factors of the condition number κ .

Original: Lemma 10 (Incoherence supermartingale) .
Revised: Lemma 10 (Incoherence supermartingale) .

Original: Let g i ( X ) =
Revised: Let g i ( X ) =

Original: e Ti X ( X T X ) − 1 X T e i .
Revised: e Ti X ( X T X ) − 1 X T e i .

Original: Define g max =(1 − 2 ρ ) 2 max
Revised: Define g max =(1 − 2 ρ ) 2 max

Original: i g i ( Z ) .
Revised: i g i ( Z ) .

Original: For a fixed i ∈ [ n ] with sufficiently small c > 0 , the following with learning rate α < c/ ( ρd 2 log d ) is a supermartingale 
Revised: For a fixed i ∈ [ n ] with sufficiently small c > 0 , the following with learning rate α < c/ ( ρd 2 log d ) is a supermartingale [Equation] where ζ =

Original: [Equation]
Revised: [Equation]

Original:  meaning that E [ G i ( t +1) | X t , . .
Revised: 2 ρ [Equation] < 1 , meaning that E [ G i ( t +1) | X t , . .

Original: , X 0 ] ≤ G it holds for all t ∈ { 0 , 1 , 2 , . . . } .
Revised: , X 0 ] ≤ G it holds for all t ∈ { 0 , 1 , 2 , . . . } .

Original: Moreover, there exist sufficiently large constants C dev , C var > 0 with no dependence on g max , n, t such that
Revised: Moreover, there exist sufficiently large constants C dev , C var > 0 with no dependence on g max , n, t such that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof is long but straightforward; it is deferred to Section E.5.
Revised: The proof is long but straightforward; it is deferred to Section E.5.

Original: Lemma 11 (Incoherence concentration) .
Revised: Lemma 11 (Incoherence concentration) .

Original: Let the initial point satisfy max
Revised: Let the initial point satisfy max

Original: i g i ( X 0 )
Revised: i g i ( X 0 )

Original: ≤ 12 g max .
Revised: ≤ 12 g max .

Original: Then, there exists a sufficiently small constant c > 0 such that for all learning rates α <
Revised: Then, there exists a sufficiently small constant c > 0 such that for all learning rates α <

Original: c/ ( ρd 2 log d ) , we have
Revised: c/ ( ρd 2 log d ) , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let σ 2 = (cid:80) tτ
Revised: Let σ 2 = (cid:80) tτ

Original: =1 Var [ G iτ | X τ − 1 , . . .
Revised: =1 Var [ G iτ | X τ − 1 , . . .

Original: , X 0 ] and let R satisfy E
Revised: , X 0 ] and let R satisfy E

Original: [ G iτ |
Revised: [ G iτ |

Original: X τ − 1 , . . .
Revised: X τ − 1 , . . .

Original: , X 0 ]
Revised: , X 0 ]

Original: ≤ X τ + R almost surely for all τ ∈ { 1 , 2 , . . .
Revised: ≤ X τ + R almost surely for all τ ∈ { 1 , 2 , . . .

Original: , t } .
Revised: , t } .

Original: Recall via the standard Azuma–Bernstein concetration inequality for supermartingales that
Revised: Recall via the standard Azuma–Bernstein concetration inequality for supermartingales that

Original: [Equation]
Revised: [Equation]

Original: Equivalently, there exists a large enough constant C > 0 such that the following is true
Revised: Equivalently, there exists a large enough constant C > 0 such that the following is true

Original: [Equation]
Revised: [Equation]

Original: where 
Revised: where s ′ =

Original: [Equation]
Revised: C · (1 − ζ · α ) t · (cid:104)(cid:112) σ 2 log d + R log d (cid:105) .

Original: Given that g ( X 0 ) ≤ 12 g max holds by hypothesis, the desired claim is true if we can show that s ′ ≤ 12 g max .
Revised: Given that g ( X 0 ) ≤ 12 g max holds by hypothesis, the desired claim is true if we can show that s ′ ≤ 12 g max .

Original: Crucially, we observe that the variance term in s ′ does not blow-up with time t
Revised: Crucially, we observe that the variance term in s ′ does not blow-up with time t

Original: [Equation]
Revised: [Equation]

Original: due to the geometric series expansion (cid:80) tτ
Revised: due to the geometric series expansion (cid:80) tτ

Original: =

Revised: =

Original: β t − τ =
Revised: β t − τ =

Original: (1 − β t +1 ) / (1 − β ) .
Revised: (1 − β t +1 ) / (1 − β ) .

Original: Substituting the deviations term, choosing a step-size α ≤ c/ ( ρd 2 log d ) for sufficiently small c yields
Revised: Substituting the deviations term, choosing a step-size α ≤ c/ ( ρd 2 log d ) for sufficiently small c yields

Original: [Equation]
Revised: [Equation]

Original: In summary, Lemma 9 requires a step-size of α ≤ c/ ( g max d 2 log d ) to keep deviations on f small, while Lemma 11 requires a step-size of α ≤ c/ ( ρd 2 log d ) to keep deviations on g i small.
Revised: In summary, Lemma 9 requires a step-size of α ≤ c/ ( g max d 2 log d ) to keep deviations on f small, while Lemma 11 requires a step-size of α ≤ c/ ( ρd 2 log d ) to keep deviations on g i small.

Original: Therefore, it follows that a step-size α ≤ c/ (( g max + ρ ) d 2 log d ) will keep both deviations small.
Revised: Therefore, it follows that a step-size α ≤ c/ (( g max + ρ ) d 2 log d ) will keep both deviations small.

Original: Proof of Theorem 2.
Revised: Proof of Theorem 2.

Original: For a step-size α ≤ c/ (( g max + ρ ) d 2 log d ) with sufficiently small c > 0 , both concentration bounds Lemma 9 and Lemma 11 are valid.
Revised: For a step-size α ≤ c/ (( g max + ρ ) d 2 log d ) with sufficiently small c > 0 , both concentration bounds Lemma 9 and Lemma 11 are valid.

Original: Combined, we take the trivial union bound to determine the probability of failure at the ( t + 1) -th step, after succeeding after t steps:
Revised: Combined, we take the trivial union bound to determine the probability of failure at the ( t + 1) -th step, after succeeding after t steps:

Original: [Equation]
Revised: [Equation]

Original: Here, E t +1 denotes the complement of E t +1 .
Revised: Here, E t +1 denotes the complement of E t +1 .

Original: The probability of failure at the T -th step is then the cummulative probability of failing at the ( t +1) -th step, after succeeding after t steps, over all t ≤ T :
Revised: The probability of failure at the T -th step is then the cummulative probability of failing at the ( t +1) -th step, after succeeding after t steps, over all t ≤ T :

Original: [Equation]
Revised: [Equation]

Original: and this proves that E T happens with probability 1 − T/d 10 as desired.
Revised: and this proves that E T happens with probability 1 − T/d 10 as desired.

Original: E.5 Proofs of supermartingale deviations and variances
Revised: E.5 Proofs of supermartingale deviations and variances

Original: We will now verify the supermartingales and their deviations and variances in detail.
Revised: We will now verify the supermartingales and their deviations and variances in detail.

Original: We first begin by proving the following bounds on the size of the stochastic gradient.
Revised: We first begin by proving the following bounds on the size of the stochastic gradient.

Original: Lemma 12.
Revised: Lemma 12.

Original: Let X, Z ∈ R n × r satisfy ∥ XX T − ZZ T ∥ F ≤ ρ · λ min ( Z T Z ) with ρ < 1 / 2 andmax i
Revised: Let X, Z ∈ R n × r satisfy ∥ XX T − ZZ T ∥ F ≤ ρ · λ min ( Z T Z ) with ρ < 1 / 2 and

Original: e Ti X ( X T X ) − 1 Xe i ≤ g max and max i e Ti Z ( Z T Z ) − 1 Ze i ≤ g max .
Revised: [Equation] g max and max i e Ti Z ( Z T Z ) − 1 Ze i ≤ g max .

Original: Then, with respect to the randomness of the following
Revised: Then, with respect to the randomness of the following

Original: [Equation]
Revised: [Equation]

Original: where ( i, j ) ∼ [ d ] 2 is selected uniformly at random, we have:
Revised: where ( i, j ) ∼ [ d ] 2 is selected uniformly at random, we have:

Original: ∥ SG ( X ) ∥ ∗ X ≤ 8 d 2 · g 1 / 2 max · ∥ XX T − ZZ T ∥ F . 2.
Revised: ∥ SG ( X ) ∥ ∗ X ≤ 8 d 2 · g 1 / 2 max · ∥ XX T − ZZ T ∥ F . 2.

Original: ∥ SG ( X )( X T X ) − 1 ∥ ∗ X ≤ 16 d 2 · g 1 / 2 max · ρ . 3.
Revised: ∥ SG ( X )( X T X ) − 1 ∥ ∗ X ≤ 16 d 2 · g 1 / 2 max · ρ . 3.

Original: E ( ∥ SG ( X ) ∥ ∗ X ) p ≤ 2 2 p · d 2( p − 1) · g p/ 2 max · ∥ XX T − ZZ T ∥ pF . 4.
Revised: E ( ∥ SG ( X ) ∥ ∗ X ) p ≤ 2 2 p · d 2( p − 1) · g p/ 2 max · ∥ XX T − ZZ T ∥ pF . 4.

Original: E ( ∥ SG ( X )( X T X ) − 1 ∥ ∗ X ) p ≤ 2 3 p · d 2( p − 1) · g p/ 2 max · ρ p .
Revised: E ( ∥ SG ( X )( X T X ) − 1 ∥ ∗ X ) p ≤ 2 3 p · d 2( p − 1) · g p/ 2 max · ρ p .

Original: Proof.
Revised: Proof.

Original: Let us write E
Revised: Let us write E

Original: =
Revised: =

Original: XX T − ZZ T .
Revised: XX T − ZZ T .

Original: To prove (i) we have
Revised: To prove (i) we have

Original: [Equation]
Revised: [Equation]

Original: and if we write Q X = X
Revised: and if we write Q X = X

Original: ( X T X ) − 1 / 2 and Q Z
Revised: ( X T X ) − 1 / 2 and Q Z

Original: =
Revised: =

Original: Z
Revised: Z

Original: ( Z T Z ) − 1 / 2 we have
Revised: ( Z T Z ) − 1 / 2 we have

Original: E i,j = e Ti Ee j = e Ti Q X Q TX ( XX T − ZZ T ) 
Revised: E i,j = e Ti Ee j = e Ti Q X Q TX ( XX T − ZZ T ) e j − e Ti ( I

Original: [Equation]
Revised: [Equation]

Original:  we use the fact that ( I − Q X Q TX )( XX T − ZZT ) =
Revised: X [Equation] we use the fact that ( I − Q X Q TX )( XX T − ZZ

Revised: [Equation]

Original:  − ( I − Q X Q TX ) ZZ T in the first and last lines.
Revised: = − ( I − Q X Q TX ) ZZ T in the first and last lines.

Original: To prove (ii) we have
Revised: To prove (ii) we have

Original: [Equation]
Revised: [Equation]

Original: where we used Weyl’s inequality λ r ( XX T ) ≥ λ r ( ZZ T ) − ∥ XX T − ZZ T ∥ F .
Revised: where we used Weyl’s inequality λ r ( XX T ) ≥ λ r ( ZZ T ) − ∥ XX T − ZZ T ∥ F .

Original: To prove (iii) we have
Revised: To prove (iii) we have

Original: [Equation]
Revised: [Equation]

Original: where we used ( (cid:80) i x 2 i ) 1 / 2 ≥ ( (cid:80) i x pi ) 1 /p for any p ≥ 2 .
Revised: where we used ( (cid:80) i x 2 i ) 1 / 2 ≥ ( (cid:80) i x pi ) 1 /p for any p ≥ 2 .

Original: The proof of (iv) follows identically by applying the proof of (ii) to the proof of (iii).
Revised: The proof of (iv) follows identically by applying the proof of (ii) to the proof of (iii).

Original: We now prove the properties of the function value supermartingale F t .
Revised: We now prove the properties of the function value supermartingale F t .

Original: Proof of Lemma 8.
Revised: Proof of Lemma 8.

Original: Conditioning on the current iterate X t and the event E t , the new iterate X t +1 =
Revised: Conditioning on the current iterate X t and the event E t , the new iterate X t +1 =

Original: X t − αSG ( X t )( X Tt X t ) − 1 has expectation
Revised: X t − αSG ( X t )( X Tt X t ) − 1 has expectation

Original: [Equation]
Revised: [Equation]

Original: with L X
Revised: with L X

Original: =
Revised: =

Original: O (1) by evoking Lemma 3 noting that ∥ αSG ( X t ) ∥ ∗ X
Revised: O (1) by evoking Lemma 3 noting that ∥ αSG ( X t ) ∥ ∗ X

Original: =
Revised: =

Original: O (1) · (cid:112) f
Revised: O (1) · (cid:112) f

Original: ( X t ) for the step-size α ≤ c/ ( g max d 2 log d ) , since
Revised: ( X t ) for the step-size α ≤ c/ ( g max d 2 log d ) , since

Original: [Equation]
Revised: [Equation]

Original: The linear term evaluates simply as E [ SG ( X )( X T X ) − 1 ] = ∇ f ( X )( X T X ) − 1 , while the quadratic term evaluates
Revised: The linear term evaluates simply as E [ SG ( X )( X T X ) − 1 ] = ∇ f ( X )( X T X ) − 1 , while the quadratic term evaluates

Original: [Equation]
Revised: [Equation]

Original: Combined, substituting ( ∥∇ f ( X ) ∥ ∗ X ) 2 ≥ 13 · f ( X ) , it follows that we have geometric convergenceE [ f
Revised: Combined, substituting ( ∥∇ f ( X ) ∥ ∗ X ) 2 ≥ 13 · f ( X ) , it follows that we have geometric convergence

Original: [Equation]
Revised: [Equation]

Original: X t +1 )] ≤ f [Equation]
Revised: where we observe that we can pick a small enough constant c in the step-size α ≤ c/ ( g max d 2 log d ) so that

Original: [Equation]
Revised: [Equation]

Original: ) where we observe that we can pick a small enough constant c in the step-size α ≤ c/ ( g max d 2 log d ) so that
Revised: Now

Original: [Equation]
Revised: [Equation]

Original: Now, to confirm that F t is a martingale
Revised: to confirm that F t is a martingale

Original: [Equation]
Revised: [Equation]

Original: it remains to see that
Revised: it remains to see that

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows from 1 E t ≤ 1 E t − 1 .
Revised: where the last inequality follows from 1 E t ≤ 1 E t − 1 .

Original: We now bound the deviations on F t .
Revised: We now bound the deviations on F t .

Original: Conditioning on the previous iterates X t , . . .
Revised: Conditioning on the previous iterates X t , . . .

Original: , X 0 , we obseve that the f ( X t ) terms cancel:
Revised: , X 0 , we obseve that the f ( X t ) terms cancel:

Original: [Equation]
Revised: [Equation]

Original: Here we have for the linear term ⟨∇f ( X t ) , SG ( X t )( X Tt X t ) − 1 ⟩ · 1 E t ≤ ∥∇ f [Equation] X t ) ∥ ∗ X ∥ SG [Equation]
Revised: Here we have for the linear term ⟨∇

Original: ∥ ∗ X

Original: [Equation]

Original: [Equation]

Original: g

Original: [Equation]
Revised: [Equation]

Original: ) f

Original: [Equation]

Original: X

Original: [Equation]

Original: [Equation]

Original: and the quadratic term
Revised: and the quadratic term

Original: [Equation]
Revised: [Equation]

Original: Therefore, using the maximum value to bound the expectation, we have
Revised: Therefore, using the maximum value to bound the expectation [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: where again we observe that a step-size like α ≤ c/ ( g max d 2 log d )
Revised: where again we observe that a step-size like α ≤ c/ ( g max d 2 log d )

Original: =
Revised: =

Original: O ( d − 2 g − 1max ) yields the cancellation of exponents α · O ( d 4 g 2max )
Revised: O ( d − 2 g − 1max ) yields the cancellation of exponents α · O ( d 4 g 2max )

Original: =
Revised: =

Original: O ( d 2 g max ) .
Revised: O ( d 2 g max ) .

Original: Finally, we bound the variance.
Revised: Finally [Equation] we bound the variance.

Original: Conditioned on all previous iterates X t , . . .
Revised: Conditioned on all previous iterates X t , . . .

Original: X 

Revised: X

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: and also
Revised: and also

Original: [Equation]
Revised: [Equation]

Original: By the same expansion in (22) we have
Revised: By the same expansion in (22) we have

Original: [Equation]
Revised: [Equation]

Original: where again we observe that a step-size like α ≤ c/ ( g max d 2 log d )
Revised: where again we observe that a step-size like α ≤ c/ ( g max d 2 log d )

Original: =
Revised: =

Original: O ( d − 2 g − 1max ) yields the cancellation of exponents α 2 · ( d 6 g 3max )
Revised: O ( d − 2 g − 1max ) yields the cancellation of exponents α 2 · ( d 6 g 3max )

Original: =
Revised: =

Original: O ( d 2 g max ) .
Revised: O ( d 2 g max ) .

Original: We now prove properties of the incoherence martingale.
Revised: We now prove properties of the incoherence martingale.

Original: Proof of Lemma 10.
Revised: Proof of Lemma 10.

Original: Conditioning on X t and the event E t , we have for V
Revised: Conditioning on X t and the event E t , we have for V

Original: = SG ( X t )( X Tt X t ) −
Revised: = SG ( X t )( X Tt X t ) −

Original: E 
Revised: E [

Original: [Equation]
Revised: [Equation]

Original: g
Revised: for

Original: [Equation]
Revised: [Equation]

Original: V ] ⟩
Revised: O

Original: [Equation]
Revised: [Equation]

Original: Here we note that we have carefully chosen g max so that the ratio max i g i
Revised: Here we note that we have carefully chosen g max so that the ratio max i g i

Original: ( Z ) /g max
Revised: ( Z ) /g max

Original: =
Revised: =

Original: [(1 − ρ ) · ζ/ 4] 2 .
Revised: [(1 − ρ ) · ζ/ 4] 2 .

Original: It then follows that the following is a supermartingale
Revised: It then follows that the following is a supermartingale

Original: [Equation]
Revised: [Equation]

Original: Indeed [Equation] we have
Revised: Indeed, we have

Original: [Equation]
Revised: [Equation]

Original: where the final line uses 1 E t ≤ 1 E t − 1 ≤ 1 .
Revised: where the final line uses 1 E t ≤ 1 E t − 1 ≤ 1 .

Original: We now bound the deviations on G it .
Revised: We now bound the deviations on G it .

Original: Conditioning on the previous iterates X t , . . .
Revised: Conditioning on the previous iterates X t , . . .

Original: , X 0 , we obseve that the g i ( X t ) terms cancel:
Revised: , X 0 , we obseve that the g i ( X t ) terms cancel:

Original: [Equation]
Revised: [Equation]

Original: Here we have for the linear term 
Revised: Here we have for the linear term ∥∇ g

Original: [Equation]
Revised: [Equation]

Original: and ∥SG ( X ) ∥ ∗ X · 
Revised: ∥ · [Equation] T [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: t ≤ O
Revised: ≤ O

Original: [Equation]
Revised: [Equation]

Original: d 2 ·

Original: [Equation]
Revised: [Equation]

Original: g  max f ( X ))
Revised: g 1 / 2 max ) and ∥ SG ( X ) ∥ ∗ X · 1 E t ≤ O ( d 2 · (cid:112) g max f ( X ))

Original: =
Revised: =

Original: O ( d 2 √ g max ρ/κ ) noting that f max = ρ 2 /κ 2 and hence 
Revised: O ( d 2 √ g max ρ/κ ) noting that f max = ρ 2 /κ 2 and hence |⟨∇

Original: [Equation]
Revised: [Equation]

Original: We have for the quadratic term
Revised: We have for the quadratic term

Original: [Equation]
Revised: [Equation]

Original: Therefore, using the maximum value to bound the expectation
Revised: Therefore, using the maximum value to bound the expectation

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where again we observe that a step-size like α ≤ c/ ( ρd 2 log d )
Revised: where again we observe that a step-size like α ≤ c/ ( ρd 2 log d )

Original: =
Revised: =

Original: O ( d − 2 ρ − 1 ) yields the cancellation of exponents α · O ( d 4 ρ 2 )
Revised: O ( d − 2 ρ − 1 ) yields the cancellation of exponents α · O ( d 4 ρ 2 )

Original: =
Revised: =

Original: O ( d 2 ρ ).
Revised: O ( d 2 ρ )

Revised: [Equation]

Original: Finally, we bound the variance.
Revised: Finally, we bound the variance.

Original: Conditioned on all previous iterates X t , . . .
Revised: Conditioned on all previous iterates X t , . . .

Original: X 

Revised: X

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: and also
Revised: and also

Original: [Equation]
Revised: [Equation]

Original: By the same expansion in (23) we have
Revised: By the same expansion in (23) we have

Original: [Equation]
Revised: [Equation]

Original: where again we observe that a step-size like α ≤ c/ ( ρd 2 log d )
Revised: where again we observe that a step-size like α ≤ c/ ( ρd 2 log d )

Original: =
Revised: =

Original: O ( d − 2 ρ − 1 ) yields the cancellation of exponents α 2 · ( d 6 ρ 4 )
Revised: O ( d − 2 ρ − 1 ) yields the cancellation of exponents α 2 · ( d 6 ρ 4 )

Original: =
Revised: =

Original: O ( d 2 ρ 2 ) .
Revised: O ( d 2 ρ 2 ) .