Original: D IRECTIONAL B IAS H ELPS S TOCHASTIC G RADIENT D ESCENT TO G ENERALIZE IN N ONPARAMETRIC M ODEL
Revised: D IRECTIONAL B IAS H ELPS S TOCHASTIC G RADIENT D ESCENT TO G ENERALIZE IN N ONPARAMETRIC M ODEL

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This paper studies the Stochastic Gradient Descent (SGD) algorithm in kernel regression.
Revised: This paper studies the Stochastic Gradient Descent (SGD) algorithm in kernel regression.

Original: The main ﬁnding is that SGD with moderate and annealing step size converges in the direction of the eigenvector that corresponds to the largest eigenvalue of the gram matrix.
Revised: The main ﬁnding is that SGD with moderate and annealing step size converges in the direction of the eigenvector that corresponds to the largest eigenvalue of the gram matrix.

Original: On the contrary, the Gradient Descent (GD) with a moderate or small step size converges along the direction that corresponds to the smallest eigenvalue.
Revised: On the contrary, the Gradient Descent (GD) with a moderate or small step size converges along the direction that corresponds to the smallest eigenvalue.

Original: For a general squared risk minimization problem, we show that directional bias towards a larger eigenvalue of the Hessian (which is the gram matrix in our case) results in an estimator that is closer to the ground truth.
Revised: For a general squared risk minimization problem, we show that directional bias towards a larger eigenvalue of the Hessian (which is the gram matrix in our case) results in an estimator that is closer to the ground truth.

Original: Adopting this result to kernel regression, the directional bias helps the SGD estimator generalize better.
Revised: Adopting this result to kernel regression, the directional bias helps the SGD estimator generalize better.

Original: This result gives one way to explain how noise helps in generalization when learning with a nontrivial step size, which may be useful for promoting further understanding of stochastic algorithms in deep learning.
Revised: This result gives one way to explain how noise helps in generalization when learning with a nontrivial step size, which may be useful for promoting further understanding of stochastic algorithms in deep learning.

Original: The correctness of our theory is supported by simulations and experiments of Neural
Revised: The correctness of our theory is supported by simulations and experiments of Neural

Original: Network on the FashionMNIST dataset.
Revised: Network on the FashionMNIST dataset.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In this paper, we study the Stochastic Gradient Descent (SGD) algorithm in a nonparametric regression model.
Revised: In this paper, we study the Stochastic Gradient Descent (SGD) algorithm in a nonparametric regression model.

Original: Among nonparametric models, one popular choice in both statistics and machine learning communities is the kernel model that is generated by a Reproducing Kernel Hilbert Space (RKHS).
Revised: Among nonparametric models, one popular choice in both statistics and machine learning communities is the kernel model that is generated by a Reproducing Kernel Hilbert Space (RKHS).

Original: When ﬁtting kernel models to the data, SGD is computationally efﬁcient as compared to
Revised: When ﬁtting kernel models to the data, SGD is computationally efﬁcient as compared to

Original: Gradient Descent (GD) (Ma et al., 2018).
Revised: Gradient Descent (GD) (Ma et al., 2018).

Original: This motivates us to analyze the properties of SGD in the kernel model, especially for SGD with a nontrivial step size for a practical reason.
Revised: This motivates us to analyze the properties of SGD in the kernel model, especially for SGD with a nontrivial step size for a practical reason.

Original: In particular, we aim to provide a fundamental explanation of why SGD estimators generalize well.
Revised: In particular, we aim to provide a fundamental explanation of why SGD estimators generalize well.

Original: Our work is inspired by Wu et al.
Revised: Our work is inspired by Wu et al.

Original: (2021), which shows that directional bias has a signiﬁcant impact on the generalization property in the linear regression model.
Revised: (2021), which shows that directional bias has a signiﬁcant impact on the generalization property in the linear regression model.

Original: We adopt a similar concept of the directional bias, but generalize them to the nonparametric model.
Revised: We adopt a similar concept of the directional bias, but generalize them to the nonparametric model.

Original: Our result is a non-trivial extension of their approach, due to the difference in our problem setting and our SGD algorithm design.
Revised: Our result is a non-trivial extension of their approach, due to the difference in our problem setting and our SGD algorithm design.

Original: To the best of our knowledge, we are the ﬁrst to show the directional bias phenomenon of SGD and analyze how it helps generalization in nonparametric regression.
Revised: To the best of our knowledge, we are the ﬁrst to show the directional bias phenomenon of SGD and analyze how it helps generalization in nonparametric regression.

Original: Our contributions are two folded.
Revised: Our contributions are two folded.

Original: First, we study the directional bias of (S)GD in a nonparametric regression model.
Revised: First, we study the directional bias of (S)GD in a nonparametric regression model.

Original: On the one hand, nonparametric regression is well studied in both statistics and machine learning.
Revised: On the one hand, nonparametric regression is well studied in both statistics and machine learning.

Original: On the other hand, the directional bias is a relatively new concept (Wu et al., 2021) of an algorithm that affects the statistics properties, and there is no thorough understanding of the directional bias of (S)GD algorithms for the nonparametric regression model.
Revised: On the other hand, the directional bias is a relatively new concept (Wu et al., 2021) of an algorithm that affects the statistics properties, and there is no thorough understanding of the directional bias of (S)GD algorithms for the nonparametric regression model.

Original: Note that our result is closely related to those in Belkin et al.
Revised: Note that our result is closely related to those in Belkin et al.

Original: ; Liang & Rakhlin (2020): they prove that
Revised: ; Liang & Rakhlin (2020): they prove that

Original: SGD and GD algorithms both converge to the minimum norm interpolate, where the same properties are discovered for SGD and GD; whereas we investigate the solution paths before their convergence, and show that SGD and GD have different solution paths that lead to different properties.
Revised: SGD and GD algorithms both converge to the minimum norm interpolate, where the same properties are discovered for SGD and GD; whereas we investigate the solution paths before their convergence, and show that SGD and GD have different solution paths that lead to different properties.

Original: Our result helps to explain why the SGD generalizes better than GD.
Revised: Our result helps to explain why the SGD generalizes better than GD.

Original: Our second contribution is to unify the conditions to show the directional bias of (S)GD sequences in nonparametric models.
Revised: Our second contribution is to unify the conditions to show the directional bias of (S)GD sequences in nonparametric models.

Original: The main condition is the diagonal dominant gram matrix, which covers a large class of kernel functions and allows us to study their properties.
Revised: The main condition is the diagonal dominant gram matrix, which covers a large class of kernel functions and allows us to study their properties.

Original: Moreover, our SGD is different from those in
Revised: Moreover, our SGD is different from those in

Original: Wu et al
Revised: Wu et al.

Original: (2021): they deﬁne SGD in epochs while we deﬁne SGD in steps.
Revised: ): they deﬁne SGD in epochs while we deﬁne SGD in steps.

Original: The fundamental difference in the SGD algorithm requires us to develop different techniques for analyzing the SGD sequence and showing its directional bias, which has not yet been covered in the current literature.
Revised: The fundamental difference in the SGD algorithm requires us to develop different techniques for analyzing the SGD sequence and showing its directional bias, which has not yet been covered in the current literature.

Original: Main Theorems of this paper can be divided into two parts, brieﬂy summarized as follows:
Revised: Main Theorems of this paper can be divided into two parts, brieﬂy summarized as follows:

Original: First is the directional bias of SGD.
Revised: First is the directional bias of SGD.

Original: Theorem 4 shows that for a two-stage SGD with a moderate step size in the ﬁrst stage and a small step size in the second stage, an early-stopped estimator has a directional bias towards the eigenvector that corresponds to the largest eigenvalue of the gram matrix.
Revised: Theorem 5 shows that for a two-stage SGD with a moderate step size in the ﬁrst stage and a small step size in the second stage, an early-stopped estimator has a directional bias towards the eigenvector that corresponds to the largest eigenvalue of the gram matrix.

Original: Later we refer to this direction as the direction towards the largest eigenvector to simplify the statement.
Revised: Later we refer to this direction as the direction towards the largest eigenvector to simplify the statement.

Original: As a comparison, Theorem 6 shows that GD with both moderate and small step sizes has a directional bias towards the eigenvector that corresponds to the smallest eigenvalue of the gram matrix (denote it as the direction towards the smallest eigenvector).
Revised: As a comparison, Theorem 7 shows that GD with both moderate and small step sizes has a directional bias towards the eigenvector that corresponds to the smallest eigenvalue of the gram matrix (denote it as the direction towards the smallest eigenvector).

Original: From which, we conclude that SGD and GD have different direction biases in kernel regression.
Revised: From which, we conclude that SGD and GD have different direction biases in kernel regression.

Original: Second is the implication of directional bias.
Revised: Second is the implication of directional bias.

Original: The implication is very useful since it quantiﬁes the effect of the directional bias on the generalization error.
Revised: The implication is very useful since it quantiﬁes the effect of the directional bias on the generalization error.

Original: Theorem 8 considers a general problem of quadratic loss.
Revised: Theorem 9 considers a general problem of quadratic loss.

Original: It shows that the estimator biased towards the largest eigenvector of the Hessian (which is the gram matrix in our nonparametric regression) can have the smallest parameter estimation error, when compared with other estimators of the same loss.
Revised: It shows that the estimator biased towards the largest eigenvector of the Hessian (which is the gram matrix in our nonparametric regression) can have the smallest parameter estimation error, when compared with other estimators of the same loss.

Original: With this high-level idea of directional bias helps generalization, Theorem 10 compares the generalization error of SGD and GD in our problem setting.
Revised: With this high-level idea of directional bias helps generalization, Theorem 11 compares the generalization error of SGD and GD in our problem setting.

Original: In particular, it upper bounds the generalization error of SGD and lower bounds the generalization error of GD.
Revised: In particular, it upper bounds the generalization error of SGD and lower bounds the generalization error of GD.

Original: By directly comparing the error bounds, we guarantee that the generalization error of the SGD estimator is smaller than that of the
Revised: By directly comparing the error bounds, we guarantee that the generalization error of the SGD estimator is smaller than that of the

Original: GD estimator with high probability.
Revised: GD estimator with high probability.

Original: We also point out that our result might shed new light on deep learning (Belkin et al., 2018).
Revised: We also point out that our result might shed new light on deep learning (Belkin et al., 2018).

Original: By the state-of-the-art mathematical theory of Neural Networks (NN), kernel and/or nonparametric methods can approximate the functional space of neural networks, see for example the NTK theory (Jacot et al., 2018), and the Radon bounded variance space description for ReLU NN (Parhi &
Revised: By the state-of-the-art mathematical theory of Neural Networks (NN), kernel and/or nonparametric methods can approximate the functional space of neural networks, see for example the NTK theory (Jacot et al., 2018), and the Radon bounded variance space description for ReLU NN (Parhi &

Original: Nowak, 2021).
Revised: Nowak, 2021).

Original: Our technique might allow one to characterize the SGD solution path and show the generalization property in those problem settings.
Revised: Our technique might allow one to characterize the SGD solution path and show the generalization property in those problem settings.

Original: Paper organization .
Revised: Paper organization .

Original: The rest of the paper is as follows:
Revised: The rest of the paper is as follows:

Original: In Section 2, we review some relevant literature; In Section 3, we give the formulation of the nonparametric regression and deﬁne the optimization problem.
Revised: In Section 2, we review some relevant literature; In Section 3, we give the formulation of the nonparametric regression and deﬁne the optimization problem.

Original: We also formalize the algorithm that is considered in this work and make assumptions to analyze the algorithms;
Revised: We also formalize the algorithm that is considered in this work and make assumptions to analyze the algorithms;

Original: In Section 4, we state our main theory on the directional bias of SGD/GD in nonparametric regression, where we include both the directional bias result and the implication of the directional bias for generalization.
Revised: In Section 4, we state our main theory on the directional bias of SGD/GD in nonparametric regression, where we include both the directional bias result and the implication of the directional bias for generalization.

Original: Experiments are conducted to support our theory; In Section 5, we discuss the ﬁnding in this paper, and propose some future research topic.
Revised: Experiments are conducted to support our theory; In Section 5, we discuss the ﬁnding in this paper, and propose some future research topic.

Original: All the proof, experiment details, and more experiments are deferred to the appendix due to page limits.
Revised: All the proof, experiment details, and more experiments are deferred to the appendix due to page limits.

Original:  L ITERATURE R EVIEW
Revised:  L ITERATURE R EVIEW

Original: In this section, we review some relevant works.
Revised: In this section, we review some relevant works.

Original: For better understanding, we split into two subsections: Subsection 2.1 reviews the background of RKHS; Subsection 2.2 presents the state-of-the-art technique for analyzing the directional bias of the (S)GD algorithm.
Revised: For better understanding, we split into two subsections: Subsection 2.1 reviews the background of RKHS; Subsection 2.2 presents the state-of-the-art technique for analyzing the directional bias of the (S)GD algorithm.

Original: 2. RKHS
Revised: 2. RKHS

Original: Kernel methods are among the core algorithms in machine learning and statistics (Bartlett et al., 2021).
Revised: Kernel methods are among the core algorithms in machine learning and statistics (Bartlett et al., 2021).

Original: As proposed by Wahba (1990), the kernel method and RKHS serve as a uniﬁed framework of a group of nonparametric models, which extends the spline method.
Revised: As proposed by Wahba (1990), the kernel method and RKHS serve as a uniﬁed framework of a group of nonparametric models, which extends the spline method.

Original: Later, kernel models become an important component in nonparametric models.
Revised: Later, kernel models become an important component in nonparametric models.

Original: In machine learning, the kernel-based method is always referred to as the “kernel trick”.
Revised: In machine learning, the kernel-based method is always referred to as the “kernel trick”.

Original: By lifting the x variable to a high dimensional space via the kernel method, we can explore possibly nonlinear relationships between variables.
Revised: By lifting the x variable to a high dimensional space via the kernel method, we can explore possibly nonlinear relationships between variables.

Original: Moreover, to play the kernel trick, one can directly calculate the kernel function using original features.
Revised: Moreover, to play the kernel trick, one can directly calculate the kernel function using original features.

Original: This is computationally efﬁcient since we avoid calculating high dimension or inﬁnite dimension features.
Revised: This is computationally efﬁcient since we avoid calculating high dimension or inﬁnite dimension features.

Original: For applications of kernel method in machine learning, one can see kernel regression for image processing (Takeda et al., 2007), for text mining (Greene & Cunningham, 2006), and for tasks in bioinformatics (Saigo et al., 2004).
Revised: For applications of kernel method in machine learning, one can see kernel regression for image processing (Takeda et al., 2007), for text mining (Greene & Cunningham, 2006), and for tasks in bioinformatics (Saigo et al., 2004).

Original: Regarding deep learning, the kernel method is also important because it has implications for deep learning models.
Revised: Regarding deep learning, the kernel method is also important because it has implications for deep learning models.

Original: On the one hand, the kernel methods have similar benign overﬁtting behavior to neural network due to the implicit regularization and/or implicit bias phenomenon that we will review in the next subsection (Belkin et al., 2018).
Revised: On the one hand, the kernel methods have similar benign overﬁtting behavior to neural network due to the implicit regularization and/or implicit bias phenomenon that we will review in the next subsection (Belkin et al., 2018).

Original: On the other hand, the RKHS itself is closely related to Neural Networks via the Neural Tangent Kernel theory (Jacot et al., 2018).
Revised: On the other hand, the RKHS itself is closely related to Neural Networks via the Neural Tangent Kernel theory (Jacot et al., 2018).

Original: This all indicates that to understand deep learning, one should ﬁrst study kernel methods.
Revised: This all indicates that to understand deep learning, one should ﬁrst study kernel methods.

Original: 2. D IRECTIONAL B IAS
Revised: 2. D IRECTIONAL B IAS

Original: This paper analyzes the directional bias of SGD for the nonparametric regression.
Revised: This paper analyzes the directional bias of SGD for the nonparametric regression.

Original: Directional bias, also referred to as implicit bias, of an algorithm refers to that its solution path is biased towards a certain direction.
Revised: Directional bias, also referred to as implicit bias, of an algorithm refers to that its solution path is biased towards a certain direction.

Original: It works as that the algorithm prefers some directions over the others even though they may have the same objective function value.
Revised: It works as that the algorithm prefers some directions over the others even though they may have the same objective function value.

Original: Since the algorithm selects a direction by itself, instead of explicitly required by any constraint, people use the term “implicit”.
Revised: Since the algorithm selects a direction by itself, instead of explicitly required by any constraint, people use the term “implicit”.

Original: It is worth noting that the implicit regularization is related to implicit bias.
Revised: It is worth noting that the implicit regularization is related to implicit bias.

Original: The implicit regularization refers to that the converged point of an algorithm is like a regularized estimator, even if the objective function is not explicitly regularized.
Revised: The implicit regularization refers to that the converged point of an algorithm is like a regularized estimator, even if the objective function is not explicitly regularized.

Original: One can also interpret implicit regularization as the “ﬁnal result” of implicit bias.
Revised: One can also interpret implicit regularization as the “ﬁnal result” of implicit bias.

Original: In the recent work by Derezinski et al.
Revised: In the recent work by Derezinski et al.

Original: (2020), implicit regularization is used to develop an exact bound for double descent in linear regression.
Revised: (2020), implicit regularization is used to develop an exact bound for double descent in linear regression.

Original: In this way, implicit regularization/bias serves as a way to explain some deep learning phenomenons that could not be addressed by the classical empirical risk minimization (ERM) framework.
Revised: In this way, implicit regularization/bias serves as a way to explain some deep learning phenomenons that could not be addressed by the classical empirical risk minimization (ERM) framework.

Original: Therefore, it is important to study directional bias.
Revised: Therefore, it is important to study directional bias.

Original: State-of-the-art study on the directional bias of ﬁrst-order algorithms can be divided into two categories by the technique they use:
Revised: State-of-the-art study on the directional bias of ﬁrst-order algorithms can be divided into two categories by the technique they use:

Original: The ﬁrst category is the (stochastic) gradient ﬂow method, by taking an inﬁnitesimal step size in (S)GD, the parameter dynamic follows a (stochastic) differential equation.
Revised: The ﬁrst category is the (stochastic) gradient ﬂow method, by taking an inﬁnitesimal step size in (S)GD, the parameter dynamic follows a (stochastic) differential equation.

Original: Studying the solution path and the stationary point of the underlying differential equation helps to reveal the property of the parameter estimation.
Revised: Studying the solution path and the stationary point of the underlying differential equation helps to reveal the property of the parameter estimation.

Original: We list some works that use the ﬁrst method to show the directional bias result of the (stochastic) gradient descent.
Revised: We list some works that use the ﬁrst method to show the directional bias result of the (stochastic) gradient descent.

Original: Liu et al.
Revised: Liu et al.

Original: (2018) analyze the Momentum SGD (MSGD) with inﬁnitesimal step size, and show that the solution path escapes from the saddle for a nonconvex objective function.
Revised: (2018) analyze the Momentum SGD (MSGD) with inﬁnitesimal step size, and show that the solution path escapes from the saddle for a nonconvex objective function.

Original: It is worth noting that in their case, the associated stochastic differential equation deﬁnes a complicated stochastic process, thus they replace it with an appropriate diffusion process, and the analysis is done based on such diffusion approximation.
Revised: It is worth noting that in their case, the associated stochastic differential equation deﬁnes a complicated stochastic process, thus they replace it with an appropriate diffusion process, and the analysis is done based on such diffusion approximation.

Original: If one analyzes a stochastic gradient ﬂow and ﬁnds it intractable, one may consider using the technique of diffusion approximation.
Revised: If one analyzes a stochastic gradient ﬂow and ﬁnds it intractable, one may consider using the technique of diffusion approximation.

Original: Ali et al.
Revised: Ali et al.

Original: (2020) shows the stochastic gradient ﬂow for the linear regression problem min w (cid:107)
Revised: (2020) shows the stochastic gradient ﬂow for the linear regression problem min w (cid:107)

Original: X w − y
Revised: X w − y

Original: (cid:107)has a solution path close to the solution path of Ridge regression; Blanc et al.
Revised: (cid:107)has a solution path close to the solution path of Ridge regression; Blanc et al.

Original: (2020) shows the stochastic gradient ﬂow for a general loss function has a solution path close to the solution path of gradient ﬂow on the objective function plus some extra penalty terms, and they explicitly identify the penalty terms; Smith et al.
Revised: (2020) shows the stochastic gradient ﬂow for a general loss function has a solution path close to the solution path of gradient ﬂow on the objective function plus some extra penalty terms, and they explicitly identify the penalty terms; Smith et al.

Original: (2021) go one more step from the inﬁnitesimal step size to small step size, and characterize the effect of small step size as an extra penalty term in the gradient ﬂow.
Revised: (2021) go one more step from the inﬁnitesimal step size to small step size, and characterize the effect of small step size as an extra penalty term in the gradient ﬂow.

Original: Another category is analyzing the discrete (S)GD sequence.
Revised: Another category is analyzing the discrete (S)GD sequence.

Original: This technique in general just requires a moderate step size such that the algorithm converges (or nearly converges), thus it is more meaningful from a practical perspective.
Revised: This technique in general just requires a moderate step size such that the algorithm converges (or nearly converges), thus it is more meaningful from a practical perspective.

Original: We also ﬁnd some directional bias work that is based on this technique.
Revised: We also ﬁnd some directional bias work that is based on this technique.

Original: Vaskevicius et al.
Revised: Vaskevicius et al.

Original: ; Zhao et al.
Revised: ; Zhao et al.

Original: ; Fan et al.
Revised: ; Fan et al.

Original: (2021) analyze Hadamard reparameterized GD in sparse regression.
Revised: (2021) analyze Hadamard reparameterized GD in sparse regression.

Original: They divide the true parameter into strong, weak, and 0 parts, and for each part, they carefully develop the stepwise error bound for each step of GD.
Revised: They divide the true parameter into strong, weak, and 0 parts, and for each part, they carefully develop the stepwise error bound for each step of GD.

Original: They ﬁnally show that an early-stopped estimator along the solution path achieves the minimax optimal error rate for sparse regression, which indicates that the solution path is in the direction that biased towards a sparse solution.
Revised: They ﬁnally show that an early-stopped estimator along the solution path achieves the minimax optimal error rate for sparse regression, which indicates that the solution path is in the direction that biased towards a sparse solution.

Original: Recently, Wu et al.
Revised: Recently, Wu et al.

Original: (2021) show that for overparameterized linear regression, SGD with moderate step size converges to the minimum norm interpolant in the direction that corresponds to the largest eigenvalue of the design matrix.
Revised: (2021) show that for overparameterized linear regression, SGD with moderate step size converges to the minimum norm interpolant in the direction that corresponds to the largest eigenvalue of the design matrix, while GD converges in the direction that corresponds to the smallest eigenvalue.

Revised: For Neural Networks in the ‘lazy training’ regime, ?

Revised: shows that GD also converges in the direction of the smallest eigenvalue of the Neural Tangent Kernel.

Revised: Their result further reveal the mechanism of the directional bias as: GD ﬁts the direction of a larger eigenvalue faster at the beginning of the training, left the smaller eigenvalue direction unﬁtted; later the direction of smaller eigenvalue is ﬁtted, resulting in that the estimator goes in this direction.

Original:  P ROBLEM F ORMULATION
Revised:  P ROBLEM F ORMULATION

Original: We give our problem formulation in this section.
Revised: We give our problem formulation in this section.

Original: In Subsection 3.1 we deﬁne the kernel regression model and objective function; in Subsection 3.2, we give the SGD and GD algorithms; in Subsection 3.3, we state our assumption for later analysis.
Revised: In Subsection 3.1 we deﬁne the kernel regression model and objective function; in Subsection 3.2, we give the SGD and GD algorithms; in Subsection 3.3, we state our assumption for later analysis.

Original: Due to the page limit, details of the nonparametric regression, RKHS and justiﬁcations for the assumption are deferred to Appendix A and B.
Revised: Due to the page limit, details of the nonparametric regression, RKHS and justiﬁcations for the assumption are deferred to Appendix A and B.

Original: 3. K ERNEL R EGRESSION
Revised: 3. K ERNEL R EGRESSION

Original: Suppose we are given n data pairs { x i , y i } ni =1 generated from an unknown model y =
Revised: Suppose we are given n data pairs { x i , y i } ni =1 generated from an unknown model y =

Original: f
Revised: f

Original: ( x ) , where x i ∈ X ⊂ R p and y i ∈ R .
Revised: ( x ) , where x i ∈ X ⊂ R p and y i ∈ R .

Original: The goal is to estimate the unknown model f from the data.
Revised: The goal is to estimate the unknown model f from the data.

Original: To achieve the goal, one way is to ﬁnd an f that minimizes the empirical risk function
Revised: To achieve the goal, one way is to ﬁnd an f that minimizes the empirical risk function

Original: [Equation]
Revised: [Equation]

Original: where (cid:96) is the loss function.
Revised: where (cid:96) is the loss function.

Original: For the regression task, we use the squared loss (cid:96) ( y, x ) = 12 ( y − f ( x )) 2 .
Revised: For the regression task, we use the squared loss (cid:96) ( y, x ) = 12 ( y − f ( x )) 2 .

Original: One can see that problem (1) is not well-deﬁned, as there are inﬁnitely many solutions to ∀ i : f ( x i ) =
Revised: One can see that problem (1) is not well-deﬁned, as there are inﬁnitely many solutions to ∀ i : f ( x i ) =

Original: y i , and some of them do not generalize for a new test data.
Revised: y i , and some of them do not generalize for a new test data.

Original: One way to ﬁx it is to restrict f ∈ H and add regularization term in (cid:107) f (cid:107) H to problem (1) for smoothness, where H is a
Revised: One way to ﬁx it is to restrict f ∈ H and add regularization term in (cid:107) f (cid:107) H to problem (1) for smoothness, where H is a

Original: RKHS with reproducing kernel K ( · , · ) and (cid:107) · (cid:107) H is the Hilbert norm.
Revised: RKHS with reproducing kernel K ( · , · ) and (cid:107) · (cid:107) H is the Hilbert norm.

Original: Adding these restrictions and applying Representer Theorem, problem (1) with the squared loss becomes
Revised: Adding these restrictions and applying Representer Theorem, problem (1) with the squared loss becomes

Original: [Equation]
Revised: [Equation]

Original: where K Ti is the i th row of K :
Revised: where K Ti is the i th row of K :

Original: =
Revised: =

Original: K ( X, X ) =
Revised: K ( X, X ) =

Original: ( K ( x i , x j ))
Revised: ( K ( x i , x j ))

Original: i,j .
Revised: i,j .

Original: For a parameter α , the corresponding estimator in H is f ( · ) =
Revised: For a parameter α , the corresponding estimator in H is f ( · ) =

Original: (cid:80) ni
Revised: (cid:80) ni

Original: =

Revised: =

Original: α i K
Revised: α i K

Original: ( x i, · ) :
Revised: [Equation] i

Original: = α T K ( · , X ) .
Revised: [Equation]

Original: Now when K is invertible, it is trivial that any algorithm on objective function (2) (if it converges) converges at the unique minimal, that is, ˆ α = K
Revised: Now when K is invertible, it is trivial that any algorithm on objective function (2) (if it converges) converges at the unique minimal, that is, ˆ α = K

Original: ( X, X ) − 1 y , result in the RKHS functional estimator
Revised: ( X, X ) − 1 y , result in the RKHS functional estimator

Original: [Equation]
Revised: [Equation]

Original: where K ( x , X ) T =
Revised: where K ( x , X ) T =

Original: ( K ( x , x 1 ) , . . .
Revised: ( K ( x , x 1 ) , . . .

Original: , K ( x , x n )) .
Revised: , K ( x , x n )) .

Original: Estimator (3) is the minimum norm interpolant as given by following problem:
Revised: Estimator (3) is the minimum norm interpolant as given by following problem:

Original: [Equation]
Revised: [Equation]

Original: And its property has been studied in Liang & Rakhlin (2020).
Revised: And its property has been studied in Liang & Rakhlin (2020).

Original: In this paper, we compare the convergence direction of SGD and GD to ˆ α .
Revised: In this paper, we compare the convergence direction of SGD and GD to ˆ α .

Original: Speciﬁcally, we consider a two-stage SGD with a phase transition from a larger step size to a decreased step size.
Revised: Speciﬁcally, we consider a two-stage SGD with a phase transition from a larger step size to a decreased step size.

Original: Note that this matches the training scheme people always use in practice for SGD algorithms: decreasing the step size after training for a few epochs.
Revised: Note that this matches the training scheme people always use in practice for SGD algorithms: decreasing the step size after training for a few epochs.

Original: For that purpose, in the following sections, we deﬁne the one-step SGD/GD update and state our assumptions and notations for analysis.
Revised: For that purpose, in the following sections, we deﬁne the one-step SGD/GD update and state our assumptions and notations for analysis.

Original: 3. O NE STEP SGD/GD UPDATE
Revised: 3. O NE STEP SGD/GD UPDATE

Original: In this paper, we consider the SGD algorithm as follows.
Revised: In this paper, we consider the SGD algorithm as follows.

Original: For objective function (2), denote the parameter estimation at t th step as α t , then SGD update α t +1 as
Revised: For objective function (2), denote the parameter estimation at t th step as α t , then SGD update α t +1 as

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where i t is uniformly random sampled from { 1 , . .
Revised: where i t is uniformly random sampled from { 1 , . .

Original: . , n } .
Revised: . , n } .

Original: The GD update α t +1 as
Revised: The GD update α t +1 as

Original: [Equation]
Revised: [Equation]

Original: 3. A SSUMPTIONS AND N OTATIONS
Revised: 3. A SSUMPTIONS AND N OTATIONS

Original: We state our assumption on the gram matrix in a uniﬁed format.
Revised: We state our assumption on the gram matrix in a uniﬁed format.

Original: Later we show in Appendix B that some popular kernel families satisfy our assumption.
Revised: Later we show in Appendix B that some popular kernel families satisfy our assumption.

Original: Assumption 1 (Diagonal Dominant gram matrix) .
Revised: Assumption 1 (Diagonal Dominant gram matrix) .

Original: Denote by K = K
Revised: Denote by K = K

Original: ( X, X ) the gram matrix, we assume that K is diagonal dominant.
Revised: ( X, X ) the gram matrix, we assume that K is diagonal dominant.

Original: Speciﬁcally, suppose w.l.o.g. that K 1 , 1 ≥ K 2 , 2 ≥ . .
Revised: Speciﬁcally, suppose w.l.o.g. that K 1 , 1 ≥ K 2 , 2 ≥ . .

Original: ≥
Revised: ≥

Original: K n,n > 0 , then we have for a small value τ that
Revised: K n,n > 0 , then we have for a small value τ that

Original: [Equation]
Revised: [Equation]

Original: Remark 2.
Revised: Remark 2.

Original: Diagonal dominant gram matrix is common in kernel learning.
Revised: Diagonal dominant gram matrix is common in kernel learning.

Original: Mathematically, one can justify that a gram matrix is diagonal dominant by imposing proper assumptions on the kernel function K ( · , · ) and the data distribution.
Revised: Mathematically, one can justify that a gram matrix is diagonal dominant by imposing proper assumptions on the kernel function K ( · , · ) and the data distribution.

Original: In Appendix B we give a setting of data distribution such that some  popularkernels produce diagonal dominant kernel matrices with high probability.
Revised: The following proposition shows diagonal dominance for bilinear kernel. We can check for some other popular kernel to be diagonal dominant, which we defer to Appendix B due to the page limit.

Original: Remark 3.
Revised: Remark 3.

Original: Think of the kernel function as the inner product of high-dimensional features, the resulting gram matrix is diagonal dominant when the high-dimension features are sparse.
Revised: Think of the kernel function as the inner product of high-dimensional features, the resulting gram matrix is diagonal dominant when the high-dimension features are sparse.

Original: This happens for a lot of practical problems (Sch¨olkopf et al., 2002; Weston et al., 2003), for example, when linear or string kernels are applied to text data (Greene & Cunningham, 2006), when domainspeciﬁc kernels are applied to image retrieval (Tao et al., 2004) and bioinformatics (Saigo et al., 2004), and when the Global Alignment kernel is applied to most datasets (Cuturi et al., 2007; Cuturi, 2011). 
Revised: This happens for a lot of practical problems (Sch¨olkopf et al., 2002; Weston et al., 2003), for example, when linear or string kernels are applied to text data (Greene & Cunningham, 2006), when domainspeciﬁc kernels are applied to image retrieval (Tao et al., 2004) and bioinformatics (Saigo et al., 2004), and when the Global Alignment kernel is applied to most datasets (Cuturi et al., 2007; Cuturi, 2011). Proposition 4 (Lemma 1 in Wu et al. Consider the bilinear kernel K ( x , x (cid:48) ) : [Equation]

Original: It is meaningful to note that the diagonal dominance is undesired in classiﬁcation and clustering tasks.
Revised: Assume the data x i , i = 1 , . . . , n are i.i.d. uniformly distributed on the unit sphere S d − 1 , where d (cid:29) n . When d ≥ 4 log(2 n 2 /δ ) for some δ ∈ (0 , 1) .

Revised: Then with probability at least 1 − δ , we have

Revised: [Equation]

Revised: It is meaningful to note that the diagonal dominance is undesired in classiﬁcation and clustering tasks.

Original: It indicates that the data pieces are dissimilar to each other as measured by the kernel function, and thus generates very little information for classiﬁcation/clustering.
Revised: It indicates that the data pieces are dissimilar to each other as measured by the kernel function, and thus generates very little information for classiﬁcation/clustering.

Original: One may ﬁnd a lot of works on solving the issue of diagonal dominance in these cases, for example, Greene &
Revised: One may ﬁnd a lot of works on solving the issue of diagonal dominance in these cases, for example, Greene &

Original: Cunningham (2006); Kandola et al. (2003).
Revised: Cunningham (2006); Kandola et al. (2003).

Original: But for the regression task, the diagonal dominance, in other words, the dissimilarity of data points, may have beneﬁts.
Revised: But for the regression task, the diagonal dominance, in other words, the dissimilarity of data points, may have beneﬁts.

Original: One can ﬁnd similar conditions such as Restricted Isometry Property and s − goodness that describes linearly dissimilar features in a huge regression literature as Candes & Tao (2007); Cand ` es (2008); Chen & Donoho (1994).
Revised: One can ﬁnd similar conditions such as Restricted Isometry Property and s − goodness that describes linearly dissimilar features in a huge regression literature as Candes & Tao (2007); Cand ` es (2008); Chen & Donoho (1994).

Original: Such conditions are required for proving minimax optimality or exact recovery of a sparse signal in sparse settings.
Revised: Such conditions are required for proving minimax optimality or exact recovery of a sparse signal in sparse settings.

Original: In our case, we adopt the dissimilarity concept and apply it to data points in high-dimensional nonlinear feature space.
Revised: In our case, we adopt the dissimilarity concept and apply it to data points in high-dimensional nonlinear feature space.

Original: Later we will see that in the existence of diagonal dominance, the directional bias drives SGD to select a good solution that generalizes well among all solutions of the same level of empirical loss.
Revised: Later we will see that in the existence of diagonal dominance, the directional bias drives SGD to select a good solution that generalizes well among all solutions of the same level of empirical loss.

Original: In this way, our SGD estimator beneﬁts from the diagonal dominance.
Revised: In this way, our SGD estimator beneﬁts from the diagonal dominance.

Original: Notations .
Revised: Notations .

Original: We use the following notations throughout the remaining of this work.
Revised: We use the following notations throughout the remaining of this work.

Original: For the gram matrix K , denote K i,j be the element at i th row j th column of K .
Revised: For the gram matrix K , denote K i,j be the element at i th row j th column of K .

Original: Denote λ i
Revised: Denote λ i

Original: =
Revised: =

Original: K i,i =
Revised: K i,i =

Original: K
Revised: K

Original: ( x i , x i ) , and assume w.l.o.g.
Revised: ( x i , x i ) , and assume w.l.o.g.

Original: that λ 1 ≥ λ 2 ≥ . . .
Revised: that λ 1 ≥ λ 2 ≥ . . .

Original: ≥ λ n .
Revised: ≥ λ n .

Original: Denote the i th column of K as K i , let
Revised: Denote the i th column of K as K i , let

Original: K − 1 =
Revised: K − 1 =

Original: [ K 2 , . .
Revised: [ K 2 , . .

Original: . , K n ] .
Revised: . , K n ] .

Original: Assume K is full rank, denote P − 1 the projection onto column space of
Revised: Assume K is full rank, denote P − 1 the projection onto column space of

Original: K − 1 , and P 1 =
Revised: K − 1 , and P 1 =

Original: I − P − 1 .
Revised: I − P − 1 .

Original: And denote γ 1 ≥ . .
Revised: And denote γ 1 ≥ . .

Original: ≥ γ n > 0 eigenvalues of K in non-increasing order.
Revised: ≥ γ n > 0 eigenvalues of K in non-increasing order.

Original:  M AIN RESULT
Revised:  M AIN RESULT

Original: The main results are presented in two subsections: Subsection 4.1 states the different directional bias result of SGD and GD estimators; Subsection 4.2 shows that directional bias towards a certain directional leads to good generalization performance, and further applies this result to show that
Revised: The main results are presented in two subsections: Subsection 4.1 states the different directional bias result of SGD and GD estimators; Subsection 4.2 shows that directional bias towards a certain directional leads to good generalization performance, and further applies this result to show that

Original: SGD generalizes better than GD.
Revised: SGD generalizes better than GD.

Original: 4. D IRECTIONAL BIAS
Revised: 4. D IRECTIONAL BIAS

Original: Since we assumed that K is full rank, then SGD and GD algorithm on objective function (2) converges to ˆ α = K − 1 y (when they converge).
Revised: Since we assumed that K is full rank, then SGD and GD algorithm on objective function (2) converges to ˆ α = K − 1 y (when they converge).

Original: We are interested in the direction at which α t converges to ˆ α , i.e. the quantity 
Revised: We are interested in the direction at which α t converges to ˆ α , i.e. the quantity b t :=

Original: [Equation]
Revised: α t − ˆ α

Original: With assumption 1 that the gram matrix is diagonal dominant, we prove that a two-stage SGD has b t converges in the direction of the largest eigenvector of K .
Revised: With assumption 1 that the gram matrix is diagonal dominant, we prove that a two-stage SGD has b t converge in the direction of the largest eigenvector of K .

Original: Theorem 4 (Direction bias of SGD estimator) .
Revised: Theorem 5 (Direction bias of SGD estimator) .

Original: Assume Assumption 1 holds, run a two-stage SGD with a ﬁxed step size for each stage: stage 1 with step size η 1 for steps 1 , . . .
Revised: Assume Assumption 1 holds, run a two-stage SGD with a ﬁxed step size for each stage: stage 1 with step size η 1 for steps 1 , . . .

Original: , k 1 , stage 2 with step size η 2 for steps k 1 + 1 , . . .
Revised: , k 1 , stage 2 with step size η 2 for steps k 1 + 1 , . . .

Original: , k 2 , such that
Revised: , k 2 , such that

Original: [Equation]
Revised: [Equation]

Original: where C 1 , C 2 , C 3 are constants that are speciﬁed in the Appendix E. For a small (cid:15) > 

Revised: where C 1 , C 2 , C 3 are constants that are speciﬁed in the Appendix E. For a small (cid:15) >

Original: such that nτ < poly ( (cid:15) )
Revised: such that nτ < poly ( (cid:15) )

Original: there exists k 1 =
Revised: there exists k 1 =

Original: O (log 1 (cid:15) ) and k 2 such that
Revised: O (log 1 (cid:15) ) and k 2 such that

Original: [Equation]
Revised: [Equation]

Original: That is, b SGDkis close to the direction of the largest eigenvector of K .
Revised: That is, b SGDkis close to the direction of the largest eigenvector of K .

Original: Remark 5.
Revised: Remark 6.

Original: One should assume τ in Assumption 1 to be small enough for (cid:15) to be very small if one would like the resulting estimator b SGDkto have the direction that corresponds to the largest eigenvalue of K .
Revised: One should assume τ in Assumption 1 to be small enough for (cid:15) to be very small if one would like the resulting estimator b SGDkto have the direction that corresponds to the largest eigenvalue of K .

Original: Later we will see that if one only wants different directional bias of SGD and GD estimators, a moderate (cid:15) is allowed and then the assumption on τ is not that strong.
Revised: Later we will see that if one only wants different directional bias of SGD and GD estimators, a moderate (cid:15) is allowed and then the assumption on τ is not that strong.

Original: The proof of Theorem 4 is in Appendix E.
Revised: The proof of Theorem 5 is in Appendix E.

Original: Next, we see that GD has b t converges in the direction of the smallest eigenvector of K , which contrasts with the directional bias of SGD.
Revised: Next, we see that GD has b t converge in the direction of the smallest eigenvector of K , which contrasts with the directional bias of SGD.

Original: Theorem 6 (Direction bias of GD estimator) .
Revised: Theorem 7 (Direction bias of GD estimator) .

Original: Assume Assumption 1 hold, run GD with a ﬁxed step size:
Revised: Assume Assumption 1 hold, run GD with a ﬁxed step size:

Original: [Equation]
Revised: [Equation]

Original: for a small (cid:15) (cid:48) > 0 , run k
Revised: for a small (cid:15) (cid:48) > 0 , run k

Original: = O (log 1 (cid:15) (cid:48) ) steps of GD, we have
Revised: = O (log 1 (cid:15) (cid:48) ) steps of GD, we have

Original: [Equation]
Revised: [Equation]

Original: That is, b t is close to the direction of the smallest eigenvector of K .
Revised: That is, b t is close to the direction of the smallest eigenvector of K .

Original: Remark 7.
Revised: Remark 8.

Original: The assumption (on τ ) is mild for differentiating the directional bias of SGD and GD.
Revised: The assumption (on τ ) is mild for differentiating the directional bias of SGD and GD.

Original: Comparing Theorem 4 and 6, we see that as long as γ n < (1 − 2 (cid:15) )
Revised: Comparing Theorem 5 and 7, we see that as long as γ n < (1 − 2 (cid:15) )

Original: γ 1 , by taking k large enough we always have
Revised: γ 1 , by taking k large enough we always have

Original: [Equation]
Revised: [Equation]

Original: That is, one may expect b SGD kto be in the direction of larger eigenvalue compared with b GD k .
Revised: That is, one may expect b SGDkto be in the direction of larger eigenvalue compared with b GDk .

Original: In the following subsection, we see that the directional bias towards a larger eigenvalue of the kernel is good for generalization, which leads to our title that directional bias helps SGD to generalize in kernel regression.
Revised: In the following subsection, we see that the directional bias towards a larger eigenvalue of the kernel is good for generalization, which leads to our title that directional bias helps SGD to generalize in kernel regression.

Original: The proof of Theorem 6 is in Appendix F.
Revised: The proof of Theorem 7 is in Appendix F.

Original: Although there is assumption 1 in Theorem 6, it is just used to bound the step size so that GD converges; the diagonal dominant structure of K is not required for the directional bias for GD to hold.
Revised: Although there is assumption 1 in Theorem 7, it is just used to bound the step size so that GD converges; the diagonal dominant structure of K is not required for the directional bias for GD to hold.

Original: Moreover, the choice of (cid:15) (cid:48) is independent of the assumption on τ , then for an arbitrarily small (cid:15) (cid:48) > 0 , we can always run GD long enough such that the theorem holds and the estimator b GDk is arbitrarily close to the smallest eigenvector.
Revised: Moreover, the choice of (cid:15) (cid:48) is independent of the assumption on τ , then for an arbitrarily small (cid:15) (cid:48) > 0 , we can always run GD long enough such that the theorem holds and the estimator b GD k is arbitrarily close to the smallest eigenvector.

Original: 4. E FFECT OF DIRECTIONAL BIAS
Revised: 4. E FFECT OF DIRECTIONAL BIAS

Original: In this subsection, the estimator that has a directional bias towards the largest eigenvalue of the
Revised: In this subsection, the estimator that has a directional bias towards the largest eigenvalue of the

Original: Hessian is shown to give the best parameter estimation error among all estimators that have the same squared in-sample loss, see Theorem 8.
Revised: Hessian is shown to give the best parameter estimation error among all estimators that have the same squared in-sample loss, see Theorem 9.

Original: Later we deﬁne a realizable problem setting of kernel regression where the generalization error depends on a term similar to the parameter estimation error, and in this way, the directional bias helps SGD to generalize.
Revised: Later we deﬁne a realizable problem setting of kernel regression where the generalization error depends on a term similar to the parameter estimation error, and in this way, the directional bias helps SGD to generalize.

Original: Theorem 8.
Revised: Theorem 9.

Original: Consider approximately minimizing the quadratic loss
Revised: Consider approximately minimizing the quadratic loss

Original: [Equation]
Revised: [Equation]

Original: Assume there is a ground truth w ∗ such that y
Revised: Assume there is a ground truth w ∗ such that y

Original: =
Revised: =

Original: A w ∗ , for a ﬁxed level of the quadratic loss, the parameter estimation error (cid:107) w − w ∗ (cid:107) 22 has a lower bound
Revised: A w ∗ , for a ﬁxed level of the quadratic loss, the parameter estimation error (cid:107) w − w ∗ (cid:107) 22 has a lower bound

Original: [Equation]
Revised: [Equation]

Original: Moreover, the equality is obtained when w − w ∗ is in the direction of the eigenvector that corresponds to the largest eigenvalue of A T A .
Revised: Moreover, the equality is obtained when w − w ∗ is in the direction of the eigenvector that corresponds to the largest eigenvalue of A T A .

Original: Remark 9.
Revised: Remark 10.

Original: Theorem 8 implies that the directional bias towards the largest eigenvalue is good for parameter estimation.
Revised: Theorem 9 implies that the directional bias towards the largest eigenvalue is good for parameter estimation.

Original: As discussed in Remark 7, the SGD estimator is biased towards a larger eigenvalue compared to the GD estimator, then by Theorem 8 the SGD estimator better estimates the true parameter and thus generalizes better.
Revised: As discussed in Remark 8, the SGD estimator is biased towards a larger eigenvalue compared to the GD estimator, then by Theorem 9 the SGD estimator better estimates the true parameter and thus generalizes better.

Original: We formalize this statement in the following paragraphs.
Revised: We formalize this statement in the following paragraphs.

Original: The proof of Theorem 8 is in Appendix G.1.
Revised: The proof of Theorem 9 is in Appendix G.1.

Original: Suppose ∃ f ∗ ∈ H such that
Revised: Suppose ∃ f ∗ ∈ H such that

Original: y
Revised: y

Original: =
Revised: =

Original: f
Revised: f

Original: ∗ ( x ) .
Revised: ∗ ( x ) .

Original: Consider the generalization error L D ( f ) :
Revised: Consider the generalization error L D ( f ) :

Original: = (cid:107) f − f ∗ (cid:107) 2 H , for an algorithm output f alg , we decompose its generalization error as: 
Revised: = (cid:107) f − f ∗ (cid:107) 2 H , for an algorithm output f alg , we decompose its generalization error as: [Equation] (cid:123)(cid:122) (cid:125) [Equation]

Revised: =

Original: [Equation]
Revised: [Equation]

Revised: f alg )

Revised: [Equation]

Revised: estimation error

Revised: [Equation]

Original: where H s is the hypothesis class that the output of the algorithm is restricted to.
Revised: where H s is the hypothesis class that the output of the algorithm is restricted to.

Original: By formulation (2), we have our hypothesis class as
Revised: By formulation (2), we have our hypothesis class as

Original: [Equation]
Revised: [Equation]

Original: We deﬁne the a − level set of training loss:
Revised: We deﬁne the a − level set of training loss:

Original: [Equation]
Revised: [Equation]

Original: denote ∆ ∗ a :=
Revised: denote ∆ ∗ a :=

Original: inf f ∈ ν a ∆( f ) .
Revised: inf f ∈ ν a ∆( f ) .

Original: Note that the approximation error can not be improved by choice of algorithm unless we change the hypothesis class, which is, changing the problem formulation in our case.
Revised: Note that the approximation error can not be improved by choice of algorithm unless we change the hypothesis class, which is, changing the problem formulation in our case.

Original: So we just minimize the estimation error for estimators that are in the a − level set.
Revised: So we just minimize the estimation error for estimators that are in the a − level set.

Original: As shown in Appendix G.2, one can check the estimation error is given by
Revised: As shown in Appendix G.2, one can check the estimation error is given by

Original: [Equation]
Revised: [Equation]

Original: where b = α − ˆ α .
Revised: where b = α − ˆ α .

Original: By similar reasoning as Theorem 8, the estimation error is minimized when b is in the direction of the largest eigenvalue of K , so the directional bias towards a larger eigenvalue helps to generalize in kernel regression.
Revised: By similar reasoning as Theorem 9, the estimation error is minimized when b is in the direction of the largest eigenvalue of K , so the directional bias towards a larger eigenvalue helps to generalize in kernel regression.

Original: We formalize the statement for comparing the estimation error of SGD and GD in the following theorem.
Revised: We formalize the statement for comparing the estimation error of SGD and GD in the following theorem.

Original: Theorem 10 (Generalization performance) .
Revised: Theorem 11 (Generalization performance) .

Original: Follow Theorems 4 and 6, we have the following:
Revised: Follow Theorems 5 and 7, we have the following:

Original: • The output of SGD has E [∆ 1 / 2 ( f SGD )] ≤ (1 + 4 (cid:15) )
Revised: • The output of SGD has E [∆ 1 / 2 ( f SGD )] ≤ (1 + 4 (cid:15) )

Original: (∆ ∗ a ) 1 / 2 , where a is such that E [ (cid:107) K α SGD − y (cid:107) 2 ] 2 = 2 na and (cid:15) could be any positive small constant; • The output of GD has ∆( f GD ) ≥ M ∆ ∗ a , where a is the training loss of GD estimator, and M = γ 1 γ n (1 − (cid:15) (cid:48) ) > 1 is a large constant.
Revised: (∆ ∗ a ) 1 / 2 , where a is such that E [ (cid:107) K α SGD − y (cid:107) 2 ] 2 = 2 na and (cid:15) could be any positive small constant; • The output of GD has ∆( f GD ) ≥ M ∆ ∗ a , where a is the training loss of GD estimator, and M = γ 1 γ n (1 − (cid:15) (cid:48) ) > 1 is a large constant.

Original: Remark 11.
Revised: Remark 12.

Original: One read from the theorem that the SGD estimator is (8 (cid:15) + 16 (cid:15) 2 ) − optimum, while
Revised: One read from the theorem that the SGD estimator is (8 (cid:15) + 16 (cid:15) 2 ) − optimum, while

Original: GD estimator is ( M − 1) − sub-optimum.
Revised: GD estimator is ( M − 1) − sub-optimum.

Original: Combine with Theorem 6 that (cid:15) (cid:48)
Revised: Combine with Theorem 7 that (cid:15) (cid:48)

Original: k →∞ −→ 0 , we can take (cid:15) in
Revised: k →∞ −→ 0 , we can take (cid:15) in

Original: Theorem 4 such that (1 + 4 (cid:15) ) 2 < γ 1 /γ n to have ∆( f SGD ) <
Revised: Theorem 5 such that (1 + 4 (cid:15) ) 2 < γ 1 /γ n to have ∆( f SGD ) <

Original: ∆( f GD ) with high probability.
Revised: ∆( f GD ) with high probability.

Original: Thisﬁnishes ourclaim that the SGD estimator generalize better than the GD estimator.
Revised: This our that the SGD estimator generalize better than the GD estimator.

Original: The detailed proof of Theorem 10 is left to Appendix G.2.
Revised: The detailed proof of Theorem 11 is left to Appendix G.2.

Original: Numeric Study.
Revised: Numeric Study.

Original: Figure 1 shows the simulation results of kernel regression, Figure 2 shows the results of a small ResNet-like Neural Network on FashionMNIST data (Xiao et al., 2017).
Revised: Figure 1 shows the simulation results of kernel regression, Figure 2 shows the results of a small ResNet-like Neural Network on FashionMNIST data (Xiao et al., 2017).

Original: Figure 1 and Figure 2a supports the directional bias results in Theorems 4 and 6, and Figure 2b validates
Revised: Figure 1 and Figure 2a supports the directional bias results in Theorems 5 and 7, and Figure 2b validates

Original: Theorem 10.
Revised: Theorem 11.

Original: For details of the experiments and more experiments, see Appendix H.
Revised: For details of the experiments and more experiments, see Appendix H.

Original: [Figure]

Original: Remark 12.
Revised: Remark 13.

Original: The purpose of experiment using a Neural Network (Figure 2) is as following: ﬁrst, the
Revised: The purpose of experiment using a Neural Network (Figure 2) is as following: ﬁrst, the

Original: Neural Network results support our ﬁnding on kernel regression, since Neural Network is related to kernel regression through NTK theory (Jacot et al., 2018); second, our experiment indicates that our result may be empirically true for the more general deep learning framework (Belkin et al., 2018), since this experiment uses a complicated network that may not be simply explained by the kernel method.
Revised: Neural Network results support our ﬁnding on kernel regression, since Neural Network is related to kernel regression through NTK theory (Jacot et al., 2018); second, our experiment indicates that our result may be empirically true for the more general deep learning framework (Belkin et al., 2018), since this experiment uses a complicated network that may not be simply explained by the kernel method.

Original:  D ISCUSSION AND F URTHER W ORK
Revised:  D ISCUSSION AND F URTHER W ORK

Original: Our work takes one more step towards understanding the directional bias of SGD in kernel learning.
Revised: Our work takes one more step towards understanding the directional bias of SGD in kernel learning.

Original: Here we discuss some implications of our results to deep learning.
Revised: Here we discuss some implications of our results to deep learning.

Original: Implication for SGD scheme : Our result shows the directional bias applies to SGD with annealing step size.
Revised: Implication for SGD scheme : Our result shows the directional bias applies to SGD with annealing step size.

Original: Speciﬁcally, the ﬁrst stage of SGD with moderate step size should run long enough, then in the second stage by decreasing step size we have the directional bias towards the largest eigenvalue of the Hessian, which helps in benign overﬁtting.
Revised: Speciﬁcally, the ﬁrst stage of SGD with moderate step size should run long enough, then in the second stage by decreasing step size we have the directional bias towards the largest eigenvalue of the Hessian, which helps in benign overﬁtting.

Original: This explains a technique for tuning the learning rate that people use in practice: start with a large step size, run long enough until the error plateaus, then decrease the step size (He et al., 2016).
Revised: This explains a technique for tuning the learning rate that people use in practice: start with a large step size, run long enough until the error plateaus, then decrease the step size (He et al., 2016).

Original: Although this technique is always used to speed convergence, we show that it also helps in benign overﬁtting, which becomes even better.
Revised: Although this technique is always used to speed convergence, we show that it also helps in benign overﬁtting, which becomes even better.

Original: Implication for deep learning : Our assumption for analysis implies certain structures for deep learning models.
Revised: Implication for deep learning : Our assumption for analysis implies certain structures for deep learning models.

Original: Per our examples in Appendix B and our discussion in Remark 3, our assumption holds when the feature space is high dimensional and/or when features are possibly sparse.
Revised: Per our examples in Appendix B and our discussion in Remark 3, our assumption holds when the feature space is high dimensional and/or when features are possibly sparse.

Original: This matches the deep learning scenario where we have a highly overparameterized model and when the trained parameter estimator becomes sparse.
Revised: This matches the deep learning scenario where we have a highly overparameterized model and when the trained parameter estimator becomes sparse.

Original: Besides, considering that some deep learning tasks can  be approximated by kernel learning (Jacot et al., 2018), our results help in explaining why the SGD estimator can benign overﬁtting in an overparameterized deep learning.
Revised: Besides, considering that some deep learning tasks can [Figure] be approximated by kernel learning (Jacot et al., 2018), our results help in explaining why the SGD estimator can benign overﬁtting in an overparameterized deep learning.

Original: Just as stated in Belkin et al.
Revised: Just as stated in Belkin et al.

Original: (2018), to understand deep learning one needs to understand kernel learning.
Revised: (2018), to understand deep learning one needs to understand kernel learning.

Original: This work takes a step in understanding kernel learning, and we expect more steps that go beyond this work towards understanding deep learning, possibly for some complicated structure that could not be approximated by kernel learning.
Revised: This work takes a step in understanding kernel learning, and we expect more steps that go beyond this work towards understanding deep learning, possibly for some complicated structure that could not be approximated by kernel learning.

Original: Reproducibility Statement: For all theoretical results presented in this paper, we carefully state and justify the assumption, we also include the proof in Appendix.
Revised: Reproducibility Statement: For all theoretical results presented in this paper, we carefully state and justify the assumption, we also include the proof in Appendix.

Original: For all experiments, we state the implementation details in Appendix, and we include the necessary code and data for reproducing our result in the supplementary material.
Revised: For all experiments, we state the implementation details in Appendix, and we include the necessary code and data for reproducing our result in the supplementary material.

Original: A B ACKGROUND ON RKHS
Revised: A B ACKGROUND ON RKHS

Original: This section details background on RKHS in two subsections.
Revised: This section details background on RKHS in two subsections.

Original: The ﬁrst subsection includes notations, theorems, and an example of RKHS, the second section reduces the kernel regression in
Revised: The ﬁrst subsection includes notations, theorems, and an example of RKHS, the second section reduces the kernel regression in

Original: RKHS from inﬁnite dimension to ﬁnite dimension, which gives our objective function (2) .
Revised: RKHS from inﬁnite dimension to ﬁnite dimension, which gives our objective function (2) .

Original: A. N ONPARAMETRIC MODEL IN RKHS
Revised: A. N ONPARAMETRIC MODEL IN RKHS

Original: In this subsection, we give the deﬁnition and notations for our model in RKHS, and its associated norms, basis, etc.
Revised: In this subsection, we give the deﬁnition and notations for our model in RKHS, and its associated norms, basis, etc.

Original: The deﬁnitions are similar to those in Raskutti et al.
Revised: The deﬁnitions are similar to those in Raskutti et al.

Original: Given n data pairs { x i , y i } ni =1 , where x i ∈ X ⊂ R p and y i ∈ R , assume that y i s are associated with x i s through f ( x i ) , where f ( · ) is some unknown function in the reproducing kernel
Revised: Given n data pairs { x i , y i } ni =1 , where x i ∈ X ⊂ R p and y i ∈ R , assume that y i s are associated with x i s through f ( x i ) , where f ( · ) is some unknown function in the reproducing kernel

Original: Hilbert space (RKHS) of functions X
Revised: Hilbert space (RKHS) of functions X

Original: → R , our goal is to estimate the function f ( · ) from the data.
Revised: → R , our goal is to estimate the function f ( · ) from the data.

Original: Denote the RKHS where f lives as H , with reproducing kernel K : X × X
Revised: Denote the RKHS where f lives as H , with reproducing kernel K : X × X

Original: → R + (which is known to us).
Revised: → R + (which is known to us).

Original: And we associate the functions in H with probability measure Q , assume w.l.o.g.
Revised: And we associate the functions in H with probability measure Q , assume w.l.o.g.

Original: that (cid:82) R p
Revised: that (cid:82) R p

Original: f ( a ) d Q ( a )
Revised: f ( a ) d Q ( a )

Original: = 0 .
Revised: = 0 .

Original: By Mercer’s theorem, K has eigen-expansion:
Revised: By Mercer’s theorem, K has eigen-expansion:

Original: [Equation]
Revised: [Equation]

Original: Where { φ j } ∞ j =1 are orthonormal basis in L 2 ( Q ) , w.r.t.
Revised: Where { φ j } ∞ j =1 are orthonormal basis in L 2 ( Q ) , w.r.t.

Original: the usual inner product in L 2 ( Q ) as
Revised: the usual inner product in L 2 ( Q ) as

Original: [Equation]
Revised: [Equation]

Original: Now for any f ∈ H , we can expand f ( · ) = (cid:80) ∞
Revised: Now for any f ∈ H , we can expand f ( · ) = (cid:80) ∞

Original: j =1 c j φ j ( · ) , where c j = (cid:104) f ( · ) , φ j ( · ) (cid:105) L 2 ( Q ) .
Revised: j =1 c j φ j ( · ) , where c j = (cid:104) f ( · ) , φ j ( · ) (cid:105) L 2 ( Q ) .

Original: And for f ( · ) = (cid:80) ∞
Revised: And for f ( · ) = (cid:80) ∞

Original: j =1 c j φ j ( · ) , g ( · ) =
Revised: j =1 c j φ j ( · ) , g ( · ) =

Original: (cid:80) ∞
Revised: (cid:80) ∞

Original: j =1 c (cid:48)
Revised: j =1 c (cid:48)

Original: j φ j ( · ) , by Parseval’s theorem
Revised: j φ j ( · ) , by Parseval’s theorem

Original: [Equation]
Revised: [Equation]

Original: And we have another inner product that is deﬁned for RKHS H as
Revised: And we have another inner product that is deﬁned for RKHS H as

Original: [Equation]
Revised: [Equation]

Original: The reproducing property of RKHS says that ∀ f ∈ H , we have
Revised: The reproducing property of RKHS says that ∀ f ∈ H , we have

Original: [Equation]
Revised: [Equation]

Original: Cubic Splines Formulate a RKHS .
Revised: Cubic Splines Formulate a RKHS .

Original: We go over an example of RKHS for better understanding.
Revised: We go over an example of RKHS for better understanding.

Original: Consider the cubic spline of one dimension, we can show that the space of cubic splines is a RKHS.
Revised: Consider the cubic spline of one dimension, we can show that the space of cubic splines is a RKHS.

Original: One can also ﬁnd the cubic spline example in Hazimeh et al.
Revised: One can also ﬁnd the cubic spline example in Hazimeh et al.

Original: For more details on the relationship between polynomial smoothing splines and RKHS, one can check Section 1.2 of Wahba (1990) .
Revised: For more details on the relationship between polynomial smoothing splines and RKHS, one can check Section 1.2 of Wahba (1990) .

Original: Assume w.l.o.g.
Revised: Assume w.l.o.g.

Original: that x i ∈ X
Revised: that x i ∈ X

Original: =
Revised: =

Original: [0 , 1] ⊂ R .
Revised: [0 , 1] ⊂ R .

Original: The cubic spline f on X is continuous, has a continuous ﬁrst-order derivative and square integrable second order derivative.
Revised: The cubic spline f on X is continuous, has a continuous ﬁrst-order derivative and square integrable second order derivative.

Original: By Taylor’s theorem with remainder, we have
Revised: By Taylor’s theorem with remainder, we have

Original: [Equation]
Revised: [Equation]

Original: where ( t − u ) + = max { 0 , t − u } .
Revised: where ( t − u ) + = max { 0 , t − u } .

Original: Let B be the set of cubic splines f on [0 , 1] that satisﬁes the boundary condition f (0) =
Revised: Let B be the set of cubic splines f on [0 , 1] that satisﬁes the boundary condition f (0) =

Original: f (cid:48) (0)
Revised: f (cid:48) (0)

Original: = 0 , then for f ∈ B
Revised: = 0 , then for f ∈ B

Original: [Equation]
Revised: [Equation]

Original: Let G ( t, u ) =
Revised: Let G ( t, u ) =

Original: ( t − u )
Revised: ( t − u )

Original: + , then we claim that B is RKHS with reproducing kernel
Revised: + , then we claim that B is RKHS with reproducing kernel

Original: [Equation]
Revised: [Equation]

Original: and inner product
Revised: and inner product

Original: [Equation]
Revised: [Equation]

Original: as one can check the reproducing property
Revised: as one can check the reproducing property

Original: [Equation]
Revised: [Equation]

Original: A. O PTIMIZATION PROBLEM CONSIDERED
Revised: A. O PTIMIZATION PROBLEM CONSIDERED

Original: This subsection gives problem formulation of kernel regression.
Revised: This subsection gives problem formulation of kernel regression.

Original: Given data pairs { x i , y i } ni =1 and
Revised: Given data pairs { x i , y i } ni =1 and

Original: RKHS H , consider a loss function (cid:96) which is selected according to how y is connected with f ( x ) , we may estimate the model by
Revised: RKHS H , consider a loss function (cid:96) which is selected according to how y is connected with f ( x ) , we may estimate the model by

Original: [Equation]
Revised: [Equation]

Original: Example for (cid:96) includes
Revised: Example for (cid:96) includes

Original: • Squared error loss (cid:96) ( y, f ) =
Revised: • Squared error loss (cid:96) ( y, f ) =

Original: ( y − f ) 2 , which is usually used in regression; • 0-1 loss (cid:96) ( y, f )
Revised: ( y − f ) 2 , which is usually used in regression; • 0-1 loss (cid:96) ( y, f )

Original: = 1 ( y ∗ f > 0) , for binary classiﬁcation; • Logistic loss (cid:96) ( y, f )
Revised: = 1 ( y ∗ f > 0) , for binary classiﬁcation; • Logistic loss (cid:96) ( y, f )

Original: = log(1 + exp( − y ∗ f )) , also a loss function for classiﬁcation
Revised: = log(1 + exp( − y ∗ f )) , also a loss function for classiﬁcation

Original: , can be considered as a surrogate function of 0-1 loss, and is the same as negative log likelihood function in logistic regression.
Revised: , can be considered as a surrogate function of 0-1 loss, and is the same as negative log likelihood function in logistic regression.

Original: Let us come back to the nonparametric model part, to control the model smoothness, the usual practice is to add a penalty to objective (6), result in
Revised: Let us come back to the nonparametric model part, to control the model smoothness, the usual practice is to add a penalty to objective (6), result in

Original: [Equation]
Revised: [Equation]

Original: A popular choice of pen ( f ) is (cid:107) f (cid:107) 2 H , or any strictly increasing function of (cid:107) f (cid:107) 2 H .
Revised: A popular choice of pen ( f ) is (cid:107) f (cid:107) 2 H , or any strictly increasing function of (cid:107) f (cid:107) 2 H .

Original: Such method is explicitly controlling the model smoothness, and by Representer Theorem, it has solution of the form
Revised: Such method is explicitly controlling the model smoothness, and by Representer Theorem, it has solution of the form

Original: [Equation]
Revised: [Equation]

Original: Plug equation (7) into objective (6), we have the problem becomes
Revised: Plug equation (7) into objective (6), we have the problem becomes

Original: [Equation]
Revised: [Equation]

Original: Which gives the formulation (2) under loss function (cid:96) ( y, f ( x ))
Revised: Which gives the formulation (2) under loss function (cid:96) ( y, f ( x ))

Original: = ( y − f ( x )) 2 / 2 .
Revised: = ( y − f ( x )) 2 / 2 .

Original: B D IAGONAL D OMINANCE OF S OME P OPULAR K ERNELS
Revised: B D IAGONAL D OMINANCE OF S OME P OPULAR K ERNELS

Original: In this section, we justify Assumption 1 by ﬁguring out a problem setting where some popular kernels give a diagonal dominant gram matrix.
Revised: In this section, we justify Assumption 1 by ﬁguring out a problem setting where some popular kernels give a diagonal dominant gram matrix.

Original: For simplicity, we assume the following data distribution throughout this section:
Revised: For simplicity, we assume the following data distribution throughout this section:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: normalized
Revised: normalized

Original: [Equation]
Revised: [Equation]

Original: that
Revised: that

Original: [Equation]
Revised: [Equation]

Original: The direction of x i s are i.i.d.
Revised: The direction of x i s are i.i.d.

Original: uniformly distributed on the unit sphere S
Revised: uniformly distributed on the unit sphere S

Original: [Equation]
Revised: [Equation]

Original: setting
Revised: setting

Original: [Equation]
Revised: [Equation]

Original: Given assumption set (A), we can bound the inner product of data (cid:104) x i , x j (cid:105) with high probability as follows:
Revised: Given assumption set (A), we can bound the inner product of data (cid:104) x i , x j (cid:105) with high probability as follows:

Original: Lemma 13 (Lemma 1 in Wu et al.
Revised: Lemma 14 (Lemma 1 in Wu et al.

Original: Under assumption set (A) , let d ≥ 4 log(2 n 2 /δ ) for some δ ∈ (0 , 1) .
Revised: Under assumption set (A) , let d ≥ 4 log(2 n 2 /δ ) for some δ ∈ (0 , 1) .

Original: Then with probability at least 1 − δ , we have
Revised: Then with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See proof of Lemma 1 in Wu et al.
Revised: See proof of Lemma 1 in Wu et al.

Original: The bound on the inner product (cid:104) x i , x j (cid:105) induces bound on K ( x i , x j ) for some popular kernels.
Revised: The bound on the inner product (cid:104) x i , x j (cid:105) induces bound on K ( x i , x j ) for some popular kernels.

Original: We show the diagonal dominance for two groups of kernels in the following propositions, and list some examples for kernels in each group.
Revised: We show the diagonal dominance for two groups of kernels in the following propositions, and list some examples for kernels in each group.

Original: Proposition 14 (Inner product kernel) .
Revised: Proposition 15 (Inner product kernel) .

Original: The inner product kernel is deﬁned as a smooth transformation of inner product.
Revised: The inner product kernel is deﬁned as a smooth transformation of inner product.

Original: We can write it as:
Revised: We can write it as:

Original: [Equation]
Revised: [Equation]

Original: Assume assumptions (A) hold, and assume the function g : [ − 1 , 1] → R satisﬁes:
Revised: Assume assumptions (A) hold, and assume the function g : [ − 1 , 1] → R satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: that is, ∇ g is L − Lipschitz
Revised: that is, ∇ g is L − Lipschitz

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: we will have with probability 1 − δ
Revised: we will have with probability 1 − δ

Original: [Equation]
Revised: [Equation]

Original: where δ and ˜ τ the same as in Lemma 
Revised: where δ and ˜ τ the same as in Lemma 14.When

Original: 13.When g (cid:48) (0)˜ τ + L 2 ˜ τ 2 (cid:28) g (1) for a small enough ˜ τ , the gram matrix is diagonal dominant.
Revised: g (cid:48) (0)˜ τ + L 2 ˜ τ 2 (cid:28) g (1) for a small enough ˜ τ , the gram matrix is diagonal dominant.

Original: Proof.
Revised: Proof.

Original: We have the following with probability at least 1 − δ by Lemma 13.
Revised: We have the following with probability at least 1 − δ by Lemma 14.

Original: For any off-diagonal elements of K :
Revised: For any off-diagonal elements of K :

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Thus | K i,j | ≤ ( c + g (cid:48) (0))˜ τ + L 2 ˜ τ 2 .
Revised: Thus | K i,j | ≤ ( c + g (cid:48) (0))˜ τ + L 2 ˜ τ 2 .

Original: Remark 15.
Revised: Remark 16.

Original: We list some examples of inner product kernels that give diagonal dominant kernel matrices:
Revised: We list some examples of inner product kernels that give diagonal dominant kernel matrices:

Original: • Bilinear Kernel: K ( x , x (cid:48) )
Revised: • Bilinear Kernel: K ( x , x (cid:48) )

Original: = (cid:104) x , x (cid:48) (cid:105) , then | K ( x i , x j
Revised: = (cid:104) x , x (cid:48) (cid:105) , then | K ( x i , x j

Original: [Equation]
Revised: [Equation]

Original: | ≤ ˜ τ (cid:28) K ( x n , x n ) = 1 .
Revised: ≤ [Equation] τ (cid:28) K ( x n , x n ) = 1 .

Original: • Polynomial Kernel: K ( x , x (cid:48) )
Revised: • Polynomial Kernel: K ( x , x (cid:48) )

Original: =
Revised: =

Original: ( (cid:104) x , x (cid:48) (cid:105)
Revised: ( (cid:104) x , x (cid:48) (cid:105)

Original: + c ) m for m ∈ N and c ∼ O
Revised: + c ) m for m ∈ N and c ∼ O

Original: (˜ τ ) , then by Proposition
Revised: (˜ τ ) , then by Proposition

Original: [Equation]
Revised: [Equation]

Original: when (1+ m )˜ τ
Revised: when (1+ m )˜ τ

Original: [Equation]
Revised: [Equation]

Original: ˜ τ 2 (cid:28) (1+ c ) m , we have diagonal dominant gram matrix.
Revised: ˜ τ 2 (cid:28) (1+ c ) m , we have diagonal dominant gram matrix.

Original: • Hyperbolic Tangent Kernel
Revised: • Hyperbolic Tangent Kernel

Original: (Sigmoid Kernel): K ( x , x (cid:48) )
Revised: (Sigmoid Kernel): K ( x , x (cid:48) )

Original: =
Revised: =

Original: tanh( α (cid:104) x , x (cid:48) (cid:105)
Revised: tanh( α (cid:104) x , x (cid:48) (cid:105)

Original: + c ) , where α > 0 , c ≥ 0 .
Revised: + c ) , where α > 0 , c ≥ 0 .

Original: Note that Hyperbolic Tangent Kernel does not satisfy all the assumptions in Proposition 14, one can still calculate
Revised: Note that Hyperbolic Tangent Kernel does not satisfy all the assumptions in Proposition 15, one can still calculate

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: When tanh( α ˜ τ + c ) (cid:28) tanh( α + c ) (which is the case when α is large, and c, ˜ τ are small enough), we have | K ( x i , x j ) | (cid:28) K ( x n , x n ) and the gram matrix is diagonal dominant.
Revised: When tanh( α ˜ τ + c ) (cid:28) tanh( α + c ) (which is the case when α is large, and c, ˜ τ are small enough), we have | K ( x i , x j ) | (cid:28) K ( x n , x n ) and the gram matrix is diagonal dominant.

Original: Proposition 16 (Radial Basis Function (RBF) kernel) .
Revised: Proposition 17 (Radial Basis Function (RBF) kernel) .

Original: Radial Basis Function kernel depends on two data points through their distance, which is of following form
Revised: Radial Basis Function kernel depends on two data points through their distance, which is of following form

Original: [Equation]
Revised: [Equation]

Original: Assume assumptions (A) hold, when γ = − c 0 log(˜ τ ) for a constant c 0 , we have with probability 1 − δ
Revised: Assume assumptions (A) hold, when γ = − c 0 log(˜ τ ) for a constant c 0 , we have with probability 1 − δ

Original: [Equation]
Revised: [Equation]

Original: where δ and ˜ τ the same as in Lemma 13.
Revised: where δ and ˜ τ the same as in Lemma 14.

Original: That is, the Gram matrix is diagonal dominant.
Revised: That is, the Gram matrix is diagonal dominant.

Original: Proof.
Revised: Proof.

Original: Bound off-diagonal terms of K by Lemma 13:
Revised: Bound off-diagonal terms of K by Lemma 14:

Original: [Equation]
Revised: [Equation]

Original: Remark 17.
Revised: Remark 18.

Original: We note some popular kernels that are related to Radial Basis Function kernel, and show that they lead to diagonal dominance:
Revised: We note some popular kernels that are related to Radial Basis Function kernel, and show that they lead to diagonal dominance:

Original: • Gaussian Kernel: K ( x , x (cid:48) )
Revised: • Gaussian Kernel: K ( x , x (cid:48) )

Original: =
Revised: =

Original: exp( − (cid:107) x − x (cid:48) (cid:107) 22 2 σ) .
Revised: exp( − (cid:107) x − x (cid:48) (cid:107) 22 2 σ) .

Original: One can see that Gaussian Kernel is reparameterizing RBF Kernel by γ = 1 / (2 σ 2 ) .
Revised: One can see that Gaussian Kernel is reparameterizing RBF Kernel by γ = 1 / (2 σ 2 ) .

Original: Thus Gaussian gram matrix is diagonal dominant when σ 2 ∼ O ( − 1 log(˜ τ ) ) .
Revised: Thus Gaussian gram matrix is diagonal dominant when σ 2 ∼ O ( − 1 log(˜ τ ) ) .

Original: • Laplace Kernel: K
Revised: • Laplace Kernel: K

Original: ( x , x (cid:48) )
Revised: ( x , x (cid:48) )

Original: =
Revised: =

Original: exp( − (cid:107) x − x (cid:48) (cid:107) 2 σ ) for σ > 0 .
Revised: exp( − (cid:107) x − x (cid:48) (cid:107) 2 σ ) for σ > 0 .

Original: The Laplace Kernel is very similar to Gaussian Kernel, and one can check by similar steps that when σ ∼ O ( − 1 log(˜ τ ) ) , Laplace gram matrix is diagonal dominant.
Revised: The Laplace Kernel is very similar to Gaussian Kernel, and one can check by similar steps that when σ ∼ O ( − 1 log(˜ τ ) ) , Laplace gram matrix is diagonal dominant.

Original: C L EMMAS
Revised: C L EMMAS

Original: This section includes two useful lemmas for characterizing the eigenvalues of a symmetric matrix.
Revised: This section includes two useful lemmas for characterizing the eigenvalues of a symmetric matrix.

Original: Lemma 18 (Gershgorin circle theorem, restated for symmetric matrix) .
Revised: Lemma 19 (Gershgorin circle theorem, restated for symmetric matrix) .

Original: Let A ∈ R n × n be a symmetric matrix.
Revised: Let A ∈ R n × n be a symmetric matrix.

Original: Let A ij be the entry in the i − th row and the j − th column.
Revised: Let A ij be the entry in the i − th row and the j − th column.

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: Consider n Gershgorin discs
Revised: Consider n Gershgorin discs

Original: [Equation]
Revised: [Equation]

Original: The eigenvalues of A are in the union of Gershgorin discs
Revised: The eigenvalues of A are in the union of Gershgorin discs

Original: [Equation]
Revised: [Equation]

Original: Furthermore, if the union of k of the n discs that comprise G ( A ) forms a set G k ( A ) that is disjoint from the remaining n − k discs, then G k ( A ) contains exactly k eigenvalues of A , counted according to their algebraic multiplicities.
Revised: Furthermore, if the union of k of the n discs that comprise G ( A ) forms a set G k ( A ) that is disjoint from the remaining n − k discs, then G k ( A ) contains exactly k eigenvalues of A , counted according to their algebraic multiplicities.

Original: Proof.
Revised: Proof.

Original: See Horn & Johnson (2012), Chap 6.1, Theorem 6.1.1.
Revised: See Horn & Johnson (2012), Chap 6.1, Theorem 6.1.1.

Original: Lemma 19 (Cauchy interlacing theorem, restated for symmetric matrix) .
Revised: Lemma 20 (Cauchy interlacing theorem, restated for symmetric matrix) .

Original: Let B ∈ R m × m be a symmetric matrix, let y ∈ R n and a ∈ R , and let A = (cid:20) B y y T a (cid:21) .
Revised: Let B ∈ R m × m be a symmetric matrix, let y ∈ R n and a ∈ R , and let A = (cid:20) B y y T a (cid:21) .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: ≥ λ m ( A ) ≥ λ m ( B ) ≥
Revised: ≥ λ m ( A ) ≥ λ m ( B ) ≥

Original: λ m +

Revised: λ m +

Original: ( A ) .
Revised: ( A ) .

Original: Proof.
Revised: Proof.

Original: See Horn & Johnson (2012), Chap 4.3, Theorem 4.3.17.
Revised: See Horn & Johnson (2012), Chap 4.3, Theorem 4.3.17.

Original: D S PECTRUM OF GRAM MATRIX
Revised: D S PECTRUM OF GRAM MATRIX

Original: This section analyzes the eigen structure of the gram matrix.
Revised: This section analyzes the eigen structure of the gram matrix.

Original: Lemma 20 (Characterizing K 2 ) .
Revised: Lemma 21 (Characterizing K 2 ) .

Original: Under Assumption 1, we have
Revised: Under Assumption 1, we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: For (cid:104) K i , K i (cid:105)
Revised: For (cid:104) K i , K i (cid:105)

Original: [Equation]
Revised: [Equation]

Original: 2 i + ( n − 1) τ 2 ]
Revised: 2 i + ( n − 1) τ 2 ]

Original: And for (cid:104) K i , K j (cid:105) , i (cid:54) =
Revised: And for (cid:104) K i , K j (cid:105) , i (cid:54) =

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: Lemma 21 (Eigenvalue of K ) .
Revised: Lemma 22 (Eigenvalue of K ) .

Original: Under Assumption 1, we have
Revised: Under Assumption 1, we have

Original: [Equation]
Revised: [Equation]

Original: If we further assume λ j + nτ < λ j − 1 − nτ , we will have
Revised: If we further assume λ j + nτ < λ j − 1 − nτ , we will have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Use Gershgorin circle theorem, calculate
Revised: Use Gershgorin circle theorem, calculate

Original: [Equation]
Revised: [Equation]

Original: then
Revised: then

Original: [Equation]
Revised: [Equation]

Original: By Gershgorin circle theorem, the lemma claim holds.
Revised: By Gershgorin circle theorem, the lemma claim holds.

Original: Lemma 22 (Characterize P 1 K and P − 1 K ) .
Revised: Lemma 23 (Characterize P 1 K and P − 1 K ) .

Original: Recall our deﬁnition: P − 1 is the projection on column space of K − 1 =
Revised: Recall our deﬁnition: P − 1 is the projection on column space of K − 1 =

Original: [ K 2 , K 3 , . . .
Revised: [ K 2 , K 3 , . . .

Original: , K n ] , and P 1 =
Revised: , K n ] , and P 1 =

Original: I − P − 1 .
Revised: I − P − 1 .

Original: We claim the following hold
Revised: We claim the following hold

Original: [Equation]
Revised: [Equation]

Original: Assume τ is small enough that nτ ≤ O (1) , λ n − nτ ≥ c 1 > 0 and λ 1 + nτ ≤ c 2 , let c 3 :
Revised: Assume τ is small enough that nτ ≤ O (1) , λ n − nτ ≥ c 1 > 0 and λ 1 + nτ ≤ c 2 , let c 3 :

Original: = c 2 c 21 , then we have the following:
Revised: = c 2 c 21 , then we have the following:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For i (cid:54) = 1 , we calculate
Revised: For i (cid:54) = 1 , we calculate

Original: [Equation]
Revised: [Equation]

Original: thus we have equations (15) and (16).
Revised: thus we have equations (15) and (16).

Original: For (cid:107) P − 1 K
Revised: For (cid:107) P − 1 K

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: and K T − 1 K − 1 has all eigenvalues in [ γ 2 n , γ 21 ] by Cauchy interlacing theorem (Lemma 19), that is, all singular values of K − 1 are in [ c 1 , c 2 ] by our assumption.
Revised: and K T − 1 K − 1 has all eigenvalues in [ γ 2 n , γ 21 ] by Cauchy interlacing theorem (Lemma 20), that is, all singular values of K − 1 are in [ c 1 , c 2 ] by our assumption.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: So we have
Revised: So we have

Original: [Equation]
Revised: [Equation]

Original: For (cid:107) P 1 K 1 (cid:107) 2 :
Revised: For (cid:107) P 1 K 1 (cid:107) 2 :

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Lemma 23 (Spectrum of H − 1 := P − 1 KK T P − 1 ) .
Revised: Lemma 24 (Spectrum of H − 1 := P − 1 KK T P − 1 ) .

Original: Assume
Revised: Assume

Original: [Equation]
Revised: [Equation]

Original: We have the following:
Revised: We have the following:

Original: • 0 is an eigenvalue of H − 1 , corresponding eigenspace is the column space of P 1 ; • Restricted to the column space of P − 1 , the eigenvalues of H − 1 are all in the interval:
Revised: • 0 is an eigenvalue of H − 1 , corresponding eigenspace is the column space of P 1 ; • Restricted to the column space of P − 1 , the eigenvalues of H − 1 are all in the interval:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The ﬁrst claim is by construction of P 1 and P − 1 .
Revised: The ﬁrst claim is by construction of P 1 and P − 1 .

Original: For the second claim, note that H − 1 has the same eigenvalues as
Revised: For the second claim, note that H − 1 has the same eigenvalues as

Original: [Equation]
Revised: [Equation]

Original: Now the diagonal entries of H (cid:48)− 1 are:
Revised: Now the diagonal entries of H (cid:48)− 1 are:

Original: [Equation]
Revised: [Equation]

Original: ∈
Revised: ∈

Original: [Equation]
Revised: [Equation]

Original: And the off-diagonal entries of H (cid:48)− 1 are:
Revised: And the off-diagonal entries of H (cid:48)− 1 are:

Original: [Equation]
Revised: [Equation]

Original: To use Gershgorin circle theorem, calculate
Revised: To use Gershgorin circle theorem, calculate

Original: [Equation]
Revised: [Equation]

Original: Thus the Gershgorin discs:
Revised: Thus the Gershgorin discs:

Original: [Equation]
Revised: [Equation]

Original: when c 23 [2 λ 1 +( n − 2) τ ] 2 nτ 2 +[2 λ 1 +( n − 2) τ ] nτ ≤ λ 2 n − [2 λ 1 +( n − 2) τ ] nτ , the ﬁrst Gershgorin discs does not intersect with the others, so we have n − 1 nonzero eigenvalues in
Revised: when c 23 [2 λ 1 +( n − 2) τ ] 2 nτ 2 +[2 λ 1 +( n − 2) τ ] nτ ≤ λ 2 n − [2 λ 1 +( n − 2) τ ] nτ , the ﬁrst Gershgorin discs does not intersect with the others, so we have n − 1 nonzero eigenvalues in

Original: [Equation]
Revised: [Equation]

Original: E D IRECTIONAL BIAS OF SGD WITH MODERATE STEP SIZE
Revised: E D IRECTIONAL BIAS OF SGD WITH MODERATE STEP SIZE

Original: This section gives formal proof of Theorem 4 and speciﬁes the constants.
Revised: This section gives formal proof of Theorem 5 and speciﬁes the constants.

Original: The proof is done in four steps: Lemma 24 analyzes one update of SGD; Lemma 25 uses Lemma 24 to bound the ﬁrst stage updates of SGD with moderate step size; Lemma 26 again uses Lemma 24, and bounds the second stage updates of SGD with small step size; ﬁnally, Theorem 27 combines Lemma 25 and Lemmato formalize the directional bias of SGD, it is the same as Theorem 4, but restated using the constants deﬁned therein.
Revised: The proof is done in four steps: Lemma 25 analyzes one update of SGD; Lemma 26 uses Lemma 25 to bound the ﬁrst stage updates of SGD with moderate step size; Lemma 27 again uses Lemma 25, and bounds the second stage updates of SGD with small step size; ﬁnally, Theorem 28 combines Lemma 26 and Lemmato formalize the directional bias of SGD, it is the same as Theorem 5, but restated using the constants deﬁned therein.

Original: Lemma 24 (One step update of SGD) .
Revised: Lemma 25 (One step update of SGD) .

Original: Under Assumption 1, denote A t :=
Revised: Under Assumption 1, denote A t :=

Original: E [ (cid:107) P 1 b t (cid:107) 2 ] , B t :=
Revised: E [ (cid:107) P 1 b t (cid:107) 2 ] , B t :=

Original: E [ (cid:107) P − 1 b t (cid:107) 2 ] , ﬁx a constant c 4 ≥ ( λ 1 + √
Revised: E [ (cid:107) P − 1 b t (cid:107) 2 ] , ﬁx a constant c 4 ≥ ( λ 1 + √

Original: [Equation]
Revised: [Equation]

Original: 3 , then we have:
Revised: 3 , then we have:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: One step of SGD update is:
Revised: One step of SGD update is:

Original: [Equation]
Revised: [Equation]

Original: where i is uniformly random sample from [1 , . .
Revised: where i is uniformly random sample from [1 , . .

Original: . , n ] .
Revised: . , n ] .

Original: For inequalities (19) and (20), check
Revised: For inequalities (19) and (20), check

Original: [Equation]
Revised: [Equation]

Original: where P 1 K i and P 1 b t are in the same 1 dimensional linear space, thus
Revised: where P 1 K i and P 1 b t are in the same 1 dimensional linear space, thus

Original: [Equation]
Revised: [Equation]

Original: ⇒
Revised: ⇒

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: E
Revised: E

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ηE
Revised: ηE

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:107)(23)
Revised: (cid:107)(23)

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality by upper bounds (17) and (18), second inequality by nτ ≤ O (1)
Revised: where the ﬁrst inequality by upper bounds (17) and (18), second inequality by nτ ≤ O (1)

Original: Plug the term (24) into inequalities (22) and (23), take expectation on both sides, we get claims (19) and (20).
Revised: Plug the term (24) into inequalities (22) and (23), take expectation on both sides, we get claims (19) and (20).

Original: For inequality (21), check
Revised: For inequality (21), check

Original: [Equation]
Revised: [Equation]

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: where the last inequality from Jensen’s inequality
Revised: where the last inequality from Jensen’s inequality

Original: [Equation]
Revised: [Equation]

Original: Now the term
Revised: Now the term

Original: [Equation]
Revised: [Equation]

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: combine all three inequalities above, take the expectation w.r.t.
Revised: combine all three inequalities above, take the expectation w.r.t.

Original: b t , we have claim (21).
Revised: b t , we have claim (21).

Original: Lemma 25 (Long run behavior of SGD with moderate step size) .
Revised: Lemma 26 (Long run behavior of SGD with moderate step size) .

Original: Assume b 0 is away from 0 , λ 2 n > (
Revised: Assume b 0 is away from 0 , λ 2 n > (

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: < λ 21 − c 5 √ nτ where c 5 , c 6 are constants such that
Revised: < λ 21 − c 5 √ nτ where c 5 , c 6 are constants such that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Consider ﬁrst k 1 steps of SGD updates with step size η :
Revised: Consider ﬁrst k 1 steps of SGD updates with step size η :

Original: [Equation]
Revised: [Equation]

Original: Fix a β 0 ≤ A 0 , then for 0 < (cid:15) < 1 and 0 < β < β 0 such that √ nτ ≤ poly ( (cid:15)β ) , there exists k 

Revised: Fix a β 0 ≤ A 0 , then for 0 < (cid:15) < 1 and 0 < β < β 0 such that √ nτ ≤ poly ( (cid:15)β ) , there exists k

Original: = O (log 1 (cid:15)β ) satisfying:
Revised: = O (log 1 (cid:15)β ) satisfying:

Original: • B k 1 ≤ (cid:15)β • A k 1 ≤ (cid:107) b 0 (cid:107) 2 ∗ ρ k 1 1 + (cid:15)β/ 2 for some ρ 1 >• A k > β 0 for k = 0 , . . . , k 1 .
Revised: • B k 1 ≤ (cid:15)β • A k 1 ≤ (cid:107) b 0 (cid:107) 2 ∗ ρ k 1 1 + (cid:15)β/ 2 for some ρ 1 >• A k > β 0 for k = 0 , . . . , k 1 .

Original: Proof.
Revised: Proof.

Original: For this choice of η , denote q 1 = q 1 ( η ) , q − 1 = q − 1 ( η ) , ξ = ξ ( η ) .
Revised: For this choice of η , denote q 1 = q 1 ( η ) , q − 1 = q − 1 ( η ) , ξ = ξ ( η ) .

Original: By Lemma 20, we have
Revised: By Lemma 21, we have

Original: [Equation]
Revised: [Equation]

Original: Decompose the coefﬁcient matrix as
Revised: Decompose the coefﬁcient matrix as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:21)
Revised: (cid:21)

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: cos
Revised: cos

Original: [Equation]
Revised: [Equation]

Original: (cid:21)
Revised: (cid:21)

Original: Assume w.l.o
Revised: Assume w.l.o

Original: [Equation]
Revised: [Equation]

Original: g. that sin θ ≥ 0 ( since otherwise we can take θ → θ + π ), then we have
Revised: g. that sin θ ≥ 0 ( since otherwise we can take θ → θ + π ), then we have

Original: [Equation]
Revised: [Equation]

Original: We claim the following holds:
Revised: We claim the following holds:

Original: [Equation]
Revised: [Equation]

Original: which we check later
Revised: which we check later

Original: [Equation]
Revised: [Equation]

Original: Using inequalities (25)
Revised: Using inequalities (25)

Original: [Equation]
Revised: [Equation]

Original: we can upper bound B k 1 as
Revised: we can upper bound B k 1 as

Original: [Equation]
Revised: [Equation]

Original: In addition, for k = 0 , . . .
Revised: In addition, for k = 0 , . . .

Original: , k
Revised: , k

Original: [Equation]
Revised: [Equation]

Original: We now lower bound A k by mathematical induction.
Revised: We now lower bound A k by mathematical induction.

Original: We have A 0 ≥ β 0 , assume A k − 1 > β 0 , then
Revised: We have A 0 ≥ β 0 , assume A k − 1 > β 0 , then

Original: [Equation]
Revised: [Equation]

Original: For upper bound A k 1 , check
Revised: For upper bound A k 1 , check

Original: [Equation]
Revised: [Equation]

Original: We have all lemma claims proved.
Revised: We have all lemma claims proved.

Original: Now it remains to check inequalities (25).
Revised: Now it remains to check inequalities (25).

Original: First note that our choice of the upper bound on η guarantees that q − 1 < 1 .
Revised: First note that our choice of the upper bound on η guarantees that q − 1 < 1 .

Original: For inequality (25a) : By Gershgorin circle theorem, it sufﬁces to show q − 1 + ξ < 1 , q 1 − ξ > 1 , then we have ρ 1 ≥ q 1 − ξ > 1 and ρ − 1 ≤ q − 1 + ξ < 1 .
Revised: For inequality (25a) : By Gershgorin circle theorem, it sufﬁces to show q − 1 + ξ < 1 , q 1 − ξ > 1 , then we have ρ 1 ≥ q 1 − ξ > 1 and ρ − 1 ≤ q − 1 + ξ < 1 .

Original: In addition, we need the matrix to be positive deﬁnite so that ρ − 1 > 0 , we just need q 1 q − 1 > ξ 2 to make the matrix p.d..
Revised: In addition, we need the matrix to be positive deﬁnite so that ρ − 1 > 0 , we just need q 1 q − 1 > ξ 2 to make the matrix p.d..

Original: [Equation]
Revised: [Equation]

Original: which is true by our choice of η .
Revised: which is true by our choice of η .

Original: [Equation]
Revised: [Equation]

Original: which is true by our lower bound on η .
Revised: which is true by our lower bound on η .

Original: [Equation]
Revised: [Equation]

Original: which is true.
Revised: which is true.

Original: For inequality (25b) : It sufﬁces to take
Revised: For inequality (25b) : It sufﬁces to take

Original: [Equation]
Revised: [Equation]

Original: For inequality (25c) :
Revised: For inequality (25c) :

Original: We just need to show sin θ < ( ρ − 1 /ρ 1 ) k 1 .
Revised: We just need to show sin θ < ( ρ − 1 /ρ 1 ) k 1 .

Original: Calculate that ξ/ ( q 1 − q
Revised: Calculate that ξ/ ( q 1 − q

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: θ cos 2 θ − sin 2 θ , then
Revised: θ cos 2 θ − sin 2 θ , then

Original: [Equation]
Revised: [Equation]

Original: For inequality (25d) : Sufﬁce to show
Revised: For inequality (25d) : Sufﬁce to show

Original: [Equation]
Revised: [Equation]

Original: Lemma 26 (Long run behavior of SGD with small step size) .
Revised: Lemma 27 (Long run behavior of SGD with small step size) .

Original: Under the same notations and assumptions as Lemma 25 unless otherwise speciﬁed.
Revised: Under the same notations and assumptions as Lemma 26 unless otherwise speciﬁed.

Original: Consider another k 2 − k 1 steps of SGD update with step size
Revised: Consider another k 2 − k 1 steps of SGD update with step size

Original: [Equation]
Revised: [Equation]

Original: where the constant c 7 ≥ √ nτ + c 4 .
Revised: where the constant c 7 ≥ √ nτ + c 4 .

Original: Then we have for k > k 1 :
Revised: Then we have for k > k 1 :

Original: • B k ≤ (cid:15)β • A k ≤ (cid:26) qA k −, A k − 1 > β β , A k − [Equation] β
Revised: • B k ≤ (cid:15)β • A k ≤ (cid:26) qA k −, A k − 1 > β β , A k − 1 < β

Original: where q := q 1 ( η (cid:48) )
Revised: where q := q 1 ( η (cid:48) )

Original: + ξ ( η (cid:48) )
Revised: + ξ ( η (cid:48) )

Original: (cid:15) < 1 .
Revised: (cid:15) < 1 .

Original: Proof.
Revised: Proof.

Original: Denote q (cid:48) 1 = q 1 ( η (cid:48) ) , q (cid:48)− 1 = q − 1 ( η (cid:48) ) , ξ (cid:48) =
Revised: Denote q (cid:48) 1 = q 1 ( η (cid:48) ) , q (cid:48)− 1 = q − 1 ( η (cid:48) ) , ξ (cid:48) =

Original: ξ ( η (cid:48) ) , then q = q (cid:48) 1 + ξ (cid:48) (cid:15) , denote B = (cid:107) b 0 (cid:107) 2 ∗ ρ k 1 1 + (cid:15)β/ 2 .
Revised: ξ ( η (cid:48) ) , then q = q (cid:48) 1 + ξ (cid:48) (cid:15) , denote B = (cid:107) b 0 (cid:107) 2 ∗ ρ k 1 1 + (cid:15)β/ 2 .

Original: We have by proof of Lemma 25 that q (cid:48)− 1 + ξ < 1 .
Revised: We have by proof of Lemma 26 that q (cid:48)− 1 + ξ < 1 .

Original: We claim the following holds:
Revised: We claim the following holds:

Original: [Equation]
Revised: [Equation]

Original: 26a
Revised: 26a

Original: [Equation]
Revised: [Equation]

Original: 26b
Revised: 26b

Original: [Equation]
Revised: [Equation]

Original: Check inequality (26a):
Revised: Check inequality (26a):

Original: [Equation]
Revised: [Equation]

Original: which are true by assumption.
Revised: which are true by assumption.

Original: Check inequality (26b):
Revised: Check inequality (26b):

Original: [Equation]
Revised: [Equation]

Original: With (26) we can prove the lemma by mathematical induction.
Revised: With (26) we can prove the lemma by mathematical induction.

Original: Suppose B k − 1 ≤ (cid:15)β and A k − 1 ≤
Revised: Suppose B k − 1 ≤ (cid:15)β and A k − 1 ≤

Original: A k 1 ≤ B , then check
Revised: A k 1 ≤ B , then check

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: We recap Theorem 4 using our notations in previous lemmas as follows:
Revised: We recap Theorem 5 using our notations in previous lemmas as follows:

Original: Theorem 27 (Directional bias of the two-stage SGD) .
Revised: Theorem 28 (Directional bias of the two-stage SGD) .

Original: Use the two stage SGD scheme as deﬁned in
Revised: Use the two stage SGD scheme as deﬁned in

Original: Lemma 25 and 26.
Revised: Lemma 26 and 27.

Original: Assume nτ < poly ( (cid:15) ) , then there exists k 

Revised: Assume nτ < poly ( (cid:15) ) , then there exists k

Original: =
Revised: =

Original: O (log 1 (cid:15) ) and k 2 such that
Revised: O (log 1 (cid:15) ) and k 2 such that

Original: [Equation]
Revised: [Equation]

Original: where γ 1 is the largest eigenvalue of K .
Revised: where γ 1 is the largest eigenvalue of K .

Original: Proof.
Revised: Proof.

Original: In Lemma 25 let β = β 0 , then for k 

Revised: In Lemma 26 let β = β 0 , then for k

Original: =
Revised: =

Original: O (log 1 (cid:15) )
Revised: O (log 1 (cid:15) )

Original: we have B k 1 ≤ (cid:15)β 0 .
Revised: we have B k 1 ≤ (cid:15)β 0 .

Original: For the 2nd stage, by
Revised: For the 2nd stage, by

Original: Lemma 26 we can early stop at k 2 such that A k 2 ≥ β 0 and A k 2 +1 < β 0 .
Revised: Lemma 27 we can early stop at k 2 such that A k 2 ≥ β 0 and A k 2 +1 < β 0 .

Original: We then have
Revised: We then have

Original: B k 2 ≤ (cid:15)β 0 ≤ (cid:15)A k
Revised: B k 2 ≤ (cid:15)β 0 ≤ (cid:15)A k

Original: Then we check
Revised: Then we check

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: And the upper bound in the theorem is by deﬁnition of γ 1 .
Revised: And the upper bound in the theorem is by deﬁnition of γ 1 .

Original: F D IRECTIONAL BIAS OF GD WITH MODERATE OR SMALL STEP SIZE
Revised: F D IRECTIONAL BIAS OF GD WITH MODERATE OR SMALL STEP SIZE

Original: This section includes the proof of Theorem 6.
Revised: This section includes the proof of Theorem 7.

Original: We ﬁrst rewrite the GD updates as linear combination of eigenvectors.
Revised: We ﬁrst rewrite the GD updates as linear combination of eigenvectors.

Original: Then the theorem is proved using the transformed variables and ﬁnally transformed back to original parameters.
Revised: Then the theorem is proved using the transformed variables and ﬁnally transformed back to original parameters.

Original: The directional bias of GD does not require diagonal dominant gram matrix.
Revised: The directional bias of GD does not require diagonal dominant gram matrix.

Original: Reloading notations Denote the eigen decomposition of K :
Revised: Reloading notations Denote the eigen decomposition of K :

Original: [Equation]
Revised: [Equation]

Original: where the eigenvectors g
Revised: where the eigenvectors g

Original: i ’s are orthogonal.
Revised: i ’s are orthogonal.

Original: The GD update as
Revised: The GD update as

Original: [Equation]
Revised: [Equation]

Original: Denote w t := G T ( α t − ˆ α ) , we can rewrite GD update in w t :
Revised: Denote w t := G T ( α t − ˆ α ) , we can rewrite GD update in w t :

Original: [Equation]
Revised: [Equation]

Original: We recap Theorem 6 to make reading easier as follows:
Revised: We recap Theorem 7 to make reading easier as follows:

Original: Theorem 28 (Direction bias of GD) .
Revised: Theorem 29 (Direction bias of GD) .

Original: Assume α 0 is away from 0 , λ n + 2 nτ < λ n − 1 , GD with step size: η < n
Revised: Assume α 0 is away from 0 , λ n + 2 nτ < λ n − 1 , GD with step size: η < n

Original: [Equation]
Revised: [Equation]

Original: nτ
Revised: nτ

Original: [Equation]
Revised: [Equation]

Original: For a small (cid:15) > 0 , take k =
Revised: For a small (cid:15) > 0 , take k =

Original: O (log 1 (cid:15) ) , we have
Revised: O (log 1 (cid:15) ) , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For i = 1 , . . . , n , we have
Revised: For i = 1 , . . . , n , we have

Original: [Equation]
Revised: [Equation]

Original: Denote q i = 1 − ηγ 2 i
Revised: Denote q i = 1 − ηγ 2 i

Original: /n , then 0 <
Revised: /n , then 0 <

Original: q 1 ≤ . . .
Revised: q 1 ≤ . . .

Original: ≤ q n < 1 since 0 < η < n ( λ 1 + nτ )(13) ≤ n
Revised: ≤ q n < 1 since 0 < η < n ( λ 1 + nτ )(13) ≤ n

Original: [Equation]
Revised: [Equation]

Original: ≤ n
Revised: ≤ n

Original: [Equation]
Revised: [Equation]

Original: 2 i
Revised: 2 i

Original: Since λ n + nτ < λ n − 1 − nτ , we have γ n < γ n − 1 by lemma 19, it follows that q n > q n − 1 .
Revised: Since λ n + nτ < λ n − 1 − nτ , we have γ n < γ n − 1 by lemma 20, it follows that q n > q n − 1 .

Original: Denote q = q n − 1 /q n < 1 , then
Revised: Denote q = q n − 1 /q n < 1 , then

Original: [Equation]
Revised: [Equation]

Original: ( w
Revised: ( w

Original: [Equation]
Revised: [Equation]

Original: Let q 2 k ≤
Revised: Let q 2 k ≤

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: ⇐⇒
Revised: ⇐⇒

Original: [Equation]
Revised: [Equation]

Original: log
Revised: log

Original: [Equation]
Revised: [Equation]

Original: n(cid:15)
Revised: n(cid:15)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: w (
Revised: w (

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: log q = O (log 1 (cid:15) ) , we have (cid:80) n − 1 i =1 ( w ( [Equation] [Equation] )( w ( n ) k )≤ γ 2 n (cid:15) γ
Revised: log q = O (log 1 (cid:15) ) , we have (cid:80) n − 1 i =1 ( w ( i ) k )( w ( n ) k )≤ γ 2 n (cid:15) γ

Original: Thus (cid:107) K ( α k − ˆ α ) (cid:107)(cid:107) α k − ˆ α (cid:107)= (cid:107) Γ w k (cid:107)
Revised: Thus (cid:107) K ( α k − ˆ α ) (cid:107)(cid:107) α k − ˆ α (cid:107)= (cid:107) Γ w k (cid:107)

Original: [Equation]
Revised: [Equation]

Original: thus
Revised: thus

Original: [Equation]
Revised: [Equation]

Original: The lower bound of the theorem holds by deﬁnition of γ n .
Revised: The lower bound of the theorem holds by deﬁnition of γ n .

Original: G E FFECT OF DIRECTIONAL BIAS
Revised: G E FFECT OF DIRECTIONAL BIAS

Original: In this section, we provide the proof for theorems in Section 4.2.
Revised: In this section, we provide the proof for theorems in Section 4.2.

Original: There are two theorems there, so we split this section into two subsections.
Revised: There are two theorems there, so we split this section into two subsections.

Original: Subsection G.1 proves Theorem 8, for a general problem setting of squared error minimization, it provides a straightforward understanding for why directional bias towards the largest eigenvalue of the Hessian is good for generalization.
Revised: Subsection G.1 proves Theorem 9, for a general problem setting of squared error minimization, it provides a straightforward understanding for why directional bias towards the largest eigenvalue of the Hessian is good for generalization.

Original: Section G. proves Theorem 10 by giving concrete generalization bounds of SGD and GD estimators in kernel regression.
Revised: Section G. proves Theorem 11 by giving concrete generalization bounds of SGD and GD estimators in kernel regression.

Original: G. P ROOF OF T HEOREM
Revised: G. P ROOF OF T HEOREM

Original: Denote v = w − w ∗ , rewrite the objective function as
Revised: Denote v = w − w ∗ , rewrite the objective function as

Original: [Equation]
Revised: [Equation]

Original: Denote the eigen decomposition of A T A = Q Γ Q T where Q =
Revised: Denote the eigen decomposition of A T A = Q Γ Q T where Q =

Original: [ q 1 , . . . , q n ] , QQ T = Q T Q
Revised: [ q 1 , . . . , q n ] , QQ T = Q T Q

Original: =
Revised: =

Original: I and Γ = diag ([ ρ 1 , . .
Revised: I and Γ = diag ([ ρ 1 , . .

Original: . , ρ n ]) , ρ 1 ≥ . .
Revised: . , ρ n ]) , ρ 1 ≥ . .

Original: ≥ ρ n ≥ 0 .
Revised: ≥ ρ n ≥ 0 .

Original: Then (cid:107)
Revised: Then (cid:107)

Original: A v (cid:107) 

Revised: A v (cid:107)

Original: =
Revised: =

Original: n (cid:88)
Revised: n (cid:88)

Original: i =
Revised: i =

Original: ρ i ( q T i v )
Revised: ρ i ( q T i v )

Original: So (cid:107)
Revised: So (cid:107)

Original: [Equation]
Revised: [Equation]

Original: v
Revised: v

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 1 [ n (cid:88) i =
Revised: 1 [ n (cid:88) i =

Original: ( q
Revised: ( q

Original: [Equation]
Revised: [Equation]

Original: i v ) 2 ]
Revised: i v ) 2 ]

Original: =
Revised: [Equation]

Original: ρ
Revised: ρ

Original: [Equation]
Revised: [Equation]

Original: ρ 1 (cid:107) v (cid:107)
Revised: ρ 1 (cid:107) v (cid:107)

Original: The equality is achieved when v is in the direction of q 1 , and ρ 1 = (cid:107) A T A (cid:107) 2 .
Revised: The equality is achieved when v is in the direction of q 1 , and ρ 1 = (cid:107) A T A (cid:107) 2 .

Original: Take L ( w ) =
Revised: Take L ( w ) =

Original: (cid:107)
Revised: (cid:107)

Original: A v (cid:107) 22 =
Revised: A v (cid:107) 22 =

Original: a then the theorem holds.
Revised: a then the theorem holds.

Original: G. P ROOF OF T HEOREM
Revised: G. P ROOF OF T HEOREM

Original: Calculate ∆ ∗ a : Denote f ∗ =
Revised: Calculate ∆ ∗ a : Denote f ∗ =

Original: ˆ α T K ( · , X ) + ˜ f , then we have for a f ∈ H s , f = α T K ( · , X ) , let b
Revised: ˆ α T K ( · , X ) + ˜ f , then we have for a f ∈ H s , f = α T K ( · , X ) , let b

Original: = ˆ α − α , then (cid:107)
Revised: = ˆ α − α , then (cid:107)

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: H where we can check
Revised: H where we can check

Original: [Equation]
Revised: [Equation]

Original: i =
Revised: i =

Original: b i (cid:104) K ( · , x i ) , f ∗ − ˆ α T K ( · , X ) (cid:105) H = (cid:88) i
Revised: b i (cid:104) K ( · , x i ) , f ∗ − ˆ α T K ( · , X ) (cid:105) H = (cid:88) i

Original: =
Revised: =

Original: b i [ (cid:104) K ( · , x i ) , f ∗ (cid:105) H − (cid:104) K ( · , x i ) , ˆ α T K ( · , X ) (cid:105) H ] = (cid:88) i
Revised: b i [ (cid:104) K ( · , x i ) , f ∗ (cid:105) H − (cid:104) K ( · , x i ) , ˆ α T K ( · , X ) (cid:105) H ] = (cid:88) i

Original: =
Revised: =

Original: b i [ f ∗ ( x i ) − ˆ α T K ( x i , X )]
Revised: b i [ f ∗ ( x i ) − ˆ α T K ( x i , X )]

Original: ( By reproducing property ) = (cid:88) i
Revised: ( By reproducing property ) = (cid:88) i

Original: =
Revised: =

Original: b i [ y i − y i ] =
Revised: b i [ y i − y i ] =

Original: And we further calculate that (cid:107) b T K ( · , X ) (cid:107) 2 H = (cid:104) n (cid:88) i
Revised: And we further calculate that (cid:107) b T K ( · , X ) (cid:107) 2 H = (cid:104) n (cid:88) i

Original: = b i K ( · , x i ) , n (cid:88)
Revised: = b i K ( · , x i ) , n (cid:88)

Original: j = b j K ( · , x j ) (cid:105) H = n (cid:88) i,j = b i b j (cid:104) K ( · , x i ) , K ( · , x j ) (cid:105) =
Revised: j = b j K ( · , x j ) (cid:105) H = n (cid:88) i,j = b i b j (cid:104) K ( · , x i ) , K ( · , x j ) (cid:105) =

Original: n (cid:88) i,j = b i
Revised: n (cid:88) i,j = b i

Original: b j K ( x i , x j ) =
Revised: b j K ( x i , x j ) =

Original: b T K b
Revised: b T K b

Original: That is,
Revised: That is,

Original: [Equation]
Revised: [Equation]

Original: and inf
Revised: and inf

Original: [Equation]
Revised: [Equation]

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: We claim that
Revised: We claim that

Original: [Equation]
Revised: [Equation]

Original: where the equality is obtained when b is in the direction of the largest eigenvector of K .
Revised: where the equality is obtained when b is in the direction of the largest eigenvector of K .

Original: To see this, we check (cid:107) K b (cid:107) 22 ≤ γ 1 b T K b .
Revised: To see this, we check (cid:107) K b (cid:107) 22 ≤ γ 1 b T K b .

Original: Recall the eigendecomposition of K = G Γ G T
Revised: Recall the eigendecomposition of K = G Γ G T

Original: where G =
Revised: where G =

Original: [ g 1 , . . . , g n ] has orthogonal columns and Γ = diag ( γ 1 , . .
Revised: [ g 1 , . . . , g n ] has orthogonal columns and Γ = diag ( γ 1 , . .

Original: . , γ n ) .
Revised: . , γ n ) .

Original: Then (cid:107) K b (cid:107) 22 = n (cid:88) i
Revised: Then (cid:107) K b (cid:107) 22 = n (cid:88) i

Original: = γ 2 i ( g T i b )and b T K b = n (cid:88) i = γ
Revised: = γ 2 i ( g T i b )and b T K b = n (cid:88) i = γ

Original: i ( g T i b )
Revised: i ( g T i b )

Original: So we have
Revised: So we have

Original: [Equation]
Revised: [Equation]

Original: This ﬁnishes our claim on ∆ ∗ a .
Revised: This ﬁnishes our claim on ∆ ∗ a .

Original: SGD output: By Theorem 4, the SGD output has
Revised: SGD output: By Theorem 5, the SGD output has

Original: [Equation]
Revised: [Equation]

Original: Thus
Revised: Thus

Original: E [
Revised: E [

Original: [Equation]
Revised: [Equation]

Original: i =
Revised: i =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: last inequality by let (cid:15) < 1 / 4 .
Revised: last inequality by let (cid:15) < 1 / 4 .

Original: GD output: By Theorem 6, the GD output has
Revised: GD output: By Theorem 7, the GD output has

Original: [Equation]
Revised: [Equation]

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: where M > 1 by taking (cid:15) (cid:48) <
Revised: where M > 1 by taking (cid:15) (cid:48) <

Original: 1 − γ n /γ 1 .
Revised: 1 − γ n /γ 1 .

Original: H E XPERIMENTS
Revised: H E XPERIMENTS

Original: We list the implementation details of the experiments at the end of Section 4 and include more experiment results.
Revised: We list the implementation details of the experiments at the end of Section 4 and include more experiment results.

Original: For better presenting, we split into two subsections: Subsection H.1 includes the details of simulation; Subsection H.2 is about the NN experiment on FashionMNIST, including the data description, network structure, and algorithm details, also there are more experiment results in
Revised: For better presenting, we split into two subsections: Subsection H.1 includes the details of simulation; Subsection H.2 is about the NN experiment on FashionMNIST, including the data description, network structure, and algorithm details, also there are more experiment results in

Original: Subsection H.2 that are not listed in Section 4 due to page limit.
Revised: Subsection H.2 that are not listed in Section 4 due to page limit.

Original: H. S IMULATION
Revised: H. S IMULATION

Original: This subsection is corresponding to Figure 1.
Revised: This subsection is corresponding to Figure 1.

Original: Data Generation .
Revised: Data Generation .

Revised: The training data is simulated as follows:

Original: Set n = 10 , p = 100 , simulate X n × p where elements of X are i.i.d.
Revised: Set n = 10 , p = 100 , simulate X n × p where elements of X are i.i.d.

Original: N (0 , 1) ; denote i th row of X as x i , normalize x i such that it has squared (cid:96) 2 norm in [ . 49 , 1] ; set y i
Revised: N (0 , 1) ; denote i th row of X as x i , normalize x i such that it has squared (cid:96) 2 norm in [ . 49 , 1] ; set y i

Original: = (cid:80) pj =1 sin( x i,j ) + (cid:15) i where (cid:15)
Revised: = (cid:80) pj =1 sin( x i,j ) + (cid:15) i where (cid:15)

Original: i i.i.d.
Revised: i i.i.d.

Original: ∼ N (0 , .
Revised: ∼ N (0 , .

Revised: The testing data is simulated in exactly the same way, except that we only simulate n = 5 testing data.

Original: Kernel Function .
Revised: Kernel Function .

Original: We set the kernel function to be the polynomial kernel
Revised: We set the kernel function to be the polynomial kernel

Original: [Equation]
Revised: K ( x 1 , x 2 ) =

Original:  x 2 (cid:105) + . 01)
Revised: ( (cid:104) x 1 , x 2 (cid:105) + . 01)

Original: SGD and GD implementation .
Revised: SGD and GD implementation .

Original:  SGD algorithm has two stages, stage 1 uses step size  .
Revised: Both SGD and GD is run for small and moderate step sizes. The moderate step size scheme for SGD is: η 1 = . 1 for the ﬁrst 50 steps, and η 2 = . 01 for the nextsteps; for GD is: η 1 = .

Original: and is run for 20 steps, stage 2 uses step size  .
Revised: 5 for the ﬁrst 50 steps, and η 2 = . for the next 1000 steps. The small step size scheme for SGD is η = 0 .

Original: 001 and is run for another 400 steps;  GD uses step size 0 . 1 and updates 420 steps to match the total number of steps of SGD.
Revised: 01 for 1050 steps; for GD is η = 0 . for 1050 steps. Note that the step size for SGD is a fraction of that for GD, this matches our Theorem 5 and 7 that the step size of GD is of magnitude n/ 2 times that of SGD.

Original: H. N EURAL N ETWORK ON F ASHION MNIST
Revised: H. N EURAL N ETWORK ON F ASHION MNIST

Original: This subsection is corresponding to Figure 2.
Revised: This subsection is corresponding to Figure 2.

Original: Dataset .
Revised: Dataset .

Original: The original FashionMNIST consist of 60 , 000 training data and 10 , 000 testing data.
Revised: The original FashionMNIST consist of 60 , 000 training data and 10 , 000 testing data.

Original: We randomly sample 1 , 500 data from original training data for training, and use all 10 , 000 original testing data for testing.
Revised: We randomly sample 1 , 500 data from original training data for training, and use all 10 , 000 original testing data for testing.

Original: All data entries are normalized to [0 , 1] .
Revised: All data entries are normalized to [0 , 1] .

Original: Network structure .
Revised: Network structure .

Original: we use a 6 -layer ResNet-like (He et al., 2016) Neural Network, and the structure is as follows
Revised: we use a 6 -layer ResNet-like (He et al., 2016) Neural Network, and the structure is as follows

Original: [Equation]
Revised: [Equation]

Original: The Residual Blocks are as Figure 7 . 6 . 3 in Zhang et al.
Revised: The Residual Blocks are as Figure 7 . 6 . 3 in Zhang et al.

Original: (2021) (without 1 × 1 convolution).
Revised: (2021) (without 1 × 1 convolution).

Original: Note that each residual block contains two 3 × 3 convolutional layers, thus total number of layers is as stated.
Revised: Note that each residual block contains two 3 × 3 convolutional layers, thus total number of layers is as stated.

Original: Algorithm .
Revised: Algorithm .

Original: We minimize the Cross Entropy Loss objective L ( w ) = 1 n (cid:80) ni
Revised: We minimize the Cross Entropy Loss objective L ( w ) = 1 n (cid:80) ni

Original: =1 l i ( w ) , where l i ( w ) is the loss function at i th sample.
Revised: =1 l i ( w ) , where l i ( w ) is the loss function at i th sample.

Original: One step SGD is as follows:
Revised: One step SGD is as follows:

Original: [Equation]
Revised: [Equation]

Original: where I is a randomly sampled subset of { 1 , . .
Revised: where I is a randomly sampled subset of { 1 , . .

Original: . , n } (uniform random sample without replacement).
Revised: . , n } (uniform random sample without replacement).

Original: We choose the batch size | I | to be 25 .
Revised: We choose the batch size | I | to be 25 .

Original: One step GD is as follows:
Revised: One step GD is as follows:

Original: [Equation]
Revised: [Equation]

Original: Both SGD and GD are run using two settings of step sizes η t .
Revised: Both SGD and GD are run using two settings of step sizes η t .

Original: The moderate step size setting is as follows:
Revised: The moderate step size setting is as follows:

Original: [Equation]
Revised: [Equation]

Original: And the small step size setting has η t = 0 .
Revised: And the small step size setting has η t = 0 .

Original: 02 , t = 1 , . .
Revised: 02 , t = 1 , . .

Original: Comparison of convergence direction .
Revised: Comparison of convergence direction .

Original: Since the loss surface is nonconvex and the Hessian varies, we follow Wu et al.
Revised: Since the loss surface is nonconvex and the Hessian varies, we follow Wu et al.

Original: (2021) to measure the convergence direction by Relative Rayleigh Quotient(RRQ), which normalizes the Rayleigh Quotient by the maximum eigenvalue of the Hessian as follows
Revised: (2021) to measure the convergence direction by Relative Rayleigh Quotient(RRQ), which normalizes the Rayleigh Quotient by the maximum eigenvalue of the Hessian as follows

Original: [Equation]
Revised: [Equation]

Original: where L ( w ) is the loss function on the whole training set.
Revised: where L ( w ) is the loss function on the whole training set.

Original: A high RRQ indicates that the convergence direction of w is close to a larger eigenvector of the Hessian.
Revised: A high RRQ indicates that the convergence direction of w is close to a larger eigenvector of the Hessian.

Original: Comparison of test accuracy .
Revised: Comparison of test accuracy .

Original: We set 20 different random seeds.
Revised: We set 20 different random seeds.

Original: For each random seed, we run:
Revised: For each random seed, we run:

Original: SGD with moderate step size, GD with moderate step size, SGD with small step size, GD with small step size.
Revised: SGD with moderate step size, GD with moderate step size, SGD with small step size, GD with small step size.

Original: For each algorithm, we evaluate its test accuracy once every 500 steps, and use the average of the last 5 values as its test accuracy.
Revised: For each algorithm, we evaluate its test accuracy once every 500 steps, and use the average of the last 5 values as its test accuracy.

Original: We list the test accuracy in Table 
Revised: We list the test accuracy in Table 1.

Revised: Experiment

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: We also use one-side Wilcoxon signed-rank test to check if the test accuracy of different algorithm are signiﬁcantly different, the result is in Table 2.
Revised: [Table] [Table] GD [Table]

Revised: SGD + small LR 82.

Revised: GD

Revised: [Table]

Revised: small LR 78.

Revised: Experiment #

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: LR 83.

Revised: GD

Revised: [Table]

Revised: LR 82.

Revised: SGD

Revised: [Table]

Revised: LR 82.

Revised: [Table]

Revised: small

Revised: [Table]

Revised: [Table]

Revised: We also use one-side Wilcoxon signed-rank test to check if the test accuracy of different algorithm are signiﬁcantly different, the result is in Table 2.

Original: All the p-values are signiﬁcant at 0 .
Revised: All the p-values are signiﬁcant at 0 .

Original: 01 level, so we reject the null hypothesis and conclude that the SGD with moderate step size has test accuracy signiﬁcantly higher than all other algorithms. 
Revised: 01 level, so we reject the null hypothesis and conclude that the SGD with moderate step size has test accuracy signiﬁcantly higher than all other algorithms. Null Hypothesis on [Table] Accuracy

Original: [Table]
Revised: [Table]

Original: Additional experiments .
Revised: Additional experiments .

Original: We conduct more experiments using different step sizes.
Revised: We conduct more experiments using different step sizes.

Original: The initial step size is taken in { 1 , 0 . 5 , 0 . 2 , 0 . 1 , 0 . 02 , 0 . 01 , 0 .
Revised: The initial step size is taken in { 1 , 0 . 5 , 0 . 2 , 0 . 1 , 0 . 02 , 0 . 01 , 0 .

Original: 001 } , and the step size is divided by a factor of 10 after 5000 steps.
Revised: 001 } , and the step size is divided by a factor of 10 after 5000 steps.

Original: The test accuracy is in Figure 3, where we see that SGD with step size 0 .
Revised: The test accuracy is in Figure 3, where we see that SGD with step size 0 .

Original: 2 has the best test accuracy, and GD with step size 0 .
Revised: 2 has the best test accuracy, and GD with step size 0 .

Original: 5 performs better than GD with any other step sizes, but is still worse than the best SGD.
Revised: 5 performs better than GD with any other step sizes, but is still worse than the best SGD.