Original: On the Double Descent of Random Features Models Trained with SGD
Revised: On the Double Descent of Random Features Models Trained with SGD

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress
Revised: AffiliationAddress

Original: Abstract
Revised: Abstract

Original: We study generalization properties of random features (RF) regression in highdimensions optimized by stochastic gradient descent (SGD).
Revised: We study generalization properties of random features (RF) regression in highdimensions optimized by stochastic gradient descent (SGD).

Original: In this regime, wederive precise non-asymptotics error bounds of RF regression under both constantand polynomial-decay step-size SGD setting, and observe the double descentphenomenon both theoretically and empirically.
Revised: In this regime, wederive precise non-asymptotics error bounds of RF regression under both constantand polynomial-decay step-size SGD setting, and observe the double descentphenomenon both theoretically and empirically.

Original: Our analysis shows how to copewith multiple randomness sources of initialization, label noise, and data sampling(as well as stochastic gradients) with no closed-form solution, and also goesbeyond the commonly-used Gaussian/spherical data assumption.
Revised: Our analysis shows how to copewith multiple randomness sources of initialization, label noise, and data sampling(as well as stochastic gradients) with no closed-form solution, and also goesbeyond the commonly-used Gaussian/spherical data assumption.

Original: Our theoreticalresults demonstrate that, with SGD training, RF regression still generalizes well forinterpolation learning, and is able to characterize the double descent behavior bythe unimodality of variance and monotonic decrease of bias.
Revised: Our theoreticalresults demonstrate that, with SGD training, RF regression still generalizes well forinterpolation learning, and is able to characterize the double descent behavior bythe unimodality of variance and monotonic decrease of bias.

Original: Besides, we also provethat the constant step-size SGD setting incurs no loss in convergence rate whencompared to the exact minimum-norm interpolator, as a theoretical justification ofusing SGD in practice.
Revised: Besides, we also provethat the constant step-size SGD setting incurs no loss in convergence rate whencompared to the exact minimum-norm interpolator, as a theoretical justification ofusing SGD in practice.

Original:  Introduction
Revised:  Introduction

Original: Over-parameterized models, e g., linear/kernel regression [1, 2, 3, 4] and neural networks [5, 6,
Revised: Over-parameterized models, e g., linear/kernel regression [1, 2, 3, 4] and neural networks [5, 6,

Original: 7],still generalize well even if the labels are pure noise [8].
Revised: 7],still generalize well even if the labels are pure noise [8].

Original: Such high-capacity models have receivedsignificant attention recently as they go against with classical generalization theory.
Revised: Such high-capacity models have receivedsignificant attention recently as they go against with classical generalization theory.

Original: A paradigm forunderstanding this important phenomenon is double descent [9], in which the test error first decreaseswith increasing number of model parameters in the under-parameterized regime.
Revised: A paradigm forunderstanding this important phenomenon is double descent [9], in which the test error first decreaseswith increasing number of model parameters in the under-parameterized regime.

Original: They large error isyielded until interpolating the data, which is called the interpolation threshold.
Revised: They large error isyielded until interpolating the data, which is called the interpolation threshold.

Original: Finally, the test errordecreases again in the over-parameterized regime.
Revised: Finally, the test errordecreases again in the over-parameterized regime.

Original: Our work partakes in this research vein and studies the random features (RF) model [10], as a simplified version of neural networks, in the context of double descent phenomenon.
Revised: Our work partakes in this research vein and studies the random features (RF) model [10], as a simplified version of neural networks, in the context of double descent phenomenon.

Original: Briefly, RF modelsamples random features { ω i } mi =1 from a specific distribution, corresponding to a kernel function.
Revised: Briefly, RF modelsamples random features { ω i } mi =1 from a specific distribution, corresponding to a kernel function.

Original: Wethen construct an explicit map: x ∈ R d (cid:55)→ σ ( W x ) ∈ R m , where W = [ ω 1 , · · · , ω m ] ⊤ ∈ R m × d isthe random features matrix and σ ( · ) is the nonlinear (activation) function determined by the kernel.
Revised: Wethen construct an explicit map: x ∈ R d (cid:55)→ σ ( W x ) ∈ R m , where W = [ ω 1 , · · · , ω m ] ⊤ ∈ R m × d isthe random features matrix and σ ( · ) is the nonlinear (activation) function determined by the kernel.

Original: As a result, the RF model can be viewed as training a two-layer neural network where the weights inthe are and then fixed (a.k.a.
Revised: As a result, the RF model can be viewed as training a two-layer neural network where the weights inthe are and then fixed (a.k.a.

Original: the random features) and only the outputlayer is optimized, striking a trade-off between practical performance and accessibility to analysis[4, 11].
Revised: the random features) and only the outputlayer is optimized, striking a trade-off between practical performance and accessibility to analysis[4, 11].

Original: An RF model becomes an over-parameterized model if we take the number of randomfeatures m larger than that of training data n .
Revised: An RF model becomes an over-parameterized model if we take the number of randomfeatures m larger than that of training data n .

Original: The literature on RF under the over-parameterizedregime can be split into various camps according to different assumptions on the formulation oftarget function, data distribution, and activation functions [4, 12, 11, 13, 14, 15] (see comparisons in Table 1 in Appendix A).
Revised: The literature on RF under the over-parameterizedregime can be split into various camps according to different assumptions on the formulation oftarget function, data distribution, and activation functions [4, 12, 11, 13, 14, 15] (see comparisons in Table 1 in Appendix A).

Original: The existing theoretical results demonstrate that the excess risk curveexhibits double descent.
Revised: The existing theoretical results demonstrate that the excess risk curveexhibits double descent.

Original: Nevertheless, the analysis framework of previous work on RF regression mainly relies on the leastsquares closed-form solution, including minimal-norm interpolator and ridge regressor.
Revised: Nevertheless, the analysis framework of previous work on RF regression mainly relies on the leastsquares closed-form solution, including minimum-norm interpolator and ridge regressor.

Original: Besides,they often assume the data with specific distribution, e.g., to be Gaussian or uniformly spread ona sphere.

Original: Such dependency on the analytic solutionand specific data distribution in fact mismatchpractical neural networks optimized by stochastic gradient descent (SGD) based algorithms ongeneral data distribution.
Revised: Suchdependency on the analytic solution in fact mismatches practical neural networks optimized bystochastic gradient descent (SGD) based algorithms.

Original: Our work precisely bridges this gap:
Revised: Our work precisely bridges this gap:

Original: We provide a new analysis frameworkfor the generalization properties of RF models trained with SGD, also accommodating adaptive(i.e. , polynomial decay) step-size selection, and provide non-asymptotic results in under-/overparameterized regimes on (relatively) general data distribution and activation functions.
Revised: We providea new analysis framework for the generalization properties of RF models trained with SGD andgeneral activation functions, also accommodating adaptive (i.e., polynomial decay) step-size selection,and provide non-asymptotic results in under-/over-parameterized regimes.

Original: We make thefollowing contributions and findings:
Revised: We make the followingcontributions and findings:

Original: First, we characterize statistical properties of covariance operators/matrices in RF, including Σ m :=1 m E x [ σ ( W x / √ d ) σ ( W x / √ d ) ⊤ ] and its expectation version (cid:101) Σ m :=
Revised: First, we characterize statistical properties of covariance operators/matrices in RF, including Σ m :=1 m E x [ σ ( W x / √ d ) σ ( W x / √ d ) ⊤ ] and its expectation version (cid:101) Σ m :=

Original: E W [Σ m ] .
Revised: E W [Σ m ] .

Original: We demonstratethat, under Gaussian initialization, if the activation function σ ( · ) :
Revised: We demonstratethat, under Gaussian initialization, if the activation function σ ( · ) :

Original: R (cid:55)→ R is Lipschitz continuous,
Revised: R (cid:55)→ R is Lipschitz continuous,

Original: Tr(Σ m ) is a sub-exponential random variable with O (1) sub-exponential norm; (cid:101) Σ m has only twodistinct eigenvalues at O (1) and O (1 /m ) order, respectively.
Revised: Tr(Σ m ) is a sub-exponential random variable with O (1) sub-exponential norm; (cid:101) Σ m has only twodistinct eigenvalues at O (1) and O (1 /m ) order, respectively.

Original: Such analysis on the spectra of Σ mand (cid:101) Σ m (without spectral decay assumption) is helpful to obtain sharp error bounds for excess risk.
Revised: Such analysis on the spectra of Σ mand (cid:101) Σ m (without spectral decay assumption) is helpful to obtain sharp error bounds for excess risk.

Revised: This is different from the least squares setting based on effective dimension [2, 16].

Original: Second, based on the bias-variance decomposition in stochastic approximation, we take into accountmultiple randomness sources of initialization, label noise, and data sampling as well as stochasticgradients.
Revised: Second, based on the bias-variance decomposition in stochastic approximation, we take into accountmultiple randomness sources of initialization, label noise, and data sampling as well as stochasticgradients.

Original: We  derive non-asymptotic error bounds under the optimization effect: the error bounds forbias and variance as a function of the radio m/n are monotonic decreasing and unimodal, respectively.
Revised: We (partly) disentangle these randomness sources and derive non-asymptotic error boundsunder the optimization effect: the error bounds for bias and variance as a function of the radiom/n are monotonic decreasing and unimodal, respectively.

Original: Importantly, our analysis holds for both constant and polynomial-decay step-size SGD setting, and isvalid under (relatively) general assumptions on data distribution and activation functions.
Revised: Importantly, our analysis holds for bothconstant and polynomial-decay step-size SGD setting, and is valid under sub-Gaussian data andgeneral activation functions.

Original: Third, our non-asymptotic results show that, RF regression trained with SGD still generalizes well forinterpolation learning, and is able to capture the double descent behavior.
Revised: Third, our non-asymptotic results show that, RF regression trained with SGD still generalizes well forinterpolation learning, and is able to capture the double descent behavior.

Original: In addition, we demonstratethat the constant step-size SGD setting incurs no loss on the convergence rate of excess risk whencompared to the exact least-squares closed form solution.
Revised: In addition, we demonstratethat the constant step-size SGD setting incurs no loss on the convergence rate of excess risk whencompared to the exact least-squares closed form solution.

Original: Our empirical evaluations support ourtheoretical results and findings.
Revised: Our empirical evaluations support ourtheoretical results and findings.

Original: Our analysis (technical challenges are discussed in Section 4) sheds light on the effect of SGD onhigh dimensional RF models in under-/over-parameterized regimes, and bridges the gap between theminimal-norm solution and numerical iteration solution in terms of optimization and generalizationon double descent.
Revised: Our analysis (technical challenges are discussed in Section 4) sheds light on the effect of SGD onhigh dimensional RF models in under-/over-parameterized regimes, and bridges the gap between theminimum-norm solution and numerical iteration solution in terms of optimization and generalizationon double descent.

Original: It would be helpful for understanding large dimensional machine learning andneural network models more generally.
Revised: It would be helpful for understanding large dimensional machine learning andneural network models more generally.

Original:  Related Work and Problem Setting
Revised:  Related Work and Problem Setting

Original: This section reviews relevant works and introduces our problem setting of RF regression with SGD.
Revised: This section reviews relevant works and introduces our problem setting of RF regression with SGD.

Original: Notation: The notation a ⊗ a denotes the tensor product of a vector a .
Revised: Notation: The notation a ⊗ a denotes the tensor product of a vector a .

Original: For two operators/matrices,
Revised: For two operators/matrices,

Original: A ≼ B means B − A is positive semi-definite (PSD).
Revised: A ≼ B means B − A is positive semi-definite (PSD).

Original: For any two positive sequences { a t } st =1 and{ b t } st =1 , the notation a t ≲ b t means that there exists a positive constant C independent of s such thata t ≤ Cb t , and analogously for ∼ , ≳ , and ≾ .
Revised: For any two positive sequences { a t } st =1 and{ b t } st =1 , the notation a t ≲ b t means that there exists a positive constant C independent of s such thata t ≤ Cb t , and analogously for ∼ , ≳ , and ≾ .

Original: For any a, b ∈ R , a ∧ b denotes the minimum of a and b .
Revised: For any a, b ∈ R , a ∧ b denotes the minimum of a and b .

Original: 2.1 Related works
Revised: 2.1 Related works

Original: A flurry of research papers are devoted to analysis of over-parameterized models on optimization[16, 17, 18], generalization (or their combination) under neural tangent kernel [19, 20, 21] andmean-field analysis regime [22, 23].
Revised: A flurry of research papers are devoted to analysis of over-parameterized models on optimization[17, 18, 19], generalization (or their combination) under neural tangent kernel [20, 21, 22] andmean-field analysis regime [23, 24].

Original: We take a unified perspective on optimization and generalizationbut work in the high-dimensional setting to fully capture the double descent behavior.
Revised: We take a unified perspective on optimization and generalizationbut work in the high-dimensional setting to fully capture the double descent behavior.

Original: By highdimensional setting, we mean that m , n , and d increase proportionally, large and comparable[4, 12, 13, 11].
Revised: By highdimensional setting, we mean that m , n , and d increase proportionally, large and comparable[4, 12, 13, 11].

Original: Double descent in random features model:Characterizing the double descent of the RF model oftenderives from random matrix theory (RMT) in high dimensional statistics [1, 4, 12, 13, 24] and from the replica method [11, 25, 14].
Revised: Double descent in random features model: the replica method [11, 26, 14].

Original: Under specific assumptions on data distribution, activation functions,target function, and initialization, these results show that the generalization error/excess risk increasewhen m/n < 1 , diverge when m/n → 1 , and then decrease when m/n > 1 .
Revised: Under specific assumptions on data distribution, activation functions,target function, and initialization, these results show that the generalization error/excess risk increasewhen m/n < 1 , diverge when m/n → 1 , and then decrease when m/n > 1 .

Original: Further, refinedresults are developed on the analysis of variance due to multiple randomness sources [11, 26, 15].
Revised: Further, refinedresults are developed on the analysis of variance due to multiple randomness sources [11, 27, 15].

Original: We refer to comparisons in Table 1 in Appendix A for further details.
Revised: We refer to comparisons in Table 1 in Appendix A for further details.

Original: Technically speaking, since
Revised: Technically speaking, since

Original: RF (least-squares) regression involves with inverse random matrices, these two classes of methodsattempt to achieve a similar target: how to disentangle the nonlinear activation function by the
Revised: RF (least-squares) regression involves with inverse random matrices, these two classes of methodsattempt to achieve a similar target: how to disentangle the nonlinear activation function by the

Original: Gaussian equivalence conjecture.
Revised: Gaussian equivalence conjecture.

Original: RMT utilizes calculus of deterministic equivalents (or resolvents)for random matrices and replica methods focus on some specific scalar parameters that allows forcircumventing the expectation computation.
Revised: RMT utilizes calculus of deterministic equivalents (or resolvents)for random matrices and replica methods focus on some specific scalar parameters that allows forcircumventing the expectation computation.

Original: In fact, most of the above methods can be asymptoticallyequivalent to the Gaussian covariate model [27].
Revised: In fact, most of the above methods can be asymptoticallyequivalent to the Gaussian covariate model [28].

Original: Non-asymptotic stochastic approximation: Many papers on linear least-squares regression [28, 29],kernel regression [30, 31], random features [32] with SGD often work in the under-parameterizedregime, where d is finite and much smaller than n .
Revised: Non-asymptotic stochastic approximation: Many papers on linear least-squares regression [29, 30],kernel regression [31, 32], random features [33] with SGD often work in the under-parameterizedregime, where d is finite and much smaller than n .

Original: In the over-parameterized regime, under GDsetting, the excess risk of least squares is controlled by the smallest positive eigenvalue in [33] byvirtue of random matrix theory.
Revised: In the over-parameterized regime, under GDsetting, the excess risk of least squares is controlled by the smallest positive eigenvalue in [34] byvirtue of random matrix theory.

Original: Under the averaged constant step-size SGD setting, the excess riskin [34] on least squares in high dimensions can be independent of d , and is further improved toconverge with n in [35].
Revised: Under the averaged constant step-size SGD setting, the excess riskin [35] on least squares in high dimensions can be independent of d , and is further improved toconverge with n in [16].

Original: This convergence rate is also demonstrated under the min [36] or last-iterate[37] setting in step-size SGD for noiseless least squares.
Revised: This convergence rate is also demonstrated under the min [36] or last-iterate[37] setting in step-size SGD for noiseless least squares.

Original: We also notice a concurrent work [38] onlast-iterate SGD with decaying step-size on least squares.
Revised: We also notice a concurrent work [38] onlast-iterate SGD with decaying step-size on least squares.

Original: Besides, the existence of multiple descent[39, 40] beyond double descent and SGD as implicit regularizer [41, 42] can be traced to the abovetwo lines of work.
Revised: Besides, the existence of multiple descent[39, 40] beyond double descent and SGD as implicit regularizer [41, 42] can be traced to the abovetwo lines of work.

Original: Our work shares some similar technical tools with [30] and [35] but differs fromthem in several aspects.
Revised: Our work shares some similar technical tools with [31] and [16] but differs fromthem in several aspects.

Original: We detail the differences in Section 4.
Revised: We detail the differences in Section 4.

Original: 2.2 The problem setting
Revised: 2.2 The problem setting

Original: We study the standard problem setting for RF least-squares regression and adopt the relevant terminologies from learning theory: cf. , [43, 30, 32, 24] for details.
Revised: We study the standard problem setting for RF least-squares regression and adopt the relevant terminologies from learning theory: cf. , [43, 31, 33, 25] for details.

Original: Let X ⊆
Revised: Let X ⊆

Original: R
Revised: R

Original: d be a metric spaceand Y ⊆ R .
Revised: d be a metric spaceand Y ⊆ R .

Original: The training data { ( x i , y i ) } ni =1 is assumed to be independently drawn from a nondegenerate unknown Borel probability measure ρ on X × Y .
Revised: The training data { ( x i , y i ) } ni =1 is assumed to be independently drawn from a nondegenerate unknown Borel probability measure ρ on X × Y .

Original: The target function of ρ is defined byf ρ ( x ) =
Revised: The target function of ρ is defined byf ρ ( x ) =

Original: (cid:82) Y y d ρ ( y | x ) , where ρ ( · | x ) is the conditional distribution of ρ at x ∈ X .
Revised: (cid:82) Y y d ρ ( y | x ) , where ρ ( · | x ) is the conditional distribution of ρ at x ∈ X .

Original: RF least squares regression: We study the RF regression problem with the squared loss as follows:
Revised: RF least squares regression: We study the RF regression problem with the squared loss as follows:

Original: [Equation]
Revised: [Equation]

Original: where the optimization vector θ ∈ R m and the feature mapping φ ( x ) is defined as
Revised: where the optimization vector θ ∈ R m and the feature mapping φ ( x ) is defined as

Original: [Equation]
Revised: [Equation]

Original: where W = [ ω 1 , · · · , ω m ] ⊤ ∈ R m × d with W ij ∼ N (0 , 1) corresponds to such two-layer neuralnetwork initialized with random Gaussian weights.
Revised: where W = [ ω 1 , · · · , ω m ] ⊤ ∈ R m × d with W ij ∼ N (0 , 1) corresponds to such two-layer neuralnetwork initialized with random Gaussian weights.

Original: Then, the corresponding hypothesis space H is areproducing kernel Hilbert space
Revised: Then, the corresponding hypothesis space H is areproducing kernel Hilbert space

Original: [Equation]
Revised: [Equation]

Original: with ∥ f ∥ 2 L 2 ρX = (cid:82) X | f ( x ) | 2 d ρ X ( x ) =
Revised: with ∥ f ∥ 2 L 2 ρX = (cid:82) X | f ( x ) | 2 d ρ X ( x ) =

Original: [Equation]
Revised: [Equation]

Original: f, Σ m f ⟩ H with the covariance operator Σ m : R m → R m
Revised: f, Σ m f ⟩ H with the covariance operator Σ m : R m → R m

Original: [Equation]
Revised: [Equation]

Original: actually defined in H that is isomorphic to R m .
Revised: actually defined in H that is isomorphic to R m .

Original: This is the usually (uncentered) covariance matrixin finite dimensions, 1 i.e., Σ m =
Revised: This is the usually (uncentered) covariance matrixin finite dimensions, 1 i.e., Σ m =

Original: E x [ φ ( x ) ⊗ φ ( x )] .
Revised: E x [ φ ( x ) ⊗ φ ( x )] .

Original: Define J m : R m →
Revised: Define J m : R m →

Original: L 2 ρ X such that
Revised: L 2 ρ X such that

Original: ( J m v )( · ) =
Revised: ( J m v )( · ) =

Original: ⟨ v , φ ( · ) ⟩ , ∀ v ∈ R m , we have Σ m =
Revised: ⟨ v , φ ( · ) ⟩ , ∀ v ∈ R m , we have Σ m =

Original: J ∗ m J m , where J ∗ m denotes the adjoint operator of J m .
Revised: J ∗ m J m , where J ∗ m denotes the adjoint operator of J m .

Original: Clearly, Σ m is random with respect to W , and thus its deterministic version is defined as (cid:101) Σ m = 122 E x , W [ φ ( x ) ⊗ φ ( x )] .
Revised: Clearly, Σ m is random with respect to W , and thus its deterministic version is defined as (cid:101) Σ m = 122 E x , W [ φ ( x ) ⊗ φ ( x )] .

Original: SGD with averaging: Regarding the stochastic approximation, we consider the polynomial-decaystep-size SGD with iterate averaging at each iteration t , after a training sample ( x t , y t ) ∼ ρ isobserved, we update the decision variable as below (initialized at θ 0 )
Revised: SGD with averaging: Regarding the stochastic approximation, we consider the polynomial-decaystep-size SGD with iterate averaging at each iteration t , after a training sample ( x t , y t ) ∼ ρ isobserved, we update the decision variable as below (initialized at θ 0 )

Original: [Equation]
Revised: [Equation]

Original: This averaged SGD with bath size = 1 is commonly used in theory [30, 35, 44] for ease of analysis,which captures the key idea of SGD by combining stochastic gradients and data sampling.
Revised: This averaged SGD with bath size = 1 is commonly used in theory [31, 16, 44] for ease of analysis,which captures the key idea of SGD by combining stochastic gradients and data sampling.

Original: Herethe step-size is given by γ t :
Revised: Herethe step-size is given by γ t :

Original: = γ 0 t − ζ with ζ ∈ [0 , 1) , following [30], which naturally holds for theconstant step-size case by taking ζ = 0 .
Revised: = γ 0 t − ζ with ζ ∈ [0 , 1) , following [31], which naturally holds for theconstant step-size case by taking ζ = 0 .

Original: The final output is defined as the average of the iterates:¯ θ n := 1 n (cid:80) n − 1 t =0 θ t .
Revised: The final output is defined as the average of the iterates:¯ θ n := 1 n (cid:80) n − 1 t =0 θ t .

Original: Here we sum up { θ t } n − 1 t =0 with n terms for notational simplicity.
Revised: Here we sum up { θ t } n − 1 t =0 with n terms for notational simplicity.

Original: The optimalitycondition for Eq.
Revised: The optimalitycondition for Eq.

Original: (4) implies E ( x ,y ) ∼ ρ
Revised: (4) implies E ( x ,y ) ∼ ρ

Original: [( y −⟨ θ ∗ , φ ( x ) ⟩ ) φ ( x )] = 0 , which corresponds to f ∗ =
Revised: [( y −⟨ θ ∗ , φ ( x ) ⟩ ) φ ( x )] = 0 , which corresponds to f ∗ =

Original: J m θ ∗if we assume that f ∗ = argmin f ∈H E ( f ) exists (see Assumption 2 in the next section).
Revised: J m θ ∗if we assume that f ∗ = argmin f ∈H E ( f ) exists (see Assumption 2 in the next section).

Original: Likewise,we have f t =
Revised: Likewise,we have f t =

Original: J m θ t and ¯ f n = J m ¯ θ n .
Revised: J m θ t and ¯ f n = J m ¯ θ n .

Original: In this paper, we study the averaged excess risk E ∥
Revised: In this paper, we study the averaged excess risk E ∥

Original: ¯ f n − f ∗ ∥
Revised: ¯ f n − f ∗ ∥

Original: 2 L 2 ρX instead of E
Revised: 2 L 2 ρX instead of E

Original: ∥
Revised: ∥

Original: ¯ f n − f ρ ∥ 2 L 2 ρX , thatfollows [30, 45, 32, 24], as f ∗ is the best possible solution in H and the mis-specification error∥ f ∗ − f ρ ∥ 2 L 2 ρX pales into insignificance.
Revised: ¯ f n − f ρ ∥ 2 L 2 ρX , thatfollows [31, 45, 33, 25], as f ∗ is the best possible solution in H and the mis-specification error∥ f ∗ − f ρ ∥ 2 L 2 ρX pales into insignificance.

Original: Note that the expectation used here is considered withrespect to the random features matrix W , and
Revised: Note that the expectation used here is considered withrespect to the random features matrix W , and

Original: the distribution of the training data { ( x t , y t ) } nt =1 (notethat ∥
Revised: the distribution of the training data { ( x t , y t ) } nt =1 (notethat ∥

Original: ¯ f n − f ∗ ∥
Revised: ¯ f n − f ∗ ∥

Original: 2 L 2 ρX is itself a different expectation over ρ X ).
Revised: 2 L 2 ρX is itself a different expectation over ρ X ).

Original:  Main results
Revised:  Main results

Original: In this section, we present our main theoretical results on the generalization properties employingerror bounds for bias and variance of RF regression in high dimensions optimized by averaged SGD.
Revised: In this section, we present our main theoretical results on the generalization properties employingerror bounds for bias and variance of RF regression in high dimensions optimized by averaged SGD.

Original: 3.1 Assumptions
Revised: 3.1 Assumptions

Original: Before we present our result, we list the assumptions used in this paper, refer to Appendix B for morediscussions.
Revised: Before we present our result, we list the assumptions used in this paper, refer to Appendix B for morediscussions.

Original: Assumption 1.
Revised: Assumption 1.

Original: [46, 1, high dimensional setting] We work in the large d, n, m regime with c ⩽{ d/n, m/n } ⩽ C for some constants c, C > 0 such that m, n, d are large and comparable.
Revised: [46, 1, high dimensional setting] We work in the large d, n, m regime with c ⩽{ d/n, m/n } ⩽ C for some constants c, C > 0 such that m, n, d are large and comparable.

Original: Thedata point x ∈ R d is assumed to satisfy ∥ x ∥ 22 ∼ O ( d ) and the sample covariance operatorΣ
Revised: Thedata point x ∈ R d is assumed to satisfy ∥ x ∥ 22 ∼ O ( d ) and the sample covariance operatorΣ

Original: d := E x [ x ⊗ x ] with bounded spectral norm ∥ Σ d ∥ 2 (finite and independent of d ).
Revised: d := E x [ x ⊗ x ] with bounded spectral norm ∥ Σ d ∥ 2 (finite and independent of d ).

Original: Assumption 2.
Revised: Assumption 2.

Original: There exists f ∗ ∈ H such that f ∗ = argmin f ∈H E ( f ) with bounded Hilbert norm.
Revised: There exists f ∗ ∈ H such that f ∗ = argmin f ∈H E ( f ) with bounded Hilbert norm.

Original: Remark:
Revised: Remark:

Original: This bounded Hilbert norm assumption is commonly used in [47, 40, 48] even though nand d tend to infinity.
Revised: This bounded Hilbert norm assumption is commonly used in [47, 40, 48] even though nand d tend to infinity.

Original: Bach [49] provides an example satisfying this assumption, cf. , Appendix B.
Revised: Bach [49] provides an example satisfying this assumption, ∥ f

Original: This is a standard assumption in learning theory, e.g., [45, 32]. Assuming the existence of f ∗ ∈ Himplies that ∥ f ∥ H is bounded in Eq. (2), which is in fact indispensable and standard.
Revised: ∥ H ⩽ 4 π in highdimensions, see Appendix B.

Original: Assumption 3.
Revised: Assumption 3.

Original: The activation function σ ( · ) is assumed to be Lipschitz continuous.
Revised: The activation function σ ( · ) is assumed to be Lipschitz continuous.

Original: Assumption 4 (Fourth moment condition) .
Revised: Assumption 4 (Fourth moment condition) .

Original: Assume there exists some positive constants r ′ , r ⩾ 1 ,such that for any PSD operator A , it holds that
Revised: Assume there exists some positive constants r ′ , r ⩾ 1 ,such that for any PSD operator A , it holds that

Original: [Equation]
Revised: [Equation]

Original: Remark:
Revised: Remark:

Original: This assumption follows [35] that requires the data are drawn from some not-too-heavytailed distribution, e.g., Σ − 1 2 m x has sub-Gaussian tail, common in high dimensional statistics.
Revised: This assumption follows [16] that requires the data are drawn from some not-too-heavytailed distribution, e.g., Σ − 1 2 m x has sub-Gaussian tail, common in high dimensional statistics.

Original: Thiscondition is weaker than most previous work on double descent that requires the data to be Gaussian[1, 11, 26, 12], or uniformly spread on a sphere [4, 50], see comparisons in Table 1 in Appendix A.
Revised: Thiscondition is weaker than most previous work on double descent that requires the data to be Gaussian[1, 11, 27, 12], or uniformly spread on a sphere [4, 50], see comparisons in Table 1 in Appendix A.

Original: Besides, a special case of this assumption with A :
Revised: Besides, a special case of this assumption with A :

Original: = I is proved by Lemma 3, and thus thisassumption can be regarded as a natural extension, with more discussions in Appendix B.
Revised: = I is proved by Lemma 3, and thus thisassumption can be regarded as a natural extension, with more discussions in Appendix B.

Original: Assumption 5 (Noise condition) .
Revised: Assumption 5 (Noise condition) .

Original: There exists τ > 0 such that Ξ :=
Revised: There exists τ > 0 such that Ξ :=

Original: E x [ ε 2 φ ( x ) ⊗ φ ( x )]
Revised: E x [ ε 2 φ ( x ) ⊗ φ ( x )]

Original: ≼ τ 2 Σ m ,where the noise ε :=
Revised: ≼ τ 2 Σ m ,where the noise ε :=

Original: y − f ∗ ( x ) .
Revised: y − f ∗ ( x ) .

Original: Remark:
Revised: Remark:

Original: This noise assumption is standard in [30, 35] and holds for the standard noise modely =
Revised: This noise assumption is standard in [31, 16] and holds for the standard noise modely =

Original: f ∗ ( x ) + ε with E [ ε ] = 0 and V [ ε ] < ∞ [1].
Revised: f ∗ ( x ) + ε with E [ ε ] = 0 and V [ ε ] < ∞ [1].

Original: 3.2 Properties of covariance operators
Revised: 3.2 Properties of covariance operators

Original: Before we present the main results, we study statistical properties of Σ m and (cid:101) Σ m by the followinglemmas (with proof deferred to Appendix C), that will be needed for our main result.
Revised: Before we present the main results, we study statistical properties of Σ m and (cid:101) Σ m by the followinglemmas (with proof deferred to Appendix C), that will be needed for our main result.

Revised: This isdifferent from the least squares setting [2, 16] that introduces the effective dimension to separate theentire space into a “head” subspace where the error decays more quickly than the complement “tail”subspace.

Revised: Instead, the following lemma shows that (cid:101) Σ m has only two distinct eigenvalues at O (1) and

Revised: O (1 /m ) order, respectively.

Revised: Such fast eigenvalue decay can avoid extra data spectrum assumptionfor tight bound.

Original: Lemma 1.
Revised: Lemma 1.

Original: Under Assumption 1 and 3, if we further require σ ( · ) is single-output, then the expectedcovariance operator (cid:101) Σ m :=
Revised: Under Assumption 1 and 3, if we further require σ ( · ) is single-output, then the expectedcovariance operator (cid:101) Σ m :=

Original: E x , W [ φ ( x ) ⊗ φ ( x )]
Revised: E x , W [ φ ( x ) ⊗ φ ( x )]

Original: ∈
Revised: ∈

Original: R m × m has the same diagonal elements and thesame non-diagonal element
Revised: R m × m has the same diagonal elements and thesame non-diagonal element

Original: [Equation]
Revised: [Equation]

Original: Accordingly, (cid:101) Σ m has only two distinct eigenvalues
Revised: Accordingly, (cid:101) Σ m has only two distinct eigenvalues

Original: [Equation]
Revised: [Equation]

Original: Remark: Lemma 1 implies tr( (cid:101) Σ m ) < ∞ .
Revised: Remark: Lemma 1 implies tr( (cid:101) Σ m ) < ∞ .

Original: In fact, E x V [ σ ( z )] > 0 holds almost surely as σ ( · ) isnot a constant, and thus (cid:101)Σ m is positive definite.
Revised: In fact, E x V [ σ ( z )] > 0 holds almost surely as σ ( · ) is nota constant, and thus (cid:101)

Original: For description simplicity, we mainly focus on thesingle-output activation function in this paper, but it can be easily extended to a multiple-outputversion, as discussed in below. Here we take two examples with different activation functions σ ( · ) for demonstration. The calculationdetails can be found in Appendix C.1.2.
Revised: Σ m is positive definite.

Original: 1) If we choose the ReLU activation σ ( x ) = max { x, 0 } , RF actually approximates the first-order arccosine kernel [51] with φ ( x ) ∈ R m .
Revised: Here we take the ReLU activation σ ( x ) = max { x, 0 } as one example, RF actually approximates thefirst-order arc-cosine kernel [51] with φ ( x ) ∈ R m .

Original: We have ( (cid:101) Σ m ) ii = 1 2 md Tr(Σ d ) and ( (cid:101) Σ m ) ij = 1 2 mdπ Tr(Σ d )by recalling Σ d :=
Revised: We have ( (cid:101) Σ m ) ii = 1 2 md Tr(Σ d ) and ( (cid:101) Σ m ) ij =1 2 mdπ Tr(Σ d ) by recalling Σ d :=

Original: E x [ xx ⊤ ] and Tr(Σ d ) /d ∼ O (1) .
Revised: E x [ xx ⊤ ] and Tr(Σ d ) /d ∼ O (1) .

Original: 2) If we choose σ ( x ) =

Original: [cos( x ) , sin( x )] ⊤ , a multiple-output version, RF actually approximates theGaussian kernel with φ ( x ) ∈ R 2 m inEq. (1), resulting in (cid:101) Σ m ∈ R 2 m × 2 m . In this case, (cid:101) Σ m = S 1 ⊕ S 2is a block diagonal matrix, where ⊕ is the direct sum. By denoting ϑ := ∥ x ∥ 22 /d , the matrixS 1 ∈ R m × m has the same diagonal elements [ S 1 ] ii = 12 m E x [1 + exp ( − 2 ϑ )] , and the same nondiagonal elements 1 m E x [exp ( − ϑ )] .
Revised: More examples can be found in

Original: The matrix S 2 is diagonal with [ S 2 ] ii = 12 m E x [1 − exp ( − 2 ϑ )] . In this case (cid:101) Σ m admits three distinct eigenvalues: the largest eigenvalue at O (1) order, and theremaining two eigenvalues at O (1 /m ) order.
Revised: Appendix C.1.2.

Original: Lemma 2.
Revised: Lemma 2.

Original: Under Assumptions 1 and 3, random variables ∥ Σ m ∥ 2 , ∥
Revised: Under Assumptions 1 and 3, random variables ∥ Σ m ∥ 2 , ∥

Original: Σ m − (cid:101) Σ m ∥ 2 , and Tr(Σ m ) aresub-exponential, and have sub-exponential norm at O (1) order.
Revised: Σ m − (cid:101) Σ m ∥ 2 , and Tr(Σ m ) aresub-exponential, and have sub-exponential norm at O (1) order.

Original: Lemma 3. Under Assumptions 1,and 3, there exists a constant r > 0 such that E W (cid:0) Σ 2 m (cid:1) ≼ E x , W [ φ ( x ) ⊗ φ ( x ) ⊗ φ ( x ) ⊗ φ ( x )] ≼ r Tr( (cid:101) Σ m ) (cid:101) Σ m .
Revised: Remark:

Original: Lemma 4.

Original: Under Assumptions 1 and 3, we have Tr[ (cid:101) Σ − 1 m E W

Original: (Σ 2 m )] ∼ O (1) . We remark here that Lemma 3 is a special case of Assumption 4 if we take A : = I and r := 1+ O (cid:0) 1 m (cid:1) ;and Lemma 4 is a direct corollary of Lemma 3.
Revised: This lemma characterizes the sub-exponential property of covariance operator Σ m , whichis a fundamental result for our proof since the bias and variance involve them.

Original: 3.3 Results for error bounds
Revised: 3.3 Results for error bounds

Original: Recall the definition of the noise ε =
Revised: Recall the definition of the noise ε =

Original: [ ε 1 , · · · , ε n ] ⊤ with ε t =
Revised: [ ε 1 , · · · , ε n ] ⊤ with ε t =

Original: y t − f ∗ ( x t ) , t = 1 , 2 , . . . , n , theaveraged excess risk can be expressed as
Revised: y t − f ∗ ( x t ) , t = 1 , 2 , . . . , n , theaveraged excess risk can be expressed as

Original: [Equation]
Revised: [Equation]

Original: where ¯ η n := 1 n (cid:80) n − 1 t =0 η t with the centered SGD iterate η t :=
Revised: where ¯ η n := 1 n (cid:80) n − 1 t =0 η t with the centered SGD iterate η t :=

Original: f t − f ∗ .
Revised: f t − f ∗ .

Original: Following the standardbias-variance decomposition in stochastic approximation [30, 29, 35], it admits
Revised: Following the standardbias-variance decomposition in stochastic approximation [31, 30, 16], it admits

Original: [Equation]
Revised: [Equation]

Original: where the first term corresponds to the bias
Revised: where the first term corresponds to the bias

Original: [Equation]
Revised: [Equation]

Revised: bias

Revised: [Equation]

Original: and the second term corresponds to the variance
Revised: and the second term corresponds to the variance

Original: [Equation]
Revised: [Equation]

Original: Accordingly, we have f t = η bias t [Equation] to E [Equation]
Revised: Accordingly, we have f t = η bias t + η var t + f ∗ due to E ε ¯ f n = ¯ η bias n + f ∗ and ∥ f ∥ 2 L 2 ρX = ⟨ f, Σ m f ⟩ .

Original: Proposition 1.
Revised: Proposition 1.

Original: Based on the above setting, the averaged excess risk admits the following biasvariance decomposition
Revised: Based on the above setting, the averaged excess risk admits the following biasvariance decomposition

Original: [Equation]
Revised: [Equation]

Original: By (partly) decoupling the multiple randomness sources of initialization, label noise, and datasampling (as well as stochastic gradients), we give precise non-asymptotic error bounds for bias andvariance as below.
Revised: By (partly) decoupling the multiple randomness sources of initialization, label noise, and datasampling (as well as stochastic gradients), we give precise non-asymptotic error bounds for bias andvariance as below.

Original: Theorem 1.
Revised: Theorem 1.

Original: (Error bound for bias) Under Assumptions 2, 1, 3, 4 with r ′ ⩾ 1 , if the step-sizeγ t := γ 0 t − ζ with ζ ∈ [0 , 1) satisfiesγ 0 ≲
Revised: (Error bound for bias) Under Assumptions 2, 1, 3, 4 with r ′ ⩾ 1 , if the step-sizeγ t := γ 0 t − ζ with ζ ∈ [0 , 1) satisfies

Original: [Equation]
Revised: [Equation]

Original: ∼ O
Revised: ≲

Original: [Equation]
Revised: [Equation]

Original: 1) , the Bias in Proposition 1 holds by
Revised: m ) ∼ O (1) , the Bias in Proposition 1 holds by

Original: [Equation]
Revised: [Equation]

Original: Remark:
Revised: Remark:

Original: The error bound for Bias is monotonically decreasing at O ( n ζ − 1 ) rate.
Revised: The error bound for Bias is monotonically decreasing at O ( n ζ − 1 ) rate.

Original: For the constantstep-size setting, it converges at O (1 /n ) rate, which is better than O ( (cid:112) log n/n ) in [24] relying onclosed-form solution under correlated features with polynomial decay on Σ d .
Revised: For the constantstep-size setting, it converges at O (1 /n ) rate, which is better than O ( (cid:112) log n/n ) in [25] relying onclosed-form solution under correlated features with polynomial decay on Σ d .

Revised: Besides, our resulton bias matches the exact formulation in [11] under the closed-form solution, i.e., monotonicallydecreasing bias.

Revised: One slight difference is, their result on bias tends to a constant under the overparameterized regime while our bias result can converge to zero.

Original: Theorem 2.
Revised: Theorem 2.

Original: (Error bound for variance)
Revised: (Error bound for variance)

Original: Under Assumptions 1, 3, 4 with r ′ ⩾ 1 , and Assumption5 with τ > 0 , if the step-size γ t := γ 0 t − ζ with ζ ∈ [0 , 1) satisfies γ 0 ≲ 
Revised: Under Assumptions 1, 3, 4 with r ′ ⩾ 1 , and Assumption5 with τ > 0 , if the step-size γ t := γ 0 t − ζ with ζ ∈ [0 , 1) satisfies γ 0 ≲ 1 r ′

Original: [Equation] ) ∼ O (1) , the
Revised: Tr( (cid:101) Σ m ) ∼ O (1) , the

Original: Variance defined in Proposition 1 holds
Revised: Variance defined in Proposition 1 holds

Original: [Equation]
Revised: [Equation]

Original: Remark: 
Revised: Remark: We make the following remarks: i )

Original: The error bound for Variance is demonstrated to be unimodal: increasing with m in theunder-parameterized regime and decreasing with m in the over-parameterized regime, and finallyconverge to a constant order (that depends on noise parameter τ 2 ), which matches recent resultsrelying on closed-form solution for (refined) variance, e.g., [11, 26, 15].
Revised: The error bound for Variance is demonstrated to be unimodal: increasing with m in the underparameterized regime and decreasing with m in the over-parameterized regime, and finally converge to a constant order (that depends on noise parameter τ 2 ), which matches recent results relying onclosed-form solution for (refined) variance, e g.,

Revised: ii

Revised: ) when compared to least squares, our result can degenerate to this setting by choosing m :=

Revised: d andthe constant step-size SGD.

Original:  More discussion on ourderived results refers toAppendix A.
Revised: Our upper bound is able to match the lower bound in [1, Corollary 1]with the same order; can recover the result of [16] by taking the effective dimension k ∗ = min { n, d }(no data spectrum assumption is required here). More discussion on our derived results refers to

Revised: Appendix A.

Original:  Proof Outline and Discussion
Revised:  Proof Outline and Discussion

Original: In this section, we first introduce the structure of the proofs with high level ideas, and then discussour work with previous literature in terms of the used techniques and the obtained results.
Revised: In this section, we first introduce the structure of the proofs with high level ideas, and then discussour work with previous literature in terms of the used techniques and the obtained results.

Original: 4.1 Proof outline
Revised: 4.1 Proof outline

Original: We (partly) disentangle the multiple randomness sources on the data X , the random features matrix
Revised: We (partly) disentangle the multiple randomness sources on the data X , the random features matrix

Original: W , the noise ε , make full use of statistical properties of covariance operators Σ m and (cid:101) Σ m in
Revised: W , the noise ε , make full use of statistical properties of covariance operators Σ m and (cid:101) Σ m in

Original: Section 3.2, and provide the respective (bias and variance) upper bounds in terms of multiplerandomness sources, as shown in Figure 1.
Revised: Section 3.2, and provide the respective (bias and variance) upper bounds in terms of multiplerandomness sources, as shown in Figure 1.

Original: Bias : To bound Bias , we need some auxiliary notations.
Revised: Bias : To bound Bias , we need some auxiliary notations.

Original: Recall Σ m =
Revised: Recall Σ m =

Original: E x [ φ ( x ) ⊗ φ ( x )] and(cid:101) Σ m =
Revised: E x [ φ ( x ) ⊗ φ ( x )] and(cid:101) Σ m =

Original: E x , W [ φ ( x ) ⊗ φ ( x )] , define
Revised: E x , W [ φ ( x ) ⊗ φ ( x )] , defineη bX t

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: with the average ¯ η bX n := 1 n (cid:80) n − 1 t =

Revised: with the average ¯ η bX n := 1 n (cid:80) n − 1 t =

Original: ¯ η bX t and ¯ η bXW n := 1 n (cid:80) n − 1 t =

Revised: ¯ η bX t and ¯ η bXW n := 1 n (cid:80) n − 1 t =

Original: ¯ η bXW t .
Revised: ¯ η bXW t .

Original: Accordingly, η bX t can be regardedas a "deterministic" version of η bias t : we omit the randomness on X (data sampling, stochasticgradients) by replacing [ φ ( x ) φ ( x ) ⊤ ] with its expectation Σ m .
Revised: Accordingly, η bX t can be regardedas a "deterministic" version of η bias t : we omit the randomness on X (data sampling, stochastic [Figure] gradients) by replacing [ φ ( x ) φ ( x ) ⊤ ] with its expectation Σ m .

Original: Likewise, η bXW t is a deterministicversion of η vX t by replacing Σ m with its expectation (cid:101) Σ m (randomness on initialization).
Revised: Likewise, η bXW t is a deterministicversion of η vX t by replacing Σ m with its expectation (cid:101) Σ m (randomness on initialization).

Original: By Minkowski inequality, the Bias can be decomposed as Bias ≲ B1 + B2 + B3 , where B1 :
Revised: By Minkowski inequality, the Bias can be decomposed as Bias ≲ B1 + B2 + B3 , where B1 :

Original: = 238 E
Revised: = 237 E

Original: X , W (cid:2) ⟨ ¯ η bias n − ¯ η bX n , Σ m (¯ η bias n − ¯ η bX n ) ⟩ (cid:3) and B2 :=
Revised: X , W (cid:2) ⟨ ¯ η bias n − ¯ η bX n , Σ m (¯ η bias n − ¯ η bX n ) ⟩ (cid:3) and B2 :=

Original: E W (cid:2) ⟨ ¯ η bX n − ¯ η bXW n , Σ m (¯ η bX n − ¯ η bXW n ) ⟩ (cid:3) and B3 :=⟨ ¯ η bXW n , (cid:101) Σ m ¯ η bXW n ⟩ .
Revised: E W (cid:2) ⟨ ¯ η bX n − ¯ η bXW n , Σ m (¯ η bX n − ¯ η bXW n ) ⟩ (cid:3) and B3 :=⟨ ¯ η bXW n , (cid:101) Σ m ¯ η bXW n ⟩ .

Original: Here B3 is a deterministic quantity that is closely connected to model (intrinsic) biaswithout any randomness; while B1 and B2 evaluate the effect of randomness from X and W on thebias, respectively.
Revised: Here B3 is a deterministic quantity that is closely connected to model (intrinsic) biaswithout any randomness; while B1 and B2 evaluate the effect of randomness from X and W on thebias, respectively.

Original: The error bounds for them can be directly found in Figure 1.
Revised: The error bounds for them can be directly found in Figure 1.

Original: To bound B3 , we directly focus on its formulation by virtue of spectrum decomposition and integralestimation.
Revised: To bound B3 , we directly focus on its formulation by virtue of spectrum decomposition and integralestimation.

Original: To bound B2 , we have B2 = 1 n 2 E W (cid:13)(cid:13)(cid:13) Σ 12 m (cid:80) n − 1 t =0 ( η bX t − η bXW t ) (cid:13)(cid:13)(cid:13) 2 , where the key partη bX t − η bXW t can be estimated by Lemma 6.
Revised: To bound B2 , we have B2 = 1 n 2 E W (cid:13)(cid:13)(cid:13) Σ 12 m (cid:80) n − 1 t =0 ( η bX t − η bXW t ) (cid:13)(cid:13)(cid:13) 2 , where the key partη bX t − η bXW t can be estimated by Lemma 6.

Original: To bound B1 , it can be further decomposed as (here weuse inaccurate expression for description simplicity) B1 ≲ (cid:80) t ∥ η bX t − η bXW t ∥ 

Revised: To bound B1 , it can be further decomposed as (here weuse inaccurate expression for description simplicity) B1 ≲ (cid:80) t ∥ η bX t − η bXW t ∥

Original: + (cid:80) t E X
Revised: + (cid:80) t E X

Original: ∥
Revised: ∥

Original: H t ∥ 2 in
Revised: H t ∥ 2 in

Original: Lemma 7, where H t − 1 :=
Revised: Lemma 7, where H t − 1 :=

Original: [Σ m − φ ( x t ) ⊗ φ ( x t )]
Revised: [Σ m − φ ( x t ) ⊗ φ ( x t )]

Original: η bX t − 1 .
Revised: η bX t − 1 .

Original: The first term can be upper bounded by 247 (cid:80) t ∥ η bX t − η bXW t ∥
Revised: The first term can be upper bounded by 246 (cid:80) t ∥ η bX t − η bXW t ∥

Original: 22 ≲ Tr(Σ m ) n ζ ∥ f ∗ ∥ 2 in Lemma 8, and the second term admits (
Revised: 22 ≲ Tr(Σ m ) n ζ ∥ f ∗ ∥ 2 in Lemma 8, and the second term admits (

Original: cid:80) t E X ∥
Revised: cid:80) t E X ∥

Original: H t ∥ 2 ≲
Revised: H t ∥ 2 ≲

Original: Tr(Σ m )
Revised: Tr(Σ m )

Original: ∥ f ∗ ∥ 2 in Lemma 9.
Revised: ∥ f ∗ ∥ 2 in Lemma 9.

Original: Variance : To bound Variance , we need some auxiliary notations.
Revised: Variance : To bound Variance , we need some auxiliary notations.

Original: [Equation]
Revised: [Equation]

Original:  with the averaged quantities ¯ η vX n := 1 n (cid:80) n − 1 t =0 ¯ η vX t , ¯ η vXW n := 1 n (cid:80) n − 1 t =

Revised: := ( I − γ t (cid:101) Σ m ) η vXW t − 1 + γ t ε t φ ( x t ) , η vXW= 0 . with the averaged quantities ¯ η vX n := 1 n (cid:80) n − 1 t =0 ¯ η vX t , ¯ η vXW n := 1 n (cid:80) n − 1 t =

Original: ¯ η vXW t .
Revised: ¯ η vXW t .

Original: Accordingly, η vX t can beregarded as a “semi-stochastic” version of η var t : we keep the randomness due to the noise ε t but omitthe randomness on X (data sampling) by replacing [ φ ( x ) φ ( x ) ⊤ ] with its expectation Σ m .
Revised: Accordingly, η vX t can beregarded as a “semi-stochastic” version of η var t : we keep the randomness due to the noise ε t but omitthe randomness on X (data sampling) by replacing [ φ ( x ) φ ( x ) ⊤ ] with its expectation Σ m .

Original: Likewise,η vXW t can be regarded as a “semi-stochastic” version of η vX t by replacing Σ m with its expectation (cid:101) Σ m(randomness on initialization).
Revised: Likewise,η vXW t can be regarded as a “semi-stochastic” version of η vX t by replacing Σ m with its expectation (cid:101) Σ m(randomness on initialization).

Original: The proof outline of Variance is similar to Bias as above mentioned,and thus is deferred to Appendix G due to page limit.
Revised: The proof outline of Variance is similar to Bias as above mentioned,and thus is deferred to Appendix G due to page limit.

Original: 4.2 Discussion with previous work
Revised: 4.2 Discussion on techniques

Original: Difference in techniques: Our proof framework follows [30] that focuses on kernel regressionwith stochastic approximation in the under-parameterized regimes ( d is regarded as finite and muchsmaller than n ).
Revised: Our proof framework follows [31] that focuses on kernel regression with stochastic approximationin the under-parameterized regimes ( d is regarded as finite and much smaller than n ).

Original: Nevertheless, even in the under-parameterized regime, their results can not bedirectly extended to random features model due to the extra randomness on W , coupling with otherrandomness sources on noise and data sampling, which makes their proof framework invalid: their[Figure] results depend on [28, Lemma 1] by taking conditional expectation to bridge the connection between263 E [ ∥ α t ∥ 2 ] and E ⟨ α t , Σ m α t ⟩ .
Revised: Nevertheless,even in the under-parameterized regime, their results can not be directly extended to random featuresmodel due to the extra randomness on W , coupling with other randomness sources on noise and datasampling, which makes their proof framework invalid: their results depend on [29, Lemma 1] bytaking conditional expectation to bridge the connection between E [ ∥ α t ∥ 2 ] and E ⟨ α t , Σ m α t ⟩ .

Original: This is valid for B1 but expires on other quantities.
Revised: This isvalid for B1 but expires on other quantities.

Original:  Some technical tools used in this paper follow [35] that focuses on linear regression with constantstep-size SGD for benign overfitting.
Revised: [Figure] Some technical tools used in this paper follow [16] that focuses on linear regression with constantstep-size SGD for benign overfitting.

Original: However, our results differ from it in 1) tackling multiplerandomness, e.g., stochastic gradients, random features (Gaussian initialization), by introducinganother type of error decomposition and several deterministic/randomness covariance operators.
Revised: However, our results differ from it in 1) tackling multiplerandomness, e.g., stochastic gradients, random features (Gaussian initialization), by introducinganother type of error decomposition and several deterministic/randomness covariance operators.

Original: Weprove nice statistical properties of them for proof, which gets rid of data spectrum assumption in [35].2) tackling non-constant step-size SGD setting by introducing new integral estimation techniques.
Revised: Weprove nice statistical properties of them for proof, which gets rid of data spectrum assumption in [16].2) tackling non-constant step-size SGD setting by introducing new integral estimation techniques.

Original: Original techniques on constant step-size in [35] are invalid due to non-homogeneous update rules.
Revised: Original techniques on constant step-size in [16] are invalid due to non-homogeneous update rules.

Original: The above two points make our proof relatively more intractable and largely different.
Revised: The above two points make our proof relatively more intractable and largely different.

Original: Besides, theirresults demonstrate that linear regression with SGD generalizes well (converges with n ) but has fewfindings on double descent.
Revised: Besides, theirresults demonstrate that linear regression with SGD generalizes well (converges with n ) but has fewfindings on double descent.

Original: Instead, our result depends on n and m (where d is implicitly included inm ), and is able to explain double descent.
Revised: Instead, our result depends on n and m (where d is implicitly included inm ), and is able to explain double descent.

Original: Comparison with previous work: Compared to [12] on RF optimized by gradient descent underthe Gaussian data in an asymptotic view, our non-asymptotic result holds for more general datadistribution under the SGD setting. In fact, our data assumption is weaker than most previous workassuming the data to be Gaussian, uniformly spread on a sphere, or isotropic/correlated features (withspectral decay assumption), except [13].
Revised: Here we take the estimation for the variance in [16] as an example to illustrate this. Variance [Equation]

Original: Nevertheless, we extend their asymptotic results relyingon the least-squares closed-form solution to non-asymptotic results under the SGD setting,whichtakes the effect of optimization into consideration. Besides, our result coincides several findings withrefined variance decomposition in [11, 26, 15], e.g., the interaction effect can dominate the variance(between samples and initialization); the unimodality of variance is a prevalent phenomenon.
Revised: In the least squares setting, the effective dimension to tackle I − ( I − γ Σ d ) n − t ; while our result is basedon fast eigenvalue decay of (cid:101) Σ m in Lemma 1 can direct to bound this. Besides, the homogeneousmarkov chain under the constant step-size setting is employed [16] for ( I − γ Σ d ) n − t , which isnaturally invalid under our decaying step-size setting. Instead, we introduce integral estimationtechniques to tackle adaptive step-size, see Appendix E for details.

Original:  Numerical Validation
Revised:  Numerical Validation

Original: In this section, we provide some numerical experiments in Figure 2 to support our theoretical resultsand findings.
Revised: In this section, we provide some numerical experiments in Figure 2 to support our theoretical resultsand findings.

Original: Note that our results go beyond Gaussian data assumption and can be empiricallyvalidated on real-world datasets.
Revised: Note that our results go beyond Gaussian data assumption and can be empiricallyvalidated on real-world datasets.

Original: More experiments can be found in Appendix H.
Revised: More experiments can be found in Appendix H.

Original: 5.1 Behavior of RF for interpolation learning
Revised: 5.1 Behavior of RF for interpolation learning

Original: Here we evaluate the test mean square error (MSE) of RFF regression on the MNIST data set[52], following the experimental setting of [13, 53], to study the generalization performance ofminimum-norm solution, see Figure 2(a).
Revised: Here we evaluate the test mean square error (MSE) of RFF regression on the MNIST data set[52], following the experimental setting of [13, 53], to study the generalization performance ofminimum-norm solution, see Figure 2(a).

Original: More results on regression dataset refer to Appendix H.
Revised: More results on regression dataset refer to Appendix H.

Original: Experimental settings: We take digit 3 vs. 7 as an example, and randomly select 300 trainingdata in these two classes, resulting in n = 600 for training.
Revised: Experimental settings: We take digit 3 vs. 7 as an example, and randomly select 300 trainingdata in these two classes, resulting in n = 600 for training.

Original: Hence, our setting with n = 600 ,d = 784 , and tuning m satisfies our realistic high dimensional assumption.
Revised: Hence, our setting with n = 600 ,d = 784 , and tuning m satisfies our realistic high dimensional assumption.

Original: The Gaussian kernelk ( x , x ′ ) = exp( −∥ x − x ′ ∥
Revised: The Gaussian kernelk ( x , x ′ ) = exp( −∥ x − x ′ ∥

Original: 22 / (2 σ 20 )) is used, where the kernel width σ 0 is chosen as σ 20 =
Revised: 22 / (2 σ 20 )) is used, where the kernel width σ 0 is chosen as σ 20 =

Original: d in high dimensional settings such that ∥ x ∥ 22 /d ∼ O (1) in Assumption 1.
Revised: d inhigh dimensional settings such that ∥ x ∥ 22 /d ∼ O (1) in Assumption 1.

Original: In our experiment, the initialstep-size is set to γ 0 = 

Revised: In our experiment, the initial step-size is set to γ 0 =

Original: and we take the initial point θ 0 near the min-norm solution 2 corruptedwith zero-mean, unit-variance Gaussian noise.
Revised: and we take the initial point θ 0 near the min-norm solution 2 corruptedwith zero-mean, unit-variance Gaussian noise.

Original: The experiments are repeated 10 times and the test
Revised: The experiments are repeated 10 times and the test

Original: MSE (mean ± std.) can be regarded as a function of the ratio m/n by tuning m .
Revised: MSE (mean ± std.) can be regarded as a function of the ratio m/n by tuning m .

Original: Results on differentinitialization and more epochs of SGD refer to Appendix H.
Revised: Results on differentinitialization and more epochs of SGD refer to Appendix H.

Original: SGD vs. minimal-norm solution: Figure 2(a) shows the test MSE of RF regression with averaged
Revised: SGD vs. minimal-norm solution: Figure 2(a) shows the test MSE of RF regression with averaged

Original: SGD (we take ζ = 0 .
Revised: SGD (we take ζ = 0 .

Original: 5 as an example; red line) and minimal-norm solution (blue line).
Revised: 5 as an example; red line) and minimal-norm solution (blue line).

Original: First, weobserve the double descent phenomenon: a phase transition on the two sides of the interpolationthreshold at 2 m = n when these two algorithms are employed.
Revised: First, weobserve the double descent phenomenon: a phase transition on the two sides of the interpolationthreshold at 2 m = n when these two algorithms are employed.

Original: Second, in terms of test error, RFwith averaged SGD is slightly inferior to that with min-norm solution, but still generalizes well.
Revised: Second, in terms of test error, RFwith averaged SGD is slightly inferior to that with min-norm solution, but still generalizes well.

Original: 5.2 Behavior of our error bounds
Revised: 5.2 Behavior of our error bounds

Original: We have experimentally validate the phase transition and corresponding double descent in the previoussection, and here we aim to semi-quantitatively assess our derived bounds for Bias and Variance ,see Figure 2(b) and 2(c).
Revised: We have experimentally validate the phase transition and corresponding double descent in the previoussection, and here we aim to semi-quantitatively assess our derived bounds for Bias and Variance ,see Figure 2(b) and 2(c).

Original: Results of these quantities on different step-size refer to Appendix H.
Revised: Results of these quantities on different step-size refer to Appendix H.

Original: Experimental settings: Since the target function f ∗ , the covariance operators Σ d , Σ m , and thenoise ε are unknown on the MNIST data set, our experimental evaluation need some assumptionsto calculate Bias and Variance .
Revised: Experimental settings: Since the target function f ∗ , the covariance operators Σ d , Σ m , and thenoise ε are unknown on the MNIST data set, our experimental evaluation need some assumptionsto calculate Bias and Variance .

Original: First, we assume the label noise ε ∼ N (0 , 1) , which can in turnobtain f ∗ ( x ) on both training and test data due to f ∗ ( x ) =
Revised: First, we assume the label noise ε ∼ N (0 , 1) , which can in turnobtain f ∗ ( x ) on both training and test data due to f ∗ ( x ) =

Original: y − ε .
Revised: y − ε .

Original: Second, the covariance matrices Σ dand Σ m are estimated by the related sample covariance matrices.
Revised: Second, the covariance matrices Σ dand Σ m are estimated by the related sample covariance matrices.

Original: When using the Gaussian kernel, thecovariance matrix (cid:101) Σ m can be directly computed, see the remark in Lemma 1, where the expectationon x is approximated by Monte Carlo sampling with n training samples.
Revised: When using the Gaussian kernel, thecovariance matrix (cid:101) Σ m can be directly computed, see the remark in Lemma 1, where the expectationon x is approximated by Monte Carlo sampling with n training samples.

Original: Accordingly, based on theabove results, we are ready to calculate η bias t in Eq.
Revised: Accordingly, based on theabove results, we are ready to calculate η bias t in Eq.

Original: (5), η bX t , and η bXW t in Eq.
Revised: (5), η bX t , and η bXW t in Eq.

Original: (7), respectively, whichis further used to approximately compute B1 :
Revised: (7), respectively, whichis further used to approximately compute B1 :

Original: = E X , W (cid:2) ⟨ ¯ η bias n − ¯ η bX n , Σ m (¯ η bias n − ¯ η bX n ) ⟩ (cid:3) (red line)and B2 :=
Revised: = E X , W (cid:2) ⟨ ¯ η bias n − ¯ η bX n , Σ m (¯ η bias n − ¯ η bX n ) ⟩ (cid:3) (

Original:  E W (cid:2) ⟨ ¯ η bX n − ¯ η bXW n , Σ m (¯ η bX n −¯ η bXW n ) ⟩ (cid:3) (blue line) andB3 := ⟨ ¯ η bXW n , (cid:101) Σ m ¯ η bXW n ⟩ (green line).
Revised: and B2 := E W (cid:2) ⟨ ¯ η bX n − ¯ η bXW n , Σ m ( η bX n − η bXW n ) ⟩ (cid:3) ( ) and

Original: The(approximate) computation for Variance can be similarachieved bythis process.
Revised: (approximate similar by

Original: Error bounds for bias: Figure 2(b) shows the trends of (scaled) B1 , B2 , and B3 .
Revised: Error bounds for bias: Figure 2(b) shows the trends of (scaled) B1 , B2 , and B3 .

Original: Recall our errorbound: B1 , B3 ∼ O ( n ζ − 1 ) ;
Revised: Recall our errorbound: B1 , B3 ∼ O ( n ζ − 1 ) ;

Original: B2 ∼ O
Revised: B2 ∼ O

Original: ( mn (2 ζ − 2) ) for m < n and O (1 /m ) for m > n .
Revised: ( mn (2 ζ − 2) ) for m < n and O (1 /m ) for m > n .

Original: We find that,all of them monotonically decreases when m increases from the under-parameterized regime to theover-parameterized regime.
Revised: We find that,all of them monotonically decreases when m increases from the under-parameterized regime to theover-parameterized regime.

Original: When m < n , B2 converges faster than B1 and B3 .
Revised: When m < n , B2 converges faster than B1 and B3 .

Original: These experimentalresults coincide with our error bound on them.
Revised: These experimentalresults coincide with our error bound on them.

Original: Error bounds for variance: Figure 2(c) shows the trends of (scaled) V1 , V2 , and V3 .
Revised: Error bounds for variance: Figure 2(c) shows the trends of (scaled) V1 , V2 , and V3 .

Original: Recall our errorbound: in the under-parameterized regime, V1 , V2 , and V3 increases with m at a certain O ( n ζ − 1 m )rate; and in the over-parameterized regime, V1 and V2 are in O (1) order while V3 decreases withm .
Revised: Recall our errorbound: in the under-parameterized regime, V1 , V2 , and V3 increases with m at a certain O ( n ζ − 1 m )rate; and in the over-parameterized regime, V1 and V2 are in O (1) order while V3 decreases withm .

Original: Figure 2(c) shows that, when 2 m < n , V1 and V2 monotonically increases with m and thenremain unchanged when 2 m > n .
Revised: Figure 2(c) shows that, when 2 m < n , V1 and V2 monotonically increases with m and thenremain unchanged when 2 m > n .

Original: Besides, V3 is observed to be unimodal: firstly increasing when2 m < n , reaching to the peak at 2 m = n , and then decreasing when 2 m > n , which admits thephase transition at 2 m = n .
Revised: Besides, V3 is observed to be unimodal: firstly increasing when2 m < n , reaching to the peak at 2 m = n , and then decreasing when 2 m > n , which admits thephase transition at 2 m = n .

Original: Accordingly, these findings accord with our theoretical results, and alsomatches refined results in [11, 26, 15]: the unimodality of variance is a prevalent phenomenon.
Revised: Accordingly, these findings accord with our theoretical results, and alsomatches refined results in [11, 27, 15]: the unimodality of variance is a prevalent phenomenon.

Original:  Conclusion
Revised:  Conclusion

Original: We present non-asymptotic results for RF regression under the averaged SGD setting for understandingbenign overfitting under the optimization effect.
Revised: We present non-asymptotic results for RF regression under the averaged SGD setting for understandingbenign overfitting under the optimization effect.

Original: Our theoretical and empirical results demonstrate that,the error bounds for variance and bias can be unimodal and monotonically decreasing, respectively,which is able to recover the double descent phenomenon.
Revised: Our theoretical and empirical results demonstrate that,the error bounds for variance and bias can be unimodal and monotonically decreasing, respectively,which is able to recover the double descent phenomenon.

Original: Regarding to constant/adaptive step-sizesetting, there is no difference between the constant step-size case and the exact minimal-norm solutionon the convergence rate; while the polynomial-decay step-size case will slow down the learning rate,but does not change the error bound for variance in over-parameterized regime that converges to O (1)order, that depends on noise parameter(s).
Revised: Regarding to constant/adaptive step-sizesetting, there is no difference between the constant step-size case and the exact minimal-norm solutionon the convergence rate; while the polynomial-decay step-size case will slow down the learning rate,but does not change the error bound for variance in over-parameterized regime that converges to O (1)order, that depends on noise parameter(s).

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] We clearly discuss the limitation of this work in Appendix A.2.
Revised: [Yes] We clearly discuss the limitationof this work in Appendix A.2.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] Our work
Revised: [No] Our work

Original: is theoretical and generally will have no negative societal impacts.
Revised: is theoretical and generally will have no negative societal impacts.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] The assump-
Revised: [Yes] The assump-

Original: tions are clearly stated and well discussed.
Revised: tions are clearly stated and well discussed.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] All of the proofs can
Revised: [Yes] All of the proofs can

Original: be found in the Appendix.
Revised: be found in the Appendix.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type

Original: of GPUs, internal cluster, or cloud provider)?
Revised: of GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Revised: [Table]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]