Original: T UNING F REQUENCY B IAS IN N EURAL N ETWORK T RAINING WITH N ONUNIFORM D ATA
Revised: T UNING F REQUENCY B IAS IN N EURAL N ETWORK T RAINING WITH N ONUNIFORM D ATA

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Small generalization errors of over-parameterized neural networks (NNs) can be partially explained by the frequency biasing phenomenon, where gradient-based algorithms minimize the low-frequency misfit before reducing the high-frequency residuals.
Revised: Small generalization errors of over-parameterized neural networks (NNs) can be partially explained by the frequency biasing phenomenon, where gradient-based algorithms minimize the low-frequency misfit before reducing the high-frequency residuals.

Original: Using the Neural Tangent Kernel (NTK), one can provide a theoretical analysis for training where data are drawn from constant or piecewise-constant probability densities.
Revised: Using the Neural Tangent Kernel (NTK), one can provide a theoretical analysis for training where data are drawn from constant or piecewise-constant probability densities.

Original: Since most training data sets are not drawn from such distributions, we use the NTK model and a data-dependent quadrature rule to theoretically quantify the frequency biasing of NN training given nonuniform data.
Revised: Since most training data sets are not drawn from such distributions, we use the NTK model and a data-dependent quadrature rule to theoretically quantify the frequency biasing of NN training given nonuniform data.

Original: By replacing the loss function with a selected Sobolev norm, we can amplify or dampen the intrinsic frequency biasing in NN training.
Revised: By replacing the loss function with a selected Sobolev norm, we can amplify or dampen the intrinsic frequency biasing in NN training.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural networks (NNs) are often trained in supervised learning on a small data set.
Revised: Neural networks (NNs) are often trained in supervised learning on a small data set.

Original: They are observed to provide accurate predictions for a large number of test examples that are not seen during training.
Revised: They are observed to provide accurate predictions for a large number of test examples that are not seen during training.

Original: A mystery is how training can achieve small generalization errors in an overparameterized
Revised: A mystery is how training can achieve small generalization errors in an overparameterized

Original: NN and a so-called “double-descent” risk curve (Belkin et al., 2019).
Revised: NN and a so-called “double-descent” risk curve (Belkin et al., 2019).

Original: In recent years, a potential answer has emerged called “frequency biasing,” which is the phenomenon that in the early epochs of training, an overparameterized NN finds a low-frequency fit of the training data while higher frequencies are learned in later epochs (Rahaman et al., 2019; Yang & Salman, 2019; Xu, 2020).
Revised: In recent years, a potential answer has emerged called “frequency biasing,” which is the phenomenon that in the early epochs of training, an overparameterized NN finds a low-frequency fit of the training data while higher frequencies are learned in later epochs (Rahaman et al., 2019; Yang & Salman, 2019; Xu, 2020).

Original: In addition to generalization errors, it is often useful to understand the convergence rate for each spectral component of the data mismatch to study the robustness of the NN under noises.
Revised: In addition to generalization errors, it is often useful to understand the convergence rate for each spectral component of the data mismatch to study the robustness of the NN under noises.

Original: Currently, frequency biasing is theoretically understood via the Neural Tangent Kernel (NTK)
Revised: Currently, frequency biasing is theoretically understood via the Neural Tangent Kernel (NTK)

Original: (Jacot et al., 2018) for uniform training data (Arora et al., 2019; Cao et al., 2019; Basri et al., 2019) and data distributed according to a piecewise constant probability measure (Basri et al., 2020).
Revised: (Jacot et al., 2018) for uniform training data (Arora et al., 2019; Cao et al., 2019; Basri et al., 2019) and data distributed according to a piecewise constant probability measure (Basri et al., 2020).

Original: However, most training data sets in practice are highly clustered and not uniform.
Revised: However, most training data sets in practice are highly clustered and not uniform.

Original: Yet, frequency biasing is still observed during NN training (Fridovich-Keil et al., 2021), even though the theory is absent.
Revised: Yet, frequency biasing is still observed during NN training (Fridovich-Keil et al., 2021), even though the theory is absent.

Original: This paper proves that frequency biasing is present when there is nonuniform training data by using a new viewpoint based on a data-dependent quadrature rule.
Revised: This paper proves that frequency biasing is present when there is nonuniform training data by using a new viewpoint based on a data-dependent quadrature rule.

Original: We use this theory to propose new loss functions for NN training that accelerate its convergence and improve stability with respect to noise.
Revised: We use this theory to propose new loss functions for NN training that accelerate its convergence and improve stability with respect to noise.

Original: A NN function is a map N : R d
Revised: A NN function is a map N : R d

Original: → R given by
Revised: → R given by

Original: [Equation]
Revised: [Equation]

Original: where W i ∈
Revised: where W i ∈

Original: R m i ×
Revised: R m i ×

Original: m i − 1 are weights, m 

Revised: m i − 1 are weights, m

Original: = d , b i ∈ R
Revised: = d , b i ∈ R

Original: m i are biases, and N L is the number of layers.
Revised: m i are biases, and N L is the number of layers.

Original: Here, σ is the activation function and applied entry-wise to a vector, i.e., σ ( a ) j = σ ( a j ) .
Revised: Here, σ is the activation function and applied entry-wise to a vector, i.e., σ ( a ) j = σ ( a j ) .

Original: In supervised learning, the goal of NN training is to learn weights and bias terms in a NN function, which we denote by N ( x ) , given training data ( x i , y i ) for 1 ≤ i ≤ n , where x i ∈ R d and y ∈ R .
Revised: In supervised learning, the goal of NN training is to learn weights and bias terms in a NN function, which we denote by N ( x ) , given training data ( x i , y i ) for 1 ≤ i ≤ n , where x i ∈ R d and y ∈ R .

Original: In this paper, we consider ReLU NNs, which are NNs for which the activation function is the ReLU function given by ReLU( t ) =
Revised: In this paper, we consider ReLU NNs, which are NNs for which the activation function is the ReLU function given by ReLU( t ) =

Original: max( t, 0) .
Revised: max( t, 0) .

Original: The ReLU activation function has a useful multiplicative property that ReLU( αt ) =
Revised: The ReLU activation function has a useful multiplicative property that ReLU( αt ) =

Original: α ReLU( t ) for any α > 0 .
Revised: α ReLU( t ) for any α > 0 .

Original: Due to this property, we assume that the training samples are normalized so that x 1 , . . .
Revised: Due to this property, we assume that the training samples are normalized so that x 1 , . . .

Original: , x n ∈ S d − 1 .
Revised: , x n ∈ S d − 1 .

Original: To introduce a continuous perspective, we assume that there is an underlying function g : S d − 

Revised: To introduce a continuous perspective, we assume that there is an underlying function g : S d −

Original: → R such that y i
Revised: → R such that y i

Original: = g ( x i ) for 1 ≤ i ≤ n and the x i ’s are distributed according to a distribution µ ( x ) .
Revised: = g ( x i ) for 1 ≤ i ≤ n and the x i ’s are distributed according to a distribution µ ( x ) .

Original: The training procedure is often a gradient-based optimization algorithm that minimizes the residual in the squared L 2 ( dµ ) norm, i.e.,
Revised: The training procedure is often a gradient-based optimization algorithm that minimizes the residual in the squared L 2 ( dµ ) norm, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where A d is the Lebesgue measure of the hypersphere S d − 1 and W represents the weights and bias terms.
Revised: where A d is the Lebesgue measure of the hypersphere S d − 1 and W represents the weights and bias terms.

Original: Similar to most theoretical studies investigating frequency biasing, we restrict ourselves to 2-layer NNs (Arora et al., 2019; Basri et al., 2019; Su & Yang, 2019; Cao et al., 2019).
Revised: Similar to most theoretical studies investigating frequency biasing, we restrict ourselves to 2-layer NNs (Arora et al., 2019; Basri et al., 2019; Su & Yang, 2019; Cao et al., 2019).

Original: To study NN training, it is common to consider the dynamics of Φ( W ) as one optimizes the coefficients in W .
Revised: To study NN training, it is common to consider the dynamics of Φ( W ) as one optimizes the coefficients in W .

Original: For example, the gradient flow of the NN weights is given by d W dt = − ∂ Φ ∂ W .
Revised: For example, the gradient flow of the NN weights is given by d W dt = − ∂ Φ ∂ W .

Original: Define the residual z ( x ; W ) =
Revised: Define the residual z ( x ; W ) =

Original: g ( x ) − N ( x ; W ) .
Revised: g ( x ) − N ( x ; W ) .

Original: Applying gradient flow with the population loss gives us
Revised: Applying gradient flow with the population loss gives us

Original: [Equation]
Revised: [Equation]

Original: where K ( x , x ′ ; W ) = (cid:10) ∂
Revised: where K ( x , x ′ ; W ) = (cid:10) ∂

Original: N ( x ; W ) ∂ W ,
Revised: N ( x ; W ) ∂ W ,

Original: [Equation]
Revised: [Equation]

Original: (cid:11) .
Revised: (cid:11) .

Original: Under the assumptions that the weights do not change much during training, one can consider the NTK given the underlying time-independent distribution of W , i.e., K ∞ ( x , x ′ ) =
Revised: Under the assumptions that the weights do not change much during training, one can consider the NTK given the underlying time-independent distribution of W , i.e., K ∞ ( x , x ′ ) =

Original: E W [ K ( x , x ′ ; W )]
Revised: E W [ K ( x , x ′ ; W )]

Original: (Du et al., 2018).
Revised: (Du et al., 2018).

Original: Based on eq.
Revised: Based on eq.

Original: (2), one can understand the decay of the residual by studying the reproducing kernel Hilbert space (RKHS) through a spectral decomposition of the integral operator L defined by ( L z )( x ) = ´
Revised: (2), one can understand the decay of the residual by studying the reproducing kernel Hilbert space (RKHS) through a spectral decomposition of the integral operator L defined by ( L z )( x ) = ´

Original: K ∞ ( x , x ′ )
Revised: K ∞ ( x , x ′ )

Original: z ( x ′ ) dµ ( x ′ ) .
Revised: z ( x ′ ) dµ ( x ′ ) .

Original: Most results in the literature require µ ( x ) to be the uniform distribution over the sphere so that the eigenfunctions of L are spherical harmonics and the eigenvalues have explicit forms (Cao et al., 2019; Basri et al., 2019; Scetbon & Harchaoui, 2021).
Revised: Most results in the literature require µ ( x ) to be the uniform distribution over the sphere so that the eigenfunctions of L are spherical harmonics and the eigenvalues have explicit forms (Cao et al., 2019; Basri et al., 2019; Scetbon & Harchaoui, 2021).

Original: These explicit formulas for the eigenvalues and eigenfunctions of L rely on the Funk–Hecke theorem, which provides a formula allowing one to express an integral over a hypersphere by an integral over an interval (Seeley, 1966).
Revised: These explicit formulas for the eigenvalues and eigenfunctions of L rely on the Funk–Hecke theorem, which provides a formula allowing one to express an integral over a hypersphere by an integral over an interval (Seeley, 1966).

Original: The frequency biasing of NN training can be explained by the fact that low-degree spherical harmonic polynomials are eigenfunctions of L associated with large eigenvalues (Basri et al., 2019).
Revised: The frequency biasing of NN training can be explained by the fact that low-degree spherical harmonic polynomials are eigenfunctions of L associated with large eigenvalues (Basri et al., 2019).

Original: Thus, for uniform training data, the optimization of the weights and biases of an NN tends to fit the low-frequency components of the residual first.
Revised: Thus, for uniform training data, the optimization of the weights and biases of an NN tends to fit the low-frequency components of the residual first.

Original: When µ ( x ) is nonuniform, it is difficult to analyze the spectral properties of L and thus the frequency biasing properties of NN training.
Revised: When µ ( x ) is nonuniform, it is difficult to analyze the spectral properties of L and thus the frequency biasing properties of NN training.

Original: Since the Funk–Hecke formula no longer holds, there are only a few special cases where frequency biasing is understood (Williams & Rasmussen, 2006,
Revised: Since the Funk–Hecke formula no longer holds, there are only a few special cases where frequency biasing is understood (Williams & Rasmussen, 2006,

Original: Sec. 4.3).
Revised: Sec. 4.3).

Original: Although one may derive asymptotic bounds for the eigenvalues (Widom, 1963; 1964;
Revised: Although one may derive asymptotic bounds for the eigenvalues (Widom, 1963; 1964;

Original: Bach & Jordan, 2002)
Revised: Bach & Jordan, 2002)

Original: , it is hard to obtain formulas for the eigenfunctions, and one usually relies on numerical approximations (Baker, 1977).
Revised: , it is hard to obtain formulas for the eigenfunctions, and one usually relies on numerical approximations (Baker, 1977).

Original: For the ReLU-based NTK, (Basri et al., 2020) provided explicit eigenfunctions assuming that the µ ( x ) is piecewise constant on S 1 , but this analysis does not generalize to higher dimension.
Revised: For the ReLU-based NTK, (Basri et al., 2020) provided explicit eigenfunctions assuming that the µ ( x ) is piecewise constant on S 1 , but this analysis does not generalize to higher dimension.

Original: To study the frequency biasing of NN training, one needs to understand both the eigenvalues and eigenfunctions of L , and this remains a significant challenge for a general µ ( x ) due to the absence of the Funk–Hecke formula.
Revised: To study the frequency biasing of NN training, one needs to understand both the eigenvalues and eigenfunctions of L , and this remains a significant challenge for a general µ ( x ) due to the absence of the Funk–Hecke formula.

Original: To overcome this challenge, we take a different point-of-view.
Revised: To overcome this challenge, we take a different point-of-view.

Original: While it is standard to discretize the integral in eq.
Revised: While it is standard to discretize the integral in eq.

Original: (1) using a Monte Carlo-like average, we discretize it using a data-dependent quadrature rule where the nodes are at the training data.
Revised: (1) using a Monte Carlo-like average, we discretize it using a data-dependent quadrature rule where the nodes are at the training data.

Original: That is, we investigate frequency biasing of
Revised: That is, we investigate frequency biasing of

Original: NN training when minimizing the residual in the standard squared L 2 norm:
Revised: NN training when minimizing the residual in the standard squared L 2 norm:

Original: [Equation]
Revised: [Equation]

Original: where c 1 , . . .
Revised: where c 1 , . . .

Original: , c n are the quadrature weights associated with the (nonuniform) input data x 1 , . . .
Revised: , c n are the quadrature weights associated with the (nonuniform) input data x 1 , . . .

Original: , x n .
Revised: , x n .

Original: If x 1 , . .
Revised: If x 1 , . .

Original: , x n are drawn from a uniform distribution over the hypersphere, then one can select c i =
Revised: , x n are drawn from a uniform distribution over the hypersphere, then one can select c i =

Original: A d /n for 1 ≤ i ≤ n ; otherwise, one can choose any quadrature weights so that the integration rule is accurate (see section 4.2).
Revised: A d /n for 1 ≤ i ≤ n ; otherwise, one can choose any quadrature weights so that the integration rule is accurate (see section 4.2).

Original: If x 1 , . .
Revised: If x 1 , . .

Original: , x n are drawn at random from µ ( x ) , then it is often reasonable to select c i = 1 / ( np ( x i )) , where dµ ( x ) =
Revised: , x n are drawn at random from µ ( x ) , then it is often reasonable to select c i = 1 / ( np ( x i )) , where dµ ( x ) =

Original: p ( x ) d x .
Revised: p ( x ) d x .

Original: While c 1 , . . .
Revised: While c 1 , . . .

Original: , c n depend on x 1 , . . .
Revised: , c n depend on x 1 , . . .

Original: , x n , the continuous expression for (cid:101) Φ( W ) is always unaltered in eq.
Revised: , x n , the continuous expression for (cid:101) Φ( W ) is always unaltered in eq.

Original: Therefore, we can use the
Revised: Therefore, we can use the

Original: Funk–Hecke formula to analyze the eigenvalues and eigenfunctions of ˜ L defined by ( ˜ L z )( x ) = ´
Revised: Funk–Hecke formula to analyze the eigenvalues and eigenfunctions of ˜ L defined by ( ˜ L z )( x ) = ´

Original: S d − 1 K ∞ ( x , x ′ )
Revised: S d − 1 K ∞ ( x , x ′ )

Original: z ( x ′ )
Revised: z ( x ′ )

Original: d x ′ , revealing the frequency bias.
Revised: d x ′ , revealing the frequency bias.

Original: We address that by choosing eq.
Revised: We address that by choosing eq.

Original: (3) as a loss function instead of eq.
Revised: (3) as a loss function instead of eq.

Original: (1), we are enforcing the frequency bias of NN training as eq.
Revised: (1), we are enforcing the frequency bias of NN training as eq.

Original: (1) does not ensure such spectral property.
Revised: (1) does not ensure such spectral property.

Original: To further tune the NN frequency biasing during training, we also propose to minimize the residual in a squared Sobolev H s norm for a carefully selected s ∈ R .
Revised: To further tune the NN frequency biasing during training, we also propose to minimize the residual in a squared Sobolev H s norm for a carefully selected s ∈ R .

Original: Unlike the L 2 norm (the case of s = 0 ), the H s norm for s ̸ = 0 has its own frequency bias.
Revised: Unlike the L 2 norm (the case of s = 0 ), the H s norm for s ̸ = 0 has its own frequency bias.

Original: For s > 0 , H s penalizes high frequencies more than low, while for s < 0 , low frequencies are penalized the most.
Revised: For s > 0 , H s penalizes high frequencies more than low, while for s < 0 , low frequencies are penalized the most.

Original: We implement the squared
Revised: We implement the squared

Original: H s norm using a quadrature rule, which induces a different integral operator L s .
Revised: H s norm using a quadrature rule, which induces a different integral operator L s .

Original: We analyze the eigenvalues and eigenfunctions of L s , and consequently, the frequency biasing in the NN training using the Funk–Hecke formula.
Revised: We analyze the eigenvalues and eigenfunctions of L s , and consequently, the frequency biasing in the NN training using the Funk–Hecke formula.

Original: Given our new understanding of frequency biasing, we select s so that the H s norm amplifies, dampens, counterbalances, or reverses the natural frequency biasing from an overparameterized NN training.
Revised: Given our new understanding of frequency biasing, we select s so that the H s norm amplifies, dampens, counterbalances, or reverses the natural frequency biasing from an overparameterized NN training.

Original: Contributions.
Revised: Contributions.

Original: Here are our three main contributions to analyzing and tuning NN frequency bias.
Revised: Here are our three main contributions to analyzing and tuning NN frequency bias.

Original: (1) From our quadrature point-of-view, we analyze the frequency biasing in training a 2-layer overparameterized ReLU NN with nonuniform training data.
Revised: (1) From our quadrature point-of-view, we analyze the frequency biasing in training a 2-layer overparameterized ReLU NN with nonuniform training data.

Original: In Theorem 2, we show that the theory of frequency biasing in (Basri et al., 2019) for uniform training data continues to hold in the nonuniform case up to quadrature errors.
Revised: In Theorem 2, we show that the theory of frequency biasing in (Basri et al., 2019) for uniform training data continues to hold in the nonuniform case up to quadrature errors.

Original: In Theorem 3, we provide control of the quadrature errors.
Revised: In Theorem 3, we provide control of the quadrature errors.

Original: (2) We use our understanding of frequency biasing to modify the usual squared L 2 loss function to a squared H s norm.
Revised: (2) We use our understanding of frequency biasing to modify the usual squared L 2 loss function to a squared H s norm.

Original: By selecting s , we can amplify or dampen the intrinsic frequency biasing in NN training, accelerate the convergence of gradient-based optimization procedures, and separate out noises of particular frequencies.
Revised: By selecting s , we can amplify or dampen the intrinsic frequency biasing in NN training, accelerate the convergence of gradient-based optimization procedures, and separate out noises of particular frequencies.

Original: (3) A potential issue with the H s norm is the difficulties of implementing with high-dimensional training data.
Revised: (3) A potential issue with the H s norm is the difficulties of implementing with high-dimensional training data.

Original: Using an image dataset of dimension 28 2 = 784 , we show how to use an encoderdecoder architecture to implement a practical version of the squared H s norm loss and adjust the frequency biasing in NN training to suppress noises of different frequencies.
Revised: Using an image dataset of dimension 28 2 = 784 , we show how to use an encoderdecoder architecture to implement a practical version of the squared H s norm loss and adjust the frequency biasing in NN training to suppress noises of different frequencies.

Original: (see Figure 4).
Revised: (see Figure 4).

Original:  P RELIMINARIES AND NOTATION
Revised:  P RELIMINARIES AND NOTATION

Original: For d > 1 , let g : S d − 1 → R be a square-integrable function defined on S d − 1 .
Revised: For d > 1 , let g : S d − 1 → R be a square-integrable function defined on S d − 1 .

Original: The function g can be expressed in a spherical harmonic expansion given by
Revised: The function g can be expressed in a spherical harmonic expansion given by

Original: [Equation]
Revised: [Equation]

Original: where Y ℓ,p is the spherical harmonic function of degree ℓ and order p .
Revised: where Y ℓ,p is the spherical harmonic function of degree ℓ and order p .

Original: Here, N ( d, ℓ ) is the number of spherical harmonic functions of degree ℓ so that N ( d, 0) =
Revised: Here, N ( d, ℓ ) is the number of spherical harmonic functions of degree ℓ so that N ( d, 0) =

Original: 1 and N ( d, ℓ ) =
Revised: 1 and N ( d, ℓ ) =

Original: 2 ℓ + d − 2)Γ( ℓ + d − 2) Γ( ℓ +1)Γ( d − 1) for ℓ ≥ 1 .
Revised: 2 ℓ + d − 2)Γ( ℓ + d − 2) Γ( ℓ +1)Γ( d − 1) for ℓ ≥ 1 .

Original: The set { Y ℓ,p } ℓ ≥ 0 , 1 ≤ p ≤ N ( d,ℓ ) is an orthonormal basis for L 2 ( S d − 1 ) .
Revised: The set { Y ℓ,p } ℓ ≥ 0 , 1 ≤ p ≤ N ( d,ℓ ) is an orthonormal basis for L 2 ( S d − 1 ) .

Original: Let H dℓ be the span of { Y ℓ,p } N ( d,ℓ ) p =1 , and Π dℓ =
Revised: Let H dℓ be the span of { Y ℓ,p } N ( d,ℓ ) p =1 , and Π dℓ =

Original: (cid:76) ℓj =

Revised: (cid:76) ℓj =

Original: H dj be the space of spherical harmonics of degree ≤ ℓ .
Revised: H dj be the space of spherical harmonics of degree ≤ ℓ .

Original: Given distinct training data { x i } ni =1 from S d − 1 and evaluations y i
Revised: Given distinct training data { x i } ni =1 from S d − 1 and evaluations y i

Original: = g ( x i ) for 1 ≤ i ≤ n , our goal is to understand the intrinsic frequency-biasing behavior of training a 2-layer ReLU NN given by
Revised: = g ( x i ) for 1 ≤ i ≤ n , our goal is to understand the intrinsic frequency-biasing behavior of training a 2-layer ReLU NN given by

Original: [Equation]
Revised: [Equation]

Original: where m is the number of neurons, w 1 , . .
Revised: where m is the number of neurons, w 1 , . .

Original: . , w m and a 1 , . .
Revised: . , w m and a 1 , . .

Original: . , a m are weights, and b 1 , . . .
Revised: . , a m are weights, and b 1 , . . .

Original: , b m are the bias terms.
Revised: , b m are the bias terms.

Original: We begin with the same setup as in Basri et al.
Revised: We begin with the same setup as in Basri et al.

Original: (2020): assuming that (1) w 1 , . . .
Revised: (2020): assuming that (1) w 1 , . . .

Original: , w m are initialized independently and identically distributed (i.i.d.) from Gaussian random variables with covariance matrix κ 2 I , where κ > 0 , (2) the bias terms are initialized to zero, and (3) a 1 , . . .
Revised: , w m are initialized independently and identically distributed (i.i.d.) from Gaussian random variables with covariance matrix κ 2 I , where κ > 0 , (2) the bias terms are initialized to zero, and (3) a 1 , . . .

Original: , a m are initialized i.i.d.
Revised: , a m are initialized i.i.d.

Original: as +1 with probability 12 and − 1 otherwise, and { a r } are not updated during training.
Revised: as +1 with probability 12 and − 1 otherwise, and { a r } are not updated during training.

Original: We use a gradient-based optimization scheme to train for the weights and biases and aim to minimize the residual defined by a symmetric positive definite (SPD) matrix P , which can be neatly written as
Revised: We use a gradient-based optimization scheme to train for the weights and biases and aim to minimize the residual defined by a symmetric positive definite (SPD) matrix P , which can be neatly written as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where y =
Revised: where y =

Original: ( g ( x 1 ) , . . .
Revised: ( g ( x 1 ) , . . .

Original: , g ( x n )) ⊤ and u = ( N ( x 1 ) , . .
Revised: , g ( x n )) ⊤ and u = ( N ( x 1 ) , . .

Original: N ( x n ))
Revised: N ( x n ))

Original: ⊤ .
Revised: ⊤ .

Original: For example, we have P =
Revised: For example, we have P =

Original: A d n − 1 I in eq.
Revised: A d n − 1 I in eq.

Original: (1) and P = diag( c 1 , . . .
Revised: (1) and P = diag( c 1 , . . .

Original: , c n ) in eq.
Revised: , c n ) in eq.

Original: Recall that W represents all the weights and biases of the NN.
Revised: Recall that W represents all the weights and biases of the NN.

Original: Given the loss function, we train the NN based on the gradient descent algorithm:
Revised: Given the loss function, we train the NN based on the gradient descent algorithm:

Original: [Equation]
Revised: [Equation]

Original: where k is the iteration number and η > 0 is the learning rate.
Revised: where k is the iteration number and η > 0 is the learning rate.

Original: The matrix P induces an inner product ⟨ ξ , ζ ⟩ P = ξ ⊤ P ζ , which leads to a finite-dimensional Hilbert space with the norm ∥ ξ ∥ P = (cid:112) ⟨ ξ , ξ ⟩ P .
Revised: The matrix P induces an inner product ⟨ ξ , ζ ⟩ P = ξ ⊤ P ζ , which leads to a finite-dimensional Hilbert space with the norm ∥ ξ ∥ P = (cid:112) ⟨ ξ , ξ ⟩ P .

Original: Given a matrix A ∈ R n × n , we define its operator norm ∥
Revised: Given a matrix A ∈ R n × n , we define its operator norm ∥

Original: A ∥ P = sup ξ ∈ R n , ∥ ξ ∥ P =1 ∥
Revised: A ∥ P = sup ξ ∈ R n , ∥ ξ ∥ P =1 ∥

Original: A ξ ∥ P .
Revised: A ξ ∥ P .

Original: We also define a finite positive number that depends on P :
Revised: We also define a finite positive number that depends on P :

Original: [Equation]
Revised: [Equation]

Original: Note that by the third expression in eq.
Revised: Note that by the third expression in eq.

Original: (8), we also have M P =
Revised: (8), we also have M P =

Original: (cid:13)(cid:13) P
Revised: (cid:13)(cid:13) P

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13) 2 = (cid:112)
Revised: (cid:13)(cid:13) 2 = (cid:112)

Original: ∥ P ∥ 2 .
Revised: ∥ P ∥ 2 .

Original: Furthermore, we define the matrix H ∞ ∈ R n × n by
Revised: Furthermore, we define the matrix H ∞ ∈ R n × n by

Original: [Equation]
Revised: [Equation]

Original: Note that due to the introduction of the biases, H ∞ is slightly different than the one in (Du et al., 2018; Arora et al., 2019).
Revised: Note that due to the introduction of the biases, H ∞ is slightly different than the one in (Du et al., 2018; Arora et al., 2019).

Original: In fact, in contrast with (Du et al., 2018), H ∞ defined in eq.
Revised: In fact, in contrast with (Du et al., 2018), H ∞ defined in eq.

Original: (9) is SPD regardless of the distribution of the training data, as shown in the supplementary material.
Revised: (9) is SPD regardless of the distribution of the training data, as shown in the supplementary material.

Original: Proposition 1.
Revised: Proposition 1.

Original: If x 1 , . .
Revised: If x 1 , . .

Original: , x n are distinct, then H ∞ in eq.
Revised: , x n are distinct, then H ∞ in eq.

Original: (9) is SPD.
Revised: (9) is SPD.

Original: As a consequence of Proposition 1, the matrix H ∞ P has positive eigenvalues, which we denote by λ n − 1 ≥ · · · ≥ λ 0 > 0 .
Revised: As a consequence of Proposition 1, the matrix H ∞ P has positive eigenvalues, which we denote by λ n − 1 ≥ · · · ≥ λ 0 > 0 .

Original: In fact, let Λ = diag
Revised: In fact, let Λ = diag

Original: ( λ 0 , . . .
Revised: ( λ 0 , . . .

Original: , λ n − 1 ) .
Revised: , λ n − 1 ) .

Original: Then, H ∞ P is self-adjoint in ( R n , ⟨· , ·⟩ P ) (see appendix A) and can be diagonalized as
Revised: Then, H ∞ P is self-adjoint in ( R n , ⟨· , ·⟩ P ) (see appendix A) and can be diagonalized as

Original: [Equation]
Revised: [Equation]

Original: where P 1 / 2 H ∞ P 1 / 2 =
Revised: where P 1 / 2 H ∞ P 1 / 2 =

Original: V − 1 ΛV is SPD and therefore diagonalizable.
Revised: V − 1 ΛV is SPD and therefore diagonalizable.

Original: One can view H ∞ as coming from sampling a continuous kernel K ∞ : S d − 1 × S d − 1 → R given by
Revised: One can view H ∞ as coming from sampling a continuous kernel K ∞ : S d − 1 × S d − 1 → R given by

Original: [Equation]
Revised: [Equation]

Original: where ⟨· , ·⟩ is the ℓ 2 inner-product.
Revised: where ⟨· , ·⟩ is the ℓ 2 inner-product.

Original: We can view H ∞ as a discretization of K ∞ .
Revised: We can view H ∞ as a discretization of K ∞ .

Original: The eigenvalues and eigenfunctions of K ∞ are known explicitly via the Funk–Hecke formula (Basri et al., 2019): ˆ
Revised: The eigenvalues and eigenfunctions of K ∞ are known explicitly via the Funk–Hecke formula (Basri et al., 2019): ˆ

Original: [Equation]
Revised: [Equation]

Original: The explicit formulas for µ ℓ with ℓ ≥ 0 are given in the supplementary material.
Revised: The explicit formulas for µ ℓ with ℓ ≥ 0 are given in the supplementary material.

Original: We find that µ ℓ >for all ℓ and µ ℓ is asymptotically O ( ℓ − d ) for large ℓ (Basri et al., 2019; Bietti & Mairal, 2019).
Revised: We find that µ ℓ >for all ℓ and µ ℓ is asymptotically O ( ℓ − d ) for large ℓ (Basri et al., 2019; Bietti & Mairal, 2019).

Original:  T RAINING CONVERGENCE WITH A GENERAL LOSS FUNCTION
Revised:  T RAINING CONVERGENCE WITH A GENERAL LOSS FUNCTION

Original: Given the NN model in eq.
Revised: Given the NN model in eq.

Original: (5) and a general loss function Φ P in eq.
Revised: (5) and a general loss function Φ P in eq.

Original: (6), we are interested in the convergence rate of NN training.
Revised: (6), we are interested in the convergence rate of NN training.

Original: We study this by analyzing the convergence rate for each harmonic component.
Revised: We study this by analyzing the convergence rate for each harmonic component.

Original: We start by presenting a convergence result that holds for any SPD matrix P .
Revised: We start by presenting a convergence result that holds for any SPD matrix P .

Original: It says that up to an error ϵ , which can be made arbitrarily small by taking κ small enough and m large enough, the residual of the NN at the k th iteration is approximately ( I − 2 η H ∞ P ) k y .
Revised: It says that up to an error ϵ , which can be made arbitrarily small by taking κ small enough and m large enough, the residual of the NN at the k th iteration is approximately ( I − 2 η H ∞ P ) k y .

Original: Theorem 1.
Revised: Theorem 1.

Original: In eq.
Revised: In eq.

Original: (5) , suppose that w 1 , . . .
Revised: (5) , suppose that w 1 , . . .

Original: , w m are initialized i.i.d.
Revised: , w m are initialized i.i.d.

Original: from Gaussian random variables with covariance matrix κ 2 I , b 1 , . . .
Revised: from Gaussian random variables with covariance matrix κ 2 I , b 1 , . . .

Original: , b m are initialized to zero, and a 1 , . . .
Revised: , b m are initialized to zero, and a 1 , . . .

Original: , a m are initialized i.i.d.
Revised: , a m are initialized i.i.d.

Original: as +1 with probability 1 / 2 and − 1 otherwise.
Revised: as +1 with probability 1 / 2 and − 1 otherwise.

Original: Suppose the NN is trained with training data ( x i , y i ) for 1 ≤ i ≤ n , loss function Φ P in eq.
Revised: Suppose the NN is trained with training data ( x i , y i ) for 1 ≤ i ≤ n , loss function Φ P in eq.

Original: (6) for a SPD matrix P , and the training procedure is the gradient descent update rule eq.
Revised: (6) for a SPD matrix P , and the training procedure is the gradient descent update rule eq.

Original: (7) with step size η .
Revised: (7) with step size η .

Original: Let N k be the NN function after the k th iteration and u ( k )
Revised: Let N k be the NN function after the k th iteration and u ( k )

Original: =
Revised: =

Original: ( N k ( x 1 ) , . . .
Revised: ( N k ( x 1 ) , . . .

Original: , N k ( x n )) , where N 0 is the initial NN function.
Revised: , N k ( x n )) , where N 0 is the initial NN function.

Original: Let an accuracy goal 0 < ϵ < 1 , a probability of failure 0 < δ < 1 , and a time span T > 0 be given.
Revised: Let an accuracy goal 0 < ϵ < 1 , a probability of failure 0 < δ < 1 , and a time span T > 0 be given.

Original: Then, there exist constants C 1 , C 2 > 0 that depend only on the dimension d such that if 0 < η ≤ 1 / (2 M 2 P n ) (see eq.
Revised: Then, there exist constants C 1 , C 2 > 0 that depend only on the dimension d such that if 0 < η ≤ 1 / (2 M 2 P n ) (see eq.

Original: (8) ), κ ≤ C 1 ϵM − 1 P (cid:112) δ/n , and m satisfies
Revised: (8) ), κ ≤ C 1 ϵM − 1 P (cid:112) δ/n , and m satisfies

Original: [Equation]
Revised: [Equation]

Original: 13) then with probability ≥ 1 − δ , we have
Revised: 13) then with probability ≥ 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: Here, H ∞ follows eq.
Revised: Here, H ∞ follows eq.

Original: (9) , y =( g ( x 1 ) , . . .
Revised: (9) , y =( g ( x 1 ) , . . .

Original: , g ( x n )) ⊤ , and λ 0 is the smallest eigenvalue of H ∞ P .
Revised: , g ( x n )) ⊤ , and λ 0 is the smallest eigenvalue of H ∞ P .

Original: We defer the proof to the supplementary material, which uses techniques from (Su & Yang, 2019).
Revised: We defer the proof to the supplementary material, which uses techniques from (Su & Yang, 2019).

Original: The main idea behind the proof is that I − 2 η H ∞ P is close to the transition matrix for the residual y − u ( k ) when m is large.
Revised: The main idea behind the proof is that I − 2 η H ∞ P is close to the transition matrix for the residual y − u ( k ) when m is large.

Original: By taking κ small, we can control the size of u (0) and therefore obtain y − u ( k )
Revised: By taking κ small, we can control the size of u (0) and therefore obtain y − u ( k )

Original: ≈ ( I − 2 η H ∞ P ) k
Revised: ≈ ( I − 2 η H ∞ P ) k

Original: ( y − u (0))
Revised: ( y − u (0))

Original: ≈ ( I − 2 η H ∞ P ) k y .
Revised: ≈ ( I − 2 η H ∞ P ) k y .

Original: Furthermore, M 2 P n  is an upper bound on λ n − 1 , the maximum eigenvalue of H ∞ P . Therefore, our step size η does not vanish as the number of training data points n → ∞ , provided that M P
Revised: As η decreases, the gradient descent algorithm gets closer to the gradient flow algorithm (Du et al., 2018), which allows us to more accurately quantify the frequency biasing (see section 4). For a fixed n , if y is an eigenvector associated to λ min ( H ∞ P ) , then (cid:13)(cid:13)

Original: = O ( n −1 / 2  ).
Revised: ( I − 2 η H ∞ P )

Original: This is the case when P = A d n − 1 I , which corresponds to eq. So, we overcome the vanishing convergence rate issue due to the diminishing λ 0 as n → ∞ that appears in previous analyses of frequency biasing (Arora et al., 2019; Basri et al., 2019).
Revised: k y

Original: Also see (Su & Yang, 2019, Thm. 2), where the issue is discussed in details.
Revised: (cid:13)(cid:13) = exp ( − k log((1 − 2 ηλ min ( H ∞ P )) − 1 ))

Original: As η decreases, the gradient descent algorithm gets closer to the gradient flow algorithm (Du et al., 2018), which allows us to more accurately quantify the frequency biasing (see section 4).  F REQUENCY BIASING WITH AN L 2 - BASED LOSS FUNCTION
Revised: ∥ y ∥ , where log((1 − 2 ηλ min ( H ∞ P )) − 1 ) is the convergence rate (Su & Yang, 2019). If we assume λ min ( P ) =

Revised: O (1 /n ) , which is the case of eq.

Revised: (1), then as shown in (Su & Yang, 2019, Thm. 2) and (Nguyen et al., 2021), we expect that λ min ( H ∞ P ) → 0 as n → ∞ .

Revised: Hence, as n → ∞ , there exists a labeling y , depending on n , such that the convergence rate vanishes.

Revised: However, as suggested by (Su & Yang, 2019; Cao et al., 2019), for a fixed bandlimited target function g , its convergence rate in early epochs stays constant as n → ∞ .

Revised: We make similar observations in section 4.

Revised:  F REQUENCY BIASING WITH AN L 2 - BASED LOSS FUNCTION

Original: The mean-squared loss function in eq.
Revised: The mean-squared loss function in eq.

Original: (1) corresponds to setting c
Revised: (1) corresponds to setting c

Original: i =
Revised: i =

Original: A d /n for 1 ≤ i ≤ n in eq.
Revised: A d /n for 1 ≤ i ≤ n in eq.

Original: When µ is uniform, eq.
Revised: When µ is uniform, eq.

Original: (1) and eq.
Revised: (1) and eq.

Original: (3) are equivalent; when µ is nonuniform, we introduce a quadrature rule with nodes x 1 , . . .
Revised: (3) are equivalent; when µ is nonuniform, we introduce a quadrature rule with nodes x 1 , . . .

Original: , x n and weights c 1 , . . .
Revised: , x n and weights c 1 , . . .

Original: , c n to approximate the L 2 loss function eq.
Revised: , c n to approximate the L 2 loss function eq.

Original: The weights are selected so that for low-frequency functions f : S d − 1 → R , the quadrature errorE c ( f ) = ˆ
Revised: The weights are selected so that for low-frequency functions f : S d − 1 → R , the quadrature error

Original: [Equation]
Revised: [Equation]

Original: is relatively small.
Revised: is relatively small.

Original: A reasonable quadrature rule has positive weights for numerical stability and satisfies (cid:80) ni
Revised: A reasonable quadrature rule has positive weights for numerical stability and satisfies (cid:80) ni

Original: =1 c i =
Revised: =1 c i =

Original: A d so that it exactly integrates constants.
Revised: A d so that it exactly integrates constants.

Original: The continuous squared L 2 loss function based on the Lebesgue measure is then discretized to be the square of a weighted discrete ℓ 2 norm (see eq.
Revised: The continuous squared L 2 loss function based on the Lebesgue measure is then discretized to be the square of a weighted discrete ℓ 2 norm (see eq.

Original: Hence, we take P = D c = diag ( c 1 , . . .
Revised: Hence, we take P = D c = diag ( c 1 , . . .

Original: , c n ) , which is SPD as the c i ’s are positive.
Revised: , c n ) , which is SPD as the c i ’s are positive.

Original: For a vector v ∈ R n , we write ∥ v ∥ 2 c = v ⊤ D c v and set c max = max 1 ≤ i ≤ n { c i } .
Revised: For a vector v ∈ R n , we write ∥ v ∥ 2 c = v ⊤ D c v and set c max = max 1 ≤ i ≤ n { c i } .

Original: We now apply Theorem 1 to study the frequency bias of NN training with the squared L 2 loss eq.
Revised: We now apply Theorem 1 to study the frequency bias of NN training with the squared L 2 loss eq.

Original: We state these results in terms of quadrature errors.
Revised: We state these results in terms of quadrature errors.

Original: Recall our continuous setup where we assume that the training data is taken from a function g : S d − 1 → R so that y i
Revised: Recall our continuous setup where we assume that the training data is taken from a function g : S d − 1 → R so that y i

Original: = g ( x i ) for 1 ≤ i ≤ n .
Revised: = g ( x i ) for 1 ≤ i ≤ n .

Original: We further assume that g is bandlimited with bandlimit L where g = g 0 + · · · + g L and g ℓ ∈ H dℓ for 0 ≤ ℓ ≤ L .
Revised: We further assume that g is bandlimited with bandlimit L where g = g 0 + · · · + g L and g ℓ ∈ H dℓ for 0 ≤ ℓ ≤ L .

Original: With 1 ≤ i ≤ n , j, ℓ ≥ 0 , and 1 ≤ p ≤ N ( d, ℓ ) , we define quadrature errors as e aj,ℓ,p =
Revised: With 1 ≤ i ≤ n , j, ℓ ≥ 0 , and 1 ≤ p ≤ N ( d, ℓ ) , we define quadrature errors as [Equation]

Original: E c ( g j Y ℓ,p ) , e bi,ℓ,p = E c K ∞ ( x i , · ) Y ℓ,p ) , e cj,ℓ = E c ( g j g ℓ ) , e di,ℓ = E c ( K ∞ ( x i , · ) g ℓ ) .
Revised: [Equation]

Original: We interpret g ℓ = 0 when ℓ > L and function products are interpreted as the pointwise product.
Revised: We interpret g ℓ = 0 when ℓ > L and function products are interpreted as the pointwise product.

Original: 4. A FREQUENCY - BASED FORMULA FOR THE TRAINING ERROR
Revised: 4. A FREQUENCY - BASED FORMULA FOR THE TRAINING ERROR

Original: We obtain a similar result to (Arora et al., 2019, Thm. 4.1) when usingthe standard squared L 2 loss function  eq.
Revised: We obtain a similar result to (Arora et al., 2019, Thm. 4.1) when using loss function ˜ Φ in eq.

Original: (3), except our step size does not depend on λ 0 .

Original: Instead of expressing the training error in terms of the eigenvalues and eigenvectors of H ∞ D c , we directly relate the training error to the frequency component ofthe target function g and the eigenvalues of the continuous kernel K ∞ .
Revised: Instead of expressing the training error using the spectrum of H ∞ D c , we directly relate the training error to the frequency components of g and the eigenvalues of the continuous kernel K ∞ .

Original: Theorem 2.
Revised: Theorem 2.

Original: Under the same setup and assumptions of Theorem 1, let P = D c and M P = √ c max .
Revised: Under the same setup and assumptions of Theorem [Equation] let P = D c and M P = √ c max .

Original: If g : S d − 1 → R is a bandlimited function with bandlimit L and 1 − 2 ηµ ℓ > 0 for all 0 ≤ ℓ ≤ L (see eq.
Revised: If g : S d − 1 → R is a bandlimited function with bandlimit L and 1 − 2 ηµ ℓ > 0 for all 0 ≤ ℓ ≤ L (see eq.

Original: (12) ), then with probability ≥ 1 − δ we have
Revised: [Equation] then with probability ≥ 1 − δ we have

Original: [Equation]
Revised: [Equation]

Original: where ε 1 ( k ) and ε 2 satisfy
Revised: where ε 1 ( k ) and ε 2 satisfy

Original: [Equation]
Revised: [Equation]

Revised: limited understanding of the spectral property of the target function given its values at { x i } ni =1 , making frequency bias a void topic.

Revised: Hence, we assume the existence of a good quadrature rule for theoretical purposes.

Original: The proof of Theorem 2 is postponed to the supplementary material.
Revised: The proof of Theorem 2 is postponed to the supplementary material.

Original: The idea is that by Theorem 1, we know y − u ( k )
Revised: The idea is that by Theorem 1, we know y − u ( k )

Original: =
Revised: =

Original: ( I − 2 η H ∞ D c )
Revised: ( I − 2 η H ∞ D c )

Original: k y + ε 3 ( k ) .
Revised: k y + ε 3 ( k ) .

Original: Using Funk–Hecke formula and quadrature, we have that 1 − 2 ηµ ℓ are roughly the eigenvalues of I − 2 η H ∞ D c and y ℓ =
Revised: Using Funk–Hecke formula and quadrature, we have that 1 − 2 ηµ ℓ are roughly the eigenvalues of I − 2 η H ∞ D c and y ℓ =

Original: ( g ℓ ( x 1 ) , . . . , g ℓ ( x n ))
Revised: ( g ℓ ( x 1 ) , . . . , g ℓ ( x n ))

Original: ⊤ are associated eigenvectors.
Revised: ⊤ are associated eigenvectors.

Original: Hence, y − u ( k ) ≈ (cid:80) Lℓ =0 (1 − 2 ηµ ℓ ) k y ℓ .
Revised: Hence, y − u ( k ) ≈ (cid:80) Lℓ =0 (1 − 2 ηµ ℓ ) k y ℓ .

Original: This can be made precise by introducing ε 2 .
Revised: This can be made precise by introducing ε 2 .

Original: Finally, up to some quadrature error ε 1 , we have ⟨ g j , g ℓ ⟩ L 2 ≈ A d n − 1 y ⊤ j y ℓ , which gives us eq.
Revised: Finally, up to some quadrature error ε 1 , we have ⟨ g j , g ℓ ⟩ L 2 ≈ A d n − 1 y ⊤ j y ℓ , which gives us eq.

Original: Since (cid:80) ni =1 c i =
Revised: Since (cid:80) ni =1 c i =

Original: A d , For a fixed data distribution µ , we expect that c max =
Revised: A d , For a fixed data distribution µ , we expect that c max =

Original: O ( n − 1 ) as n → ∞ so that η does not decay as n → ∞ .
Revised: O ( n − 1 ) as n → ∞ so that η does not decay as n → ∞ .

Original: Up to a quadrature error, ∥ y − u ( k ) ∥ c is close to the L 2 norm of the residual function g − N k .
Revised: Up to a quadrature error, ∥ y − u ( k ) ∥ c is close to the L 2 norm of the residual function g − N k .

Original: Explicit formulas for the eigenvalues { µ ℓ } (see eq.
Revised: Explicit formulas for the eigenvalues { µ ℓ } (see eq.

Original: (12)) are given in (Basri et al., 2019), and it was shown that µ ℓ = O ( ℓ − d ) (Bietti & Mairal, 2019).
Revised: (12)) are given in (Basri et al., 2019), and it was shown that µ ℓ = O ( ℓ − d ) (Bietti & Mairal, 2019).

Original: Theorem 2 demonstrates the frequency bias in NN training as the rate of convergence for frequency 0 ≤ ℓ ≤ L is 1 − 2 ηµ ℓ , which is close to 1 when ℓ is large.
Revised: Theorem 2 demonstrates the frequency bias in NN training as the rate of convergence for frequency 0 ≤ ℓ ≤ L is 1 − 2 ηµ ℓ , which is close to 1 when ℓ is large.

Original: As η → 0 , we have (1 − 2 ηµ ℓ ) 2 t/η → e − 4 µ ℓ t , which is the rate of convergence for frequency using gradient flow.
Revised: As η → 0 , we have (1 − 2 ηµ ℓ ) 2 t/η → e − 4 µ ℓ t , which is the rate of convergence for frequency using gradient flow.

Original: Therefore, we expect that NN training approximates the low-frequency content of g faster than its high-frequency one, which is similar to the case of training with uniform data (Basri et al., 2019).
Revised: Therefore, we expect that NN training approximates the low-frequency content of g faster than its high-frequency one, which is similar to the case of training with uniform data (Basri et al., 2019).

Original: 4. E STIMATING THE QUADRATURE ERRORS
Revised: 4. E STIMATING THE QUADRATURE ERRORS

Original: We now quantify the quadrature errors in Theorem 2.
Revised: We now quantify the quadrature errors in Theorem 2.

Original: If we can design a quadrature rule at the training data x 1 , . .
Revised: If we can design a quadrature rule at the training data x 1 , . .

Original: , x n such that the quadrature error satisfies
Revised: , x n such that the quadrature error satisfies

Original: [Equation]
Revised: [Equation]

Original: for some constant γ n,ℓ ≥ 0 , then we can bound the terms in eq.
Revised: for some constant γ n,ℓ ≥ 0 , then we can bound the terms in eq.

Original: We expect that for each fixed ℓ , γ n,ℓ → 0 as n → ∞ as this is saying that integrals can be calculated more accurately for a large number of quadrature nodes.
Revised: We expect that for each fixed ℓ , γ n,ℓ → 0 as n → ∞ as this is saying that integrals can be calculated more accurately for a large number of quadrature nodes.

Original: In practice, it can require a large amount of training data to make γ n,ℓ small when d is large.
Revised: In practice, it can require a large amount of training data to make γ n,ℓ small when d is large.

Original: Under the reasonable assumption that our quadrature rule satisfies eq.
Revised: Under the reasonable assumption that our quadrature rule satisfies eq.

Original: (18), we can bound the quadrature errors appearing in Theorem 2.
Revised: (18), we can bound the quadrature errors appearing in Theorem 2.

Original: Theorem 3.
Revised: Theorem 3.

Original: Under the same assumptions of Theorem 2, and that the quadrature rule satisfies eq.
Revised: Under the same assumptions of Theorem 2, and that the quadrature rule satisfies eq.

Original: (18) , there exist constants C 1 , C 2 > 0 only depending on the dimension d such that the terms | ε 1 ( k ) | and | ε 2 | in Theorem 2 satisfy |
Revised: (18) , there exist constants C 1 , C 2 > 0 only depending on the dimension d such that the terms | ε 1 ( k ) | and | ε 2 | in Theorem 2 satisfy |

Original: [Equation]
Revised: [Equation]

Original: for all k ≥ 0 , ℓ ≥ 1 , where g = g 0 + · · · + g L with g j ∈ H dj and γ n,ℓ satisfies eq.
Revised: for all k ≥ 0 , ℓ ≥ 1 , where g = g 0 + · · · + g L with g j ∈ H dj and γ n,ℓ satisfies eq.

Original: The proof is in the supplementary material.
Revised: The proof is in the supplementary material.

Original: Theorem 3 states that ε 1 ( k ) and ε 2 can be made arbitrarily small if the quadrature errors converge to 0 as the number of nodes n → ∞ .
Revised: Theorem 3 states that ε 1 ( k ) and ε 2 can be made arbitrarily small if the quadrature errors converge to 0 as the number of nodes n → ∞ .

Original: In particular, if there is a sequence { ℓ n } that increases to ∞ such that the quadrature rule is exact for all functions
Revised: In particular, if there is a sequence { ℓ n } that increases to ∞ such that the quadrature rule is exact for all functions

Original: h ∈ Π dℓ n , i.e., E c ( h ) = 0 , where ℓ n → ∞ (see e.g. (Mhaskar et al., 2000)), the rates of convergence of ε 1 ( k ) and ε 2 are both O (1 /ℓ n ) for a fixed g .
Revised: h ∈ Π dℓ n , i.e., E c ( h ) = 0 , where ℓ n → ∞ (see e.g. (Mhaskar et al., 2000)), the rates of convergence of ε 1 ( k ) and ε 2 are both O (1 /ℓ n ) for a fixed g .

Original: Without the quadrature being exact, we still have nice convergence provided the quadrature errors are small, as the following corollary shows.
Revised: Without the quadrature being exact, we still have nice convergence provided the quadrature errors are small, as the following corollary shows.

Original: Corollary 1.
Revised: Corollary 1.

Original: Suppose there exists a sequence ℓ n → ∞ such that γ n,ℓ n → 0 as n → ∞ .
Revised: Suppose there exists a sequence ℓ n → ∞ such that γ n,ℓ n → 0 as n → ∞ .

Original: Then, for a fixed L ≥ 0 , we have max k ≥ 0 ,g ∈ Π dL | ε 1 ( k ) | /
Revised: Then, for a fixed L ≥ 0 , we have max k ≥ 0 ,g ∈ Π dL | ε 1 ( k ) | /

Original: ∥ g ∥ 2 L 2 and max g ∈ Π dL | ε 2 | / ∥ g ∥
Revised: ∥ g ∥ 2 L 2 and max g ∈ Π dL | ε 2 | / ∥ g ∥

Original: L 2 → 0 as n → ∞ .
Revised: L 2 → 0 as n → ∞ .

Original: Corollary 1 shows that as n increases, the quadrature errors ε 1 ( k ) and ε 2 converge to zero.
Revised: Corollary 1 shows that as n increases, the quadrature errors ε 1 ( k ) and ε 2 converge to zero.

Original: Moreover, this convergence is uniform in the sense that it does not depend on the specific choice of g ∈ Π dL .
Revised: Moreover, this convergence is uniform in the sense that it does not depend on the specific choice of g ∈ Π dL .

Original: Here, we normalize ε 1 ( k ) and ε 2 by ∥ g ∥ 2 L 2 and ∥ g ∥ L 2 , respectively, to obtain the “relative” quadrature errors that do not scale when g is multiplied by a scalar (see (17)).
Revised: Here, we normalize ε 1 ( k ) and ε 2 by ∥ g ∥ 2 L 2 and ∥ g ∥ L 2 , respectively, to obtain the “relative” quadrature errors that do not scale when g is multiplied by a scalar (see (17)).

Original:  F REQUENCY BIASING WITH A S OBOLEV - NORM LOSS FUNCTION
Revised:  F REQUENCY BIASING WITH A S OBOLEV - NORM LOSS FUNCTION

Original: The frequency biasing during the training of an overparameterized NN has several consequences.
Revised: The frequency biasing during the training of an overparameterized NN has several consequences.

Original: In some situations, worse convergence rates for high-frequency components of a function are beneficial since the NN training procedure is less sensitive to the oscillatory noise in the data, acting as a low-pass filter.
Revised: In some situations, worse convergence rates for high-frequency components of a function are beneficial since the NN training procedure is less sensitive to the oscillatory noise in the data, acting as a low-pass filter.

Original: This significantly improves the generalization error of overparameterized NNs.
Revised: This significantly improves the generalization error of overparameterized NNs.

Original: However, in other situations, NN training struggles to accurately learn the high-frequency content of g , resulting in slow convergence.
Revised: However, in other situations, NN training struggles to accurately learn the high-frequency content of g , resulting in slow convergence.

Original: To precisely control the frequency biasing of NN training, we propose to train a NN with a loss function that has intrinsic spectral bias.
Revised: To precisely control the frequency biasing of NN training, we propose to train a NN with a loss function that has intrinsic spectral bias.

Original: Let D ′
Revised: Let D ′

Original: ( S d − 1 ) be the space of distributions on S d − 1 .
Revised: ( S d − 1 ) be the space of distributions on S d − 1 .

Original: Given s ∈ R , consider L s : D ′ ( S d − 1 ) →
Revised: Given s ∈ R , consider L s : D ′ ( S d − 1 ) →

Original: D ′ ( S d − 1 ) , where L s = (cid:0) I + ( − ∆) 1 / 2 (cid:1) s and ∆ is the Laplace–Beltrami operator on the sphere.
Revised: D ′ ( S d − 1 ) , where L s = (cid:0) I + ( − ∆) 1 / 2 (cid:1) s and ∆ is the Laplace–Beltrami operator on the sphere.

Original: We follow (Barcel´o et al., 2021) and define the spherical Sobolev space H s ( S d − 1 ) =
Revised: We follow (Barcel´o et al., 2021) and define the spherical Sobolev space H s ( S d − 1 ) =

Original: { f ∈ D ′ ( S d − 1 ) :
Revised: { f ∈ D ′ ( S d − 1 ) :

Original: L s f ∈ L 2 ( S d − 1 ) } , equipped with a norm equivalent to eq.
Revised: L s f ∈ L 2 ( S d − 1 ) } , equipped with a norm equivalent to eq.

Original: (1.24) in (Barcel´o et al., 2021),
Revised: (1.24) in (Barcel´o et al., 2021),

Original: [Equation]
Revised: [Equation]

Original: where ˆ f ℓ,p are the spherical harmonic coefficients of f (see eq.
Revised: where ˆ f ℓ,p are the spherical harmonic coefficients of f (see eq.

Original: (4)) and N ( d, ℓ ) is given in Section 2.
Revised: (4)) and N ( d, ℓ ) is given in Section 2.

Original: We propose to set the loss function to be 12 ∥ g − N∥ 2 H s in replace of 12 ∥ g − N∥ 2 L 2 in eq.
Revised: We propose to set the loss function to be 12 ∥ g − N∥ 2 H s in replace of 12 ∥ g − N∥ 2 L 2 in eq.

Original: When s = 0 , it reduces to the L 2 norm.
Revised: When s = 0 , it reduces to the L 2 norm.

Original: If s > 0 , the high-frequency spherical harmonic coefficients are amplified by (1 + ℓ ) 2 s .
Revised: If s > 0 , the high-frequency spherical harmonic coefficients are amplified by (1 + ℓ ) 2 s .

Original: The high-frequency components of the residual are then penalized more in the loss function, and one can expect the NN training to learn the high-frequency components faster with the squared H s loss function than the case of eq.
Revised: The high-frequency components of the residual are then penalized more in the loss function, and one can expect the NN training to learn the high-frequency components faster with the squared H s loss function than the case of eq.

Original: Similarly, if s < 0 , the high-frequency spherical harmonic coefficients are dampened by (1+ ℓ ) 2 s and one expects the NN training captures the high-frequency components of the residual more slowly with the squared H s loss function.
Revised: Similarly, if s < 0 , the high-frequency spherical harmonic coefficients are dampened by (1+ ℓ ) 2 s and one expects the NN training captures the high-frequency components of the residual more slowly with the squared H s loss function.

Original: However, when s < 0 , the training is more robust to high-frequency noise in the data.
Revised: However, when s < 0 , the training is more robust to high-frequency noise in the data.

Original: By tuning the parameter s , we can control the frequency biasing in NN training (see Theorem 4).
Revised: By tuning the parameter s , we can control the frequency biasing in NN training (see Theorem 4).

Original: The choice of s for a particular application can be determined from theory or by cross-validation.
Revised: The choice of s for a particular application can be determined from theory or by cross-validation.

Original: First, we justify that the residual function is indeed in H s .
Revised: First, we justify that the residual function is indeed in H s .

Original: Since we assume that g is bandlimited, g ∈ H s for all s ∈ R .
Revised: Since we assume that g is bandlimited, g ∈ H s for all s ∈ R .

Original: Proposition 2 shows that we could consider s < 3 / 2 for ReLU-based NN.
Revised: Proposition 2 shows that we could consider s < 3 / 2 for ReLU-based NN.

Original: Proposition 2.
Revised: Proposition 2.

Original: Suppose N : S d − 1 → R is a 2-layer ReLU
Revised: Suppose N : S d − 1 → R is a 2-layer ReLU

Original: NN (see eq.
Revised: NN (see eq.

Original: Then, we have
Revised: Then, we have

Original: N ∈ H s ( S d − 1 ) for all s < 3 / 2 .
Revised: N ∈ H s ( S d − 1 ) for all s < 3 / 2 .

Original: Moreover, if s ≥ 3 / 2 , N ∈ H s ( S d − 1 )
Revised: Moreover, if s ≥ 3 / 2 , N ∈ H s ( S d − 1 )

Original: if and only if N is affine.
Revised: if and only if N is affine.

Original: The proof is deferred to the supplementary material.
Revised: The proof is deferred to the supplementary material.

Original: When s ≥ 3 / 2 , the residual function N − g may not be in H s .
Revised: When s ≥ 3 / 2 , the residual function N − g may not be in H s .

Original: However, we can still truncate the sum in eq.
Revised: However, we can still truncate the sum in eq.

Original: (19) to a maximum frequency ℓ max to train the NN, although the sum can no longer be interpreted as an approximation of some Sobolev norm at the continuous level.
Revised: (19) to a maximum frequency ℓ max to train the NN, although the sum can no longer be interpreted as an approximation of some Sobolev norm at the continuous level.

Original: We discretize the Sobolev-based loss function as
Revised: We discretize the Sobolev-based loss function as

Original: [Equation]
Revised: [Equation]

Original: where u and y follow eq.
Revised: where u and y follow eq.

Original: (6), and P s = (cid:80)
Revised: (6), and P s = (cid:80)

Original: [Equation]
Revised: [Equation]

Original: ) p =1 (1+ ℓ ) 2 s P ℓ,p , P ℓ,p =
Revised: ) p =1 (1+ ℓ ) 2 s P ℓ,p , P ℓ,p =

Original: a ℓ,p a ⊤ ℓ,p , and ( a ℓ,p )
Revised: a ℓ,p a ⊤ ℓ,p , and ( a ℓ,p )

Original: i =
Revised: i =

Original: c i Y ℓ,p ( x i ) .
Revised: c i Y ℓ,p ( x i ) .

Original: We assume that P s is SPD, which requires that ( ℓ max + 1) 2 ≥ n .
Revised: We assume that P s is SPD, which requires that ( ℓ max + 1) 2 ≥ n .

Original: Next, we present our convergence theorem for Sobolev training.
Revised: Next, we present our convergence theorem for Sobolev training.

Original: Theorem 4.
Revised: Theorem 4.

Original: Suppose g ∈ Π dL
Revised: Suppose g ∈ Π dL

Original: and Φ s is the loss function in eq.
Revised: and Φ s is the loss function in eq.

Original: (20) , where P s is SPD and ℓ max ≥ L .
Revised: (20) , where P s is SPD and ℓ max ≥ L .

Original: Under the assumptions of Theorem 1, if 1 − 2 ηµ ℓ (1 + ℓ ) 2 s > 0 for all 0 ≤ ℓ ≤ L , then with probability ≥ 1 − δ over the random initialization, we have
Revised: Under the assumptions of Theorem 1, if 1 − 2 ηµ ℓ (1 + ℓ ) 2 s > 0 for all 0 ≤ ℓ ≤ L , then with probability ≥ 1 − δ over the random initialization, we have

Original: [Equation]
Revised: [Equation]

Original: where y ℓ = ( g ℓ ( x 1 ) , . .
Revised: where y ℓ = ( g ℓ ( x 1 ) , . .

Original: . , g ℓ ( x n ))
Revised: . , g ℓ ( x n ))

Original: ⊤ and ε 1 satisfies
Revised: ⊤ and ε 1 satisfies

Original: [Equation]
Revised: [Equation]

Original: Compared to Theorem 2, Theorem 4 says that up to the level of quadrature errors, the convergence rate of the degree- ℓ component is 1 − 2 ηµ ℓ (1 + ℓ ) 2 s .
Revised: Compared to Theorem 2, Theorem 4 says that up to the level of quadrature errors, the convergence rate of the degree- ℓ component is 1 − 2 ηµ ℓ (1 + ℓ ) 2 s .

Original: In particular, since µ ℓ = O ( ℓ − d ) , there is an s ∗ > 0 , which depends on d , such that (1 + ℓ ) 2 s ∗ µ ℓ can be bounded from above and below by positive constant that are independent of ℓ for all ℓ ≥ 0 .
Revised: In particular, since µ ℓ = O ( ℓ − d ) , there is an s ∗ > 0 , which depends on d , such that (1 + ℓ ) 2 s ∗ µ ℓ can be bounded from above and below by positive constant that are independent of ℓ for all ℓ ≥ 0 .

Original: This means for any s > s ∗ , we expect to reverse the frequency biasing behavior of NN training.
Revised: This means for any s > s ∗ , we expect to reverse the frequency biasing behavior of NN training.

Original: Figure 2 shows the reversal of frequency biasing as s increases from − 1 to 4 (see section 6.1).
Revised: Figure 2 shows the reversal of frequency biasing as s increases from − 1 to 4 (see section 6.1).

Original: cos θ s i n θ 10101010101010 - 10epochs l o ss   6 7 8 91010frequency ℓ nu m b e r
Revised: cos θ s i n θ 10101010101010 - 10epochs l o ss   6 7 8 91010frequency ℓ nu m b e r

Original: o
Revised: o

Original: f
Revised: f

Original: it e r a ti on s
Revised: it e r a ti on s

Original: [Figure]
Revised: [Figure]

Original: training with a squared H s loss function.
Revised: training with a squared H s loss function.

Original: The blue-to-red rainbow corresponds to low-to-high frequency losses | (cid:98) N ( ℓ ) − ˆ g ( ℓ ) | for the frequency index ℓ ranging from 1 (blue) to 9 (red) with nonuniform training data, respectively.
Revised: The blue-to-red rainbow corresponds to low-to-high frequency losses | (cid:98) N ( ℓ ) − ˆ g ( ℓ ) | for the frequency index ℓ ranging from 1 (blue) to 9 (red) with nonuniform training data, respectively.

Original: Here, an overparameterized 2-layer ReLU NN N ( x ) is trained for 5000 epochs to learn function g ( x ) given the H s loss with − 1 ≤ s ≤ 4 .
Revised: Here, an overparameterized 2-layer ReLU NN N ( x ) is trained for 5000 epochs to learn function g ( x ) given the H s loss with − 1 ≤ s ≤ 4 .

Original: The inversion of the rainbow demonstrates the reversal of frequency bias.
Revised: The inversion of the rainbow demonstrates the reversal of frequency bias.

Original:  E XPERIMENTS AND DISCUSSION
Revised:  E XPERIMENTS AND DISCUSSION

Original: This section presents three experiments with synthetic and real-world datasets to investigate the frequency biasing of NN training using squared L 2 loss and squared H s loss.
Revised: This section presents three experiments with synthetic and real-world datasets to investigate the frequency biasing of NN training using squared L 2 loss and squared H s loss.

Original: The first two experiments learn functions on S 1 and S 2 , respectively.
Revised: The first two experiments learn functions on S 1 and S 2 , respectively.

Original: In the third test, we train an autoencoder on the MNIST dataset for a denoising task.
Revised: In the third test, we train an autoencoder on the MNIST dataset for a denoising task.

Original: One can find more details in the supplementary material.
Revised: One can find more details in the supplementary material.

Original: 6. L EARNING TRIGONOMETRIC POLYNOMIALS ON THE UNIT CIRCLE
Revised: 6. L EARNING TRIGONOMETRIC POLYNOMIALS ON THE UNIT CIRCLE

Original: First, we consider learning a function on S 1 .
Revised: First, we consider learning a function on S 1 .

Original: We create a set of n = 1140 nonuniform data { x i } ni =1 , as seen in Figure 1, and compute the quadrature weights { c i } ni =1 for the loss function (cid:101) Φ in eq.
Revised: We create a set of n = 1140 nonuniform data { x i } ni =1 , as seen in Figure 1, and compute the quadrature weights { c i } ni =1 for the loss function (cid:101) Φ in eq.

Original: (3) by solving a constrained quadratic program (see appendix F.1).
Revised: (3) by solving a constrained quadratic program (see appendix F.1).

Original: We train a 2-layer ReLU NN to learn g ( x ) =
Revised: We train a 2-layer ReLU NN to learn g ( x ) =

Original: ˜ g
Revised: ˜ g

Original: ( θ ) = (cid:80) 9 ℓ =1 sin( ℓθ ) , where x = (cos θ, sin θ ) .
Revised: ( θ ) = (cid:80) 9 ℓ =1 sin( ℓθ ) , where x = (cos θ, sin θ ) .

Original: We define the frequency loss | (cid:98) N ( ℓ ) − ˆ g ( ℓ ) | where (cid:98) N and ˆ g are the Fourier coefficients of N and g , respectively (see appendix F.1).
Revised: We define the frequency loss | (cid:98) N ( ℓ ) − ˆ g ( ℓ ) | where (cid:98) N and ˆ g are the Fourier coefficients of N and g , respectively (see appendix F.1).

Original: In Figure 1, we plot the frequency loss for ℓ = 1 , 5 , 9 in different colors to illustrate how well the NN fits each frequency component.
Revised: In Figure 1, we plot the frequency loss for ℓ = 1 , 5 , 9 in different colors to illustrate how well the NN fits each frequency component.

Original: The solid and dash lines correspond to the loss function Φ in eq.
Revised: The solid and dash lines correspond to the loss function Φ in eq.

Original: (1) and (cid:101) Φ in eq.
Revised: (1) and (cid:101) Φ in eq.

Original: (3), respectively.
Revised: (3), respectively.

Original: Our observations collaborate the theoretical statements in Theorem 2.
Revised: Our observations collaborate the theoretical statements in Theorem 2.

Original: Figure 1 also shows that it takes asymptotically O ( ℓ 2 ) iterations to learn the ℓ th frequency sin( ℓθ ) given the loss function (cid:101) Φ .
Revised: Figure 1 also shows that it takes asymptotically O ( ℓ 2 ) iterations to learn the ℓ th frequency sin( ℓθ ) given the loss function (cid:101) Φ .

Original: A similar plot appears in Basri et al.
Revised: A similar plot appears in Basri et al.

Original: (2019) for uniform training data.
Revised: (2019) for uniform training data.

Original: We also use the squared H s norm as the loss function to learn g .
Revised: We also use the squared H s norm as the loss function to learn g .

Original: After 5000 epochs, we plot the ℓ th frequency loss with ℓ ranging from 1 (blue) to 9 (red) in Figure 2, given different s values.
Revised: After 5000 epochs, we plot the ℓ th frequency loss with ℓ ranging from 1 (blue) to 9 (red) in Figure 2, given different s values.

Original: As s increases, the higher-frequency components are learned faster.
Revised: As s increases, the higher-frequency components are learned faster.

Original: When s > 2 , the frequency biasing is reversed in the sense that higher-frequency parts are learned faster than the lower-frequency ones rather than a low-frequency biasing under the squared L 2 loss (see Theorem 2).
Revised: When s > 2 , the frequency biasing is reversed in the sense that higher-frequency parts are learned faster than the lower-frequency ones rather than a low-frequency biasing under the squared L 2 loss (see Theorem 2).

Original: The gradually changing “rainbow” in Figure 2 shows that the smoothing property of an overparameterized NN can be compensated by the H s loss function for large enough s , corroborating Theorem 4.
Revised: The gradually changing “rainbow” in Figure 2 shows that the smoothing property of an overparameterized NN can be compensated by the H s loss function for large enough s , corroborating Theorem 4.

Original: epochs l o ss      6 10 50.
Revised: epochs l o ss      6 10 50.

Original: epochs l o ss      6 10 5 0.
Revised: epochs l o ss      6 10 5 0.

Original: epochs l o ss
Revised: epochs l o ss

Original: [Figure]
Revised: [Figure]

Original: 6. L EARNING SPHERICAL HARMONICS ON THE UNIT SPHERE
Revised: 6. L EARNING SPHERICAL HARMONICS ON THE UNIT SPHERE

Original: Similar to the previous example in S 1 , we design an experiment on S 2 .
Revised: Similar to the previous example in S 1 , we design an experiment on S 2 .

Original: We utilize a data set { x i } 2500 i =1 in (Wright & Michaels, 2015), which comes with carefully designed positive quadrature weights { c i } 2500 i =1 .
Revised: We utilize a data set { x i } 2500 i =1 in (Wright & Michaels, 2015), which comes with carefully designed positive quadrature weights { c i } 2500 i =1 .

Original: We test the squared H s loss function in NN training with a target function g ( x ) = (cid:80) 30 ℓ =0 ,ℓ even Y ℓ, 0 defined on S 2 that involves many high-frequency components.
Revised: We test the squared H s loss function in NN training with a target function g ( x ) = (cid:80) 30 ℓ =0 ,ℓ even Y ℓ, 0 defined on S 2 that involves many high-frequency components.

Original: The results are shown in Figure 3 with different s values.
Revised: The results are shown in Figure 3 with different s values.

Original: The natural low-frequency bias of NN in the case of
Revised: The natural low-frequency bias of NN in the case of

Original: L 2 -based training (i.e., s = 0 ) is enhanced when s = − 1 , and is totally reversed when s = 2 . 5 .
Revised: L 2 -based training (i.e., s = 0 ) is enhanced when s = − 1 , and is totally reversed when s = 2 . 5 .

Original: 6. A UTOENCODER ON THE MNIST DATASET
Revised: 6. A UTOENCODER ON THE MNIST DATASET

Original: The idea of Sobolev training is also useful for high-dimensional training data.
Revised: The idea of Sobolev training is also useful for high-dimensional training data.

Original: In Figure 4, we present the results of the autoencoder for image denoising using the MNIST dataset (LeCun et al., 2010).
Revised: In Figure 4, we present the results of the autoencoder for image denoising using the MNIST dataset (LeCun et al., 2010).

Original: The outputs of the autoencoder are presented when trained with the squared H s norm as the loss function.
Revised: The outputs of the autoencoder are presented when trained with the squared H s norm as the loss function.

Original: We contaminate the dataset with random low-frequency noise (top row) and highfrequency noise (bottom row).
Revised: We contaminate the dataset with random low-frequency noise (top row) and highfrequency noise (bottom row).

Original: When high-frequency noise is present, the H s loss function generally performs better with s < 0 , while the case of s > 0 helps image deblurring when the input image suffers from low-frequency noise.
Revised: When high-frequency noise is present, the H s loss function generally performs better with s < 0 , while the case of s > 0 helps image deblurring when the input image suffers from low-frequency noise.

Original: This corroborates our discussion in Section 5.
Revised: This corroborates our discussion in Section 5.

Original: In appendix G, we theoretically justify this phenomenon by studying the frequency bias in operator learning.
Revised: In appendix G, we theoretically justify this phenomenon by studying the frequency bias in operator learning.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: A frequency bias phenomenon is observed in NN training with nonuniform training data.
Revised: A frequency bias phenomenon is observed in NN training with nonuniform training data.

Original: Instead of the standard mean-squared loss function in eq.
Revised: Instead of the standard mean-squared loss function in eq.

Original: (1), we propose the use of a different loss function in eq.
Revised: (1), we propose the use of a different loss function in eq.

Original: (3), which involves quadrature weights and has a natural continuous analog.
Revised: (3), which involves quadrature weights and has a natural continuous analog.

Original: With eq.
Revised: With eq.

Original: (3), we rigorously analyze frequency bias with nonuniform training data using the Funk–Hecke formula.
Revised: (3), we rigorously analyze frequency bias with nonuniform training data using the Funk–Hecke formula.

Original: By changing the loss function to a squared Sobolev-based one, we can control the frequency biasing in
Revised: By changing the loss function to a squared Sobolev-based one, we can control the frequency biasing in

Original: NN training, which can accelerate NN training convergence and improve robustness under noises.
Revised: NN training, which can accelerate NN training convergence and improve robustness under noises.

Original: S UPPLEMENTARY M ATERIAL
Revised: S UPPLEMENTARY M ATERIAL

Original: This is the supplementary material for the paper titled “Tuning Frequency Bias in Neural Network
Revised: This is the supplementary material for the paper titled “Tuning Frequency Bias in Neural Network

Original: Training with Nonuniform Data.”
Revised: Training with Nonuniform Data.”

Original: The supplementary material is organized as follows.
Revised: The supplementary material is organized as follows.

Original: In Appendix A, we recall the notations used in the paper and introduce additional concepts for our analysis.
Revised: In Appendix A, we recall the notations used in the paper and introduce additional concepts for our analysis.

Original: In Appendix B, we prove that H ∞ is a symmetric and positive definite matrix.
Revised: In Appendix B, we prove that H ∞ is a symmetric and positive definite matrix.

Original: In Appendix C, we prove Theorem 1 of the paper, while the proofs of results in Section 4 and Section 5 are given in Appendix D and Appendix E, respectively.
Revised: In Appendix C, we prove Theorem 1 of the paper, while the proofs of results in Section 4 and Section 5 are given in Appendix D and Appendix E, respectively.

Original: In Appendix F, we provide further details of our three experiments.
Revised: In Appendix F, we provide further details of our three experiments.

Original: The observation made in the third experiment is then briefly justified in Appendix G, in which we study frequency bias in a general operator learning setting.
Revised: The observation made in the third experiment is then briefly justified in Appendix G, in which we study frequency bias in a general operator learning setting.

Original: In Appendix H, we formally discuss the computation of positive quadrature weights.
Revised: In Appendix H, we formally discuss the computation of positive quadrature weights.

Original: A P RELIMINARIES AND NOTATION
Revised: A P RELIMINARIES AND NOTATION

Original: For d > 1 , let g : S d − 1 → R be a square-integrable function defined on S d − 1 .
Revised: For d > 1 , let g : S d − 1 → R be a square-integrable function defined on S d − 1 .

Original: The function g has a spherical harmonic expansion given in Section 2.
Revised: The function g has a spherical harmonic expansion given in Section 2.

Original: We denote the space of harmonic functions of degree ℓ as H dℓ , which is the span of { Y ℓ,p } N ( d,ℓ ) p =1 .
Revised: We denote the space of harmonic functions of degree ℓ as H dℓ , which is the span of { Y ℓ,p } N ( d,ℓ ) p =1 .

Original: We further denote the space of spherical harmonics of degree ≤ ℓ as Π dℓ =
Revised: We further denote the space of spherical harmonics of degree ≤ ℓ as Π dℓ =

Original: (cid:76) ℓj =

Revised: (cid:76) ℓj =

Original: H dj .
Revised: H dj .

Original: Given distinct training data { x i } ni =1 from S d − 1 and evaluations y i
Revised: Given distinct training data { x i } ni =1 from S d − 1 and evaluations y i

Original: = g ( x i ) for 1 ≤ i ≤ n , our goal is to understand the intrinsic frequency-biasing behavior of training a 2-layer ReLU NN given in eq.
Revised: = g ( x i ) for 1 ≤ i ≤ n , our goal is to understand the intrinsic frequency-biasing behavior of training a 2-layer ReLU NN given in eq.

Original: It is important for the theory that we initialize the weights as independently and identically distributed (iid) Gaussian random variables with a covariance matrix κ 2 I , the bias terms are initialized to zero, and the coefficients, i.e., a 1 , . .
Revised: It is important for the theory that we initialize the weights as independently and identically distributed (iid) Gaussian random variables with a covariance matrix κ 2 I , the bias terms are initialized to zero, and the coefficients, i.e., a 1 , . .

Original: . , a m , are initialized iid as +1 with probability 1 / 2 and − 1 otherwise.
Revised: . , a m , are initialized iid as +1 with probability 1 / 2 and − 1 otherwise.

Original: During the training process, the values of { a r } are not updated.
Revised: During the training process, the values of { a r } are not updated.

Original: We train with the loss function given in eq.
Revised: We train with the loss function given in eq.

Original: (6) so that the gradient descent algorithm for NN training is given by eq.
Revised: (6) so that the gradient descent algorithm for NN training is given by eq.

Original: An important object in understanding the frequency biasing of NN training is the symmetric and positive definite matrix H ∞ ∈ R n × n in eq.
Revised: An important object in understanding the frequency biasing of NN training is the symmetric and positive definite matrix H ∞ ∈ R n × n in eq.

Original: Since H ∞ and P are symmetric positive definite matrices (see Proposition 1), H ∞ P has positive real eigenvalues.
Revised: Since H ∞ and P are symmetric positive definite matrices (see Proposition 1), H ∞ P has positive real eigenvalues.

Original: To see this, note that H ∞ P = H ∞
Revised: To see this, note that H ∞ P = H ∞

Original: P 1 / 2 P 1 / 2 =
Revised: P 1 / 2 P 1 / 2 =

Original: P − 1 / 2 ( P 1 / 2 H ∞ P 1 / 2 )
Revised: P − 1 / 2 ( P 1 / 2 H ∞ P 1 / 2 )

Original: P 1 / 2 .
Revised: P 1 / 2 .

Original: This means that H ∞ P and
Revised: This means that H ∞ P and

Original: P 1 / 2 H ∞ P 1 / 2 are similar.
Revised: P 1 / 2 H ∞ P 1 / 2 are similar.

Original: Since the matrix P 1 / 2 H ∞ P 1 / 2 is symmetric positive definite, H ∞ P has positive eigenvalues.
Revised: Since the matrix P 1 / 2 H ∞ P 1 / 2 is symmetric positive definite, H ∞ P has positive eigenvalues.

Original: We denote the eigenvalues of H ∞ P by λ n − 1 ≥ · · · ≥ λ 0 > 0 , which partially govern the frequency biasing phenomena.
Revised: We denote the eigenvalues of H ∞ P by λ n − 1 ≥ · · · ≥ λ 0 > 0 , which partially govern the frequency biasing phenomena.

Original: It is worth mentioning that although H ∞ P is not symmetric, it is self-adjoint in the inner product space induced by P because
Revised: It is worth mentioning that although H ∞ P is not symmetric, it is self-adjoint in the inner product space induced by P because

Original: [Equation]
Revised: [Equation]

Original: It is convenient to analyze the eigenvalues and eigenvectors of H ∞ P via the zonal kernel K ∞ : S d − 1 × S d − 1 → R given in eq.
Revised: It is convenient to analyze the eigenvalues and eigenvectors of H ∞ P via the zonal kernel K ∞ : S d − 1 × S d − 1 → R given in eq.

Original: The key is the Funk–Hecke formula (Seeley, 1966).
Revised: The key is the Funk–Hecke formula (Seeley, 1966).

Original: Theorem 5 (Funk–Hecke) .
Revised: Theorem 5 (Funk–Hecke) .

Original: Suppose K : [ − 1 , 1] → R is measurable and K ( t )(1 − t 2 ) ( d − 3) / 2 is integrable on [ − 1 , 1] .
Revised: Suppose K : [ − 1 , 1] → R is measurable and K ( t )(1 − t 2 ) ( d − 3) / 2 is integrable on [ − 1 , 1] .

Original: Then, for any h ∈ H dℓ , we have ˆ
Revised: Then, for any h ∈ H dℓ , we have ˆ

Original: [Equation]
Revised: [Equation]

Original: where P ℓ,d is the ultraspherical polynomial given by
Revised: where P ℓ,d is the ultraspherical polynomial given by

Original: [Equation]
Revised: [Equation]

Original: Applying Theorem
Revised: Applying Theorem

Original: [Equation]
Revised: [Equation]

Original: we have that ˆ
Revised: we have that ˆ

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: µ
Revised: µ

Original: [Table]
Revised: [Table]

Original: ℓ
Revised: ℓ

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original:  
Revised:  

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: Here, the exclamation mark means a factorial and (cid:0) ·· (cid:1) denotes the binomial coefficient.
Revised: Here, the exclamation mark means a factorial and (cid:0) ·· (cid:1) denotes the binomial coefficient.

Original: Given x , w r , and b r in eq.
Revised: Given x , w r , and b r in eq.

Original: (5), we write ˜ x
Revised: (5), we write ˜ x

Original: = 1 √ 2 ( x , 1) ∈ S d and ˜
Revised: = 1 √ 2 ( x , 1) ∈ S d and ˜

Original: w r = ( w r , b r ) ∈ R d +1 .
Revised: w r = ( w r , b r ) ∈ R d +1 .

Original: Therefore, we have ReLU( w ⊤ r x + b r ) = √ 2ReLU( ˜ w ⊤ r ˜ x ) and the NN function can be rewritten as
Revised: Therefore, we have ReLU( w ⊤ r x + b r ) = √ 2ReLU( ˜ w ⊤ r ˜ x ) and the NN function can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: By replacing the expectation over random initialization of ˜ w by ˜ w ( t ) , we define the instantiations of H ∞ at the k th iteration by H ( k ) , where
Revised: By replacing the expectation over random initialization of ˜ w by ˜ w ( t ) , we define the instantiations of H ∞ at the k th iteration by H ( k ) , where

Original: [Equation]
Revised: [Equation]

Original: ) where 1 is an indicator function.
Revised: ) where 1 is an indicator function.

Original: B T HE MATRIX H ∞ IS SYMMETRIC AND POSITIVE DEFINITE
Revised: B T HE MATRIX H ∞ IS SYMMETRIC AND POSITIVE DEFINITE

Original: Proposition 1 states that the matrix H ∞ defined by eq.
Revised: Proposition 1 states that the matrix H ∞ defined by eq.

Original: (9) is symmetric and positive definite.
Revised: (9) is symmetric and positive definite.

Original: While the symmetry of H ∞ is immediate from its closed-form expression, the fact that it is positive definite requires a more detailed analysis.
Revised: While the symmetry of H ∞ is immediate from its closed-form expression, the fact that it is positive definite requires a more detailed analysis.

Original: The proof idea is similar to that of Theorem 3.1 of (Du et al., 2018), in which the matrix H ∞ is associated with a 2-layer ReLU NN without biases.
Revised: The proof idea is similar to that of Theorem 3.1 of (Du et al., 2018), in which the matrix H ∞ is associated with a 2-layer ReLU NN without biases.

Original: However, our H ∞ is associated with a 2-layer ReLU NN with biases.
Revised: However, our H ∞ is associated with a 2-layer ReLU NN with biases.

Original: While (Du et al., 2018) requires no two training data points are parallel, we allow the existence of x i 1 = − x i 2 for some i 

Revised: While (Du et al., 2018) requires no two training data points are parallel, we allow the existence of x i 1 = − x i 2 for some i

Original: and i 2 .
Revised: and i 2 .

Original: Hence, our proof employs on a pair of nodes denoted by x i 1 , x i 2 .
Revised: Hence, our proof employs on a pair of nodes denoted by x i 1 , x i 2 .

Original: Proof of Proposition 1.
Revised: Proof of Proposition 1.

Original: For a measurable function f : R d → R d +1 , we define a norm of f as
Revised: For a measurable function f : R d → R d +1 , we define a norm of f as

Original: [Equation]
Revised: [Equation]

Original: and let H be the space of measurable functions such that ∥ f ∥ H < ∞ .
Revised: and let H be the space of measurable functions such that ∥ f ∥ H < ∞ .

Original: It can be shown that H is a
Revised: It can be shown that H is a

Original: Hilbert space with respect to the inner product ⟨ f, g ⟩ H =
Revised: Hilbert space with respect to the inner product ⟨ f, g ⟩ H =

Original: E w ∼N ( 0 ,κ 2 I ) (cid:2)
Revised: E w ∼N ( 0 ,κ 2 I ) (cid:2)

Original: f ( w ) ⊤ g ( w ) (cid:3) .
Revised: f ( w ) ⊤ g ( w ) (cid:3) .

Original: For each x i , 1 ≤ i ≤ n , we define the function ϕ
Revised: For each x i , 1 ≤ i ≤ n , we define the function ϕ

Original: i by
Revised: i by

Original: [Equation]
Revised: [Equation]

Original: Then, ϕ i ∈ H for all i , and H ∞ ij = ⟨ ϕ i , ϕ j ⟩ H .
Revised: Then, ϕ i ∈ H for all i , and H ∞ ij = ⟨ ϕ i , ϕ j ⟩ H .

Original: We prove that H ∞ is positive definite by showing that { ϕ i } ni =1 is a linearly independent set in H .
Revised: We prove that H ∞ is positive definite by showing that { ϕ i } ni =1 is a linearly independent set in H .

Original: To show { ϕ i } ni =1 is a linearly independent set, we show that
Revised: To show { ϕ i } ni =1 is a linearly independent set, we show that

Original: [Equation]
Revised: [Equation]

Original: almost every w
Revised: almost every w

Original: [Equation]
Revised: [Equation]

Original: 26) implies that α i = 0 for 1 ≤ i ≤ n .
Revised: 26) implies that α i = 0 for 1 ≤ i ≤ n .

Original: We fix some 1 ≤ i 1 ≤ n and, without loss of generality, assume that x i 2 = − x i 1 .
Revised: We fix some 1 ≤ i 1 ≤ n and, without loss of generality, assume that x i 2 = − x i 1 .

Original: 2 Define the set D j = { w ∈ R d | w ⊤ x j = 0 } for 1 ≤ j ≤ n .
Revised: 2 Define the set D j = { w ∈ R d | w ⊤ x j = 0 } for 1 ≤ j ≤ n .

Original: As a result, D i 1 = D i 2 .
Revised: As a result, D i 1 = D i 2 .

Original: Since each D j is a hyperplane passing through the origin and D i 1 ̸ = D j for any j ̸
Revised: Since each D j is a hyperplane passing through the origin and D i 1 ̸ = D j for any j ̸

Original: = i 1 , i 2 , ∃ z ∈ D i 1 such that z / ∈ D j for any j ̸ = i 1 , i 2 .
Revised: = i 1 , i 2 , ∃ z ∈ D i 1 such that z / ∈ D j for any j ̸ = i 1 , i 2 .

Original: For a positive radius R > 0 , let
Revised: For a positive radius R > 0 , let

Original: B R = B ( z , R ) be the ball centered at z of radius R .
Revised: B R = B ( z , R ) be the ball centered at z of radius R .

Original: Define a partition of B R into two sets denoted by B + R and B − R (possibly missing a subset of B R that has zero Lebesgue measure), where
Revised: Define a partition of B R into two sets denoted by B + R and B − R (possibly missing a subset of B R that has zero Lebesgue measure), where

Original: [Equation]
Revised: [Equation]

Original: Since D j is closed for each 1 ≤ j ≤ n , B R is eventually disjoint from D j as R → 0 .
Revised: Since D j is closed for each 1 ≤ j ≤ n , B R is eventually disjoint from D j as R → 0 .

Original: Hence, we have
Revised: Hence, we have

Original: [Equation]
Revised: [Equation]

Original: where |·| denotes the Euclidean distance.
Revised: where |·| denotes the Euclidean distance.

Original: Then, for any j ̸ = i 1 , i 2 , we have
Revised: Then, for any j ̸ = i 1 , i 2 , we have

Original: [Equation]
Revised: [Equation]

Original: Consequently, we find that
Revised: Consequently, we find that

Original: [Equation]
Revised: [Equation]

Original: Now, consider the integral of ϕ i 1 and ϕ i 2 .
Revised: Now, consider the integral of ϕ i 1 and ϕ i 2 .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: lim
Revised: lim

Original: [Equation]
Revised: [Equation]

Original: By applying these limiting expressions to (cid:80) nj
Revised: By applying these limiting expressions to (cid:80) nj

Original: =
Revised: =

Original: α j ϕ j ( w ) = 0 , we find that α i 1 ˜ x i 1 − α i 2 ˜ x i 2 = 0 .
Revised: α j ϕ j ( w ) = 0 , we find that α i 1 ˜ x i 1 − α i 2 ˜ x i 2 = 0 .

Original: Since the last entries of both ˜ x i 1 and ˜ x i 2 are 1 / √ 2 , we have α i 

Revised: Since the last entries of both ˜ x i 1 and ˜ x i 2 are 1 / √ 2 , we have α i

Original: = α
Revised: = α

Original: i 2 .
Revised: i 2 .

Original: Thus, we have α i 1 =because x i 1 ̸ = 0 .
Revised: Thus, we have α i 1 =because x i 1 ̸ = 0 .

Original: Since i 1 is arbitrary, the statement of the proposition follows.
Revised: Since i 1 is arbitrary, the statement of the proposition follows.

Original: It is clear that the proof of Proposition 1 is also true if we assume that each entry of w r is initialized from an iid sub-Gaussian distribution with zero mean and whose support is the entire R , and we update the definition of H ∞ according to eq.
Revised: It is clear that the proof of Proposition 1 is also true if we assume that each entry of w r is initialized from an iid sub-Gaussian distribution with zero mean and whose support is the entire R , and we update the definition of H ∞ according to eq.

Original: C T HE CONVERGENCE OF NEURAL NETWORK TRAINING
Revised: C T HE CONVERGENCE OF NEURAL NETWORK TRAINING

Original: In this section, we develop the theory for learning a NN with a general loss function Φ P defined by a positive definite matrix P in eq.
Revised: In this section, we develop the theory for learning a NN with a general loss function Φ P defined by a positive definite matrix P in eq.

Original: In particular, we prove Theorem 1, which states that provided the learning rate is sufficiently small, the weights are initialized without too much variance, and the
Revised: In particular, we prove Theorem 1, which states that provided the learning rate is sufficiently small, the weights are initialized without too much variance, and the

Original: NN is sufficiently wide, then the residual in the first few epochs can be described with the matrix
Revised: NN is sufficiently wide, then the residual in the first few epochs can be described with the matrix

Original: H ∞ P .
Revised: H ∞ P .

Original: While our proof is similar to that of (Su & Yang, 2019), the argument is distinct in three essential ways: (1) Our proof applies to any loss function defined by a positive definite matrix P , which requires us to use a different Hilbert space ( R n , ⟨· , ·⟩ P ) .
Revised: While our proof is similar to that of (Su & Yang, 2019), the argument is distinct in three essential ways: (1) Our proof applies to any loss function defined by a positive definite matrix P , which requires us to use a different Hilbert space ( R n , ⟨· , ·⟩ P ) .

Original: (2) While the result in (Su & Yang, 2019) bounds the residual using the minimum eigenvalue of H ∞ , we estimate the residual as a matrixvector product of ( I − 2 η H ∞ P ) k y , which allows us to analyze the training error using all eigenvalues of H ∞ P .
Revised: (2) While the result in (Su & Yang, 2019) bounds the residual using the minimum eigenvalue of H ∞ , we estimate the residual as a matrixvector product of ( I − 2 η H ∞ P ) k y , which allows us to analyze the training error using all eigenvalues of H ∞ P .

Original: (3) We use a different NN function that incorporates the bias terms and we do not assume that we initialize the weights in a way that makes N 0 = 0 .
Revised: (3) We use a different NN function that incorporates the bias terms and we do not assume that we initialize the weights in a way that makes N 0 = 0 .

Original: Before we prove the theorem, we define some useful quantities.
Revised: Before we prove the theorem, we define some useful quantities.

Original: Let A be the set of indices such that the coefficients a r are initialized to 1 and let B be the set initialized to − 1 .
Revised: Let A be the set of indices such that the coefficients a r are initialized to 1 and let B be the set initialized to − 1 .

Original: We then decompose
Revised: We then decompose

Original: H ( k ) into two parts, where H ( k ) is defined in eq.
Revised: H ( k ) into two parts, where H ( k ) is defined in eq.

Original: (25), so that H ( k ) =
Revised: (25), so that H ( k ) =

Original: H + ( k ) + H − ( k ) with
Revised: H + ( k ) + H − ( k ) with

Original: [Equation]
Revised: [Equation]

Original: Similarly, we define two other matrices
Revised: Similarly, we define two other matrices

Original: ˜ H + ( k ) and ˜ H − ( k ) as
Revised: ˜ H + ( k ) and ˜ H − ( k ) as

Original: [Equation]
Revised: [Equation]

Original: Unfortunately, ˜ H + ( k ) and ˜ H − ( k ) are not necessarily symmetric and they differ from H + ( k ) and
Revised: Unfortunately, ˜ H + ( k ) and ˜ H − ( k ) are not necessarily symmetric and they differ from H + ( k ) and

Original: H − ( k ) up to sign flips.
Revised: H − ( k ) up to sign flips.

Original: To simplify the notation later, we also define two auxiliary matrices L ( k ) and M ( k ) as
Revised: To simplify the notation later, we also define two auxiliary matrices L ( k ) and M ( k ) as

Original: [Equation]
Revised: [Equation]

Original: We now prove that I − 2 η H ( k ) P is close to the transition matrix for the residual, up to sign flips, i.e., y − u ( k + 1)
Revised: We now prove that I − 2 η H ( k ) P is close to the transition matrix for the residual, up to sign flips, i.e., y − u ( k + 1)

Original: ≈ ( I − 2 η H ( k ) P )( y − u ( k )) .
Revised: ≈ ( I − 2 η H ( k ) P )( y − u ( k )) .

Original: Lemma 1.
Revised: Lemma 1.

Original: Let z ( k ) =
Revised: Let z ( k ) =

Original: y − u ( k ) be the residual after the k th iteration.
Revised: y − u ( k ) be the residual after the k th iteration.

Original: For any k ≥ 0 and η > 0 , we have
Revised: For any k ≥ 0 and η > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: where the inequalities are entry-wise.
Revised: where the inequalities are entry-wise.

Original: Proof.
Revised: Proof.

Original: First, by the gradient descent update rule, we have
Revised: First, by the gradient descent update rule, we have

Original: [Equation]
Revised: [Equation]

Original: where the ( d + 1) × n Jacobian matrix is given by
Revised: where the ( d + 1) × n Jacobian matrix is given by

Original: [Equation]
Revised: [Equation]

Original: and the gradient of the loss function Φ P defined in eq.
Revised: and the gradient of the loss function Φ P defined in eq.

Original: (6) with respect to u is given by
Revised: (6) with respect to u is given by

Original: [Equation]
Revised: [Equation]

Original: which is a vector of length n .
Revised: which is a vector of length n .

Original: Hence, it follows that
Revised: Hence, it follows that

Original: [Equation]
Revised: [Equation]

Original: where ( Pz ( k ))
Revised: where ( Pz ( k ))

Original: p denotes the p th element of Pz ( k ) .
Revised: p denotes the p th element of Pz ( k ) .

Original: Using the property of ReLU that
Revised: Using the property of ReLU that

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: ReLU
Revised: ReLU

Original: [Equation]
Revised: [Equation]

Original: Hence, we have
Revised: Hence, we have

Original: [Equation]
Revised: [Equation]

Original: This proves the first inequality.
Revised: This proves the first inequality.

Original: The second inequality can be shown with a similar argument.
Revised: The second inequality can be shown with a similar argument.

Original: In particular, if there is no sign flip of the weights, then ˜ H ( k ) =
Revised: In particular, if there is no sign flip of the weights, then ˜ H ( k ) =

Original: ˜ H + ( k ) + ˜ H − ( k ) =
Revised: ˜ H + ( k ) + ˜ H − ( k ) =

Original: H ( k ) and the inequalities in Lemma 1 are equalities.
Revised: H ( k ) and the inequalities in Lemma 1 are equalities.

Original: Next, using Lemma 1, we can derive an expression for y − u ( k ) using H ∞ , up to an error term.
Revised: Next, using Lemma 1, we can derive an expression for y − u ( k ) using H ∞ , up to an error term.

Original: Lemma 2.
Revised: Lemma 2.

Original: For any 0 < η < 1 / (2 M 2 P n ) and any k ≥ 0 , we have that
Revised: For any 0 < η < 1 / (2 M 2 P n ) and any k ≥ 0 , we have that

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For k ≥
Revised: For k ≥

Original: [Equation]
Revised: [Equation]

Original: we define r ( k ) by
Revised: we define r ( k ) by

Original: [Equation]
Revised: [Equation]

Original: Then, by Lemma 1 we have
Revised: Then, by Lemma 1 we have

Original: [Equation]
Revised: [Equation]

Original: Note that eq.
Revised: Note that eq.

Original: (31) is a first-order non-homogeneous recurrence relation for y − u ( k ) , which has an analytic solution.
Revised: (31) is a first-order non-homogeneous recurrence relation for y − u ( k ) , which has an analytic solution.

Original: Thus, we can expand y − u ( k ) for k ≥ 1 as
Revised: Thus, we can expand y − u ( k ) for k ≥ 1 as

Original: [Equation]
Revised: [Equation]

Original: Moreover
Revised: Moreover

Original: [Equation]
Revised: [Equation]

Original: we can write the product of of the matrices as (Su & Yang, 2019)
Revised: we can write the product of of the matrices as (Su & Yang, 2019)

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Combining eq.
Revised: Combining eq.

Original: (33) and eq.
Revised: (33) and eq.

Original: (34), we obtain eq.
Revised: (34), we obtain eq.

Original: (29) where
Revised: (29) where

Original: [Equation]
Revised: [Equation]

Original: Finally, we note that
Revised: Finally, we note that

Original: [Equation]
Revised: [Equation]

Original: and that
Revised: and that

Original: [Equation]
Revised: [Equation]

Original: We can then bound ∥ H ( t ) P ∥ P using M P defined in eq.
Revised: We can then bound ∥ H ( t ) P ∥ P using M P defined in eq.

Original: (8) and ∥ H ( t ) ∥ 

Revised: (8) and ∥ H ( t ) ∥

Original: as
Revised: as

Original: [Equation]
Revised: [Equation]

Original: By requiring that η < 1 / (2 M 2 P n ) , we have λ min ( I − 2 η H ( t ) P ) > 0 for all t .
Revised: By requiring that η < 1 / (2 M 2 P n ) , we have λ min ( I − 2 η H ( t ) P ) > 0 for all t .

Original: Hence, I − 2 η H ( t ) P is positive definite in ( R n , ⟨· , ·⟩ P ) , and according to eq.
Revised: Hence, I − 2 η H ( t ) P is positive definite in ( R n , ⟨· , ·⟩ P ) , and according to eq.

Original: (35), we have ∥ I − 2 η H ( t ) P ∥
Revised: (35), we have ∥ I − 2 η H ( t ) P ∥

Original: P = λ max ( I − 2 η H ( t ) P ) < 1 .
Revised: P = λ max ( I − 2 η H ( t ) P ) < 1 .

Original: The upper bound in eq.
Revised: The upper bound in eq.

Original: (30) follows from the triangle inequality and our estimate on r k in eq.
Revised: (30) follows from the triangle inequality and our estimate on r k in eq.

Original: The residual terms in Lemma 2 can be made small by controlling ∥ M ( t ) P
Revised: The residual terms in Lemma 2 can be made small by controlling ∥ M ( t ) P

Original: ∥ P + ∥ L ( t ) P ∥ P and ∥ ( H (0) − H ( t ))
Revised: ∥ P + ∥ L ( t ) P ∥ P and ∥ ( H (0) − H ( t ))

Original: P ∥ P .
Revised: P ∥ P .

Original: Their upper bounds are given in Lemma 3.
Revised: Their upper bounds are given in Lemma 3.

Original: First, we define
Revised: First, we define

Original: [Equation]
Revised: [Equation]

Original: for some
Revised: for some

Original: [Equation]
Revised: [Equation]

Original: ≤ t
Revised: ≤ t

Original: [Equation]
Revised: [Equation]

Original: ≤ t (cid:9) to be the set of indices of the weights that have changed sign at least once by the k th iteration.
Revised: ≤ t (cid:9) to be the set of indices of the weights that have changed sign at least once by the k th iteration.

Original: Lemma 3.
Revised: Lemma 3.

Original: For all t ≥ 0 , we have
Revised: For all t ≥ 0 , we have

Original: [Equation]
Revised: [Equation]

Original: i ( t ) | 2 .
Revised: i ( t ) | 2 .

Original: Moreover, for any 0 < δ < 1 , with probability at least 1 − δ , we have
Revised: Moreover, for any 0 < δ < 1 , with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First
Revised: First

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: The estimate for ∥ L ( t ) P ∥ 2 P is exactly the same and obtained by replacing A with B .
Revised: The estimate for ∥ L ( t ) P ∥ 2 P is exactly the same and obtained by replacing A with B .

Original: We also have
Revised: We also have

Original: [Equation]
Revised: [Equation]

Original: This proves the first inequality.
Revised: This proves the first inequality.

Original: Since H ij (0) is the average of m iid random variables bounded in [0 , 1] , by Hoeffding’s inequality (Hoeffding, 1963), for any t > 0 and any 1 ≤
Revised: Since H ij (0) is the average of m iid random variables bounded in [0 , 1] , by Hoeffding’s inequality (Hoeffding, 1963), for any t > 0 and any 1 ≤

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: j ≤ n , we have
Revised: j ≤ n , we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Set t = (cid:112)
Revised: Set t = (cid:112)

Original: [Equation]
Revised: [Equation]

Original: δ ) .
Revised: δ ) .

Original: With probability at least 1 − δ/n 2 , we have
Revised: With probability at least 1 − δ/n 2 , we have

Original: [Equation]
Revised: [Equation]

Original: Hence, by a union bound, we know that with probability at least 1 − δ , we have
Revised: Hence, by a union bound, we know that with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: The last estimate follows from the definitions of M P (see eq.
Revised: The last estimate follows from the definitions of M P (see eq.

Original: (8) and eq.
Revised: (8) and eq.

Original: Now, we state and prove our initial control of the decay of the residual.
Revised: Now, we state and prove our initial control of the decay of the residual.

Original: Lemma 4.
Revised: Lemma 4.

Original: Let ϵ > 0 , κ > 0 , 0 < δ < 1 and T > 0 be given.
Revised: Let ϵ > 0 , κ > 0 , 0 < δ < 1 and T > 0 be given.

Original: There exist constants C m , C ′ m >such that if 0 ≤ η ≤ 1 / (2 M 2 P n ) and m satisfies
Revised: There exist constants C m , C ′ m >such that if 0 ≤ η ≤ 1 / (2 M 2 P n ) and m satisfies

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: then with probability at least 1 − δ , we have the following for all 0 ≤ k ≤ T :
Revised: then with probability at least 1 − δ , we have the following for all 0 ≤ k ≤ T :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Set δ ′ =
Revised: Set δ ′ =

Original: δ/ 3 .
Revised: δ/ 3 .

Original: For any R > 0 and r = 1 , . . .
Revised: For any R > 0 and r = 1 , . . .

Original: , m , since w r (0)
Revised: , m , since w r (0)

Original: ⊤ x i ∼ N (0 , κ 2 ) , we have
Revised: ⊤ x i ∼ N (0 , κ 2 ) , we have

Original: [Equation]
Revised: [Equation]

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: R }
Revised: R }

Original: [Equation]
Revised: [Equation]

Original: < 2 R √ πκ.
Revised: < 2 R √ πκ.

Original: By Hoeffding’s inequality (Hoeffding, 1963), for any t > 0 we have
Revised: By Hoeffding’s inequality (Hoeffding, 1963), for any t > 0 we have

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: Thus, if we set t = (cid:112) m log( n/δ ′ ) then we find that with probability at least 1 − δ ′ /n
Revised: Thus, if we set t = (cid:112) m log( n/δ ′ ) then we find that with probability at least 1 − δ ′ /n

Original: we have m (cid:88)
Revised: we have m (cid:88)

Original: [Equation]
Revised: [Equation]

Original: By a union bound, we have with probability at least 1 − δ ′ , n
Revised: By a union bound, we have with probability at least 1 − δ ′ , n

Original: [Equation]
Revised: [Equation]

Original: By combining this with Lemma 3, we have that with probability at least 1 − 2 δ ′ ,
Revised: By combining this with Lemma 3, we have that with probability at least 1 − 2 δ ′ ,

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Since the i th entry of u (0) has mean 0 and variance ≤ κ 2 , we have E [( u (0))
Revised: Since the i th entry of u (0) has mean 0 and variance ≤ κ 2 , we have E [( u (0))

Original: 2 i ] ≤ κ 2 , where ( u (0))
Revised: 2 i ] ≤ κ 2 , where ( u (0))

Original: i is the i th entry.
Revised: i is the i th entry.

Original: Hence, we have E (cid:2)
Revised: Hence, we have E (cid:2)

Original: ∥ u (0) ∥ 2 P (cid:3) ≤ M 2 P nκ 2 .
Revised: ∥ u (0) ∥ 2 P (cid:3) ≤ M 2 P nκ 2 .

Original: By Markov’s inequality, with probability at least 1 − δ ′ , we have
Revised: By Markov’s inequality, with probability at least 1 − δ ′ , we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: By a union bound, we know that eqs.
Revised: By a union bound, we know that eqs.

Original: (41) to (43) hold with probability of at least 1 − 3 δ ′ .
Revised: (41) to (43) hold with probability of at least 1 − 3 δ ′ .

Original: The theorem now follows using induction, where the base case when k = 0 is obvious.
Revised: The theorem now follows using induction, where the base case when k = 0 is obvious.

Original: Assume eq.
Revised: Assume eq.

Original: (39) holds for t = 0 , . . .
Revised: (39) holds for t = 0 , . . .

Original: , k − 1 , where 1 ≤ k ≤ T .
Revised: , k − 1 , where 1 ≤ k ≤ T .

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: where the first inequality follows from the fact that I − 2 η H ∞
Revised: where the first inequality follows from the fact that I − 2 η H ∞

Original: P is positive semidefinite in ( R n , ⟨· , ·⟩ P ) with the maximum eigenvalue being 1 − 2 ηλ 0 , and the second inequality follows by bounding the power series.
Revised: P is positive semidefinite in ( R n , ⟨· , ·⟩ P ) with the maximum eigenvalue being 1 − 2 ηλ 0 , and the second inequality follows by bounding the power series.

Original: By the definition of λ 0 , we have
Revised: By the definition of λ 0 , we have

Original: [Equation]
Revised: [Equation]

Original: By Lemma 2 and 3, we have
Revised: By Lemma 2 and 3, we have

Original: [Equation]
Revised: [Equation]

Original: with
Revised: with

Original: [Equation]
Revised: [Equation]

Original: where we used the fact that | S i ( t ) | is a nondecreasing function of t and the triangle inequality ∥
Revised: where we used the fact that | S i ( t ) | is a nondecreasing function of t and the triangle inequality ∥

Original: ( H ∞ − H ( t ))
Revised: ( H ∞ − H ( t ))

Original: P
Revised: P

Original: ∥
Revised: ∥

Original: P ≤ ∥
Revised: P ≤ ∥

Original: ( H (0) − H ∞ )
Revised: ( H (0) − H ∞ )

Original: P ∥ P + ∥
Revised: P ∥ P + ∥

Original: ( H ( t ) − H (0))
Revised: ( H ( t ) − H (0))

Original: P ∥ P .
Revised: P ∥ P .

Original: Here, we also combined λ − 1 0 ∥
Revised: Here, we also combined λ − 1 0 ∥

Original: ( H ( t ) − H (0))
Revised: ( H ( t ) − H (0))

Original: P ∥ P ∥ y − u (0)
Revised: P ∥ P ∥ y − u (0)

Original: ∥ P with the last term on the right-hand side of eq.
Revised: ∥ P with the last term on the right-hand side of eq.

Original: (30) and applied Lemma 3, eq.
Revised: (30) and applied Lemma 3, eq.

Original: (44), and eq.
Revised: (44), and eq.

Original: (45) to obtain the last term on the right-hand side of eq.
Revised: (45) to obtain the last term on the right-hand side of eq.

Original: To control | S i ( k ) | , we first bound the change of the weights.
Revised: To control | S i ( k ) | , we first bound the change of the weights.

Original: For any 0 ≤ t ≤ k − 1 and 1 ≤ r ≤ m , the change of weights in one iteration can be bounded by
Revised: For any 0 ≤ t ≤ k − 1 and 1 ≤ r ≤ m , the change of weights in one iteration can be bounded by

Original: [Equation]
Revised: [Equation]

Original: where the inequalities follow from eq.
Revised: where the inequalities follow from eq.

Original: (27) and eq.
Revised: (27) and eq.

Original: Hence, the total change of the weights can be bounded by
Revised: Hence, the total change of the weights can be bounded by

Original: [Equation]
Revised: [Equation]

Original: where R T =
Revised: where R T =

Original: M P (cid:112) n 2 m (cid:0) λ − 1 0 ∥
Revised: M P (cid:112) n 2 m (cid:0) λ − 1 0 ∥

Original: y − u (0) ∥
Revised: y − u (0) ∥

Original: P + 2 ηTϵ (cid:1) .
Revised: P + 2 ηTϵ (cid:1) .

Original: Recall that S i ( k ) is the set of indices of weights that have gone through at least one sign flip by iteration k .
Revised: Recall that S i ( k ) is the set of indices of weights that have gone through at least one sign flip by iteration k .

Original: Thus, if r ∈ S i ( k ) , we have ∥ ˜ w r ( t ) − ˜ w r
Revised: Thus, if r ∈ S i ( k ) , we have ∥ ˜ w r ( t ) − ˜ w r

Original: [Equation]
Revised: [Equation]

Original: ∥ 2 ≥ (cid:12)(cid:12) ˜
Revised: ∥ 2 ≥ (cid:12)(cid:12) ˜

Original: [Equation]
Revised: [Equation]

Original: r
Revised: r

Original: [Equation]
Revised: [Equation]

Original: ⊤ ˜ x
Revised: ⊤ ˜ x

Original: i (cid:12)(cid:12) for some 0 ≤ t ≤ k as the sign flip leads to (cid:12)(cid:12)
Revised: i (cid:12)(cid:12) for some 0 ≤ t ≤ k as the sign flip leads to (cid:12)(cid:12)

Original: ˜ w r (0)
Revised: ˜ w r (0)

Original: ⊤ ˜ x
Revised: ⊤ ˜ x

Original: i (cid:12)(cid:12) ≤ (cid:12)(cid:12)
Revised: i (cid:12)(cid:12) ≤ (cid:12)(cid:12)

Original: ˜ w r (0)
Revised: ˜ w r (0)

Original: ⊤ ˜ x
Revised: ⊤ ˜ x

Original: i − ˜ w r ( t )
Revised: i − ˜ w r ( t )

Original: ⊤ ˜ x i (cid:12)(cid:12) .
Revised: ⊤ ˜ x i (cid:12)(cid:12) .

Original: This gives us
Revised: This gives us

Original: [Equation]
Revised: [Equation]

Original: 48) where [ m ] = { 1 , . . .
Revised: 48) where [ m ] = { 1 , . . .

Original: , m } .
Revised: , m } .

Original: Hence, there exists a constant C > 0 such that
Revised: Hence, there exists a constant C > 0 such that

Original: [Equation]
Revised: [Equation]

Original: where the first inequality follows from eq.
Revised: where the first inequality follows from eq.

Original: (41), eq.
Revised: (41), eq.

Original: (46), and eq.
Revised: (46), and eq.

Original: (48), and the second inequality follows from eq.
Revised: (48), and the second inequality follows from eq.

Original: (43) and eq.
Revised: (43) and eq.

Original: Finally, eq. (39) follows from the way we define m .
Revised: Finally, eq. (39) follows from the way we define m .

Original: By taking
Revised: By taking

Original: C m large enough, we guarantee that 2 M 2 P nA 1 , 8 M 2 P nA 2 A 4 < ϵ/ 3 .
Revised: C m large enough, we guarantee that 2 M 2 P nA 1 , 8 M 2 P nA 2 A 4 < ϵ/ 3 .

Original: By taking C ′ m large enough, we guarantee that 8 M 2 P nA 3 A 4 < ϵ/ 3 .
Revised: By taking C ′ m large enough, we guarantee that 8 M 2 P nA 3 A 4 < ϵ/ 3 .

Original: Hence, eq. (39) follows.
Revised: Hence, eq. (39) follows.

Original: Lemma 4 gives us an estimate of the residual y − u ( k ) in terms of the initial residual y − u (0) .
Revised: Lemma 4 gives us an estimate of the residual y − u ( k ) in terms of the initial residual y − u (0) .

Original: However, in analyzing the frequency bias, we hope to express the residual in terms of y only.
Revised: However, in analyzing the frequency bias, we hope to express the residual in terms of y only.

Original: This can be done by controlling the size of u (0) .
Revised: This can be done by controlling the size of u (0) .

Original: First, we note that the proof of eq.
Revised: First, we note that the proof of eq.

Original: (43) does not rely on the assumptions on m in Lemma 4.
Revised: (43) does not rely on the assumptions on m in Lemma 4.

Original: Hence, it holds for any n, m ≥ 1 , κ > 0 , 0 < δ < 1 , and positive definite matrix P .
Revised: Hence, it holds for any n, m ≥ 1 , κ > 0 , 0 < δ < 1 , and positive definite matrix P .

Original: Now we are ready to prove our first main result Theorem 1.
Revised: Now we are ready to prove our first main result Theorem 1.

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: By eq.
Revised: By eq.

Original: (43), with probability at least 1 − δ/ 2 , we have
Revised: (43), with probability at least 1 − δ/ 2 , we have

Original: [Equation]
Revised: [Equation]

Original: By taking κ ≤ ϵ (cid:112) δ/ 2 n/ (2 M P ) , we guarantee that
Revised: By taking κ ≤ ϵ (cid:112) δ/ 2 n/ (2 M P ) , we guarantee that

Original: [Equation]
Revised: [Equation]

Original: By the way we pick κ and m , for some constant C ′ > 0 that only depends on d , we have
Revised: By the way we pick κ and m , for some constant C ′ > 0 that only depends on d , we have

Original: [Equation]
Revised: [Equation]

Original: Hence, by taking C 2 in eq.
Revised: Hence, by taking C 2 in eq.

Original: (13) large enough, we guarantee that m satisfies the assumptions in
Revised: (13) large enough, we guarantee that m satisfies the assumptions in

Original: Lemma 4 with ϵ, κ, T, and δ to be ϵ/ 2 , κ , T , and δ/ 2 , respectively.
Revised: Lemma 4 with ϵ, κ, T, and δ to be ϵ/ 2 , κ , T , and δ/ 2 , respectively.

Original: Then, we have eq.
Revised: Then, we have eq.

Original: (39) is true with probability of at least 1 − δ/ 2 , for which
Revised: (39) is true with probability of at least 1 − δ/ 2 , for which

Original: ∥ ϵ ( k ) ∥
Revised: ∥ ϵ ( k ) ∥

Original: P ≤ ϵ/ 2 .
Revised: P ≤ ϵ/ 2 .

Original: The result follows from the triangle inequality and union bound.
Revised: The result follows from the triangle inequality and union bound.

Original: Notably, there are other initialization schemes that allow us to avoid using κ .
Revised: Notably, there are other initialization schemes that allow us to avoid using κ .

Original: One of the examples is to initialize the weights at odd indices w 2 p +1 and a 2 p +1 randomly and set w 2 p +2 =
Revised: One of the examples is to initialize the weights at odd indices w 2 p +1 and a 2 p +1 randomly and set w 2 p +2 =

Original: w 2 p
Revised: w 2 p

Original: +1 , a 2 p +2 = − a 2 r +1 , assuming m is even (Su & Yang, 2019).
Revised: +1 , a 2 p +2 = − a 2 r +1 , assuming m is even (Su & Yang, 2019).

Original: This initialization scheme guarantees that u (0) = 

Revised: This initialization scheme guarantees that u (0) =

Original: and hence we do not need to introduce κ to control the initialization size ∥ u (0) ∥ P .
Revised: and hence we do not need to introduce κ to control the initialization size ∥ u (0) ∥ P .

Original: In addition, if we assume that each entry of w r is initialized from an iid sub-Gaussian distribution with zero mean and whose support is the entire R , then since H ∞ is still SPD (see the remark at the end of Appendix B) and the Hoeffding’s inequality holds, the proof does not break down and a result similar to Theorem 1 can be shown.
Revised: In addition, if we assume that each entry of w r is initialized from an iid sub-Gaussian distribution with zero mean and whose support is the entire R , then since H ∞ is still SPD (see the remark at the end of Appendix B) and the Hoeffding’s inequality holds, the proof does not break down and a result similar to Theorem 1 can be shown.

Original: Following the same steps of proof, we can study the case when the gradient descent steps are slightly perturbed.
Revised: Following the same steps of proof, we can study the case when the gradient descent steps are slightly perturbed.

Original: That is, Suppose we perturb the output of the NN by δ u j at the j th iteration.
Revised: That is, Suppose we perturb the output of the NN by δ u j at the j th iteration.

Original: Then, we expect that the residual of the NN at the k th iteration is approximately
Revised: Then, we expect that the residual of the NN at the k th iteration is approximately

Original: [Equation]
Revised: [Equation]

Original: Since the maximum eigenvalue of I − 2 η H ∞ P is less than one, we can then control the errors in this approximation using arguments similar to previous lemmas.
Revised: Since the maximum eigenvalue of I − 2 η H ∞ P is less than one, we can then control the errors in this approximation using arguments similar to previous lemmas.

Original: In Theorem 1, we showed how the parameters of the NN should depend on the desired maximum error ϵ .
Revised: In Theorem 1, we showed how the parameters of the NN should depend on the desired maximum error ϵ .

Original: Sometimes, it is also very useful to understand the dependence of ϵ on the parameters η, T, n, m , etc.
Revised: Sometimes, it is also very useful to understand the dependence of ϵ on the parameters η, T, n, m , etc.

Original: Therefore, we present the following result to show this dependency.
Revised: Therefore, we present the following result to show this dependency.

Original: Theorem 1 ′ .
Revised: Theorem 1 ′ .

Original: In eq.
Revised: In eq.

Original: (5) , suppose that w 1 , . . .
Revised: (5) , suppose that w 1 , . . .

Original: , w m are initialized iid from Gaussian random variables with covariance matrix κ 2 I , b 1 , . . .
Revised: , w m are initialized iid from Gaussian random variables with covariance matrix κ 2 I , b 1 , . . .

Original: , b m are initialized to zero, and a 1 , . . .
Revised: , b m are initialized to zero, and a 1 , . . .

Original: , a m are initialized iid as +1 with probability 1 / 2 and − 1 otherwise.
Revised: , a m are initialized iid as +1 with probability 1 / 2 and − 1 otherwise.

Original: Suppose the NN is trained with training data ( x i , y i ) for 1 ≤ i ≤ n , loss function Φ P in eq.
Revised: Suppose the NN is trained with training data ( x i , y i ) for 1 ≤ i ≤ n , loss function Φ P in eq.

Original: (6) for a symmetric positive definite matrix P , and the training procedure is the gradient descent update rule eq.
Revised: (6) for a symmetric positive definite matrix P , and the training procedure is the gradient descent update rule eq.

Original: (7) with step size η ≤ 1 / (2 M 2 P n ) .
Revised: (7) with step size η ≤ 1 / (2 M 2 P n ) .

Original: Let
Revised: Let

Original: N k be the NN function after the k th iteration and u ( k ) =
Revised: N k be the NN function after the k th iteration and u ( k ) =

Original: ( N k ( x 1 ) , . . .
Revised: ( N k ( x 1 ) , . . .

Original: , N k ( x n )) , where N 0 is the initial NN function.
Revised: , N k ( x n )) , where N 0 is the initial NN function.

Original: Let a probability of failure 0 < δ < 1 be given.
Revised: Let a probability of failure 0 < δ < 1 be given.

Original: Then, there exists a constant
Revised: Then, there exists a constant

Original: C > 0 that depends only on the dimension d such that with probability ≥ 1 − δ , the following statement holds: for any k ≥ 1 , if we define ϵ k by
Revised: C > 0 that depends only on the dimension d such that with probability ≥ 1 − δ , the following statement holds: for any k ≥ 1 , if we define ϵ k by

Original: [Equation]
Revised: [Equation]

Original: then we have
Revised: then we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The result follows immediately from eq.
Revised: The result follows immediately from eq.

Original: (49) and eq.
Revised: (49) and eq.

Original: While this paper is primarily concerned with learning a continuous function using loss functions that are adapted from MSE, we briefly discuss the changes that are needed to study classifiers trained by cross-entropy.
Revised: While this paper is primarily concerned with learning a continuous function using loss functions that are adapted from MSE, we briefly discuss the changes that are needed to study classifiers trained by cross-entropy.

Original: If the classification task has p classes, then our NN has p outputs, each of which is then passed through a softmax layer and represents an estimate of the likelihood that the input belongs to the corresponding class.
Revised: If the classification task has p classes, then our NN has p outputs, each of which is then passed through a softmax layer and represents an estimate of the likelihood that the input belongs to the corresponding class.

Original: More information on the NN architecture and the cross-entropy loss function can be found in (Kurbiel, 2021).
Revised: More information on the NN architecture and the cross-entropy loss function can be found in (Kurbiel, 2021).

Original: Let N j ( x ; W ) be the j th entry of the outputs of the
Revised: Let N j ( x ; W ) be the j th entry of the outputs of the

Original: NN and let u j ( x ; W ) =
Revised: NN and let u j ( x ; W ) =

Original: softmax ( N j ( x ; W )) , 1 ≤ j ≤ p .
Revised: softmax ( N j ( x ; W )) , 1 ≤ j ≤ p .

Original: Let g j ( x ) be the ground-truth of the j th entry of the outputs and let z j ( x ; W )
Revised: Let g j ( x ) be the ground-truth of the j th entry of the outputs and let z j ( x ; W )

Original: =
Revised: =

Original: g j ( x ) − u j ( x ; W ) be the residual.
Revised: g j ( x ) − u j ( x ; W ) be the residual.

Original: Assume we use the gradient flow and have access to data on the entire domain.
Revised: Assume we use the gradient flow and have access to data on the entire domain.

Original: Then, to derive a formula analogous to eq.
Revised: Then, to derive a formula analogous to eq.

Original: (2), for each fixed x , we have
Revised: (2), for each fixed x , we have

Original: [Equation]
Revised: [Equation]

Original: where L is the cross-entropy loss function computed as an integral over S d − 1 and in the last step we used the fact that ( d/d N i ′ )
Revised: where L is the cross-entropy loss function computed as an integral over S d − 1 and in the last step we used the fact that ( d/d N i ′ )

Original: L ( x ′ ; W ) =
Revised: L ( x ′ ; W ) =

Original: u i ′ ( x
Revised: u i ′ ( x

Original: [Equation]
Revised: [Equation]

Original: ; W
Revised: ; W

Original: [Equation]
Revised: [Equation]

Original: − g i
Revised: − g i

Original: [Equation]
Revised: [Equation]

Original: ( x
Revised: ( x

Original: [Equation]
Revised: [Equation]

Original: = − z
Revised: = − z

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: ( x
Revised: ( x

Original: [Equation]
Revised: [Equation]

Original: ; W ) Kurbiel (2021).
Revised: ; W ) Kurbiel (2021).

Original: Hence, eq.
Revised: Hence, eq.

Original: (2) now becomes
Revised: (2) now becomes

Original: [Equation]
Revised: [Equation]

Original: where we also know that ( d/d N i )
Revised: where we also know that ( d/d N i )

Original: u j ( x ; W ) =
Revised: u j ( x ; W ) =

Original: u i ( x ; W )( I { i = j } − u j ( x ; W )) .
Revised: u i ( x ; W )( I { i = j } − u j ( x ; W )) .

Original: Again, we define
Revised: Again, we define

Original: H ∞
Revised: H ∞

Original: i,i ′ as the discretization of K i,i ′ in expectation over random initialization of W .
Revised: i,i ′ as the discretization of K i,i ′ in expectation over random initialization of W .

Original: Note that H ∞
Revised: Note that H ∞

Original: i,i coincides with H ∞ that we used extensively in this paper.
Revised: i,i coincides with H ∞ that we used extensively in this paper.

Original: However, the entries of H ∞ i,i ′ might differ with H ∞ in signs.
Revised: However, the entries of H ∞ i,i ′ might differ with H ∞ in signs.

Original: This potentially causes difficulties in analyzing the spectrum of H ∞ i,i ′ .
Revised: This potentially causes difficulties in analyzing the spectrum of H ∞ i,i ′ .

Original: Now, by the formula above, we expect that the residual can approximately be written as
Revised: Now, by the formula above, we expect that the residual can approximately be written as

Original: [Equation]
Revised: [Equation]

Original: where u j ( k ) =
Revised: where u j ( k ) =

Original: u j ( x ; W ( k )) and ‘ ◦ ’ is the Hadamard product.
Revised: u j ( x ; W ( k )) and ‘ ◦ ’ is the Hadamard product.

Original: Suppose we define the vectorized residual z = (cid:2)
Revised: Suppose we define the vectorized residual z = (cid:2)

Original: z ⊤ 1 , . . .
Revised: z ⊤ 1 , . . .

Original: , z ⊤ p (cid:3) ⊤ and define the np × np block matrix J ( k ) by
Revised: , z ⊤ p (cid:3) ⊤ and define the np × np block matrix J ( k ) by

Original: J ( k ) ( in +1):( i +1) n, ( jn +1):( j +1) n = diag (cid:0) u i ( k ) ◦
Revised: J ( k ) ( in +1):( i +1) n, ( jn +1):( j +1) n = diag (cid:0) u i ( k ) ◦

Original: ( I { i = j } − u j ( k )) (cid:1) for i, j = 0 , . . .
Revised: ( I { i = j } − u j ( k )) (cid:1) for i, j = 0 , . . .

Original: , p − 1 .
Revised: , p − 1 .

Original: Let H ∞ be the np × np block matrix such that H ∞ ( in +1):( i +1) n, ( i ′ n +1):( i ′ +1) n = H ∞
Revised: Let H ∞ be the np × np block matrix such that H ∞ ( in +1):( i +1) n, ( i ′ n +1):( i ′ +1) n = H ∞

Original: i,i ′ for i, i ′ =
Revised: i,i ′ for i, i ′ =

Original: . , p − 1 .
Revised: . , p − 1 .

Original: Then, eq.
Revised: Then, eq.

Original: (52) can be written compactly as
Revised: (52) can be written compactly as

Original: [Equation]
Revised: [Equation]

Original: where ‘ ⊗ ’ is the Kronecker product.
Revised: where ‘ ⊗ ’ is the Kronecker product.

Original: Frequency biasing can be analyzed by studying the dynamics of z ( k ) based on eq.
Revised: Frequency biasing can be analyzed by studying the dynamics of z ( k ) based on eq.

Original: However, the fact that J depends on k is expected to add complication to the analysis.
Revised: However, the fact that J depends on k is expected to add complication to the analysis.

Original: D T HE THEORY OF FREQUENCY BIASING WITH AN L 2 - BASED LOSS FUNCTION
Revised: D T HE THEORY OF FREQUENCY BIASING WITH AN L 2 - BASED LOSS FUNCTION

Original: In this section, we prove the results stated in Section 4, where we are concerned with the frequency biasing behavior of NN training when using the squared L 2 norm as the loss function.
Revised: In this section, we prove the results stated in Section 4, where we are concerned with the frequency biasing behavior of NN training when using the squared L 2 norm as the loss function.

Original: We theoretically show the frequency biasing phenomena in this setting, up to a quadrature error.
Revised: We theoretically show the frequency biasing phenomena in this setting, up to a quadrature error.

Original: D. A CONSEQUENCE OF T HEOREM
Revised: D. A CONSEQUENCE OF T HEOREM

Original: Given a bandlimited function g : S d − 1 → R with bandlimit L , we can uniquely decompose g into a spherical harmonic expansion as g ( x ) = (cid:80) Lℓ =

Revised: Given a bandlimited function g : S d − 1 → R with bandlimit L , we can uniquely decompose g into a spherical harmonic expansion as g ( x ) = (cid:80) Lℓ =

Original: g ℓ ( x ) , where g ℓ ( x ) ∈ H dℓ .
Revised: g ℓ ( x ) , where g ℓ ( x ) ∈ H dℓ .

Original: Here, H dℓ is the space of the restriction of (real) homogeneous harmonic polynomials of degree ℓ on S d − 1 .
Revised: Here, H dℓ is the space of the restriction of (real) homogeneous harmonic polynomials of degree ℓ on S d − 1 .

Original: That is, g
Revised: That is, g

Original: [Equation]
Revised: [Equation]

Original: ( x ) =
Revised: ( x ) =

Original: N ( d
Revised: N ( d

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: ˆ
Revised: ˆ

Original: [Equation]
Revised: [Equation]

Original: Y
Revised: Y

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: ˆ
Revised: ˆ

Original: [Equation]
Revised: [Equation]

Original: g (
Revised: g (

Original: [Equation]
Revised: [Equation]

Original: Y
Revised: Y

Original: [Equation]
Revised: [Equation]

Original: where N ( d, ℓ ) is given in section 2 and Y ℓ,p are the spherical harmonic function of degree ℓ and order p .
Revised: where N ( d, ℓ ) is given in section 2 and Y ℓ,p are the spherical harmonic function of degree ℓ and order p .

Original: As a consequence of Theorem 1, we can consider the NN training error with the squared
Revised: As a consequence of Theorem 1, we can consider the NN training error with the squared

Original: L 2 -based loss function.
Revised: L 2 -based loss function.

Original: Proof of Theorem 2.
Revised: Proof of Theorem 2.

Original: By Theorem 1, for every k = 0 , . . .
Revised: By Theorem 1, for every k = 0 , . . .

Original: , T , we can write
Revised: , T , we can write

Original: [Equation]
Revised: [Equation]

Original: To estimate the first term, we first note that the matrix I − 2 η H ∞ D c is positive semidefinite in ( R n , ⟨· , ·⟩ c ) and
Revised: To estimate the first term, we first note that the matrix I − 2 η H ∞ D c is positive semidefinite in ( R n , ⟨· , ·⟩ c ) and

Original: ∥ I − 2 η H ∞ D c
Revised: ∥ I − 2 η H ∞ D c

Original: ∥ c ≤ 1 − 2 ηλ 0 (see Theorem 1).
Revised: ∥ c ≤ 1 − 2 ηλ 0 (see Theorem 1).

Original: Since g ( x ) = (cid:80) Lℓ =

Revised: Since g ( x ) = (cid:80) Lℓ =

Original: g ℓ ( x ) , we have
Revised: g ℓ ( x ) , we have

Original: [Equation]
Revised: [Equation]

Original: where y ℓ =
Revised: where y ℓ =

Original: [ g ℓ ( x 1 ) , . . . , g ℓ ( x n )]
Revised: [ g ℓ ( x 1 ) , . . . , g ℓ ( x n )]

Original: ⊤ ∈ R n .
Revised: ⊤ ∈ R n .

Original: By the Funk–Hecke formula and the quadrature rule, we have
Revised: By the Funk–Hecke formula and the quadrature rule, we have

Original: [Equation]
Revised: [Equation]

Original: where e dp,ℓ is a quadrature error (see eq.
Revised: where e dp,ℓ is a quadrature error (see eq.

Original: Therefore, in vectorized form, we have
Revised: Therefore, in vectorized form, we have

Original: H ∞ D c y ℓ = µ ℓ y ℓ + e dℓ or, equivalently, ( I − 2 η H ∞ D c ) y ℓ
Revised: H ∞ D c y ℓ = µ ℓ y ℓ + e dℓ or, equivalently, ( I − 2 η H ∞ D c ) y ℓ

Original: = (1 − 2 ηµ ℓ ) y ℓ − 2 η e dℓ , where e dℓ =
Revised: = (1 − 2 ηµ ℓ ) y ℓ − 2 η e dℓ , where e dℓ =

Original: ( e d 1 ,ℓ , . . .
Revised: ( e d 1 ,ℓ , . . .

Original: , e dn,ℓ )
Revised: , e dn,ℓ )

Original: ⊤ .
Revised: ⊤ .

Original: By applying I − 2 η H ∞ D c to y ℓ for k times, we find that
Revised: By applying I − 2 η H ∞ D c to y ℓ for k times, we find that

Original: [Equation]
Revised: [Equation]

Original: The second term, ε ℓ 2 , can be easily bounded from above to obtain
Revised: The second term, ε ℓ 2 , can be easily bounded from above to obtain

Original: [Equation]
Revised: [Equation]

Original: The inequality above shows that ( I − 2 η H ∞ D c ) k y ℓ is close to (1 − 2 ηµ ℓ ) k y ℓ .
Revised: The inequality above shows that ( I − 2 η H ∞ D c ) k y ℓ is close to (1 − 2 ηµ ℓ ) k y ℓ .

Original: We define
Revised: We define

Original: [Equation]
Revised: [Equation]

Original: to be the quantity in the statement of Theorem 2, which measures the accuracy of approximating the eigenvalues and eigenvectors of
Revised: to be the quantity in the statement of Theorem 2, which measures the accuracy of approximating the eigenvalues and eigenvectors of

Original: H ∞
Revised: H ∞

Original: D c using the eigenvalues and eigenfunctions of the continuous kernel K ∞ .
Revised: D c using the eigenvalues and eigenfunctions of the continuous kernel K ∞ .

Original: Hence, we have
Revised: Hence, we have

Original: [Equation]
Revised: [Equation]

Original: Using the triangle inequality
Revised: Using the triangle inequality

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Recall that (cid:80) ni
Revised: Recall that (cid:80) ni

Original: =1 c i =
Revised: =1 c i =

Original: A d , the surface area of S d − 1 .
Revised: A d , the surface area of S d − 1 .

Original: Next, we can write
Revised: Next, we can write

Original: [Equation]
Revised: [Equation]

Original: 56) where we used the fact that g j and g ℓ are orthogonal in L 2 ( S d − 1 ) for j ̸ = ℓ .
Revised: 56) where we used the fact that g j and g ℓ are orthogonal in L 2 ( S d − 1 ) for j ̸ = ℓ .

Original: Combining eq.
Revised: Combining eq.

Original: (55) and eq.
Revised: (55) and eq.

Original: (56), we can write
Revised: (56), we can write

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: The result follows from eq.
Revised: The result follows from eq.

Original: (54) and eq.
Revised: (54) and eq.

Original: Similarly, suppose we use Theorem 1 ′ , we can write down Theorem 2 in a form such that ϵ 3 ( k ) depends on the parameters of the NN.
Revised: Similarly, suppose we use Theorem 1 ′ , we can write down Theorem 2 in a form such that ϵ 3 ( k ) depends on the parameters of the NN.

Original: Theorem 2 ′ .
Revised: Theorem 2 ′ .

Original: Under the same setup and assumptions of Theorem 1 ′ , let P =
Revised: Under the same setup and assumptions of Theorem 1 ′ , let P =

Original: D
Revised: D

Original: c
Revised: c

Original: and M P = √ c max .
Revised: and M P = √ c max .

Original: If g : S d − 1 → R is a bandlimited function with bandlimit L and 1 − 2 ηµ ℓ > 0 for all 0 ≤ ℓ ≤ L (see eq.
Revised: If g : S d − 1 → R is a bandlimited function with bandlimit L and 1 − 2 ηµ ℓ > 0 for all 0 ≤ ℓ ≤ L (see eq.

Original: (12) ), then with probability ≥ 1 − δ
Revised: (12) ), then with probability ≥ 1 − δ

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where | ε 3 ( k ) | is bounded by eq.
Revised: where | ε 3 ( k ) | is bounded by eq.

Original: (51) , and ε 1 ( k ) and ε 2 satisfy
Revised: (51) , and ε 1 ( k ) and ε 2 satisfy

Original: [Equation]
Revised: [Equation]

Original: We remark that the left-hand side of eq.
Revised: We remark that the left-hand side of eq.

Original: (17) can be rewritten as
Revised: (17) can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: where N k is the neural network at the k th iteration.
Revised: where N k is the neural network at the k th iteration.

Original: Hence, eq.
Revised: Hence, eq.

Original: (17) can also be written as
Revised: (17) can also be written as

Original: [Equation]
Revised: [Equation]

Original: where ε 1 , ε 2 , ε 3 are as in Theorem 2, and | ε 4 ( k ) | ≤ (cid:112) | E c (( g − N k ) 2 ) | .
Revised: where ε 1 , ε 2 , ε 3 are as in Theorem 2, and | ε 4 ( k ) | ≤ (cid:112) | E c (( g − N k ) 2 ) | .

Original: Moreover, we assumed g is bandlimited and as we see in Proposition 2, the spherical harmonic coefficients of N k decay fast as the frequency ℓ → ∞ .
Revised: Moreover, we assumed g is bandlimited and as we see in Proposition 2, the spherical harmonic coefficients of N k decay fast as the frequency ℓ → ∞ .

Original: This shows that the size of | ε 4 ( k ) | can also be controlled by γ n,ℓ in eq.
Revised: This shows that the size of | ε 4 ( k ) | can also be controlled by γ n,ℓ in eq.

Original: Therefore, we derived a frequency bias statement of the generalization error with respect to the uniform distribution on S d − 1 .
Revised: Therefore, we derived a frequency bias statement of the generalization error with respect to the uniform distribution on S d − 1 .

Original: D. F REQUENCY BIASING UP TO AN APPROXIMATION ERROR
Revised: D. F REQUENCY BIASING UP TO AN APPROXIMATION ERROR

Original: Theorem 2 shows that we theoretically have frequency biasing up to the level of quadrature errors ε 1 ( k ) and ε 2 .
Revised: Theorem 2 shows that we theoretically have frequency biasing up to the level of quadrature errors ε 1 ( k ) and ε 2 .

Original: If the quadrature errors are large, then we may not observe frequency biasing in practice.
Revised: If the quadrature errors are large, then we may not observe frequency biasing in practice.

Original: Here, we show that the quadrature errors can be made arbitrarily small by taking enough samples in the training data.
Revised: Here, we show that the quadrature errors can be made arbitrarily small by taking enough samples in the training data.

Original: Recall that our quadrature rule satisfies eq.
Revised: Recall that our quadrature rule satisfies eq.

Original: Next, we pass the quadrature error to the approximation error, which may not necessarily be tight.
Revised: Next, we pass the quadrature error to the approximation error, which may not necessarily be tight.

Original: However, this allows us to use the existing theory of spherical harmonics approximation to show the decay of quadrature errors.
Revised: However, this allows us to use the existing theory of spherical harmonics approximation to show the decay of quadrature errors.

Original: Lemma 5.
Revised: Lemma 5.

Original: Let f : S d − 1 → R be a function and dist ( f, Π dℓ ) =
Revised: Let f : S d − 1 → R be a function and dist ( f, Π dℓ ) =

Original: min h ∈ Π dℓ ∥ f − h ∥
Revised: min h ∈ Π dℓ ∥ f − h ∥

Original: L ∞ .
Revised: L ∞ .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: (60) for any integer ℓ ≥ 0 .
Revised: (60) for any integer ℓ ≥ 0 .

Original: Proof.
Revised: Proof.

Original: Let h ℓ = arg
Revised: Let h ℓ = arg

Original: min h ∈
Revised: min h ∈

Original: Π dℓ ∥
Revised: Π dℓ ∥

Original: f − h ∥
Revised: f − h ∥

Original: L ∞ .
Revised: L ∞ .

Original: By the triangle inequality, we have (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ˆ
Revised: By the triangle inequality, we have (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ˆ

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: ˆ
Revised: ˆ

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where we used the fact that c i > 0 for i = 1 , . .
Revised: where we used the fact that c i > 0 for i = 1 , . .

Original: . , n and the fact that ∥ h ℓ ∥ L ∞ ≤ 2 ∥ f ∥ L ∞ .
Revised: . , n and the fact that ∥ h ℓ ∥ L ∞ ≤ 2 ∥ f ∥ L ∞ .

Original: Next, we focus on controlling the minimum approximation error dist ( f, Π dℓ ) on the right-hand side of eq.
Revised: Next, we focus on controlling the minimum approximation error dist ( f, Π dℓ ) on the right-hand side of eq.

Original: To do so, we prove the following lemma.
Revised: To do so, we prove the following lemma.

Original: Lemma 6.
Revised: Lemma 6.

Original: Let f ij ( ξ ) =
Revised: Let f ij ( ξ ) =

Original: K ∞ ( x i , ξ ) g j ( ξ ) and g jp ( ξ ) =
Revised: K ∞ ( x i , ξ ) g j ( ξ ) and g jp ( ξ ) =

Original: g j ( ξ ) g p ( ξ ) .
Revised: g j ( ξ ) g p ( ξ ) .

Original: Then, for a constant C >that only depends on d , we have
Revised: Then, for a constant C >that only depends on d , we have

Original: [Equation]
Revised: [Equation]

Original: for all ℓ ≥ 1 .
Revised: for all ℓ ≥ 1 .

Original: Proof.
Revised: Proof.

Original: For 1 ≤ a < b ≤ d where a, b ∈ N , and t ∈ [ − π, π ) , let Q a,b,t denote the action on S d −of rotation by the angle t in the ( x a , x b ) -plane.
Revised: For 1 ≤ a < b ≤ d where a, b ∈ N , and t ∈ [ − π, π ) , let Q a,b,t denote the action on S d −of rotation by the angle t in the ( x a , x b ) -plane.

Original: For an integer α ≥ 1 , we define the operator on functions on S d − 1 by
Revised: For an integer α ≥ 1 , we define the operator on functions on S d − 1 by

Original: [Equation]
Revised: [Equation]

Original: where T ( Q ) f ( x ) =
Revised: where T ( Q ) f ( x ) =

Original: f ( Q − 1 x ) .
Revised: f ( Q − 1 x ) .

Original: If f ∈ C ( S d − 1 ) , then we define for t > 0 that
Revised: If f ∈ C ( S d − 1 ) , then we define for t > 0 that

Original: [Equation]
Revised: [Equation]

Original: By (Dai & Xu, 2013, Thm.
Revised: By (Dai & Xu, 2013, Thm.

Original: 4.4.2), we have
Revised: 4.4.2), we have

Original: [Equation]
Revised: [Equation]

Original: where c 1 > 0 is some constant that only depends on α .
Revised: where c 1 > 0 is some constant that only depends on α .

Original: Then it is sufficient to bound ω
Revised: Then it is sufficient to bound ω

Original: α
Revised: α

Original: ( f ij ; ℓ − 1 ) and ω α
Revised: ( f ij ; ℓ − 1 ) and ω α

Original: ( g jp ; ℓ − 1 ) to finish the proof.
Revised: ( g jp ; ℓ − 1 ) to finish the proof.

Original: First, we aim to bound the term dist ( f ij , Π dℓ ) where f ij ( ξ ) =
Revised: First, we aim to bound the term dist ( f ij , Π dℓ ) where f ij ( ξ ) =

Original: K ∞ ( x i , ξ ) g j ( ξ ) .
Revised: K ∞ ( x i , ξ ) g j ( ξ ) .

Original: We fix 1 ≤ i ≤ n , 1 ≤ a < b ≤ d where i, a, b ∈ N , and choose θ such that | θ | ≤ ℓ − 1 .
Revised: We fix 1 ≤ i ≤ n , 1 ≤ a < b ≤ d where i, a, b ∈ N , and choose θ such that | θ | ≤ ℓ − 1 .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: We then define
Revised: We then define

Original: [Equation]
Revised: [Equation]

Original: We then have
Revised: We then have

Original: [Equation]
Revised: [Equation]

Original: We control the two terms separately.
Revised: We control the two terms separately.

Original: First, to control the second term, we write
Revised: First, to control the second term, we write

Original: [Equation]
Revised: [Equation]

Original: based on the definition of K ∞ in eq.
Revised: based on the definition of K ∞ in eq.

Original: For some constant c 2 > 0 that depends only on d , we have
Revised: For some constant c 2 > 0 that depends only on d , we have

Original: [Equation]
Revised: [Equation]

Original: where D a,b := x a ∂ b − x b ∂ a and the two inequalities follow from (Dai & Xu, 2013, Lem. 4.2.
Revised: where D a,b := x a ∂ b − x b ∂ a and the two inequalities follow from (Dai & Xu, 2013, Lem. 4.2.

Original: (iii)) and (Dai & Xu, 2013, Lem. 4.2.4), respectively.
Revised: (iii)) and (Dai & Xu, 2013, Lem. 4.2.4), respectively.

Original: Next, we control the first term of eq.
Revised: Next, we control the first term of eq.

Original: (61) by writing
Revised: (61) by writing

Original: [Equation]
Revised: [Equation]

Original: We fix ξ and define ξ ′ =
Revised: We fix ξ and define ξ ′ =

Original: Q − 1 a,b,θ ξ .
Revised: Q − 1 a,b,θ ξ .

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: Next, by the triangle inequality for angles
Revised: Next, by the triangle inequality for angles

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Since arccos is monotone and | ( d/dt ) arccos( t ) | > 1 for all t , by the fundamental theorem of calculus, we must have
Revised: Since arccos is monotone and | ( d/dt ) arccos( t ) | > 1 for all t , by the fundamental theorem of calculus, we must have

Original: [Equation]
Revised: [Equation]

Original: As a result
Revised: As a result

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: By eq.
Revised: By eq.

Original: (64) and eq.
Revised: (64) and eq.

Original: (65), we find that
Revised: (65), we find that

Original: [Equation]
Revised: [Equation]

Original: Putting eqs.
Revised: Putting eqs.

Original: (61) to (63) and (66) together, we obtain
Revised: (61) to (63) and (66) together, we obtain

Original: [Equation]
Revised: [Equation]

Original: Since a , b , and θ are chosen arbitrarily
Revised: Since a , b , and θ are chosen arbitrarily

Original: [Equation]
Revised: [Equation]

Original: this bound also holds for ω 1 ( f ij ; ℓ − 1 ) .
Revised: this bound also holds for ω 1 ( f ij ; ℓ − 1 ) .

Original: Second, we aim to bound the term dist ( g jp , Π dℓ ) where g jp ( ξ ) =
Revised: Second, we aim to bound the term dist ( g jp , Π dℓ ) where g jp ( ξ ) =

Original: g j ( ξ ) g p ( ξ ) .
Revised: g j ( ξ ) g p ( ξ ) .

Original: As before, we fix the indices a, b ∈ N where 1 ≤ a < b ≤ d and θ such that | θ | ≤ ℓ − 1 .
Revised: As before, we fix the indices a, b ∈ N where 1 ≤ a < b ≤ d and θ such that | θ | ≤ ℓ − 1 .

Original: Similarly
Revised: Similarly

Original: [Equation]
Revised: [Equation]

Original: we define
Revised: we define

Original: [Equation]
Revised: [Equation]

Original: By eq.
Revised: By eq.

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Since a , b , and θ are arbitrary numbers, this bound also holds for ω 1 ( g jp ; ℓ − 1 ) .
Revised: Since a , b , and θ are arbitrary numbers, this bound also holds for ω 1 ( g jp ; ℓ − 1 ) .

Original: Using these lemmas, we can prove that Theorem 3 asymptotically controls the quadrature errors.
Revised: Using these lemmas, we can prove that Theorem 3 asymptotically controls the quadrature errors.

Original: Proof of Theorem 3.
Revised: Proof of Theorem 3.

Original: First, we control ε 1 ( k ) as
Revised: First, we control ε 1 ( k ) as

Original: [Equation]
Revised: [Equation]

Original: where the third and the forth inequalities follow from Lemma 5 and Lemma 6, respectively.
Revised: where the third and the forth inequalities follow from Lemma 5 and Lemma 6, respectively.

Original: The final upper bound for | ε 1 ( k ) | holds since (cid:80)
Revised: The final upper bound for | ε 1 ( k ) | holds since (cid:80)

Original: Lj =0 (cid:80)
Revised: Lj =0 (cid:80)

Original: Lp
Revised: Lp

Original: =

Revised: =

Original: ( j + p ) =
Revised: ( j + p ) =

Original: O ( L 3 ) .
Revised: O ( L 3 ) .

Original: Next, there exists C 2 >such that we can control ε 2 by writing
Revised: Next, there exists C 2 >such that we can control ε 2 by writing

Original: [Equation]
Revised: [Equation]

Original: where the second and the third inequalities follow from Lemma 5 and Lemma 6, respectively.
Revised: where the second and the third inequalities follow from Lemma 5 and Lemma 6, respectively.

Original: The proof is complete.
Revised: The proof is complete.

Original: Proof of Corollary 1.
Revised: Proof of Corollary 1.

Original: By Theorem 3, it suffices to show that max 0 ≤ j ≤ L ∥ g j ∥ L ∞ ≤ C ∥
Revised: By Theorem 3, it suffices to show that max 0 ≤ j ≤ L ∥ g j ∥ L ∞ ≤ C ∥

Original: g ∥
Revised: g ∥

Original: L 2 for some constant C > 0 that does not depend on g .
Revised: L 2 for some constant C > 0 that does not depend on g .

Original: Since H di ⊥ H dj in L 2 for i ̸
Revised: Since H di ⊥ H dj in L 2 for i ̸

Original: = j , we have ∥
Revised: = j , we have ∥

Original: g j ∥
Revised: g j ∥

Original: L 2 ≤ ∥ g ∥
Revised: L 2 ≤ ∥ g ∥

Original: L 2 for 0 ≤ j ≤ L .
Revised: L 2 for 0 ≤ j ≤ L .

Original: The claim follows from the fact that ∥·∥ L 2 and ∥·∥ L ∞ are equivalent in Π dL .
Revised: The claim follows from the fact that ∥·∥ L 2 and ∥·∥ L ∞ are equivalent in Π dL .

Original: E T HE THEORY OF FREQUENCY BIASING WITH A H s - BASED LOSS FUNCTION
Revised: E T HE THEORY OF FREQUENCY BIASING WITH A H s - BASED LOSS FUNCTION

Original: This section presents detailed proofs for Proposition 2 and Theorem 4 in Section 5, which concerns the frequency biasing behavior of NN training using the H s loss function.
Revised: This section presents detailed proofs for Proposition 2 and Theorem 4 in Section 5, which concerns the frequency biasing behavior of NN training using the H s loss function.

Original: E. A 2- LAYER R E LU NEURAL NETWORK IS IN H s ( S d − 1 ) FOR s < 3 /
Revised: E. A 2- LAYER R E LU NEURAL NETWORK IS IN H s ( S d − 1 ) FOR s < 3 /

Original: We prove that a 2-layer ReLU NN map is contained in H s ( S d − 1 ) for any s < 3 / 2 (see Proposition 2).
Revised: We prove that a 2-layer ReLU NN map is contained in H s ( S d − 1 ) for any s < 3 / 2 (see Proposition 2).

Original: Proof of Proposition 2.
Revised: Proof of Proposition 2.

Original: Since N can be written as
Revised: Since N can be written as

Original: [Equation]
Revised: [Equation]

Original: it suffices to prove that f ( x ) =
Revised: it suffices to prove that f ( x ) =

Original: ReLU( w ⊤ x + b ) is in H s for all w ∈ R d , b ∈ R .
Revised: ReLU( w ⊤ x + b ) is in H s for all w ∈ R d , b ∈ R .

Original: Since ReLU( a x ) = a ReLU( x ) for any a > 0 , we can assume that ∥ w ∥ 2 = 1 .
Revised: Since ReLU( a x ) = a ReLU( x ) for any a > 0 , we can assume that ∥ w ∥ 2 = 1 .

Original: Moreover, since the Sobolev spaces are rotationally invariant, we can assume that w = (1 , 0 , . . . , 0) ⊤ .
Revised: Moreover, since the Sobolev spaces are rotationally invariant, we can assume that w = (1 , 0 , . . . , 0) ⊤ .

Original: Then, f can be written as
Revised: Then, f can be written as

Original: [Equation]
Revised: [Equation]

Original: x 1 + b ) .
Revised: x 1 + b ) .

Original: If b ≤ − 1 or b ≥ 1 , then f ( x ) is a constant, and thus f ∈ H s ( S d − 1 ) for all s ∈ R .
Revised: If b ≤ − 1 or b ≥ 1 , then f ( x ) is a constant, and thus f ∈ H s ( S d − 1 ) for all s ∈ R .

Original: We assume − 1 < b < 1 .
Revised: We assume − 1 < b < 1 .

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: We define the function
Revised: We define the function

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Here, ffl
Revised: Here, ffl

Original: [Equation]
Revised: [Equation]

Original: ´
Revised: ´

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: ξ ) d ξ is the averaged integral, where | C ( x , t ) | is the
Revised: ξ ) d ξ is the averaged integral, where | C ( x , t ) | is the

Original: Lebesgue measure of C ( x , t ) .
Revised: Lebesgue measure of C ( x , t ) .

Original: Then, by (Barcel´o et al., 2020, Thm. 1.1), we have f ∈ H s ( S d − 1 )
Revised: Then, by (Barcel´o et al., 2020, Thm. 1.1), we have f ∈ H s ( S d − 1 )

Original: if and only if S s ( f ) is integrable.
Revised: if and only if S s ( f ) is integrable.

Original: We now show that S s ( f ) is integrable on both E 1 = { x ∈ S
Revised: We now show that S s ( f ) is integrable on both E 1 = { x ∈ S

Original: d − 1 | x 1 > − b } and E 2 =
Revised: d − 1 | x 1 > − b } and E 2 =

Original: { x ∈ S
Revised: { x ∈ S

Original: d − 1 | x 1 < − b } if s < 3 / 2 .
Revised: d − 1 | x 1 < − b } if s < 3 / 2 .

Original: First, we define the function h ( x ) = x
Revised: First, we define the function h ( x ) = x

Original: [Equation]
Revised: [Equation]

Original: + b.
Revised: + b.

Original: Then, h ∈ H s ( S d − 1 ) .
Revised: Then, h ∈ H s ( S d − 1 ) .

Original: If we can show that
Revised: If we can show that

Original: [Equation]
Revised: [Equation]

Original: ( h − f )( x
Revised: ( h − f )( x

Original: [Equation]
Revised: [Equation]

Original: is integrable on E 1 , then we have ˆ
Revised: is integrable on E 1 , then we have ˆ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Assume x ∈ E 1 , and let ρ be the minimum angular distance between x and any point in the set
Revised: Assume x ∈ E 1 , and let ρ be the minimum angular distance between x and any point in the set

Original: S = { ξ ∈ S d − 1 | ξ 1 = − b } , i.e., ρ = min ξ ∈ S arccos( ξ · x ) .
Revised: S = { ξ ∈ S d − 1 | ξ 1 = − b } , i.e., ρ = min ξ ∈ S arccos( ξ · x ) .

Original: Then, for 0 < t ≤ ρ , we clearly have
Revised: Then, for 0 < t ≤ ρ , we clearly have

Original: I t ( h − f )( x ) = 

Revised: I t ( h − f )( x ) =

Original: = ( h − f )( x ) .
Revised: = ( h − f )( x ) .

Original: Assume ρ < t < π .
Revised: Assume ρ < t < π .

Original: We divide C ( x , t ) into two parts C 1 and Cup to a Lebesgue null set, where C i =
Revised: We divide C ( x , t ) into two parts C 1 and Cup to a Lebesgue null set, where C i =

Original: C ( x , t ) ∩ E i .
Revised: C ( x , t ) ∩ E i .

Original: Then, h − f = 0 on C 1 .
Revised: Then, h − f = 0 on C 1 .

Original: Next, by Li (2011), we know that the measure of C 2 satisfies
Revised: Next, by Li (2011), we know that the measure of C 2 satisfies

Original: [Equation]
Revised: [Equation]

Original: where I is the regularized incomplete beta function and B is the incomplete beta function.
Revised: where I is the regularized incomplete beta function and B is the incomplete beta function.

Original: Here and throughout the proof, the constants in the big-
Revised: Here and throughout the proof, the constants in the big-

Original: Θ notations are independent of ρ or t , but possibly depend on b and d , which are fixed in the proof.
Revised: Θ notations are independent of ρ or t , but possibly depend on b and d , which are fixed in the proof.

Original: Moreover, we have the formula
Revised: Moreover, we have the formula

Original: [Equation]
Revised: [Equation]

Original: where F is the hypergeometric function that converges to 1 as t → ρ + (DLMF, sect.
Revised: where F is the hypergeometric function that converges to 1 as t → ρ + (DLMF, sect.

Original: 8.17, sect. 15.2).
Revised: 8.17, sect. 15.2).

Original: Hence, we have
Revised: Hence, we have

Original: [Equation]
Revised: [Equation]

Original: Now, by the way we defined h − f , there exist constants R 1 , R 2 > 0 such that
Revised: Now, by the way we defined h − f , there exist constants R 1 , R 2 > 0 such that

Original: [Equation]
Revised: [Equation]

Original: This gives us
Revised: This gives us

Original: [Equation]
Revised: [Equation]

Original: Now, we have
Revised: Now, we have

Original: [Equation]
Revised: [Equation]

Original: To integrate S s ( h − f ) over E 1 , we first change the coordinates and integrate over S d − 2 by fixing a ρ .
Revised: To integrate S s ( h − f ) over E 1 , we first change the coordinates and integrate over S d − 2 by fixing a ρ .

Original: The resulting integral is still in Θ( ρ 2 − 2 s + 1) .
Revised: The resulting integral is still in Θ( ρ 2 − 2 s + 1) .

Original: We then integrate over ρ and the result follows from the fact that a function in Θ( ρ 2 − 2 s + 1) is integrable near ρ = 

Revised: We then integrate over ρ and the result follows from the fact that a function in Θ( ρ 2 − 2 s + 1) is integrable near ρ =

Original: if and only if s < 3 / 2 .
Revised: if and only if s < 3 / 2 .

Original: This proves S s ( h − f ) is integrable on E 

Revised: This proves S s ( h − f ) is integrable on E

Original: if and only if s < 3 / 2 .
Revised: if and only if s < 3 / 2 .

Original: Note that if S s ( h − f ) is not integrable on E 1 , then S s ( f ) is neither integrable.
Revised: Note that if S s ( h − f ) is not integrable on E 1 , then S s ( f ) is neither integrable.

Original: This proves the proposition when s ≥ 3 / 2 .
Revised: This proves the proposition when s ≥ 3 / 2 .

Original: To see S s ( f ) is integrable over E 2 when s < 3 / 2 , we note that f can be rewritten as ˜ f − ˜ h , where
Revised: To see S s ( f ) is integrable over E 2 when s < 3 / 2 , we note that f can be rewritten as ˜ f − ˜ h , where

Original: ˜ f
Revised: ˜ f

Original: ( x ) = ReLU( − x 1 − b ) and ˜ h
Revised: ( x ) = ReLU( − x 1 − b ) and ˜ h

Original: ( x ) = − x 1 − b .
Revised: ( x ) = − x 1 − b .

Original: By the same argument, we have that
Revised: By the same argument, we have that

Original: S s ( f ) =
Revised: S s ( f ) =

Original: S s ( ˜ f − ˜ h ) is integrable on E 2 , which completes the proof.
Revised: S s ( ˜ f − ˜ h ) is integrable on E 2 , which completes the proof.

Original: E. F REQUENCY BIASING WITH A SQUARED S OBOLEV NORM AS THE LOSS FUNCTION
Revised: E. F REQUENCY BIASING WITH A SQUARED S OBOLEV NORM AS THE LOSS FUNCTION

Original: In this section, we prove Theorem 4 on Sobolev training.
Revised: In this section, we prove Theorem 4 on Sobolev training.

Original: Recall that we compute the H s -based loss based on eq.
Revised: Recall that we compute the H s -based loss based on eq.

Original: Proof of Theorem 4.
Revised: Proof of Theorem 4.

Original: Fix some 0 ≤ ℓ ≤ L .
Revised: Fix some 0 ≤ ℓ ≤ L .

Original: We can write
Revised: We can write

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where ω ℓ = (1 + ℓ ) 2 s and we used the fact that
Revised: where ω ℓ = (1 + ℓ ) 2 s and we used the fact that

Original: [Equation]
Revised: [Equation]

Original: otherwise .
Revised: otherwise .

Original: Next, we have
Revised: Next, we have

Original: [Equation]
Revised: [Equation]

Original: where the last equality follows from the Funk–Hecke formula.
Revised: where the last equality follows from the Funk–Hecke formula.

Original: Hence, the first term in eq.
Revised: Hence, the first term in eq.

Original: (71) can be written as 
Revised: (71) can be written as 

Original: [Equation]
Revised: [Equation]

Original: Moreover, the second term in eq.
Revised: Moreover, the second term in eq.

Original: (71) can be written as
Revised: (71) can be written as

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: ) where
Revised: ) where

Original: [Equation]
Revised: [Equation]

Original: Applying eq.
Revised: Applying eq.

Original: (72) recursively
Revised: (72) recursively

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Now, since H ∞ P s is self-adjoint and positive definite in ( R n , ⟨· , ·⟩ P s ) , by the way we pick η , we guarantee that I −
Revised: Now, since H ∞ P s is self-adjoint and positive definite in ( R n , ⟨· , ·⟩ P s ) , by the way we pick η , we guarantee that I −

Original: 2 η H ∞
Revised: 2 η H ∞

Original: P s is positive definite in ( R n , ⟨· , ·⟩ P s ) and hence
Revised: P s is positive definite in ( R n , ⟨· , ·⟩ P s ) and hence

Original: [Equation]
Revised: [Equation]

Original: This gives us ∥ ε 1 ∥ P s ≤ (cid:80) Lℓ =0 µ − 1 ℓ (cid:13)(cid:13)
Revised: This gives us ∥ ε 1 ∥ P s ≤ (cid:80) Lℓ =0 µ − 1 ℓ (cid:13)(cid:13)

Original: ε ℓ 1 (cid:13)(cid:13) P s , and the result follows from Theorem 1.
Revised: ε ℓ 1 (cid:13)(cid:13) P s , and the result follows from Theorem 1.

Original: While Theorem 4 captures the frequency biasing in squared H s loss training up to quadrature errors, analyzing the quadrature errors can be task-specific.
Revised: While Theorem 4 captures the frequency biasing in squared H s loss training up to quadrature errors, analyzing the quadrature errors can be task-specific.

Original: Therefore, studying the quadrature rules could be a direction of future research.
Revised: Therefore, studying the quadrature rules could be a direction of future research.

Original: Suppose we use Theorem 1 ′ in the proof, we can write down Theorem 4 in a form such that ϵ 2 ( k ) depends on the parameters of the NN.
Revised: Suppose we use Theorem 1 ′ in the proof, we can write down Theorem 4 in a form such that ϵ 2 ( k ) depends on the parameters of the NN.

Original: Theorem 4 ′ .
Revised: Theorem 4 ′ .

Original: Suppose g ∈ Π dL
Revised: Suppose g ∈ Π dL

Original: and Φ s is the loss function in eq.
Revised: and Φ s is the loss function in eq.

Original: (20) , where P s is positive definite and ℓ max ≥ L .
Revised: (20) , where P s is positive definite and ℓ max ≥ L .

Original: Under the assumptions of Theorem 1 ′ , if 1 − 2 ηµ ℓ (1 + ℓ ) 2 s > 0 for all 0 ≤ ℓ ≤ L , then with probability ≥ 1 − δ over the random initialization, we have
Revised: Under the assumptions of Theorem 1 ′ , if 1 − 2 ηµ ℓ (1 + ℓ ) 2 s > 0 for all 0 ≤ ℓ ≤ L , then with probability ≥ 1 − δ over the random initialization, we have

Original: [Equation]
Revised: [Equation]

Original: where ∥ ε 2 ( k ) ∥
Revised: where ∥ ε 2 ( k ) ∥

Original: P s is bounded by eq.
Revised: P s is bounded by eq.

Original: (51) with M P = M P s , and y ℓ
Revised: (51) with M P = M P s , and y ℓ

Original: =
Revised: =

Original: ( g ℓ ( x 1 ) , . . . , g ℓ ( x n )) ⊤ and ε 1 satisfies
Revised: ( g ℓ ( x 1 ) , . . . , g ℓ ( x n )) ⊤ and ε 1 satisfies

Original: [Equation]
Revised: [Equation]

Original: Moreover, we note that the remark after the proof of Theorem 2 applies here as well.
Revised: Moreover, we note that the remark after the proof of Theorem 2 applies here as well.

Original: In particular, using the relation that
Revised: In particular, using the relation that

Original: [Equation]
Revised: [Equation]

Original: we can rewrite eq.
Revised: we can rewrite eq.

Original: (21) into
Revised: (21) into

Original: [Equation]
Revised: [Equation]

Original: where | ε 1 | ≤ ∥ ε 1 ∥ c , | ε 2 ( k ) | ≤ ∥ ε 2 ( k ) ∥
Revised: where | ε 1 | ≤ ∥ ε 1 ∥ c , | ε 2 ( k ) | ≤ ∥ ε 2 ( k ) ∥

Original: c are as in Theorem 4, and | ε 3 ( k ) | ≤ (cid:112) | E c ( g − N k ) 2 | .
Revised: c are as in Theorem 4, and | ε 3 ( k ) | ≤ (cid:112) | E c ( g − N k ) 2 | .

Original: By eq.
Revised: By eq.

Original: (57), we can directly relate (cid:13) (cid:13)(cid:13)(cid:80) Lℓ =0 (cid:0)
Revised: (57), we can directly relate (cid:13) (cid:13)(cid:13)(cid:80) Lℓ =0 (cid:0)

Original: 1 − 2 ηµ ℓ (1 + ℓ ) 2 s (cid:1)
Revised: 1 − 2 ηµ ℓ (1 + ℓ ) 2 s (cid:1)

Original: k y ℓ (cid:13) (cid:13)(cid:13) c to (cid:113)(cid:80)
Revised: k y ℓ (cid:13) (cid:13)(cid:13) c to (cid:113)(cid:80)

Original: Lℓ =

Revised: Lℓ =

Original: (1 − 2 ηµ ℓ (1 + ℓ ) 2 s ) 2 k ∥ g ℓ ∥ 2 L 2 up to some quadrature errors.
Revised: (1 − 2 ηµ ℓ (1 + ℓ ) 2 s ) 2 k ∥ g ℓ ∥ 2 L 2 up to some quadrature errors.

Original: Therefore, the generalization error with respect to the uniform distribution can be written as
Revised: Therefore, the generalization error with respect to the uniform distribution can be written as

Original: [Equation]
Revised: [Equation]

Original: where E NTK is an error that can be made arbitrarily small be taking κ small enough and m large enough and E quad involves only quadrature errors and can be made arbitrarily small if we make the quadrature rule accurate enough.
Revised: where E NTK is an error that can be made arbitrarily small be taking κ small enough and m large enough and E quad involves only quadrature errors and can be made arbitrarily small if we make the quadrature rule accurate enough.

Original: F E XPERIMENTAL DETAILS
Revised: F E XPERIMENTAL DETAILS

Original: We now present the details of the three experiments in Section 6.
Revised: We now present the details of the three experiments in Section 6.

Original: F. L EARNING TRIGONOMETRIC POLYNOMIALS ON THE UNIT CIRCLE
Revised: F. L EARNING TRIGONOMETRIC POLYNOMIALS ON THE UNIT CIRCLE

Original: In section 6.1, we train a NN with data derived from sampling a trigonometric polynomial at nonuniform points on the unit circle.
Revised: In section 6.1, we train a NN with data derived from sampling a trigonometric polynomial at nonuniform points on the unit circle.

Original: The n = 1140 nonuniform data points { x i } for this test are generated by taking the union of three sets of equally spaced points, as shown in Figure 1.
Revised: The n = 1140 nonuniform data points { x i } for this test are generated by taking the union of three sets of equally spaced points, as shown in Figure 1.

Original: The data set { (cos( θ i ) , sin( θ i )) } contains 100 equally spaced nodes sampled from θ ∈ (0 , 2 π ] , superimposed with 40 equally spaced nodes sampled from θ ∈ [0 , 0 .
Revised: The data set { (cos( θ i ) , sin( θ i )) } contains 100 equally spaced nodes sampled from θ ∈ (0 , 2 π ] , superimposed with 40 equally spaced nodes sampled from θ ∈ [0 , 0 .

Original: 3 π ] and 1000 equally spaced nodes sampled from θ ∈ [1 . 4 π, 1 . 8 π ] (see Figure 1, left).
Revised: 3 π ] and 1000 equally spaced nodes sampled from θ ∈ [1 . 4 π, 1 . 8 π ] (see Figure 1, left).

Original: We construct the quadrature weights c j by minimizing (cid:80) nj =1 c 2 j , under the constraints that the c i ’s are positive and the quadrature rule is exact on Π 255 , i.e.,
Revised: We construct the quadrature weights c j by minimizing (cid:80) nj =1 c 2 j , under the constraints that the c i ’s are positive and the quadrature rule is exact on Π 255 , i.e.,

Original: E c ( f ) = 0 for all f ∈ Π 255 .
Revised: E c ( f ) = 0 for all f ∈ Π 255 .

Original: Note that due to linearity of the quadrature rule, it suffices to check a finite set of linear constraints
Revised: Note that due to linearity of the quadrature rule, it suffices to check a finite set of linear constraints

Original: [Equation]
Revised: [Equation]

Original: This computation method is proposed in (Mhaskar et al., 2000).
Revised: This computation method is proposed in (Mhaskar et al., 2000).

Original: Here, 55 is selected to be an integer close to the maximum degree ℓ such that there exists a positive quadrature rule exact on Π dℓ .
Revised: Here, 55 is selected to be an integer close to the maximum degree ℓ such that there exists a positive quadrature rule exact on Π dℓ .

Original: In the upper-bounds of the quadrature errors in Theorem 3, there exist terms
Revised: In the upper-bounds of the quadrature errors in Theorem 3, there exist terms

Original: [Equation]
Revised: [Equation]

Original: where L is a constant bandlimit of the target function.
Revised: where L is a constant bandlimit of the target function.

Original: By requiring the quadrature rule to be exact on Π 2 ℓ 0 , we guarantee that γ n,ℓ 0 = 0 .
Revised: By requiring the quadrature rule to be exact on Π 2 ℓ 0 , we guarantee that γ n,ℓ 0 = 0 .

Original: Hence, by requiring ℓ 0 to be large, we can heuristically make γ n,ℓ smaller for a moderate ℓ .
Revised: Hence, by requiring ℓ 0 to be large, we can heuristically make γ n,ℓ smaller for a moderate ℓ .

Original: This allows us to show that the upper-bounds in eq.
Revised: This allows us to show that the upper-bounds in eq.

Original: (76) are small by balancing the term that involves ℓ − 1 , which vanishes as ℓ → ∞ , and the term that involves γ n,ℓ , which increases as ℓ → ∞ .
Revised: (76) are small by balancing the term that involves ℓ − 1 , which vanishes as ℓ → ∞ , and the term that involves γ n,ℓ , which increases as ℓ → ∞ .

Original: This justifies why we compute the quadrature rule by requiring it to be exact on spherical harmonics of degree as large as possible.
Revised: This justifies why we compute the quadrature rule by requiring it to be exact on spherical harmonics of degree as large as possible.

Original: The experiment consists of two parts.
Revised: The experiment consists of two parts.

Original: First, we compare the effects of training with the loss function Φ based on eq.
Revised: First, we compare the effects of training with the loss function Φ based on eq.

Original: (1) against the squared L 2 loss function (cid:101) Φ in eq.
Revised: (1) against the squared L 2 loss function (cid:101) Φ in eq.

Original: (3), where
Revised: (3), where

Original: [Equation]
Revised: [Equation]

Original: We define the target function to be
Revised: We define the target function to be

Original: [Equation]
Revised: [Equation]

Original: where ˜ g ( θ ) =
Revised: where ˜ g ( θ ) =

Original: g ( x ( θ )) .
Revised: g ( x ( θ )) .

Original: We set up two 2-layer ReLU-activated NNs with 5 × 10 4 hidden neurons in each layer and train them using the same training data and gradient descent procedure, except with different loss functions Φ and (cid:101) Φ .
Revised: We set up two 2-layer ReLU-activated NNs with 5 × 10 4 hidden neurons in each layer and train them using the same training data and gradient descent procedure, except with different loss functions Φ and (cid:101) Φ .

Original: In Figure 5, we showed the evolution of the NNs trained using Φ and ˜Φ , respectively.
Revised: In Figure 5, we showed the evolution of the NNs trained using Φ and ˜Φ , respectively.

Original: While the NN trained with Φ approximates the function very well in the region where the training data are dense (i.e., where θ ∈ [1 . 4 π, 1 . 8 π ] ≈ [4 . 40 , 5 . 65] ), the NN trained with ˜ Φ provides a better overall approximation on the entire domain and demonstrates frequency bias much more clearly.
Revised: While the NN trained with Φ approximates the function very well in the region where the training data are dense (i.e., where θ ∈ [1 . 4 π, 1 . 8 π ] ≈ [4 . 40 , 5 . 65] ), the NN trained with ˜ Φ provides a better overall approximation on the entire domain and demonstrates frequency bias much more clearly.

Original: To evaluate the frequency loss, we collect 100 uniform samples from N ( x ) and g ( x ) and compute the Fourier coefficients (cid:98) N ( ℓ ) and ˆ g ( ℓ ) such that functions
Revised: To evaluate the frequency loss, we collect 100 uniform samples from N ( x ) and g ( x ) and compute the Fourier coefficients (cid:98) N ( ℓ ) and ˆ g ( ℓ ) such that functions

Original: [Equation]
Revised: [Equation]

Original: ≈ 

Revised: ≈

Original: (cid:88) ℓ
Revised: (cid:88) ℓ

Original: = ˆ g ( ℓ ) e iℓθ , where (cid:101) N ( θ ) =
Revised: = ˆ g ( ℓ ) e iℓθ , where (cid:101) N ( θ ) =

Original: N ( x ( θ )) .
Revised: N ( x ( θ )) .

Original: The frequency loss | (cid:98) N ( ℓ ) − ˆ g ( ℓ ) | estimates how g ( x ) is approximated by
Revised: The frequency loss | (cid:98) N ( ℓ ) − ˆ g ( ℓ ) | estimates how g ( x ) is approximated by

Original: N ( x ) at the frequency ℓ when training with the different loss functions (see Figure 1, middle).
Revised: N ( x ) at the frequency ℓ when training with the different loss functions (see Figure 1, middle).

Original: In addition, we also train the NN to learn each individual frequency with the training data coming from g ℓ = sin( ℓθ ) and count the number of iterations it takes to obtain (cid:101) Φ( W ) < 1 . 

Revised: In addition, we also train the NN to learn each individual frequency with the training data coming from g ℓ = sin( ℓθ ) and count the number of iterations it takes to obtain (cid:101) Φ( W ) < 1 .

Original: × 10 − 3 (see Figure 1, right).
Revised: × 10 − 3 (see Figure 1, right).

Original: The second part of the experiment focuses on NN training with a discretized squared Sobolev norm as the loss function.
Revised: The second part of the experiment focuses on NN training with a discretized squared Sobolev norm as the loss function.

Original: More precisely, we fix some s ∈ R and consider the Sobolev loss function eq.
Revised: More precisely, we fix some s ∈ R and consider the Sobolev loss function eq.

Original: For S 1 , we have N ( d, ℓ ) =
Revised: For S 1 , we have N ( d, ℓ ) =

Original: if ℓ > 0 .
Revised: if ℓ > 0 .

Original: We take Y ℓ, 1 ( x ) = sin( ℓθ ) / √ 2 π and
Revised: We take Y ℓ, 1 ( x ) = sin( ℓθ ) / √ 2 π and

Original: Y ℓ, 2 ( x ) = cos( ℓθ ) / √ 2 π , where the √ 2 π factor is a normalization factor.
Revised: Y ℓ, 2 ( x ) = cos( ℓθ ) / √ 2 π , where the √ 2 π factor is a normalization factor.

Original: [Figure]
Revised: [Figure]

Original: Iteration
Revised: Iteration

Original: [Figure]
Revised: [Figure]

Original: Iteration
Revised: Iteration

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Iteration =
Revised: Iteration =

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: We set ℓ max = 30 in eq.
Revised: We set ℓ max = 30 in eq.

Original: (20) and learn g with different s values ranging from − 1 to 4 .
Revised: (20) and learn g with different s values ranging from − 1 to 4 .

Original: For each s , we compute the frequency loss | (cid:98) N ( ℓ ) − ˆ g ( ℓ ) | after different numbers of epochs.
Revised: For each s , we compute the frequency loss | (cid:98) N ( ℓ ) − ˆ g ( ℓ ) | after different numbers of epochs.

Original: As s increases, the frequency loss for higher frequencies decays faster (see Figure 6).
Revised: As s increases, the frequency loss for higher frequencies decays faster (see Figure 6).

Original: In particular, we see in Figure 2, the “rainbow” plot, that when s = − 1 , the lower-frequency losses are much smaller than the higher ones after 5000 iterations, while when s = 3 the higher frequencies are learned faster than the lower ones.
Revised: In particular, we see in Figure 2, the “rainbow” plot, that when s = − 1 , the lower-frequency losses are much smaller than the higher ones after 5000 iterations, while when s = 3 the higher frequencies are learned faster than the lower ones.

Original: F. L EARNING SPHERICAL HARMONICS ON THE UNIT SPHERE
Revised: F. L EARNING SPHERICAL HARMONICS ON THE UNIT SPHERE

Original: In section 6.2, we train a NN with data derived from sampling function defined on S 2 at nonuniform points.
Revised: In section 6.2, we train a NN with data derived from sampling function defined on S 2 at nonuniform points.

Original: The training data is a maximum determinant set of 2500 points that comes from the so-called “spherepts” dataset (Wright & Michaels, 2015).
Revised: The training data is a maximum determinant set of 2500 points that comes from the so-called “spherepts” dataset (Wright & Michaels, 2015).

Original: In this experiment, the target function is
Revised: In this experiment, the target function is

Original: [Equation]
Revised: [Equation]

Original: where Y 2 ℓ, 0 is the normalized zonal spherical harmonic function of degree 2 ℓ .
Revised: where Y 2 ℓ, 0 is the normalized zonal spherical harmonic function of degree 2 ℓ .

Original: Therefore, the spherical harmonic coefficients of g defined in eq.
Revised: Therefore, the spherical harmonic coefficients of g defined in eq.

Original: (4) satisfy ˆ g ℓ,p = 1 if p = 0 and ℓ = 2 , 4 , . .
Revised: (4) satisfy ˆ g ℓ,p = 1 if p = 0 and ℓ = 2 , 4 , . .

Original: . , 30 , and ˆ g ℓ,p = 

Revised: . , 30 , and ˆ g ℓ,p =

Original: otherwise.
Revised: otherwise.

Original: We then train an NN with the squared H s norm as the loss function (see eq.
Revised: We then train an NN with the squared H s norm as the loss function (see eq.

Original: for s = − 1 , 0 , 2 . 5 .
Revised: for s = − 1 , 0 , 2 . 5 .

Original: By Theorem 4, we need ℓ max ≥ L = 30 .
Revised: By Theorem 4, we need ℓ max ≥ L = 30 .

Original: We set ℓ max = 40 in eq.
Revised: We set ℓ max = 40 in eq.

Original: (20), assuming that the bandwidth L is not known a priori.
Revised: (20), assuming that the bandwidth L is not known a priori.

Original: We observe frequency biasing in this experiment by considering | (cid:98) N ℓ,p − ˆ g ℓ,p | after each epoch for ℓ = 4 , 10 , 20 .
Revised: We observe frequency biasing in this experiment by considering | (cid:98) N ℓ,p − ˆ g ℓ,p | after each epoch for ℓ = 4 , 10 , 20 .

Original: We confirm that low frequencies of g are captured earlier in training than high frequencies when s = − 1 and s = 

Revised: We confirm that low frequencies of g are captured earlier in training than high frequencies when s = − 1 and s =

Original: (see Figure 3, left and middle), and that this frequency biasing phenomena can be counterbalanced by taking s = 2 .(see Figure 3, right).
Revised: (see Figure 3, left and middle), and that this frequency biasing phenomena can be counterbalanced by taking s = 2 .(see Figure 3, right).

Original: F. T EST ON A UTOENCODER
Revised: F. T EST ON A UTOENCODER

Original: Autoencoders can be used as a generative model to randomly generate new data that is similar to the training data.
Revised: Autoencoders can be used as a generative model to randomly generate new data that is similar to the training data.

Original: In this experiment, we use Sobolev-based norms to improve NN training for producing new images of digits that match the MNIST dataset.
Revised: In this experiment, we use Sobolev-based norms to improve NN training for producing new images of digits that match the MNIST dataset.

Original: In our final experiment, we use the same autoencoder architecture as in (Chollet, 2016), except we train the autoencoder with a different loss function (see section 6.3).
Revised: In our final experiment, we use the same autoencoder architecture as in (Chollet, 2016), except we train the autoencoder with a different loss function (see section 6.3).

Original: Here, for training images { x i } , a standard loss function can be
Revised: Here, for training images { x i } , a standard loss function can be

Original: [Equation]
Revised: [Equation]

Original: where µ is the distribution of the training images and dist ( N ( x i ) , x i ) is a distance between the output of the NN given by N ( x ) and the image x .
Revised: where µ is the distribution of the training images and dist ( N ( x i ) , x i ) is a distance between the output of the NN given by N ( x ) and the image x .

Original: We select the distance metric to measure the difference between N ( x ) and x as
Revised: We select the distance metric to measure the difference between N ( x ) and x as

Original: [Equation]
Revised: [Equation]

Original: where ∥ · ∥ F denotes the matrix Frobenius norm.
Revised: where ∥ · ∥ F denotes the matrix Frobenius norm.

Original: The distance metric in eq.
Revised: The distance metric in eq.

Original: (77) can be viewed as a discretization of the continuous L 2 norm.
Revised: (77) can be viewed as a discretization of the continuous L 2 norm.

Original: That is, if one imagines generating a continuous function x : [0 , 1] 2 → [0 , ∞ ) that interpolates an image as well as a function that interpolates the NN, then
Revised: That is, if one imagines generating a continuous function x : [0 , 1] 2 → [0 , ∞ ) that interpolates an image as well as a function that interpolates the NN, then

Original: [Equation]
Revised: [Equation]

Original: where N pixel is the total number of pixels of the image x .
Revised: where N pixel is the total number of pixels of the image x .

Original: In this continuous viewpoint, the H s norm is given by (assuming that the continuous interpolating functions x and N ( x ) are constructed with periodic boundary conditions)
Revised: In this continuous viewpoint, the H s norm is given by (assuming that the continuous interpolating functions x and N ( x ) are constructed with periodic boundary conditions)

Original: [Equation]
Revised: [Equation]

Original: where F l , F r are the left and right 2D-DFT matrices, respectively, ( S s ) jℓ
Revised: where F l , F r are the left and right 2D-DFT matrices, respectively, ( S s ) jℓ

Original: = (1 + j 2 + ℓ 2 ) s/ 2 , and ‘ ◦ ’ is the Hadamard product.
Revised: = (1 + j 2 + ℓ 2 ) s/ 2 , and ‘ ◦ ’ is the Hadamard product.

Original: Hence, if we define vec ( A ) to be the vector obtained by reshaping a matrix A using the column major order, then we have
Revised: Hence, if we define vec ( A ) to be the vector obtained by reshaping a matrix A using the column major order, then we have

Original: [Equation]
Revised: [Equation]

Original: where ⊗ is the Kronecker product of two matrices.
Revised: where ⊗ is the Kronecker product of two matrices.

Original: Setting J s = diag ( vec ( S s ))
Revised: Setting J s = diag ( vec ( S s ))

Original: ( F r ⊗ F l ) , the loss function in our NN training can be written as
Revised: ( F r ⊗ F l ) , the loss function in our NN training can be written as

Original: [Equation]
Revised: [Equation]

Original: where I is the n -by- n identity matrix and u , y are the vectors of length n × N pixel given by u = ( vec ( N
Revised: where I is the n -by- n identity matrix and u , y are the vectors of length n × N pixel given by u = ( vec ( N

Original: [Equation]
Revised: [Equation]

Original: x 1 )) ⊤ , . .
Revised: x 1 )) ⊤ , . .

Original: , vec ( N ( x n )) ⊤ ) ⊤ and y =
Revised: , vec ( N ( x n )) ⊤ ) ⊤ and y =

Original: ( vec ( x 1 ) ⊤ , . .
Revised: ( vec ( x 1 ) ⊤ , . .

Original: , vec ( x n ) ⊤ ) ⊤ .
Revised: , vec ( x n ) ⊤ ) ⊤ .

Original: Hence, the discrete NTK matrix is given by n − 1 H ∞ ( I ⊗ J ⊤ s J s ) , where the ( i, j ) th sub-block is H ∞ ij = (cid:10) ∂ vec ( N ( x i ; W ))
Revised: Hence, the discrete NTK matrix is given by n − 1 H ∞ ( I ⊗ J ⊤ s J s ) , where the ( i, j ) th sub-block is H ∞ ij = (cid:10) ∂ vec ( N ( x i ; W ))

Original: ∂ W , ∂ vec ( N ( x j ; W ))
Revised: ∂ W , ∂ vec ( N ( x j ; W ))

Original: ∂ W (cid:11) for i, j = 1 , . . . , n .
Revised: ∂ W (cid:11) for i, j = 1 , . . . , n .

Original: Here, (cid:10) ∂ vec ( N ( x i ; W ))
Revised: Here, (cid:10) ∂ vec ( N ( x i ; W ))

Original: ∂ W , ∂ vec ( N ( x j ; W ))
Revised: ∂ W , ∂ vec ( N ( x j ; W ))

Original: ∂ W (cid:11) is interpreted as the N pixel -by- N pixel matrix whose ( i ′ , j ′ )
Revised: ∂ W (cid:11) is interpreted as the N pixel -by- N pixel matrix whose ( i ′ , j ′ )

Original: th entry is (cid:10) ∂ [ vec ( N ( x i ; W ))
Revised: th entry is (cid:10) ∂ [ vec ( N ( x i ; W ))

Original: i ′ ]
Revised: i ′ ]

Original: ∂ W , ∂ [ vec ( N ( x j ; W ))
Revised: ∂ W , ∂ [ vec ( N ( x j ; W ))

Original: j ′ ]
Revised: j ′ ]

Original: ∂ W (cid:11) .
Revised: ∂ W (cid:11) .

Original: This means that the frequency biasing behavior during NN training is directly affected by the choice of s .
Revised: This means that the frequency biasing behavior during NN training is directly affected by the choice of s .

Original: Alternatively, one can consider this problem more abstractly from an operator learning perspective, which is presented in appendix G. We remark that while eq.
Revised: Alternatively, one can consider this problem more abstractly from an operator learning perspective, which is presented in appendix G. We remark that while eq.

Original: (79) is a mathematically equivalent expression for the loss function that allows us to easily express the NTK, in practice, we implement the loss function based on eq.
Revised: (79) is a mathematically equivalent expression for the loss function that allows us to easily express the NTK, in practice, we implement the loss function based on eq.

Original: (78) using a 2D FFT protocol.
Revised: (78) using a 2D FFT protocol.

Original: We use the same autoencoder architecture as in (Chollet, 2016), except with the loss function in eq.
Revised: We use the same autoencoder architecture as in (Chollet, 2016), except with the loss function in eq.

Original: (79) for s = − 1 , 0 , 1 .
Revised: (79) for s = − 1 , 0 , 1 .

Original: We train the autoencoder using mini-batch gradient descent with batch size equal to 256 .
Revised: We train the autoencoder using mini-batch gradient descent with batch size equal to 256 .

Original: We first pollute the training images with low-frequency noise and train the NN, hoping that the trained NN will act as filter for the noise.
Revised: We first pollute the training images with low-frequency noise and train the NN, hoping that the trained NN will act as filter for the noise.

Original: We see that training the NN with eq.
Revised: We see that training the NN with eq.

Original: (79) for s = 1 gives us the best results due to the high-frequency biasing induced by the choice of the loss function.
Revised: (79) for s = 1 gives us the best results due to the high-frequency biasing induced by the choice of the loss function.

Original: Although H ∞ is low frequency biasing, the high-frequency biasing of
Revised: Although H ∞ is low frequency biasing, the high-frequency biasing of

Original: J ⊤ s J s dominates for sufficiently large s .
Revised: J ⊤ s J s dominates for sufficiently large s .

Original: In that case, the low-frequency noise barely changes the training in the earlier epochs as the low-frequency components of the residual correspond to small eigenvalues of H ∞ ( I ⊗ J ⊤ s J s ) .
Revised: In that case, the low-frequency noise barely changes the training in the earlier epochs as the low-frequency components of the residual correspond to small eigenvalues of H ∞ ( I ⊗ J ⊤ s J s ) .

Original: Similar results are discussed in (Engquist et al., 2020) and (Zhu et al., 2021) in the inverse problem and image processing contexts, respectively.
Revised: Similar results are discussed in (Engquist et al., 2020) and (Zhu et al., 2021) in the inverse problem and image processing contexts, respectively.

Original: The opposite phenomenon occurs when we add high-frequency noise (see Figure 4, bottom row).
Revised: The opposite phenomenon occurs when we add high-frequency noise (see Figure 4, bottom row).

Original: Since H ∞ by itself makes the NN training procedure bias towards low-frequencies, the output for s = 0 does already filter high-frequency noise.
Revised: Since H ∞ by itself makes the NN training procedure bias towards low-frequencies, the output for s = 0 does already filter high-frequency noise.

Original: Since J ⊤ s J s for s < 0 further biases towards lowfrequencies, one can obtain better high-frequency filters.
Revised: Since J ⊤ s J s for s < 0 further biases towards lowfrequencies, one can obtain better high-frequency filters.

Original: We observe that the best denoising results for the autoencoder comes from selecting s = − 1 (see Figure 4, bottom row).
Revised: We observe that the best denoising results for the autoencoder comes from selecting s = − 1 (see Figure 4, bottom row).

Original: G NTK AND F REQUENCY B IAS IN O PERATOR L EARNING
Revised: G NTK AND F REQUENCY B IAS IN O PERATOR L EARNING

Original: We saw how using the H s -based losses can help us tune frequency biasing in training the autoencoder (see Sec. 6.3).
Revised: We saw how using the H s -based losses can help us tune frequency biasing in training the autoencoder (see Sec. 6.3).

Original: In fact, in training the autoencoder, we are learning the identity operator on the space of images.
Revised: In fact, in training the autoencoder, we are learning the identity operator on the space of images.

Original: In this section, we briefly mention the NTK associated with operator learning and its consequences.
Revised: In this section, we briefly mention the NTK associated with operator learning and its consequences.

Original: Let L be a linear operator on L 2 ( S d − 1 ) .
Revised: Let L be a linear operator on L 2 ( S d − 1 ) .

Original: Let D = { f 1 , . . .
Revised: Let D = { f 1 , . . .

Original: , f N } be a finite subset of L 2 ( S d − 1 ) , and x 1 , . . .
Revised: , f N } be a finite subset of L 2 ( S d − 1 ) , and x 1 , . . .

Original: , x M be distinct “samplers” in S d − 1 .
Revised: , x M be distinct “samplers” in S d − 1 .

Original: Since L is linear, without loss of generality, we assume (cid:13)(cid:13)
Revised: Since L is linear, without loss of generality, we assume (cid:13)(cid:13)

Original: ( f i ( x 1 ) , . . .
Revised: ( f i ( x 1 ) , . . .

Original: , f i ( x M ))
Revised: , f i ( x M ))

Original: ⊤ (cid:13)(cid:13) 

Revised: ⊤ (cid:13)(cid:13)

Original: = 1 for i = 1 , . . .
Revised: = 1 for i = 1 , . . .

Original: , N .
Revised: , N .

Original: Given a function f , we use f ( x ) to denote the vector ( f ( x 1 ) , . . .
Revised: Given a function f , we use f ( x ) to denote the vector ( f ( x 1 ) , . . .

Original: , f ( x M )) ⊤ .
Revised: , f ( x M )) ⊤ .

Original: We consider a fully-connected two-layer
Revised: We consider a fully-connected two-layer

Original: ReLU NN, N = ( N 1 , . . . , N M ) , that takes M inputs and produces M outputs.
Revised: ReLU NN, N = ( N 1 , . . . , N M ) , that takes M inputs and produces M outputs.

Original: The goal of training is to learn the linear operator L on D .
Revised: The goal of training is to learn the linear operator L on D .

Original: That is, we want N j ( f ( x 1 ) , . . .
Revised: That is, we want N j ( f ( x 1 ) , . . .

Original: , f ( x M ))
Revised: , f ( x M ))

Original: ≈ ( L f )( x j ) .
Revised: ≈ ( L f )( x j ) .

Original: In other words, given the samples f ( x ) of a function f , we want the NN to output the values of L f at x 1 , . . .
Revised: In other words, given the samples f ( x ) of a function f , we want the NN to output the values of L f at x 1 , . . .

Original: , x M , i.e., ( L f )( x ) .
Revised: , x M , i.e., ( L f )( x ) .

Original: As before, we let P be an M × M symmetric positive definite matrix that measures the distance between two L 2 functions given their samples at x 1 , . . .
Revised: As before, we let P be an M × M symmetric positive definite matrix that measures the distance between two L 2 functions given their samples at x 1 , . . .

Original: , x M .
Revised: , x M .

Original: That is,
Revised: That is,

Original: [Equation]
Revised: [Equation]

Original: We then consider a loss function of the operator NN given by
Revised: We then consider a loss function of the operator NN given by

Original: [Equation]
Revised: [Equation]

Original: where N ( f i ) :
Revised: where N ( f i ) :

Original: =
Revised: =

Original: ( N 1 ( f i ) , . . .
Revised: ( N 1 ( f i ) , . . .

Original: , N M ( f i ))
Revised: , N M ( f i ))

Original: ⊤ in which N j ( f i ) denotes the j th output of the NN when we input f i .
Revised: ⊤ in which N j ( f i ) denotes the j th output of the NN when we input f i .

Original: That is, N j ( f i ) =
Revised: That is, N j ( f i ) =

Original: N j ( f i ( x )) .
Revised: N j ( f i ( x )) .

Original: Let w (1) i,j be the i th weight in the j th neuron of the first hidden layer, 1 ≤ i ≤ M and 1 ≤ j ≤ m , where m is the number of hidden neurons.
Revised: Let w (1) i,j be the i th weight in the j th neuron of the first hidden layer, 1 ≤ i ≤ M and 1 ≤ j ≤ m , where m is the number of hidden neurons.

Original: Let b j be the bias term of the j th neuron in the hidden layer.
Revised: Let b j be the bias term of the j th neuron in the hidden layer.

Original: Let w (2) i,j be the i th weight in the j th neuron of the output layer, so 1 ≤ i ≤ m and 1 ≤ j ≤ M .
Revised: Let w (2) i,j be the i th weight in the j th neuron of the output layer, so 1 ≤ i ≤ m and 1 ≤ j ≤ M .

Original: We assume the same initialization scheme.
Revised: We assume the same initialization scheme.

Original: That is, w (1) i,j are initialized from i.i.d.
Revised: That is, w (1) i,j are initialized from i.i.d.

Original: Gaussian and b j are initialized to 0 , and they are updated during training, whereas w (2) i,j are initialized from i.i.d.
Revised: Gaussian and b j are initialized to 0 , and they are updated during training, whereas w (2) i,j are initialized from i.i.d.

Original: Rademacher random variables and are not updated during training.
Revised: Rademacher random variables and are not updated during training.

Original: In the derivation of the NTK below, we let W to be the vector of all trainable weights and biases, w (1) i,j and b j , enumerated in an (arbitrary) fixed order.
Revised: In the derivation of the NTK below, we let W to be the vector of all trainable weights and biases, w (1) i,j and b j , enumerated in an (arbitrary) fixed order.

Original: This allows us to write
Revised: This allows us to write

Original: W = ( w 1 , . . . , w K ) ⊤ , where K
Revised: W = ( w 1 , . . . , w K ) ⊤ , where K

Original: = m ( M + 1) .
Revised: = m ( M + 1) .

Original: We assume that the gradient flow algorithm is used to train the NN, i.e., d W dt = − ∂ Φ ∂ W .
Revised: We assume that the gradient flow algorithm is used to train the NN, i.e., d W dt = − ∂ Φ ∂ W .

Original: We define the vector of labels by
Revised: We define the vector of labels by

Original: [Equation]
Revised: [Equation]

Original: Similarly, we define the vector of the NN outputs by
Revised: Similarly, we define the vector of the NN outputs by

Original: [Equation]
Revised: [Equation]

Original: ) ⊤ , u = ( u ⊤
Revised: ) ⊤ , u = ( u ⊤

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: u ⊤ N ) ⊤ .
Revised: u ⊤ N ) ⊤ .

Original: Our goal is to understand ddt ( y − u ) and write it in terms of y − u .
Revised: Our goal is to understand ddt ( y − u ) and write it in terms of y − u .

Original: To this end, we first consider
Revised: To this end, we first consider

Original: [Equation]
Revised: [Equation]

Original: y i
Revised: y i

Original: [Equation]
Revised: [Equation]

Original: u i
Revised: u i

Original: [Equation]
Revised: [Equation]

Original: Denote by w k the k th entry of W , where 1 ≤ k ≤ K .
Revised: Denote by w k the k th entry of W , where 1 ≤ k ≤ K .

Original: Then, by the chain rule, the k th entry of ∂ Φ( W ) ∂ W can be written as
Revised: Then, by the chain rule, the k th entry of ∂ Φ( W ) ∂ W can be written as

Original: [Equation]
Revised: [Equation]

Original: Hence, we have ∂
Revised: Hence, we have ∂

Original: [Equation]
Revised: [Equation]

Original: W )
Revised: W )

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Now, combining eq.
Revised: Now, combining eq.

Original: (81) with eq.
Revised: (81) with eq.

Original: (82), we have
Revised: (82), we have

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: which gives us
Revised: which gives us

Original: [Equation]
Revised: [Equation]

Original: The derivation so far does not rely on the architecture of the NN.
Revised: The derivation so far does not rely on the architecture of the NN.

Original: In fact, it holds for the loss function Φ defined in eq.
Revised: In fact, it holds for the loss function Φ defined in eq.

Original: (80) with any abstract function N ( f i ; W ) .
Revised: (80) with any abstract function N ( f i ; W ) .

Original: Now, we exploit the NN architecture to study J ⊤ i J i ′ .
Revised: Now, we exploit the NN architecture to study J ⊤ i J i ′ .

Original: Consider the ( j, j ′ )
Revised: Consider the ( j, j ′ )

Original: th entry of J ⊤ i J i ′ .
Revised: th entry of J ⊤ i J i ′ .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: where w (1) k
Revised: where w (1) k

Original: = ( w (1)1 ,k , . . .
Revised: = ( w (1)1 ,k , . . .

Original: , w (1) M,k ) is the collection of weights on the k th hidden neuron.
Revised: , w (1) M,k ) is the collection of weights on the k th hidden neuron.

Original: Now, note that w (2) k,j w (2) k,j ′ =
Revised: Now, note that w (2) k,j w (2) k,j ′ =

Original: if j = j ′ and is a Rademacher random variable when j ̸ = j ′ .
Revised: if j = j ′ and is a Rademacher random variable when j ̸ = j ′ .

Original: Hence, by the way we initialize w (1) i,j , we have
Revised: Hence, by the way we initialize w (1) i,j , we have

Original: [Equation]
Revised: [Equation]

Original: where the convergence is entry-wise.
Revised: where the convergence is entry-wise.

Original: Now, suppose we define the N × N matrix H ∞ as
Revised: Now, suppose we define the N × N matrix H ∞ as

Original: [Equation]
Revised: [Equation]

Original: That is, we define H ∞ as in the function learning case, except that the inputs are { f i ( x ) } Ni =1 instead of { x i } ni =1 .
Revised: That is, we define H ∞ as in the function learning case, except that the inputs are { f i ( x ) } Ni =1 instead of { x i } ni =1 .

Original: By eq. (85), we have
Revised: By eq. (85), we have

Original: [Equation]
Revised: [Equation]

Original: where the convergence is entrywise.
Revised: where the convergence is entrywise.

Original: Hence, if we discretize the gradient flow algorithm and apply gradient descent with step size η , then by eq.
Revised: Hence, if we discretize the gradient flow algorithm and apply gradient descent with step size η , then by eq.

Original: (83) and eq.
Revised: (83) and eq.

Original: (87), we expect that
Revised: (87), we expect that

Original: [Equation]
Revised: [Equation]

Original: or otherwise written more compactly,
Revised: or otherwise written more compactly,

Original: [Equation]
Revised: [Equation]

Original: if the variance in the initialization is small enough.
Revised: if the variance in the initialization is small enough.

Original: Given this characterization of the residual equation 89, we can see how the usage of the Sobolev loss plays a role in tuning frequency bias.
Revised: Given this characterization of the residual equation 89, we can see how the usage of the Sobolev loss plays a role in tuning frequency bias.

Original: Suppose we decompose the residual vector into
Revised: Suppose we decompose the residual vector into

Original: [Equation]
Revised: [Equation]

Original: where Y ℓ,p =
Revised: where Y ℓ,p =

Original: ( Y ℓ,p ( x 1 ) , . . .
Revised: ( Y ℓ,p ( x 1 ) , . . .

Original: , Y ℓ,p ( x M ))
Revised: , Y ℓ,p ( x M ))

Original: ⊤ is the evaluation of the spherical harmonic on S d − 1 .
Revised: ⊤ is the evaluation of the spherical harmonic on S d − 1 .

Original: Then, ˆ γ ℓ,pi measures the amount of frequency loss associated with the ( ℓ, p ) th frequency in learning the function f i .
Revised: Then, ˆ γ ℓ,pi measures the amount of frequency loss associated with the ( ℓ, p ) th frequency in learning the function f i .

Original: Let ˆ γ ℓ,p ( k ) =
Revised: Let ˆ γ ℓ,p ( k ) =

Original: (ˆ γ ℓ,pi ( k ) , . . .
Revised: (ˆ γ ℓ,pi ( k ) , . . .

Original: , ˆ γ ℓ,pN ( k ))
Revised: , ˆ γ ℓ,pN ( k ))

Original: ⊤ .
Revised: ⊤ .

Original: The size of ˆ γ ℓ,p ( k ) measures the amount of overall frequency- ℓ components (in the p th direction) in the residuals after the k th iteration.
Revised: The size of ˆ γ ℓ,p ( k ) measures the amount of overall frequency- ℓ components (in the p th direction) in the residuals after the k th iteration.