Original: F REQUENCY P RINCIPLE : F OURIER A NALYSIS S HEDS L IGHT ON D EEP N EURAL N ETWORKS
Revised: F REQUENCY P RINCIPLE : F OURIER A NALYSIS S HEDS L IGHT ON D EEP N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We study the training process of Deep Neural Networks (DNNs) from the Fourier analysis perspective.
Revised: We study the training process of Deep Neural Networks (DNNs) from the Fourier analysis perspective.

Original: We demonstrate a very universal Frequency Principle (FPrinciple) — DNNs often ﬁt target functions from low to high frequencies — on high-dimensional benchmark datasets such as MNIST/CIFAR10 and deep neural networks such as VGG16.
Revised: We demonstrate a very universal Frequency Principle (FPrinciple) — DNNs often ﬁt target functions from low to high frequencies — on high-dimensional benchmark datasets such as MNIST/CIFAR10 and deep neural networks such as VGG16.

Original: This F-Principle of DNNs is opposite to the behavior of most conventional iterative numerical schemes (e g., Jacobi method), which exhibit faster convergence for higher frequencies for various scientiﬁc computing problems.
Revised: This F-Principle of DNNs is opposite to the behavior of most conventional iterative numerical schemes (e g., Jacobi method), which exhibit faster convergence for higher frequencies for various scientiﬁc computing problems.

Original: With theories under an idealized setting, we illustrate that this F-Principle results from the smoothness/regularity of the commonly used activation functions.
Revised: With theories under an idealized setting, we illustrate that this F-Principle results from the smoothness/regularity of the commonly used activation functions.

Original: The F-Principle implies an implicit bias that DNNs tend to ﬁt training data by a low-frequency function.
Revised: The F-Principle implies an implicit bias that DNNs tend to ﬁt training data by a low-frequency function.

Original: This understanding provides an explanation of good generalization of DNNs on most real datasets and bad generalization of DNNs on parity function or a randomized dataset.
Revised: This understanding provides an explanation of good generalization of DNNs on most real datasets and bad generalization of DNNs on parity function or a randomized dataset.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Understanding the training process of Deep Neural Networks (DNNs) is a fundamental problem in the area of deep learning.
Revised: Understanding the training process of Deep Neural Networks (DNNs) is a fundamental problem in the area of deep learning.

Original: We ﬁnd a common behavior of the gradient-based training process of
Revised: We ﬁnd a common behavior of the gradient-based training process of

Original: DNNs, that is, a Frequency Principle (F-Principle):
Revised: DNNs, that is, a Frequency Principle (F-Principle):

Original: DNNs often ﬁt target functions from low to high frequencies during the training process.
Revised: DNNs often ﬁt target functions from low to high frequencies during the training process.

Original: In another word, at the early stage of training, the low-frequencies are ﬁtted and as iteration steps of training increase, the high-frequencies are ﬁtted.
Revised: In another word, at the early stage of training, the low-frequencies are ﬁtted and as iteration steps of training increase, the high-frequencies are ﬁtted.

Original: For example, when a DNN is trained to ﬁt y
Revised: For example, when a DNN is trained to ﬁt y

Original: = sin( x ) + sin(2 x ) , its output would be close to sin( x ) at early stage and as training goes on, its output would be close to sin( x ) + sin(2 x ) .
Revised: = sin( x ) + sin(2 x ) , its output would be close to sin( x ) at early stage and as training goes on, its output would be close to sin( x ) + sin(2 x ) .

Original: F-Principle was observed empirically in synthetic low-dimensional data with MSE loss during DNN training (Xu et al., 2018; Rahaman et al., 2018).
Revised: F-Principle was observed empirically in synthetic low-dimensional data with MSE loss during DNN training (Xu et al., 2018; Rahaman et al., 2018).

Original: However, in deep learning, empirical phenomena could vary from one network structure to another, from one dataset to another and could exhibit signiﬁcant difference between synthetic data and highdimensional real data.
Revised: However, in deep learning, empirical phenomena could vary from one network structure to another, from one dataset to another and could exhibit signiﬁcant difference between synthetic data and highdimensional real data.

Original: Therefore, the universality of the F-Principle remains an important problem for further study.
Revised: Therefore, the universality of the F-Principle remains an important problem for further study.

Original: Especially for high-dimensional real problems, because the computational cost of high-dimensional Fourier transform is prohibitive in practice, it is of great challenge to demonstrate the F-Principle.
Revised: Especially for high-dimensional real problems, because the computational cost of high-dimensional Fourier transform is prohibitive in practice, it is of great challenge to demonstrate the F-Principle.

Original: On the other hand, the mechanism underlying the F-Principle and its implication to the application of DNNs, e.g., design of DNN-based PDE solver, as well as their generalization ability are also important open problems to be addressed.
Revised: On the other hand, the mechanism underlying the F-Principle and its implication to the application of DNNs, e.g., design of DNN-based PDE solver, as well as their generalization ability are also important open problems to be addressed.

Original: In this work, weﬁrst design two methods, i.e., projection and ﬁltering methods, to show that the F-Principle exists in the training process of DNNs fordifferent high-dimensional benchmarks, e.g., MNIST (LeCun, 1998), CIFAR10 (Krizhevsky et al., 2010).
Revised: In this work, we design two methods, i.e., projection and ﬁltering methods, to show that the FPrinciple exists in the training process of DNNs for high-dimensional benchmarks, i.e., MNIST (LeCun, 1998), CIFAR10 (Krizhevsky et al., 2010).

Original: The settings we have considered are i) different DNN architectures, e.g., fully-connected network, convolutional neural network (CNN), and VGG16 (Simonyan & Zisserman, 2014); ii) different activation functions, e.g., tanh and rectiﬁed linear unit (ReLU); iii) different loss functions, e.g., cross entropy, mean squared error (MSE), and loss energy functional in variation problems.
Revised: The settings we have considered are i) different DNN architectures, e.g., fully-connected network, convolutional neural network (CNN), and VGG (Simonyan & Zisserman, 2014); ii) different activation functions, e.g., tanh and rectiﬁed linear unit (ReLU); iii) different loss functions, e.g., cross entropy, mean squared error (MSE), and loss energy functional in variational problems.

Original: These results demonstrate the universality of the F-Principle.
Revised: These results demonstrate the universality of the F-Principle.

Original: To facilitate the designs and applications of DNN-based schemes, we characterize a stark difference between DNNs and conventional numerical schemes on various scientiﬁc computing problems, where most of the conventional methods (e.g., Jacobi method) exhibit the opposite convergence behavior— faster convergence for higher frequencies.
Revised: To facilitate the designs and applications of DNN-based schemes, we characterize a stark difference between DNNs and conventional numerical schemes on various scientiﬁc computing problems, where most of the conventional methods (e.g., Jacobi method) exhibit the opposite convergence behavior faster convergence for higher frequencies.

Original: This difference implicates that DNN can be adopted to accelerate the convergence of low frequencies for computational problems.
Revised: This difference implies that DNN can be adopted to accelerate the convergence of low frequencies for computational problems

Original: We also intuitively explain with theories under an idealized setting how the smoothness/regularity of commonly used activation functions contributes to the F-Principle.
Revised: We also intuitively explain with theories under an idealized setting how the smoothness/regularity of commonly used activation functions contributes to the F-Principle.

Original: Note that this mechanism is rigorously demonstrated for DNNs of general settings in a subsequent work (Luo et al., 2019).
Revised: Note that this mechanism is rigorously demonstrated for DNNs of general settings in a subsequent work (Luo et al., 2019).

Original: Finally, we discuss that the F-Principle provides an understanding of good generalization of DNNs in many real datasets (Zhang et al., 2016) and poor generalization in learning the parity function (Shalev-Shwartz et al., 2017; Nye & Saxe, 2018), that is, the F-Principle which implies that DNNs prefer low frequencies, is consistent with the property of low frequencies dominance in many real datasets, e.g., MNIST/CIFAR10, but is different from the parity function whose spectrum concentrates on high frequencies.
Revised: Finally, we discuss that the F-Principle provides an understanding of good generalization of DNNs in many real datasets (Zhang et al., 2016) and poor generalization in learning the parity function (Shalev-Shwartz et al., 2017; Nye & Saxe, 2018), that is, the F-Principle which implies that DNNs prefer low frequencies, is consistent with the property of low frequencies dominance in many real datasets, e.g., MNIST/CIFAR10, but is different from the parity function whose spectrum concentrates on high frequencies.

Original: Compared with previous studies, our main contributions are as follows:
Revised: Compared with previous studies, our main contributions are as follows:

Original: By designing both the projection and ﬁltering methods, we consistently demonstrate the F-Principle forhigh-dimensional real datasets of MNIST/CIFAR10 over various architectures such as VGG16.
Revised: By designing both the projection and ﬁltering methods, we consistently demonstrate the F-Principle for MNIST/CIFAR10 over various architectures such as VGG16 and various loss functions.

Original: For the application of solving differential equations, we show that (i) conventional numerical schemes learn higher frequencies faster whereas DNNs learn lower frequencies faster by the FPrinciple, (ii) convergence of low frequencies can be greatly accelerated with DNN-based schemes.
Revised: For the application of solving differential equations, we show that (i) conventional numerical schemes learn higher frequencies faster whereas DNNs learn lower frequencies faster by the FPrinciple, (ii) convergence of low frequencies can be greatly accelerated with DNN-based schemes.

Original: We present theories under an idealized setting to illustrate how smoothness/regularity of activation function contributes to the F-Principle. 4.
Revised: We present theories under an idealized setting to illustrate how smoothness/regularity of activation function contributes to the F-Principle. 4.

Original: We discuss in detail the implication of the F-Principle to the generalization of DNNs that DNNs are implicitly biased towards a low frequency function and provide an explanation of good and poor generalization of DNNs for low and high frequency dominant target functions, respectively.
Revised: We discuss in detail the implication of the F-Principle to the generalization of DNNs that DNNs are implicitly biased towards a low frequency function and provide an explanation of good and poor generalization of DNNs for low and high frequency dominant target functions, respectively.

Original:  F REQUENCY P RINCIPLE
Revised:  F REQUENCY P RINCIPLE

Original: The concept of “frequency” is central to the understanding of F-Principle.
Revised: The concept of “frequency” is central to the understanding of F-Principle.

Original: In this paper, the “frequency” means response frequency NOT image (or input) frequency as explained in the following.
Revised: In this paper, the “frequency” means response frequency NOT image (or input) frequency as explained in the following.

Original: Image (or input) frequency (NOT used in the paper): Frequency of 2 -d function I : R 2 → R representing the intensity of an image over pixels at different locations.
Revised: Image (or input) frequency (NOT used in the paper): Frequency of 2 -d function I : R 2 → R representing the intensity of an image over pixels at different locations.

Original: This frequency corresponds to the rate of change of intensity across neighbouring pixels .
Revised: This frequency corresponds to the rate of change of intensity across neighbouring pixels .

Original: For example, an image of constant intensity possesses only the zero frequency, i.e., the lowest frequency, while a sharp edge contributes to high frequencies of the image.
Revised: For example, an image of constant intensity possesses only the zero frequency, i.e., the lowest frequency, while a sharp edge contributes to high frequencies of the image.

Original: Response frequency (used in the paper): Frequency of a general Input-Output mapping f .
Revised: Response frequency (used in the paper): Frequency of a general Input-Output mapping f .

Original: For example, consider a simpliﬁed classiﬁcation problem of partial MNIST data using only the data with labeland 1 , f ( x 1 , x 2 , · · · , x 784 ) : R 784 → { 0 , 1 } mapping 784 -d space of pixel values to 1 -d space, where x j is the intensity of the j -th pixel.
Revised: For example, consider a simpliﬁed classiﬁcation problem of partial MNIST data using only the data with labeland 1 , f ( x 1 , x 2 , · · · , x 784 ) : R 784 → { 0 , 1 } mapping 784 -d space of pixel values to 1 -d space, where x j is the intensity of the j -th pixel.

Original: Denote the mapping’s Fourier transform as ˆ f ( k 1 , k 2 , · · · , k 784 ) .
Revised: Denote the mapping’s Fourier transform as ˆ f ( k 1 , k 2 , · · · , k 784 ) .

Original: The frequency in the coordinate k j measures the rate of change of f ( x 1 , x 2 , · · · , x 784 ) with respect to x j , i.e., the intensity of the j -th pixel .
Revised: The frequency in the coordinate k j measures the rate of change of f ( x 1 , x 2 , · · · , x 784 ) with respect to x j , i.e., the intensity of the j -th pixel .

Original: If f possesses signiﬁcant high frequencies for large k j , then a small change of x j in the image might induce a large change of the output (e.g., adversarial example).
Revised: If f possesses signiﬁcant high frequencies for large k j , then a small change of x j in the image might induce a large change of the output (e.g., adversarial example).

Original: For a dataset with multiple classes, we can similarly deﬁne frequency for each output dimension.
Revised: For a dataset with multiple classes, we can similarly deﬁne frequency for each output dimension.

Revised: For real data, the response frequency is rigorously deﬁned via the standard nonuniform discrete Fourier transform (NUDFT), see Appendix A.

Revised: Frequency Principle : DNNs often ﬁt target functions from low to high (response) frequencies during the training process.

Original: An illustration of F-Principle using a function of 1-d input is in Appendix A.
Revised: An illustration of F-Principle using a function of 1-d input is in Appendix B.

Original: Frequency Principle : DNNs often ﬁt target functions from low to high (response) frequencies during the training process.
Revised: The F-Principle is rigorously deﬁned through the frequency deﬁned by the Fourier transform (Appendix A, Bracewell & Bracewell (1986)) and the converging speed deﬁned by the relative error.

Original: In the following, by using high-dimensional real datasets, we  experimentally demonstrate F-Principle at the levels of both individual frequencies (projection method) and coarsegrained frequencies (ﬁltering method).
Revised: By using high-dimensional real datasets, we then experimentally demonstrate F-Principle at the levels of both individual frequencies (projection method) and coarse-grained frequencies (ﬁltering method).

Original:  F-P RINCIPLE IN MNIST/CIFAR10 THROUGH PROJECTION METHOD
Revised:  F-P RINCIPLE IN MNIST/CIFAR10 THROUGH PROJECTION METHOD

Original: Real datasets are very different from synthetic data used in previous studies.
Revised: Real datasets are very different from synthetic data used in previous studies.

Original: In order to utilize the
Revised: In order to utilize the

Original: F-Principle to understand and better use DNNs in real datasets, it is important to verify whether the
Revised: F-Principle to understand and better use DNNs in real datasets, it is important to verify whether the

Original: F-Principle also holds in high-dimensional real datasets.
Revised: F-Principle also holds in high-dimensional real datasets.

Original: In the following experiments, we examine the F-Principle in a training dataset of { ( x i , y i ) } n − 1 i = where n is the size of dataset.
Revised: In the following experiments, we examine the F-Principle in a training dataset of { ( x i , y i ) } n − 1 i = where n is the size of dataset.

Original: x
Revised: x

Original: i ∈ R
Revised: i ∈ R

Original: d is a vector representing the image and y i ∈ { 0 , 1 } 10 is the output (a one-hot vector indicating the label for the dataset of image classiﬁcation).
Revised: d is a vector representing the image and y i ∈ { 0 , 1 } 10 is the output (a one-hot vector indicating the label for the dataset of image classiﬁcation).

Original: d is the dimension of the input ( d = 784 for MNIST and d = 32 × 32 × 3 for CIFAR10).
Revised: d is the dimension of the input ( d = 784 for MNIST and d = 32 × 32 × 3 for CIFAR10).

Original: Since the high dimensional discrete Fourier transform (DFT) requires prohibitively high computational cost, in this section, we only consider one direction in the Fourier space through a projection method for each examination.
Revised: Since the high dimensional discrete Fourier transform (DFT) requires prohibitively high computational cost, in this section, we only consider one direction in the Fourier space through a projection method for each examination.

Original: 3. E XAMINATION METHOD : P ROJECTION
Revised: 3. E XAMINATION METHOD : P ROJECTION

Original: For a dataset { ( x i , y i ) } n − 1 i =

Revised: For a dataset { ( x i , y i ) } n − 1 i =

Original: we consider one entry of 10-d output, denoted by y i ∈ R .
Revised: we consider one entry of 10-d output, denoted by y i ∈ R .

Original: The high dimensional discrete non-uniform Fourier transform of { ( x i , y i ) } n − 1 i =0 is ˆ y k = 1 n (cid:80) n − 1 i =

Revised: The high dimensional discrete non-uniform Fourier transform of { ( x i , y i ) } n − 1 i =0 is ˆ y k = 1 n (cid:80) n − 1 i =

Original: y i exp ( − i2 π k · x i ) .
Revised: y i exp ( − i2 π k · x i ) .

Original: The number of all possible k grows exponentially on dimension d .
Revised: The number of all possible k grows exponentially on dimension d .

Original: For illustration, in each examination, we consider a direction of k in the Fourier space, i.e., k = k p 1 , p 1 is a chosen and ﬁxed unit vector, hence | k | =
Revised: For illustration, in each examination, we consider a direction of k in the Fourier space, i.e., k = k p 1 , p 1 is a chosen and ﬁxed unit vector, hence | k | =

Original: k .
Revised: k .

Original: Then we have ˆ y k = 1 n (cid:80) n − 1 i =

Revised: Then we have ˆ y k = 1 n (cid:80) n − 1 i =

Original: y i exp ( − i2 π ( p 1 · x j ) k ) , which is essentially the 1 -d Fourier transform of { ( x p 1 ,i , y i ) } n − 1 i =0 , where x p 1 ,i = p 1 · x i is the projection of x i on the direction p 1 (Bracewell & Bracewell, 1986).
Revised: y i exp ( − i2 π ( p 1 · x j ) k ) , which is essentially the 1 -d Fourier transform of { ( x p 1 ,i , y i ) } n − 1 i =0 , where x p 1 ,i = p 1 · x i is the projection of x i on the direction p 1 (Bracewell & Bracewell, 1986).

Original: For each training dataset, p 1 is chosen as the ﬁrst principle component of the input space.
Revised: For each training dataset, p 1 is chosen as the ﬁrst principle component of the input space.

Original: To examine the convergence behavior of different frequency components during the training, we compute the relative difference between the DNN output and the target function for selected important frequencies k ’s at each recording step, that is, ∆ F ( k ) =
Revised: To examine the convergence behavior of different frequency components during the training, we compute the relative difference between the DNN output and the target function for selected important frequencies k ’s at each recording step, that is, ∆ F ( k ) =

Original: | ˆ h k − ˆ y k | / | ˆ y k | , where ˆ y k and ˆ h k are 

Revised: | ˆ h k − ˆ y k | / | ˆ y k | , where ˆ y k and ˆ h k are

Original: -d
Revised: -d

Original: Fourier transforms of { y i } n − 1 i =0 and the corresponding DNN output { h i } n − 1 i =0 , respectively, along p 1 .
Revised: Fourier transforms of { y i } n − 1 i =0 and the corresponding DNN output { h i } n − 1 i =0 , respectively, along p 1 .

Revised: Note that each response frequency component, ˆ h k , of DNN output evolves as the training goes.

Original: 3. MNIST/CIFAR 
Revised: 3. MNIST/CIFAR

Original: In the following, we show empirically that the F-Principle is exhibited in the selected direction during the training process of DNNs when applied to MNIST/CIFAR10 with cross-entropy loss.
Revised: In the following, we show empirically that the F-Principle is exhibited in the selected direction during the training process of DNNs when applied to MNIST/CIFAR10 with cross-entropy loss.

Original: The network for MNIST is a fully-connected tanh DNN ( 784 - 400 - 200 - 10 ) and for CIFAR10 is two ReLU convolutional layers followed by a fully-connected DNN ( 800 - 400 - 400 - 400 - 10 ).
Revised: The network for MNIST is a fully-connected tanh DNN ( 784 - 400 - 200 - 10 ) and for CIFAR10 is two ReLU convolutional layers followed by a fully-connected DNN ( 800 - 400 - 400 - 400 - 10 ).

Original: All experimental details of this paper can be found in Appendix B. We consider one of the 10-d outputs in each case using non-uniform Fourier transform.
Revised: All experimental details of this paper can be found in Appendix C. We consider one of the 10-d outputs in each case using non-uniform Fourier transform.

Original: As shown in Fig. 1(a) and 1(c), low frequencies dominate in both real datasets.
Revised: As shown in Fig. 1(a) and 1(c), low frequencies dominate in both real datasets.

Original: During the training, the evolution of relative errors of certain selected frequencies (marked by black squares in Fig.
Revised: During the training, the evolution of relative errors of certain selected frequencies (marked by black squares in Fig.

Original: 1(a) and 1(c)) is shown in Fig. 1(b) and 1(d).
Revised: 1(a) and 1(c)) is shown in Fig. 1(b) and 1(d).

Original: One can easily observe that DNNs capture low frequencies ﬁrst and gradually capture higher frequencies.
Revised: One can easily observe that DNNs capture low frequencies ﬁrst and gradually capture higher frequencies.

Original: Clearly, this behavior is consistent with the F-Principle.
Revised: Clearly, this behavior is consistent with the F-Principle.

Original: For other components of the output vector and other directions of p , similar phenomena are also observed. 
Revised: For other components of the output vector and other directions of p , similar phenomena are also observed. [Figure] t

Original: [Figure]
Revised: [Figure]

Revised: d e

Original:   F-P RINCIPLE IN MNIST/CIFAR10 THROUGH FILTERING METHOD
Revised: [Figure] li t u d e [Figure]  F-P RINCIPLE IN MNIST/CIFAR10 THROUGH FILTERING METHOD

Original: The projection method in  previous section enables us to visualize the F-Principle in one direction for each examination at the level of individual frequency components.
Revised: The projection method in the previous section enables us to visualize the F-Principle in one direction for each examination at the level of individual frequency components.

Original: However, demonstration by this method alone is insufﬁcient because it is impossible to verify  F-Principle at all potentially informative directions for high-dimensional data.
Revised: However, demonstration by this method alone is insufﬁcient because it is impossible to verify the F-Principle at all potentially informative directions for high-dimensional data.

Original: To compensate the projection method, in this section, we consider a coarse-grained ﬁltering method which is able to unravel whether, in the radially averaged sense, low frequencies converge faster than high frequencies.
Revised: To compensate the projection method, in this section, we consider a coarse-grained ﬁltering method which is able to unravel whether, in the radially averaged sense, low frequencies converge faster than high frequencies.

Original: 4. E XAMINATION METHOD : FILTERING
Revised: 4. E XAMINATION METHOD : FILTERING

Original: The idea of the ﬁltering method is as follows.
Revised: The idea of the ﬁltering method is as follows.

Original: We split the frequency domain into two parts, i.e., a low-frequency part with | k | ≤ k 0 and a high-frequency part with | k | > k 0 , where | · | is the length of a vector.
Revised: We split the frequency domain into two parts, i.e., a low-frequency part with | k | ≤ k 0 and a high-frequency part with | k | > k 0 , where | · | is the length of a vector.

Original: The DNN is trained as usual by the original dataset { ( x i , y i ) } n − 1 i =0 , such as MNIST or
Revised: The DNN is trained as usual by the original dataset { ( x i , y i ) } n − 1 i =0 , such as MNIST or

Original: CIFAR10.
Revised: CIFAR10.

Original: The DNN output is denoted as h .
Revised: The DNN output is denoted as h .

Original: During the training, we can examine the convergence of relative errors of low- and high- frequency part, using the two measures below e low=
Revised: During the training, we can examine the convergence of relative errors of low- and high- frequency part, using the two measures below e low

Original: [Equation]
Revised: [Equation]

Original: respectively, where ˆ · indicates Fourier transform, 1 k ≤ k 0 is an indicator function, i.e.,
Revised: respectively, where ˆ · indicates Fourier transform, 1 k ≤ k 0 is an indicator function, i.e.,

Original: [Equation]
Revised: [Equation]

Original: If we consistently observe e low < e high for different k 0 ’s during the training, then in a mean sense, lower frequencies are ﬁrst captured by the DNN, i.e., F-Principle.
Revised: If we consistently observe e low < e high for different k 0 ’s during the training, then in a mean sense, lower frequencies are ﬁrst captured by the DNN, i.e., F-Principle.

Original: However, because it is almost impossible to compute above quantities numerically due to high computational cost of high-dimensional Fourier transform, we alternatively use the Fourier transform of a Gaussian function ˆ G δ ( k ) , where δ is the variance of the Gaussian function G , to approximate| k | >k 0 .
Revised: However, because it is almost impossible to compute above quantities numerically due to high computational cost of high-dimensional Fourier transform, we alternatively use the Fourier transform of a Gaussian function ˆ G δ ( k ) , where δ is the variance of the Gaussian function G , to approximate| k | >k 0 .

Original: This is reasonable due to the following two reasons.
Revised: This is reasonable due to the following two reasons.

Original: First, the Fourier transform of a
Revised: First, the Fourier transform of a

Original: Gaussian is still a Gaussian, i.e., ˆ G δ ( k ) decays exponentially as | k | increases, therefore, it can approximate 1 | k |≤ k 0 by ˆ G δ ( k ) with a proper δ ( k 0 ) (referred to as δ for simplicity).
Revised: Gaussian is still a Gaussian, i.e., ˆ G δ ( k ) decays exponentially as | k | increases, therefore, it can approximate 1 | k |≤ k 0 by ˆ G δ ( k ) with a proper δ ( k 0 ) (referred to as δ for simplicity).

Original: Second, the computation of e low and e high contains the multiplication of Fourier transforms in the frequency domain, which is equivalent to the Fourier transform of a convolution in the spatial domain.
Revised: Second, the computation of e low and e high contains the multiplication of Fourier transforms in the frequency domain, which is equivalent to the Fourier transform of a convolution in the spatial domain.

Original: We can equivalently perform the examination in the spatial domain so as to avoid the almost impossible high-dimensional Fourier transform.
Revised: We can equivalently perform the examination in the spatial domain so as to avoid the almost impossible high-dimensional Fourier transform.

Original: The low frequency part can be derived by
Revised: The low frequency part can be derived by

Original: [Equation]
Revised: [Equation]

Original: where ∗ indicates convolution operator, and the high frequency part can be derived by
Revised: where ∗ indicates convolution operator, and the high frequency part can be derived by

Original: [Equation]
Revised: [Equation]

Original: Then, we can examine
Revised: Then, we can examine

Original: [Equation]
Revised: [Equation]

Original: where h low ,δ and h high ,δ are obtained from the DNN output h , which evolves as a function of training epoch, through the same decomposition.
Revised: where h low ,δ and h high ,δ are obtained from the DNN output h , which evolves as a function of training epoch, through the same decomposition.

Original: If e low < e high for different δ ’s during the training,
Revised: If e low < e high for different δ ’s during the training,

Original: F-Principle holds; otherwise, it is falsiﬁed.
Revised: F-Principle holds; otherwise, it is falsiﬁed.

Original: Next, we introduce the experimental procedure.
Revised: Next, we introduce the experimental procedure.

Original: Step One: Training .
Revised: Step One: Training .

Original: Train the DNN by the original dataset { ( x i , y i ) } n − 1 i =0 , such as MNIST or
Revised: Train the DNN by the original dataset { ( x i , y i ) } n − 1 i =0 , such as MNIST or

Original: CIFAR10.
Revised: CIFAR10.

Original: x i is an image vector, y i is a one-hot vector.
Revised: x i is an image vector, y i is a one-hot vector.

Original: Step Two: Filtering .
Revised: Step Two: Filtering .

Original: The low frequency part can be derived by
Revised: The low frequency part can be derived by

Original: [Equation]
Revised: [Equation]

Original: where C i = (cid:80) n − 1 j =

Revised: where C i = (cid:80) n − 1 j =

Original: G δ
Revised: G δ

Original: ( x i − x j ) is a normalization factor and
Revised: ( x i − x j ) is a normalization factor and

Original: [Equation]
Revised: [Equation]

Original: The high frequency part can be derived by y high ,δ i
Revised: The high frequency part can be derived by y high ,δ i

Original: [Equation]
Revised: [Equation]

Original: We also compute h 
Revised: We also compute h low

Original: [Equation]
Revised: [Equation]

Original: ,δ i and h high ,δ i for each DNN output h i .
Revised: i and h high ,δ i for each DNN output h i .

Original: Step Three: Examination .
Revised: Step Three: Examination .

Original: To quantify the convergence of h low ,δ and h high ,δ , we compute the relative error e low and e high at each training epoch through Eq.
Revised: To quantify the convergence of h low ,δ and h high ,δ , we compute the relative error e low and e high at each training epoch through Eq.

Original: 4. DNN S WITH VARIOUS SETTINGS
Revised: 4. DNN S WITH VARIOUS SETTINGS

Original: With the ﬁltering method, we show the F-Principle in the DNN training process of real datasets for commonly used large networks.
Revised: With the ﬁltering method, we show the F-Principle in the DNN training process of real datasets for commonly used large networks.

Original: For MNIST, we use a fully-connected tanh-DNN (no softmax) with
Revised: For MNIST, we use a fully-connected tanh-DNN (no softmax) with

Original: MSE loss; for CIFAR10, we use cross-entropy loss and two structures, one is small ReLU-CNN network, i.e., two convolutional layers, followed by a fully-connected multi-layer neural network with a softmax; the other is VGG16 (Simonyan & Zisserman, 2014) equipped with a 1024 fully-connected layer.
Revised: MSE loss; for CIFAR10, we use cross-entropy loss and two structures, one is small ReLU-CNN network, i.e., two convolutional layers, followed by a fully-connected multi-layer neural network with a softmax; the other is VGG16 (Simonyan & Zisserman, 2014) equipped with a 1024 fully-connected layer.

Original: These three structures are denoted as “DNN”, “CNN” and “VGG” in Fig. 2, respectively.
Revised: These three structures are denoted as “DNN”, “CNN” and “VGG” in Fig. 2, respectively.

Original: All are trained by SGD from scratch .
Revised: All are trained by SGD from scratch .

Original: More details are in Appendix B.
Revised: More details are in Appendix C.

Original: We scan a large range of δ for both datasets.
Revised: We scan a large range of δ for both datasets.

Original: As an example, results of each dataset for several δ ’s are shown in Fig.
Revised: As an example, results of each dataset for several δ ’s are shown in Fig.

Original: 2, respectively.
Revised: 2, respectively.

Original: Red color indicates small relative error.
Revised: Red color indicates small relative error.

Original: In all cases, the relative error of the low-frequency part, i.e., e low , decreases (turns red) much faster than that of the high-frequency part, i.e., e high .
Revised: In all cases, the relative error of the low-frequency part, i.e., e low , decreases (turns red) much faster than that of the high-frequency part, i.e., e high .

Original: Therefore, as analyzed above, the low-frequency part converges faster than the high-frequency part.
Revised: Therefore, as analyzed above, the low-frequency part converges faster than the high-frequency part.

Original: We also remark that, based on the above results on cross-entropy loss, the
Revised: We also remark that, based on the above results on cross-entropy loss, the

Original: F-Principle is not limited to MSE loss, which possesses a natural Fourier domain interpretation by the Parseval’s theorem.
Revised: F-Principle is not limited to MSE loss, which possesses a natural Fourier domain interpretation by the Parseval’s theorem.

Original: Note that the above results holds for both SGD and GD.
Revised: Note that the above results holds for both SGD and GD.

Original: [Figure]
Revised: [Figure]

Original:  F-P RINCIPLE IN SOLVING DIFFERENTIAL EQUATION
Revised:  F-P RINCIPLE IN SOLVING DIFFERENTIAL EQUATION

Original: Recently, DNN-based approaches have been actively explored for a variety of scientiﬁc computing problems, e.g., solving high-dimensional partial differential equations (E et al., 2017; Khoo et al., 2017;
Revised: Recently, DNN-based approaches have been actively explored for a variety of scientiﬁc computing problems, e.g., solving high-dimensional partial differential equations (E et al., 2017; Khoo et al., 2017;

Original: He et al., 2018; Fan et al., 2018) and molecular dynamics (MD) simulations (Han et al., 2017).
Revised: He et al., 2018; Fan et al., 2018) and molecular dynamics (MD) simulations (Han et al., 2017).

Original: However, the behaviors of DNNs applied to these problems are not well-understood.
Revised: However, the behaviors of DNNs applied to these problems are not well-understood.

Original: To facilitate the designs and applications of DNN-based schemes, it is important to characterize the difference between DNNs and conventional numerical schemes on various scientiﬁc computing problems.
Revised: To facilitate the designs and applications of DNN-based schemes, it is important to characterize the difference between DNNs and conventional numerical schemes on various scientiﬁc computing problems.

Original: In this section, focusing on solving Poisson’s equation, which has broad applications in mechanical engineering and theoretical physics (Evans, 2010), we highlight a stark difference of a DNN-based solver and the Jacobi method during the training/iteration, which can be explained by the F-Principle.
Revised: In this section, focusing on solving Poisson’s equation, which has broad applications in mechanical engineering and theoretical physics (Evans, 2010), we highlight a stark difference between a DNN-based solver and the Jacobi method during the training/iteration, which can be explained by the F-Principle.

Original: Consider a 1 -d Poisson’s equation:
Revised: Consider a 1 -d Poisson’s equation:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We consider the example with g ( x

Original: [Equation]
Revised: [Equation]

Original:  = [Equation] x ) which has analytic solution u ref ( x ) =
Revised: We consider the example with g ( x ) = sin( x )+4 sin(4 x ) − 8 sin(8 x )+16 sin(24 x ) which has analytic solution u ref ( x ) =

Original: g 0 ( x ) + c 1 x + c 0 , where g 0 = sin( x ) + sin(4 x ) / 4 − sin(8 x ) / 8 + sin(24 x ) / 36 , c 1 = ( g 0 ( − 1) − g 0 (1)) / 2 and c 0 = − ( g 0 ( − 1)+ g 0 (1)) / 2 .
Revised: g 0 ( x ) + c 1 x + c 0 , where g 0 = sin( x ) + sin(4 x ) / 4 − sin(8 x ) / 8 + sin(24 x ) / 36 , c 1 = ( g 0 ( − 1) − g 0 (1)) / 2 and c 0 = − ( g 0 ( − 1)+ g 0 (1)) / 2 .

Original: 1001 training samples { x i } n i =0 are evenly[Figure] spaced with grid size δx in [0 , 1] .
Revised: 1001 training samples { x i } ni =0 are evenly spaced with grid size δx in [0 , 1] .

Original: Here, we use the DNN output, h ( x ; θ ) , to ﬁt u ref ( x ) (Fig. 3(a)).
Revised: Here, we use the DNN output, h ( x ; θ ) , to ﬁt u ref ( x ) (Fig. 3(a)).

Original: A
Revised: A

Original:  DNN-based scheme is proposed by considering the following empirical loss function (E & Yu, 2018),
Revised: [Figure] DNN-based scheme is proposed by considering the following empirical loss function (E & Yu, 2018),

Original: [Equation]
Revised: [Equation]

Original: The second term in I emp ( h ) is a penalty, with constant β , arising from the Dirichlet boundary condition (7).
Revised: The second term in I emp ( h ) is a penalty, with constant β , arising from the Dirichlet boundary condition (7).

Original: After training, the DNN output well matches the analytical solution u ref .
Revised: After training, the DNN output well matches the analytical solution u ref .

Original: Focusing on the convergence of three peaks (inset of Fig. 3(a)) in the Fourier transform of u ref , as shown in
Revised: Focusing on the convergence of three peaks (inset of Fig. 3(a)) in the Fourier transform of u ref , as shown in

Original: Fig. 3(b), low frequencies converge faster than high frequencies as predicted by the F-Principle.
Revised: Fig. 3(b), low frequencies converge faster than high frequencies as predicted by the F-Principle.

Original: For comparison, we also use the Jacobi method to solve problem (6).
Revised: For comparison, we also use the Jacobi method to solve problem (6).

Original: High frequencies converge faster in the Jacobi method (Details can be found in Appendix C), as shown in Fig. 3(c).
Revised: High frequencies converge faster in the Jacobi method (Details can be found in Appendix D), as shown in Fig. 3(c).

Original: As a demonstration, we further propose that DNN can be combined with conventional numerical schemes to accelerate the convergence of low frequencies for computational problems.
Revised: As a demonstration, we further propose that DNN can be combined with conventional numerical schemes to accelerate the convergence of low frequencies for computational problems.

Original: First, we solve the Poisson’s equation in Eq.
Revised: First, we solve the Poisson’s equation in Eq.

Original: (6) by DNN with M optimization steps (or epochs), which needs to be chosen carefully, to get a good initial guess in the sense that this solution has already learned the low frequencies (large eigenvalues) part.
Revised: (6) by DNN with M optimization steps (or epochs), which needs to be chosen carefully, to get a good initial guess in the sense that this solution has already learned the low frequencies (large eigenvalues) part.

Original: Then, we use the Jacobi method with the new initial data for the further iterations.
Revised: Then, we use the Jacobi method with the new initial data for the further iterations.

Original: We use (cid:107) h − u ref (cid:107) ∞ (cid:44) max x ∈ Ω | h ( x ) − u ref ( x ) | to quantify the learning result.
Revised: We use (cid:107) h − u ref (cid:107) ∞ (cid:44) max x ∈ Ω | h ( x ) − u ref ( x ) | to quantify the learning result.

Original: As shown by green stars in Fig. 3(d), (cid:107) h − u ref (cid:107) ∞ ﬂuctuates after some running time using DNN only.
Revised: As shown by green stars in Fig. 3(d), (cid:107) h − u ref (cid:107) ∞ ﬂuctuates after some running time using DNN only.

Original: Dashed lines indicate the evolution of the Jacobi method with initial data set to the DNN output at the corresponding steps.
Revised: Dashed lines indicate the evolution of the Jacobi method with initial data set to the DNN output at the corresponding steps.

Original: If M is too small (stop too early) (left dashed line), which is equivalent to only using Jacobi , it would take long time to converge to a small error, because low frequencies converges slowly, yet.
Revised: If M is too small (stop too early) (left dashed line), which is equivalent to only using Jacobi , it would take long time to converge to a small error, because low frequencies converges slowly, yet.

Original: If M is too big (stop too late)
Revised: If M is too big (stop too late)

Original: (right dashed line), which is equivalent to using
Revised: (right dashed line), which is equivalent to using

Original: DNN only , much time would be wasted for the slow convergence of high frequencies.
Revised: DNN only , much time would be wasted for the slow convergence of high frequencies.

Original: A proper choice of M is indicated by the initial point of orange dashed line, in which low frequencies are quickly captured by the DNN, followed by fast convergence in high frequencies of the Jacobi method.
Revised: A proper choice of M is indicated by the initial point of orange dashed line, in which low frequencies are quickly captured by the DNN, followed by fast convergence in high frequencies of the Jacobi method.

Original: This example illustrates a cautionary tale that, although DNNs has clear advantage, using DNNs alone may not be the best option because of its limitation of slow convergence at high frequencies.
Revised: This example illustrates a cautionary tale that, although DNNs has clear advantage, using DNNs alone may not be the best option because of its limitation of slow convergence at high frequencies.

Original: Taking advantage of both DNNs and conventional methods to design faster schemes could be a promising direction in scientiﬁc computing problems.
Revised: Taking advantage of both DNNs and conventional methods to design faster schemes could be a promising direction in scientiﬁc computing problems.

Original:  A PRELIMINARY THEORETICAL UNDERSTANDING
Revised:  A PRELIMINARY THEORETICAL UNDERSTANDING

Original: A subsequent theoretical work (Luo et al., 2019) provides a rigorous mathematical study of the FPrinciple at different frequencies for general DNNs (e.g., multiple hidden layers, different activation functions, high-dimensional inputs).
Revised: A subsequent theoretical work (Luo et al., 2019) provides a rigorous mathematical study of the FPrinciple at different frequencies for general DNNs (e.g., multiple hidden layers, different activation functions, high-dimensional inputs).

Original: The key insight is that the regularity of DNN converts into the decay rate of a loss function in the frequency domain.
Revised: The key insight is that the regularity of DNN converts into the decay rate of a loss function in the frequency domain.

Original: For an intuitive understanding of this key insight, we present theories under an idealized setting, which connect the smoothness/regularity of the activation function with different gradient and convergence priorities in frequency domain.
Revised: For an intuitive understanding of this key insight, we present theories under an idealized setting, which connect the smoothness/regularity of the activation function with different gradient and convergence priorities in frequency domain.

Original: The activation function we consider is σ ( x ) = tanh( x ) , which is smooth in spatial domain and its derivative decays exponentially with respect to frequency in the Fourier domain.
Revised: The activation function we consider is σ ( x ) = tanh( x ) , which is smooth in spatial domain and its derivative decays exponentially with respect to frequency in the Fourier domain.

Original: For a DNN of one hidden layer with m nodes, 1-d input x and 1-d output: h ( x ) =
Revised: For a DNN of one hidden layer with m nodes, 1-d input x and 1-d output: h ( x ) =

Original: (cid:80) mj
Revised: (cid:80) mj

Original: =1 a j σ ( w j x + b j ) , a j , w j , b j ∈
Revised: =1 a j σ ( w j x + b j ) , a j , w j , b j ∈

Original: R .
Revised: R .

Original: We also use the notation θ = { θ lj } with θ 1 j = a j , θ 2 j = w j , and θ 3 j = b j , j = 1 , · · · , m .
Revised: We also use the notation θ = { θ lj } with θ 1 j = a j , θ 2 j = w j , and θ 3 j = b j , j = 1 , · · · , m .

Original: The loss at frequency k is L ( k ) =
Revised: The loss at frequency k is L ( k ) =

Original: 12 (cid:12)(cid:12)(cid:12) ˆ h ( k ) − ˆ f ( k ) (cid:12)(cid:12)(cid:12) 2 , ˆ · is the Fourier transform, f is the target function.
Revised: 12 (cid:12)(cid:12)(cid:12) ˆ h ( k ) − ˆ f ( k ) (cid:12)(cid:12)(cid:12) 2 , ˆ · is the Fourier transform, f is the target function.

Original: The total loss function is deﬁned as: L = (cid:82) + ∞
Revised: The total loss function is deﬁned as: L = (cid:82) + ∞

Original: −∞ L ( k ) d k .
Revised: −∞ L ( k ) d k .

Original: Note that according tothe Parseval’s theorem, this loss function in the Fourier domain is equal to the commonly used MSE loss.
Revised: Note that according to Parseval’s theorem, this loss function in the Fourier domain is equal to the commonly used MSE loss.

Original: We have the following theorems (The proofs are at Appendix D.).
Revised: We have the following theorems (The proofs are at Appendix E.).

Original: Deﬁne W = ( w 1 , w 2 , · · · , w m ) T ∈ R m .
Revised: Deﬁne W = ( w 1 , w 2 , · · · , w m ) T ∈ R m .

Original: Theorem 1.
Revised: Theorem 1.

Original: Considering a DNN of one hidden layer with activation function σ ( x ) = tanh( x ) , for any frequencies k 1 and k 2 such that | ˆ f ( k 1 ) | > 0 , | ˆ f ( k 2 ) | > 0 , and | k 2 | > | k 1 | > 0 , there exist positive constants c and C such that for sufﬁciently small δ , we have
Revised: Considering a DNN of one hidden layer with activation function σ ( x ) = tanh( x ) , for any frequencies k 1 and k 2 such that | ˆ f ( k 1 ) | > 0 , | ˆ f ( k 2 ) | > 0 , and | k 2 | > | k 1 | > 0 , there exist positive constants c and C such that for sufﬁciently small δ , we have

Original: [Equation]
Revised: [Equation]

Original: for all l, j (cid:111) ∩ B δ (cid:17)
Revised: for all l, j (cid:111) [Equation] B δ (cid:17)

Original: [Equation]
Revised: [Equation]

Original: where B δ ⊂ R m is a ball with radius δ centered at the origin and µ ( · ) is the Lebesgue measure.
Revised: where B δ ⊂ R m is a ball with radius δ centered at the origin and µ ( · ) is the Lebesgue measure.

Original: Theorem 1 indicates that for any two non-converged frequencies, with small weights, the lowerfrequency gradient exponentially dominates over the higher-frequency ones.
Revised: Theorem 1 indicates that for any two non-converged frequencies, with small weights, the lowerfrequency gradient exponentially dominates over the higher-frequency ones.

Original: Due tothe Parseval’s theorem, the MSE loss in the spatial domain is equivalent to the L2 loss in the Fourier domain.
Revised: Due to Parseval’s theorem, the MSE loss in the spatial domain is equivalent to the L2 loss in the Fourier domain.

Original: To intuitively understand the higher decay rate of a lower-frequency loss function, we consider the training in the Fourier domain with loss function of only two non-zero frequencies.
Revised: To intuitively understand the higher decay rate of a lower-frequency loss function, we consider the training in the Fourier domain with loss function of only two non-zero frequencies.

Original: Theorem 2.
Revised: Theorem 2.

Original: Considering a DNN of one hidden layer with activation function σ ( x ) = tanh( x ) .
Revised: Considering a DNN of one hidden layer with activation function σ ( x ) = tanh( x ) .

Original: Suppose the target function has only two non-zero frequencies k 1 and k 2 , that is, | ˆ f ( k 1 ) | > 0 , | ˆ f ( k 2 ) | > 0 , | k 2 | > | k 1 | > 0 , and | ˆ f ( k ) | = 0 for k (cid:54) =
Revised: Suppose the target function has only two non-zero frequencies k 1 and k 2 , that is, | ˆ f ( k 1 ) | > 0 , | ˆ f ( k 2 ) | > 0 , | k 2 | > | k 1 | > 0 , and | ˆ f ( k ) | = 0 for k (cid:54) =

Original: k 1 , k 2 .
Revised: k 1 , k 2 .

Original: Consider the loss function of
Revised: Consider the loss function of

Original: L = L ( k 1 ) + L ( k 2 ) with gradient descent training.
Revised: L = L ( k 1 ) + L ( k 2 ) with gradient descent training.

Original: Denote
Revised: Denote

Original: [Equation]
Revised: [Equation]

Original: that is, L ( k 1 ) decreases faster than L ( k 2 ) .
Revised: that is, L ( k 1 ) decreases faster than L ( k 2 ) .

Original: There exist positive constants c and C such that for sufﬁciently small δ , we have
Revised: There exist positive constants c and C such that for sufﬁciently small δ , we have

Original: [Equation]
Revised: [Equation]

Original: where B δ ⊂ R m is a ball with radius δ centered at the origin and µ ( · ) is the Lebesgue measure.
Revised: where B δ ⊂ R m is a ball with radius δ centered at the origin and µ ( · ) is the Lebesgue measure.

Original:  D ISCUSSIONS
Revised:  D ISCUSSIONS

Original: DNNs often generalize well for real problems (Zhang et al., 2016) but poorly for problems like ﬁtting a parity function (Shalev-Shwartz et al., 2017; Nye & Saxe, 2018) despite excellent training accuracy for all problems.
Revised: DNNs often generalize well for real problems (Zhang et al., 2016) but poorly for problems like ﬁtting a parity function (Shalev-Shwartz et al., 2017; Nye & Saxe, 2018) despite excellent training accuracy for all problems.

Original: Understanding the differences between above two types of problems, i.e., good and bad generalization performance of DNN, is critical.
Revised: Understanding the differences between above two types of problems, i.e., good and bad generalization performance of DNN, is critical.

Original: In the following, we show a qualitative difference between these two types of problems through Fourier analysis and use the F-Principle to provide an explanation different generalization performances of DNNs.
Revised: In the following, we show a qualitative difference between these two types of problems through Fourier analysis and use the F-Principle to provide an explanation different generalization performances of DNNs.

Original: For MNIST/CIFAR10, we examine ˆ y total , k =
Revised: For MNIST/CIFAR10, we examine ˆ y total , k =

Original: [Equation]
Revised: [Equation]

Original: y

Original: [Equation] i2 π k · x i ) , where { ( x i , y i ) } n [Equation] i = consists of both the training and test datasets with certain selected output component, at different directions of k in the Fourier space.
Revised: y i exp ( − i2 π k · x i ) , where { ( x i , y i ) } n total − 1 i = consists of both the training and test datasets with certain selected output component, at different directions of k in the Fourier space.

Original: We ﬁnd that ˆ y total , k concentrates on the low frequencies along those examined directions.
Revised: We ﬁnd that ˆ y total , k concentrates on the low frequencies along those examined directions.

Original: For illustration, ˆ y total , k ’s along the ﬁrst principle component are shown by green lines in Fig.
Revised: For illustration, ˆ y total , k ’s along the ﬁrst principle component are shown by green lines in Fig.

Original: 4(a, b) for MNIST/CIFAR10, respectively.
Revised: 4(a, b) for MNIST/CIFAR10, respectively.

Original: When only the training dataset is used, ˆ y train , k well overlaps with ˆ y total , k at the dominant low frequencies.
Revised: When only the training dataset is used, ˆ y train , k well overlaps with ˆ y total , k at the dominant low frequencies.

Original: For the parity function f ( x ) = (cid:81) dj
Revised: For the parity function f ( x ) = (cid:81) dj

Original: =1 x j deﬁned on Ω = {− 1 , 1 } d , its Fourier transform is ˆ f ( k )
Revised: =1 x j deﬁned on Ω = {− 1 , 1 } d , its Fourier transform is ˆ f ( k )

Original: =
Revised: =

Original: 12 d (cid:80) x ∈ Ω (cid:81) dj
Revised: 12 d (cid:80) x ∈ Ω (cid:81) dj

Original: =1 x j e − i2 π k · x = ( − i) d (cid:81) dj
Revised: =1 x j e − i2 π k · x = ( − i) d (cid:81) dj

Original: =1 sin 2 πk j .
Revised: =1 sin 2 πk j .

Original: Clearly, for k ∈ [ − 14 , 14 ] d , the power of the parity function concentrates at k ∈ {− 14 , 14 } d and vanishes as k → 0 , as illustrated in Fig. 4(c) for the direction of 1 d .
Revised: Clearly, for k ∈ [ − 14 , 14 ] d , the power of the parity function concentrates at k ∈ {− 14 , 14 } d and vanishes as k → 0 , as illustrated in Fig. 4(c) for the direction of 1 d .

Original: Given a randomly sampled training dataset S ⊂ Ω with s points, the nonuniform
Revised: Given a randomly sampled training dataset S ⊂ Ω with s points, the nonuniform

Original: [Figure]
Revised: [Figure]

Original: Fourier transform on S is computed as ˆ f S ( k )
Revised: Fourier transform on S is computed as ˆ f S ( k )

Original: =
Revised: =

Original: 1 s (cid:80) x ∈ S (cid:81) dj
Revised: 1 s (cid:80) x ∈ S (cid:81) dj

Original: =1 x j e − i2 π k · x .
Revised: =1 x j e − i2 π k · x .

Original: As shown in Fig. 4(c), ˆ f ( k ) and ˆ f S ( k ) signiﬁcantly differ at low frequencies.
Revised: As shown in Fig. 4(c), ˆ f ( k ) and ˆ f S ( k ) signiﬁcantly differ at low frequencies.

Original: By experiments, the generalization ability of DNNs can be well reﬂected by the Fourier analysis.
Revised: By experiments, the generalization ability of DNNs can be well reﬂected by the Fourier analysis.

Original: For the MNIST/CIFAR10, we observed the Fourier transform of the output of a well-trained DNN on { x i } n total − 1 i = faithfully recovers the dominant low frequencies, as illustrated in Fig. 4(a) and 4(b), respectively, indicating a good generalization performance as observed in experiments.
Revised: For the MNIST/CIFAR10, we observed the Fourier transform of the output of a well-trained DNN on { x i } n total − 1 i = faithfully recovers the dominant low frequencies, as illustrated in Fig. 4(a) and 4(b), respectively, indicating a good generalization performance as observed in experiments.

Original: However, for the parity function, we observed that the Fourier transform of the output of a well-trained DNN on { x i } i ∈ S signiﬁcantly deviates from ˆ f ( k ) at almost all frequencies, as illustrated in Fig. 4(c), indicating a bad generalization performance as observed in experiments.
Revised: However, for the parity function, we observed that the Fourier transform of the output of a well-trained DNN on { x i } i ∈ S signiﬁcantly deviates from ˆ f ( k ) at almost all frequencies, as illustrated in Fig. 4(c), indicating a bad generalization performance as observed in experiments.

Original: The F-Principle implicates that among all the functions that can ﬁt the training data, a DNN is implicitly biased during the training towards a function with more power at low frequencies.
Revised: The F-Principle implicates that among all the functions that can ﬁt the training data, a DNN is implicitly biased during the training towards a function with more power at low frequencies.

Original: If the target function has signiﬁcant high-frequency components, insufﬁcient training samples will lead to artiﬁcial low frequencies in training dataset (see red line in Fig. 4(c)), which is the wellknown aliasing effect.
Revised: If the target function has signiﬁcant high-frequency components, insufﬁcient training samples will lead to artiﬁcial low frequencies in training dataset (see red line in Fig. 4(c)), which is the wellknown aliasing effect.

Original: Based on the F-Principle, as demonstrated in Fig. 4(c), these artiﬁcial low frequency components will be ﬁrst captured to explain the training samples, whereas the high frequency components will be compromised by DNN.
Revised: Based on the F-Principle, as demonstrated in Fig. 4(c), these artiﬁcial low frequency components will be ﬁrst captured to explain the training samples, whereas the high frequency components will be compromised by DNN.

Original: For MNIST/CIFAR10, since the power of high frequencies is much smaller than that of low frequencies, artiﬁcial low frequencies caused by aliasing can be neglected.
Revised: For MNIST/CIFAR10, since the power of high frequencies is much smaller than that of low frequencies, artiﬁcial low frequencies caused by aliasing can be neglected.

Original: To conclude, the distribution of power in Fourier domain of above two types of problems exhibits signiﬁcant differences, which result in different generalization performances of
Revised: To conclude, the distribution of power in Fourier domain of above two types of problems exhibits signiﬁcant differences, which result in different generalization performances of

Original: DNNs according to the F-Principle.
Revised: DNNs according to the F-Principle.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: As widely observed in experiments, DNNs with gradient-based training show different generalization abilities for different problems. On the one hand, there are different approaches attempting to explain why the training process often leads to a DNN of good generalization ability even when the number of parameters is much larger than the number of training data (Zhang et al., 2016).
Revised: There are different approaches attempting to explain why DNNs often generalize well.

Original: For example, generalization error is related to various complexity measures (Bartlett et al., 1999;Bartlett & Mendelson, 2002; Bartlett et al., 2017a;b; Neyshabur et al., 2017;Golowich et al., 2017; Dziugaite & Roy, 2017; Neyshabur et al., 2018; E et al., 2018), local properties (sharpness/ﬂatness) of loss functions at minima (Hochreiter & Schmidhuber, 1995; Keskar et al., 2016; 
Revised: For example, generalization error is related to various complexity measures (Bartlett et al., 1999; Neyshabur et al., 2017; E et al., 2018), local properties (sharpness/ﬂatness) of loss functions at minima (Keskar et al., 2016; Wu et al., 2017), stability of optimization algorithms (Hardt et al., 2015), and implicit bias of the training process (Soudry et al., 2018;

Original: Dinh et al., 2017; Wu et al., 2017), stability of optimization algorithms (Bousquet & Elisseeff, 2002; Xu & Mannor, 2012; Hardt et al., 2015), and implicit bias of the training process (Neyshabur et al., 2014; Poggio et al., 2018; Soudry et al., 2018; Arpit et al., 2017; Xu et al., 2018).
Revised: Arpit et al., 2017; Xu et al., 2018).

Original: On the other hand, several works focus on the failure of DNNs (Shalev-Shwartz et al., 2017; Nye & Saxe, 2018), e.g., ﬁtting the parity function, in which a well-trained DNN possesses no generalization ability. 
Revised: On the other hand, several works focus on the failure of DNNs (Shalev-Shwartz et al., 2017; Nye & Saxe, 2018), e.g., ﬁtting the parity function, in which a well-trained DNN possesses no generalization ability. We propose that the Fourier analysis can provide insights into both success and failure of DNNs. F-Principle was ﬁrst discovered in (Xu et al., 2018; Rahaman et al., 2018) simultaneously through simple synthetic data and not very deep networks.

Original: In the revised version, Rahaman et al.
Revised: In the revised version, Rahaman et al.

Original: (2018)also examines the F-Principle in the MNIST dataset.
Revised: (2018) examines the F-Principle in the MNIST dataset.

Original: However, they add noise to MNIST, which contaminates the labels.
Revised: However, they add noise to MNIST, which contaminates the labels and damages the structure of real data.

Original: They only examine not very deep (6-layer) fully connected ReLU network with MSE loss, while cross-entropy loss is widely used.
Revised: They only examine not very deep (6-layer) fully connected ReLU network with MSE loss, while cross-entropy loss is widely used.

Revised: This paper veriﬁed that F-Principle holds in the training process of MNIST and CIFAR10, both CNN and fully connected networks, very deep networks (VGG16) and various loss functions, e.g., MSE Loss, cross-entropy loss and variational loss function.

Revised: In the aspect of theoretical study, based on the key mechanism found by the theoretical study in this paper, Luo et al.

Revised: (2019) shows a rigorous proof of the F-Principle for general DNNs.

Revised: The theoretical study of the gradient of tanh( x ) in the Fourier domain is adopted by Rahaman et al.

Revised: (2018), in which they generalize the analysis to ReLU and show similar results.

Original:  A I LLUSTRATION OF F-P RINCIPLE FOR 1 - D SYNTHETIC DATA  
Revised: Thm 1 is also used to analyze a nonlinear collaborative scheme for deep network training (Zhen et al., 2018). In the aspect of application, based on the study of the F-Principle in this paper, Cai et al. and Cai & Xu (2019) design DNN-based algorithms to solve high-dimensional and high-frequency problems. A R ESPONSE FREQUENCY OF TRAINING DATA { y i } n − 1 i =0 ON INPUTS { x i } n − 1 i =

Revised: In all our experiments, we consistently consider the response frequency deﬁned for the mapping function g between inputs and outputs, say R d → R and any k ∈ R d via the standard nonuniform discrete Fourier transform (NUDFT)

Revised: [Equation]

Revised: which is a natural estimator of frequency composition of g .

Revised: (More details can be found in https: //en.wikipedia.org/wiki/Non-uniform_discrete_Fourier_transform .)

Revised: As n → ∞ , ˆ g k → (cid:82) g ( x )e − i2 π k · x ν ( x ) d x , where ν ( x ) is the data distribution.

Original: {+We restrict all the evaluation of Fourier transform in our experiments to NUDFT of { y i } n − 1 i =
+}
Revised: {+We restrict all the evaluation of Fourier transform in our experiments to NUDFT of { y i } n − 1 i =
+}

Revised: at { x i } n − 1 i =0 for the following practical reasons.

Revised: (i)

Revised: The information of target function is only available at { x i } n − 1 i =0 for training.

Revised: (ii) It allows us to perform the convergence analysis.

Revised: As t → ∞ , in general, h ( x i , t ) →

Revised: y i for any i ( h ( x i , t ) is the DNN output), leading to ˆ h k

Revised: → ˆ y k for any k .

Revised: Therefore, we can analyze the convergence at different k by evaluating ∆ F ( k ) =

Revised: | ˆ h k − ˆ y k | / | ˆ y k | during the training.

Revised: If we use a different set of data points for frequency evaluation of DNN output, then ∆ F ( k ) may not converge to 0 at the end of training.

Revised: (iii) ˆ y k faithfully reﬂect the frequency structure of training data { x i , y i } n − 1 i =

Revised: Intuitively, high frequencies of ˆ y k correspond to sharp changes of output for some nearby points in the training data.

Revised: Then, by applying a Gaussian ﬁlter and evaluating still at { x i } n − 1 i

Revised: =

Revised: 0 , we obtain the low frequency part of training data with these sharp changes (high frequencies) well suppressed.

Revised: [Figure]

Revised: In practice, it is impossible to evaluate and compare the convergence of all k ∈ R d even with a proper cutoff frequency for a very large d of O (10 2 ) (MNIST) or O (10 3 ) (CIFAR10) due to curse of dimensionality.

Revised: Therefore, we propose the projection approach, i.e., ﬁxing k at a speciﬁc direction and the ﬁltering approach as detailed in Section 3 and 4, respectively.

Original: To illustrate the phenomenon of F-Principle, we use 1 -d synthetic data to show the evolution of relative training error at different frequencies during the training of DNN.
Revised: To illustrate the phenomenon of F-Principle, we use 1 -d synthetic data to show the evolution of relative training error at different frequencies during the training of DNN.

Original: we train a DNN to ﬁt a 1 -d target function f ( x ) = sin( x ) + sin(3 x ) + sin(5 x ) of three frequency components.
Revised: we train a DNN to ﬁt a 1 -d target function f ( x ) = sin( x ) + sin(3 x ) + sin(5 x ) of three frequency components.

Original: On n =evenly spaced training samples, i.e., { x i } n − 1 i =0 in [ − 3 . 14 , 3 . 14] , the discrete Fourier transform (DFT) of f ( x ) or the DNN output (denoted by h ( x ) ) is computed by ˆ f k = 1 n (cid:80) n − 1 i =

Revised: On n =evenly spaced training samples, i.e., { x i } n − 1 i =0 in [ − 3 . 14 , 3 . 14] , the discrete Fourier transform (DFT) of f ( x ) or the DNN output (denoted by h ( x ) ) is computed by ˆ f k = 1 n (cid:80) n − 1 i =

Original: f ( x i )
Revised: f ( x i )

Original: e − i2 πik/n and ˆ h k = 1 n (cid:80) n − 1 i =

Revised: e − i2 πik/n and ˆ h k = 1 n (cid:80) n − 1 i =

Original: h ( x i )e − i2 πjk/n , where k is the frequency.
Revised: h ( x i )e − i2 πjk/n , where k is the frequency.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 5(a), the target function has three important frequencies as we design (black dots at the inset in Fig. 5(a)).
Revised: 5(a), the target function has three important frequencies as we design (black dots at the inset in Fig. 5(a)).

Original: To examine the convergence behavior of different frequency components during the training with MSE, we compute the relative difference of the DNN output and the target function for the three important frequencies k ’s at each recording step, that is, ∆ F ( k ) =
Revised: To examine the convergence behavior of different frequency components during the training with MSE, we compute the relative difference between the DNN output and the target function for the three important frequencies k ’s at each recording step, that is, ∆ F ( k ) =

Original: | ˆ h k − ˆ f k | / | ˆ f k | , where | · | denotes the norm of a complex number.
Revised: | ˆ h k − ˆ f k | / | ˆ f k | , where | · | denotes the norm of a complex number.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 5(b), the DNN converges the ﬁrst frequency peak very fast, while converging the second frequency peak much slower, followed by the third frequency peak.
Revised: 5(b), the DNN converges the ﬁrst frequency peak very fast, while converging the second frequency peak much slower, followed by the third frequency peak.

Original: Next, we investigate the F-Principle on real datasets with more general loss functions other than MSE which was the only loss studied in the previous works (Xu et al., 2018; Rahaman et al., 2018).
Revised: Next, we investigate the F-Principle on real datasets with more general loss functions other than MSE which was the only loss studied in the previous works (Xu et al., 2018; Rahaman et al., 2018).

Original: All experimental details can be found in Appendix.
Revised: All experimental details can be found in Appendix.

Original: B.
Revised: C.

Original: B E XPERIMENTAL SETTINGS
Revised: C E XPERIMENTAL SETTINGS

Original: In Fig.
Revised: In Fig.

Original: 5, the parameters of the DNN is initialized by a Gaussian distribution with mean 0 and standard deviation 0 .
Revised: 5, the parameters of the DNN is initialized by a Gaussian distribution with mean 0 and standard deviation 0 .

Original: We use a tanh-DNN with widths 1 - 8000 - 1 with full batch training.
Revised: We use a tanh-DNN with widths 1 - 8000 - 1 with full batch training.

Original: The learning rate is 0 . 0002 .
Revised: The learning rate is 0 . 0002 .

Original: The DNN is trained by Adam optimizer (Kingma & Ba, 2014) with the MSE loss function.
Revised: The DNN is trained by Adam optimizer (Kingma & Ba, 2014) with the MSE loss function.

Original: In Fig.
Revised: In Fig.

Original: 1, for MNIST dataset, the training process of a tanh-DNN with widths 784 - 400 - 200 - 10 is shown in Fig. 1(a) and 1(b).
Revised: 1, for MNIST dataset, the training process of a tanh-DNN with widths 784 - 400 - 200 - 10 is shown in Fig. 1(a) and 1(b).

Original: For CIFAR10 dataset, results are shown in Fig.
Revised: For CIFAR10 dataset, results are shown in Fig.

Original: 1(c) and 1(d) of a
Revised: 1(c) and 1(d) of a

Original: ReLU-CNN, which consists of one convolution layer of 3 × 3 × 64 , a max pooling of 2 × 2 , one convolution layer of 3 × 3 × 128 , a max pooling of 2 × 2 , followed by a fully-connected DNN with widths 800 - 400 - 400 - 400 - 10 .
Revised: ReLU-CNN, which consists of one convolution layer of 3 × 3 × 64 , a max pooling of 2 × 2 , one convolution layer of 3 × 3 × 128 , a max pooling of 2 × 2 , followed by a fully-connected DNN with widths 800 - 400 - 400 - 400 - 10 .

Original: For both cases, the output layer of the network is equipped with a softmax.
Revised: For both cases, the output layer of the network is equipped with a softmax.

Original: The network output is a 10 -d vector.
Revised: The network output is a 10 -d vector.

Original: The DNNs are trained with cross entropy loss by Adam optimizer (Kingma & Ba, 2014).
Revised: The DNNs are trained with cross entropy loss by Adam optimizer (Kingma & Ba, 2014).

Original: (a, b) are for MNIST with a tanh-DNN.
Revised: (a, b) are for MNIST with a tanh-DNN.

Original: The learning rate is 0 .with batch size 10000 .
Revised: The learning rate is 0 .with batch size 10000 .

Original: After training, the training accuracy is 0 .
Revised: After training, the training accuracy is 0 .

Original: 951 and test accuracy is 0 . 963 .
Revised: 951 and test accuracy is 0 . 963 .

Original: The amplitude of the Fourier coefﬁcient with respect to the fourth output component at each frequency is shown in(a), in which the red dots are computed using the training data. Selected frequencies are marked by black squares. (b) ∆ F ( k ) at different training epochs for the selected frequencies. (c, d) are for CIFAR10 dataset.
Revised: The amplitude of the Fourier coefﬁcient with respect to the fourth output component at each frequency is shown in are for CIFAR10 dataset.

Original: We use a ReLU network of a CNN followed by a fully-connected DNN.
Revised: We use a ReLU network of a CNN followed by a fully-connected DNN.

Original: The learning rate is 0 .
Revised: The learning rate is 0 .

Original: 003 with batch size 512 .
Revised: 003 with batch size 512 .

Original: (c) and (d) are the results with respect to the ninth output component.
Revised: (c) and (d) are the results with respect to the ninth output component.

Original: After training, the training accuracy is 0 .
Revised: After training, the training accuracy is 0 .

Original: 98 and test accuracy is 0 . 72 .
Revised: 98 and test accuracy is 0 . 72 .

Original: In Fig.
Revised: In Fig.

Original: 2, for MNIST, we use a fully-connected tanh-DNN with widths 784 - 400 - 200 - 10 and MSE loss; for CIFAR10, we use cross-entropy loss and a ReLU-CNN, which consists of one convolution layer of 3 × 3 × 32 , a max pooling of 2 × 2 , one convolution layer of 3 × 3 × 64 , a max pooling of 2 × 2 , followed by a fully-connected DNN with widths 400 - 10 and the output layer of the network is equipped with a softmax.
Revised: 2, for MNIST, we use a fully-connected tanh-DNN with widths 784 - 400 - 200 - 10 and MSE loss; for CIFAR10, we use cross-entropy loss and a ReLU-CNN, which consists of one convolution layer of 3 × 3 × 32 , a max pooling of 2 × 2 , one convolution layer of 3 × 3 × 64 , a max pooling of 2 × 2 , followed by a fully-connected DNN with widths 400 - 10 and the output layer of the network is equipped with a softmax.

Original: The learning rate for MNIST and CIFAR10 is 0 .
Revised: The learning rate for MNIST and CIFAR10 is 0 .

Original: 015 and 0 .
Revised: 015 and 0 .

Original: 003 , respectively.
Revised: 003 , respectively.

Original: The networks are trained by Adam optimizer (Kingma & Ba, 2014) with batch size 10000 .
Revised: The networks are trained by Adam optimizer (Kingma & Ba, 2014) with batch size 10000 .

Original: For
Revised: For

Original: VGG16, the learning rate is 10 − 5 .
Revised: VGG16, the learning rate is 10 − 5 .

Original: The network is trained by Adam optimizer (Kingma & Ba, 2014) with batch size 500 .
Revised: The network is trained by Adam optimizer (Kingma & Ba, 2014) with batch size 500 .

Original: In Fig.
Revised: In Fig.

Original: 3, the samples are evenly spaced in [0 , 1] with sample size 1001 .
Revised: 3, the samples are evenly spaced in [0 , 1] with sample size 1001 .

Original: We use a DNN with widths 1-4000-500-400-1 and full batch training by Adam optimizer (Kingma & Ba, 2014).
Revised: We use a DNN with widths 1-4000-500-400-1 and full batch training by Adam optimizer (Kingma & Ba, 2014).

Original: The learning rate is 0 . 0005 .
Revised: The learning rate is 0 . 0005 .

Original: β is 10 .
Revised: β is 10 .

Original: The parameters of the DNN are initialized following a Gaussian distribution with mean 0 and standard deviation 0 .
Revised: The parameters of the DNN are initialized following a Gaussian distribution with mean 0 and standard deviation 0 .

Original: In Fig.
Revised: In Fig.

Original: 4, the settings of (a) and (b) are the same as the ones in Fig.
Revised: 4, the settings of (a) and (b) are the same as the ones in Fig.

Original: For (c), we use a tanh-DNN with widths 10-500-100-1, learning rate 0 .
Revised: For (c), we use a tanh-DNN with widths 10-500-100-1, learning rate 0 .

Original: 0005 under full batch-size training by Adam optimizer (Kingma & Ba, 2014).
Revised: 0005 under full batch-size training by Adam optimizer (Kingma & Ba, 2014).

Original: The parameters of the DNN are initialized by a Gaussian distribution with mean 0 and standard deviation 0 .
Revised: The parameters of the DNN are initialized by a Gaussian distribution with mean 0 and standard deviation 0 .

Original: C C ENTRAL DIFFERENCE SCHEME AND J ACOBI METHOD
Revised: D C ENTRAL DIFFERENCE SCHEME AND J ACOBI METHOD

Original: Consider a one-dimensional (1-d) Poisson’s equation:
Revised: Consider a one-dimensional (1-d) Poisson’s equation:

Original: [Equation]
Revised: [Equation]

Original: [ − 1 , 1] is uniformly discretized into n + 1 points with grid size h = 2 /n .
Revised: [ − 1 , 1] is uniformly discretized into n + 1 points with grid size h = 2 /n .

Original: The Poisson’s equation in
Revised: The Poisson’s equation in

Original: Eq.
Revised: Eq.

Original: (9) can be solved by the central difference scheme,
Revised: (9) can be solved by the central difference scheme,

Original: [Equation]
Revised: [Equation]

Original: resulting a linear system
Revised: resulting a linear system

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: A class of methods to solve this linear system is iterative schemes, for example, the Jacobi method.
Revised: A class of methods to solve this linear system is iterative schemes, for example, the Jacobi method.

Original: Let A
Revised: Let A

Original: = D − L − U , where D is the diagonal of A , and L and U are the strictly lower and upper triangular parts of − A , respectively.
Revised: = D − L − U , where D is the diagonal of A , and L and U are the strictly lower and upper triangular parts of − A , respectively.

Original: Then, we obtain
Revised: Then, we obtain

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: At step t ∈ N , the Jacobi iteration reads as
Revised: At step t ∈ N , the Jacobi iteration reads as

Original: [Equation]
Revised: [Equation]

Original: We perform the standard error analysis of the above iteration process.
Revised: We perform the standard error analysis of the above iteration process.

Original: Denote u ∗ as the true value obtained by directly performing inverse of A in Eq.
Revised: Denote u ∗ as the true value obtained by directly performing inverse of A in Eq.

Original: The error at step t +1 is e t +1 =
Revised: The error at step t +1 is e t +1 =

Original: u t +1 − u ∗ .
Revised: u t +1 − u ∗ .

Original: Then, e t +1 =
Revised: Then, e t +1 =

Original: R J e t , where R J = D − 1 ( L + U ) .
Revised: R J e t , where R J = D − 1 ( L + U ) .

Original: The converging speed of e t is determined by the eigenvalues of R J , that is, 
Revised: The converging speed of e t is determined by the eigenvalues of R J , that is, [Equation]

Original: [Equation]
Revised: [Equation]

Original: and the corresponding eigenvector v k ’s entry is
Revised: and the corresponding eigenvector v k ’s entry is

Original: [Equation]
Revised: [Equation]

Original: So we can write
Revised: So we can write

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where α tk can be understood as the magnitude of e t in the direction of v k .
Revised: where α tk can be understood as the magnitude of e t in the direction of v k .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Therefore, the converging rate of e t in the direction of v k is controlled by λ k .
Revised: Therefore, the converging rate of e t in the direction of v k is controlled by λ k .

Original: Since
Revised: Since

Original: [Equation]
Revised: [Equation]

Original: the frequencies k and ( n − k ) are closely related and converge with the same rate.
Revised: the frequencies k and ( n − k ) are closely related and converge with the same rate.

Original: Consider the frequency k < n/ 2 , λ k is larger for lower frequency.
Revised: Consider the frequency k < n/ 2 , λ k is larger for lower frequency.

Original: Therefore, lower frequency converges slower in the Jacobi method.
Revised: Therefore, lower frequency converges slower in the Jacobi method.

Original: D P ROOF OF THEOREMS
Revised: E P ROOF OF THEOREMS

Original: The activation function we consider is σ ( x ) = tanh( x ) .
Revised: The activation function we consider is σ [Equation] = [Equation]

Original: [Equation]
Revised: [Equation]

Original: For a DNN of one hidden layer with m nodes, 1-d input x and 1-d output:
Revised: For a DNN of one hidden layer with m nodes, 1-d input x and 1-d output:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where w j , a j , and b j are called parameters , in particular, w j and a j are called weights , and b j is also known as a bias .
Revised: where w j , a j , and b j are called parameters , in particular, w j and a j are called weights , and b j is also known as a bias .

Original: In the sequel, we will also use the notation θ = { θ lj } with θ 1 j = a j , θ 2 j = w j , and θ lj =
Revised: In the sequel, we will also use the notation θ = { θ lj } with θ 1 j = a j , θ 2 j = w j , and θ lj =

Original: b j , j = 1 , · · · , m .
Revised: b j , j = 1 , · · · , m .

Original: Note that ˆ σ ( k ) =−
Revised: Note that ˆ σ ( k ) =

Original: [Equation] 2) where the Fourier transformation and its inverse transformation are deﬁned as follows:
Revised: − i π sinh( πk/ 2) where the Fourier transformation and its inverse transformation are deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: The Fourier transform of σ ( w j x + b j ) with w j , b j ∈ R , j = 1 , · · · , m reads as 
Revised: The Fourier transform of σ ( w j x + b j ) with w j , b j ∈ R , j = 1 , · · · , m reads as [Equation]

Original: [Equation]
Revised: [Equation]

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: We deﬁne the amplitude deviation between DNN output and the target function f ( x ) at frequency k as
Revised: We deﬁne the amplitude deviation between DNN output and the target function f ( x ) at frequency k as

Original: [Equation]
Revised: [Equation]

Original: Write D ( k ) as D ( k ) =
Revised: Write D ( k ) as D ( k ) =

Original: A ( k )e i φ ( k ) , where A ( k ) ∈ [0 , + ∞ ) and φ ( k ) ∈
Revised: A ( k )e i φ ( k ) , where A ( k ) ∈ [0 , + ∞ ) and φ ( k ) ∈

Original: R are the amplitude and phase of D ( k ) , respectively.
Revised: R are the amplitude and phase of D ( k ) , respectively.

Original: The loss at frequency k is L ( k ) =
Revised: The loss at frequency k is L ( k ) =

Original: 12 | D ( k ) | 2 , where | · | denotes the norm of a complex number.
Revised: 12 | D ( k ) | 2 , where | · | denotes the norm of a complex number.

Original: The total loss function is deﬁned as: L = (cid:82) + ∞
Revised: The total loss function is deﬁned as: L = (cid:82) + ∞

Original: −∞ L ( k ) d k .
Revised: −∞ L ( k ) d k .

Original: Note that according tothe Parseval’s theorem, this loss function in the Fourier domain is equal to the commonly used loss of mean squared error, that is, L = (cid:82) + ∞
Revised: Note that according to Parseval’s theorem, this loss function in the Fourier domain is equal to the commonly used loss of mean squared error, that is, L = (cid:82) + ∞

Original: −∞ 12 ( h ( x ) − f ( x )) 2 d x .
Revised: −∞ 12 ( h ( x ) − f ( x )) 2 d x .

Original: For readers’ reference, we list the partial derivatives of L ( k ) with respect to parameters 
Revised: For readers’ reference, we list the partial derivatives of L ( k ) with respect to parameters [Equation]

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: The descent increment at any direction, say, with respect to parameter θ lj , is
Revised: The descent increment at any direction, say, with respect to parameter θ lj , is

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: The absolute contribution from frequency k to this total amount at θ lj is
Revised: The absolute contribution from frequency k to this total amount at θ lj is

Original: [Equation]
Revised: [Equation]

Original: where θ j (cid:44) { w j , b j , a j } , θ lj ∈ θ j , F lj ( θ j , k ) is a function with respect to θ j and k , which can be found in one of Eqs.
Revised: where θ j (cid:44) { w j , b j , a j } , θ lj ∈ θ j , F lj ( θ j , k ) is a function with respect to θ j and k , which can be found in one of Eqs.

Original: When the component at frequency k where ˆ h ( k ) is not close enough to ˆ f ( k ) , exp ( −| πk/ 2 w j | ) would dominate G lj
Revised: When the component at frequency k where ˆ h ( k ) is not close enough to ˆ f ( k ) , exp ( −| πk/ 2 w j | ) would dominate G lj

Original: ( θ j , k ) for a small w j .
Revised: ( θ j , k ) for a small w j .

Original: Through the above framework of analysis, we have the following theorem.
Revised: Through the above framework of analysis, we have the following theorem.

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: Theorem.
Revised: Theorem.

Original: Consider a one hidden layer DNN with activation function σ ( x ) =
Revised: Consider a one hidden layer DNN with activation function σ ( x ) =

Original: tanh x .
Revised: tanh x .

Original: For any frequencies k 1 and k 2 such that | ˆ f ( k 1 ) | > 0 , | ˆ f ( k 2 ) | > 0 , and | k 2 | > | k 1 | > 0 , there exist positive constants c and C such that for sufﬁciently small δ , we have 
Revised: For any frequencies k 1 and k 2 such that | ˆ f ( k 1 ) | > 0 , | ˆ f ( k 2 ) | > 0 , and | k 2 | > | k 1 | > 0 , there exist positive constants c and C such that for sufﬁciently small δ , we have [Equation] for all l

Original: [Equation]
Revised: [Equation]

Original: where B δ ⊂ R m is a ball with radius δ centered at the origin and µ ( · ) is the Lebesgue measure.
Revised: where B δ ⊂ R m is a ball with radius δ centered at the origin and µ ( · ) is the Lebesgue measure.

Original: We remark that c and C depend on k 1 , k 2 , | ˆ f ( k 1 ) | , | ˆ f ( k 2 ) | , sup | a i | , sup | b i | , and m .
Revised: We remark that c and C depend on k 1 , k 2 , | ˆ f ( k 1 ) | , | ˆ f ( k 2 ) | , sup | a i | , sup | b i | , and m .

Original: Proof.
Revised: Proof.

Original: To prove the statement, it is sufﬁcient to show that µ ( S lj,δ ) /µ ( B δ ) ≤ C exp( − c/δ ) for each l, j , where
Revised: To prove the statement, it is sufﬁcient to show that µ ( S lj,δ ) /µ ( B δ ) ≤ C exp( − c/δ ) for each l, j , where

Original: [Equation]
Revised: [Equation]

Original: We prove this for S 1 j,δ , that is, θ lj = a j .
Revised: We prove this for S 1 j,δ , that is, θ lj = a j .

Original: The proofs for θ lj = w j and b j are similar.
Revised: The proofs for θ lj = w j and b j are similar.

Original: Without loss of generality, we assume that k 1 , k 2 > 0 , b j > 0 , and w j (cid:54) = 0 , j = 1 , · · · , m .
Revised: Without loss of generality, we assume that k 1 , k 2 > 0 , b j > 0 , and w j (cid:54) = 0 , j = 1 , · · · , m .

Original: According to Eq.
Revised: According to Eq.

Original: (24), the inequality | ∂L ( k 1 )
Revised: (24), the inequality | ∂L ( k 1 )

Original: ∂a
Revised: ∂a

Original: j | ≤ | ∂L ( k 2 )
Revised: j | ≤ | ∂L ( k 2 )

Original: ∂a
Revised: ∂a

Original: j | is equivalent to
Revised: j | is equivalent to

Original: A ( k

Original: [Equation]
Revised: [Equation]

Original: Note that | ˆ h ( k ) | ≤ C (cid:80) mj
Revised: Note that | ˆ h ( k ) | ≤ C (cid:80) mj

Original: =1 | a j | | w j | exp( − πk 2 | w j | ) for k > 0 .
Revised: =1 | a j | | w j | exp( − πk 2 | w j | ) for k > 0 .

Original: Thus lim 
Revised: Thus lim W

Original: [Equation]{+→ 0 ˆ h ( k ) = 
+}
Revised: {+→ 0 ˆ h ( k ) = 
+}

Original: and 
Revised: and lim

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Therefore, lim W [Equation]
Revised: Therefore, lim W → 0 A ( k ) =

Original:  and lim 
Revised: | ˆ f ( k ) | and lim W →

Original: [Equation]
Revised: [Equation]

Original: For W ∈ B δ with sufﬁciently small δ , A ( k 1 ) > 12 | ˆ f ( k 1 ) | > 0 and A ( k 2 ) < 2 | ˆ f ( k 2 ) | .
Revised: For W ∈ B δ with sufﬁciently small δ , A ( k 1 ) > 12 | ˆ f ( k 1 ) | > 0 and A ( k 2 ) < 2 | ˆ f ( k 2 ) | .

Original: Also note that | sin( b j k 2 w j − φ ( k 2 )) | ≤ 1 and that for sufﬁciently small δ ,
Revised: Also note that | sin( b j k 2 w j − φ ( k 2 )) | ≤ 1 and that for sufﬁciently small δ ,

Original: [Equation]
Revised: [Equation]

Original: Thus, inequality (32) implies that
Revised: Thus, inequality (32) implies that

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Noticing that 2 π | x | ≤ | sin x | ( | x | ≤ π 2 ) and Eq.
Revised: Noticing that 2 π | x | ≤ | sin x | ( | x | ≤ π 2 ) and Eq.

Original: (34), we have for W ∈ S lj,δ , for some q ∈ Z ,
Revised: (34), we have for W ∈ S lj,δ , for some q ∈ Z ,

Original: [Equation]
Revised: [Equation]

Original:  that is,
Revised: ) that is,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where c 1 = 8 π | ˆ f ( k 2 ) | | ˆ f ( k 1 ) | and c 2 = π ( k 2 − k 1 ) .
Revised: where c 1 = 8 π | ˆ f ( k 2 ) | | ˆ f ( k 1 ) | and c 2 = π ( k 2 − k 1 ) .

Original: Deﬁne I :
Revised: Deﬁne I :

Original: = I + ∪ I − where
Revised: = I + ∪ I − where

Original: I + := { w j > 0 :W ∈ S 1 j,δ } ,
Revised: I + := { w j > 0 :

Original: [Equation]
Revised: [Equation]

Original: { w j
Revised: ∈

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: For w j > 0 , we have for some q ∈ Z ,
Revised: For w j > 0 , we have for some q ∈ Z ,

Original: [Equation]
Revised: [Equation]

Original: Since W ∈ B δ and c 1 exp( − c 2 /δ )
Revised: Since W ∈ B δ and c 1 exp( − c 2 /δ )

Original: + arg
Revised: + arg

Original: ( ˆ f ( k 1 )) ≤ 2 π , we have b j k 1 

Revised: ( ˆ f ( k 1 )) ≤ 2 π , we have b j k 1

Original: π + qπ ≤ w j ≤ δ .
Revised: π + qπ ≤ w j ≤ δ .

Original: Then Eq.
Revised: Then Eq.

Original: (40) only holds for some large q , more precisely, q ≥ q 0 :=
Revised: (40) only holds for some large q , more precisely, q ≥ q 0 :=

Original: b j k πδ − 2 .
Revised: b j kπδ − 2 .

Original: Thus we obtain the estimate for the (one-dimensional) Lebesgue measure of I + µ ( I + ) ≤∞ (cid:88)
Revised: Thus we obtain the estimate for the (one-dimensional) Lebesgue measure of I + µ ( [Equation] ≤

Original: [Equation]
Revised: [Equation]

Revised: (cid:12) (cid:12) b

Revised: j k− c 1 exp( − c 2 /δ )

Revised: + qπ

Revised: + arg

Revised: ( ˆ f ( k 1 ))

Revised: − b j kc 1 exp( − c 2 /δ )

Revised: + qπ

Revised: + arg

Revised: ( ˆ f ( k 1 )) (cid:12)(cid:12) (cid:12) (cid:12) (cid:12) ≤ 2 |

Revised: b

Revised: j | k 1 c 1 exp( − c 2 /δ ) · ∞ (cid:88) q = q ( qπ + arg

Original:  The similar estimate holds for µ ( I − ) , and hence µ ( I ) ≤ C exp( − c/δ ) .
Revised: ( ˆ f ( k 1 ))) 2 − ( c 1 exp( − c 2 /δ )) ≤ C exp( − c/δ ) . The similar estimate holds for µ ( I − ) , and hence µ ( I ) ≤ C exp( − c/δ ) .

Original: For W ∈ B δ , the ( m − 1) dimensional vector ( w 1 , · · · , w j − 1 , w j +1 , · · · , w m ) T is in a ball with radius δ in R m − 1 .
Revised: For W ∈ B δ , the ( m − 1) dimensional vector ( w 1 , · · · , w j − 1 , w j +1 , · · · , w m ) T is in a ball with radius δ in R m − 1 .

Original: Therefore, we ﬁnal arrive at the desired estimate
Revised: Therefore, we ﬁnal arrive at the desired estimate

Original: [Equation]
Revised: [Equation]

Original: where ω m is the volume of a unit ball in R m .
Revised: where ω m is the volume of a unit ball in R m .

Original: Theorem.
Revised: Theorem.

Original: Considering a DNN of one hidden layer with activation function σ ( x ) = tanh( x ) .
Revised: Considering a DNN of one hidden layer with activation function σ ( x ) = tanh( x ) .

Original: Suppose the target function has only two non-zero frequencies k 1 and k 2 , that is, | ˆ f ( k 1 ) | > 0 , | ˆ f ( k 2 ) | > 0 , and | k 2 | > | k 1 | > 0 , and | ˆ f ( k ) | = 0 for k
Revised: Suppose the target function has only two non-zero frequencies k 1 and k 2 , that is, | ˆ f ( k 1 ) | > 0 , | ˆ f ( k 2 ) | > 0 , and | k 2 | > | k 1 | > 0 , and | ˆ f ( k ) | = 0 for k

Original: (cid:54) =
Revised: (cid:54) =

Original: k 1 , k 2 .
Revised: k 1 , k 2 .

Original: Consider the loss function of
Revised: Consider the loss function of

Original: L = L ( k 1 ) + L ( k 2 ) with gradient descent training.
Revised: L = L ( k 1 ) + L ( k 2 ) with gradient descent training.

Original: Denote
Revised: Denote

Original: [Equation]
Revised: [Equation]

Original: that is, L ( k 1 ) decreases faster than L ( k 2 ) .
Revised: that is, L ( k 1 ) decreases faster than L ( k 2 ) .

Original: There exist positive constants c and C such that for sufﬁciently small δ , we have
Revised: There exist positive constants c and C such that for sufﬁciently small δ , we have

Original: [Equation]
Revised: [Equation]

Original: where B δ ⊂ R m is a ball with radius δ centered at the origin and µ ( · ) is the Lebesgue measure.
Revised: where B δ ⊂ R m is a ball with radius δ centered at the origin and µ ( · ) is the Lebesgue measure.

Original: Proof.
Revised: Proof.

Original: By gradient descent algorithm, we obtain
Revised: By gradient descent algorithm, we obtain

Original: [Equation]
Revised: [Equation]

Original: (cid:18) ∂L ( k 
Revised: l,j ∂L ( k 1 ) ∂θ lj ∂

Original: [Equation]
Revised: ( L ( k 1 ) + L ( k 2 )) ∂θ lj = − (cid:88) l,j (cid:18) ∂L ( k 1 ) ∂θ lj (cid:19)− (cid:88) l,j ∂L ( k 1 ) ∂θ lj ∂L ( k 2 ) ∂θ lj , ∂L ( k 2 ) ∂t = − (cid:88) l,j (cid:18) ∂L ( k 2 )

Original: ∂θ lj (cid:19) [Equation] − (cid:88) 
Revised: ∂θ lj (cid:19)− (cid:88) l,j ∂L ( k 1 )

Revised: ∂θ lj ∂L ( k 2 )

Original: [Equation]
Revised: ∂θ lj , and ∂L ∂t = ∂ ( L ( k 1 ) + L ( k 2 )) ∂t = − (cid:88) l,j (cid:18) ∂L ( k 1 ) ∂θ lj + ∂L ( k 2 )

Original: and ∂L
Revised: ∂θ lj

Original: [Equation]
Revised: (cid:19)≤ 0 .

Original: To obtain [Equation]
Revised: To obtain 0 < ∂L ( k 1 ) ∂t − ∂L ( k 2 ) ∂t = − (cid:88) l,j (cid:34)(cid:18) ∂L ( k 1 ) ∂θ lj (cid:19)− (cid:18) ∂L

Original:  it is sufﬁcient to have 
Revised: ( k 2 ) ∂θ lj (cid:19) 2 (cid:35) , (44) it is sufﬁcient to have (cid:12)(cid:12)(cid:12)(cid:12) ∂L ( k 1 )

Original: [Equation]
Revised: ∂θ lj (cid:12)(cid:12)(cid:12)(cid:12) > (cid:12)(cid:12)(cid:12)(cid:12) ∂L ( k 2 ) ∂θ lj (cid:12)(cid:12)(cid:12)(cid:12) .

Original: Eqs.
Revised: Eqs.

Original: (43, 44) also yield to
Revised: (43, 44) also yield to

Original: [Equation]
Revised: [Equation]

Original: Therefore, Eq.
Revised: Therefore, Eq.

Original: (45) is a sufﬁcient condition for S .
Revised: (45) is a sufﬁcient condition for S .

Original: Based on the theorem 1, we have proved the theorem 2. 
Revised: Based on the theorem 1, we have proved the theorem 2. [Figure]

Original: E M EMORIZING 2 - D IMAGE
Revised: F M EMORIZING 2 - D IMAGE

Original: We train a DNN to ﬁt a natural image (See Fig. 6(a)), a mapping from coordinate ( x, y ) to gray scale strength, where the latter is subtracted by its mean and then normalized by the maximal absolute value.
Revised: We train a DNN to ﬁt a natural image (See Fig. 6(a)), a mapping from coordinate ( x, y ) to gray scale strength, where the latter is subtracted by its mean and then normalized by the maximal absolute value.

Original: First, we initialize DNN parameters by a Gaussian distribution with mean 0 and standard deviation 0 .
Revised: First, we initialize DNN parameters by a Gaussian distribution with mean 0 and standard deviation 0 .

Original: 08 (initialization with small parameters).
Revised: 08 (initialization with small parameters).

Original: From the snapshots during the training process, we can see that the DNN captures the image from coarse-grained low frequencies to detailed high frequencies (Fig. 6(b)).
Revised: From the snapshots during the training process, we can see that the DNN captures the image from coarse-grained low frequencies to detailed high frequencies (Fig. 6(b)).

Original: As an illustration of the F-Principle, we study the Fourier transform of the image with respect to x for a ﬁxed y (red dashed line in Fig. 6(a), denoted as the target function f ( x ) in the spatial domain).
Revised: As an illustration of the F-Principle, we study the Fourier transform of the image with respect to x for a ﬁxed y (red dashed line in Fig. 6(a), denoted as the target function f ( x ) in the spatial domain).

Original: The DNN can well capture this 1-d slice after training as shown in Fig. 6(c).
Revised: The DNN can well capture this 1-d slice after training as shown in Fig. 6(c).

Original: Fig.
Revised: Fig.

Original: 6(d) displays the amplitudes | ˆ f ( k ) | of the ﬁrst 40 frequency components.
Revised: 6(d) displays the amplitudes | ˆ f ( k ) | of the ﬁrst 40 frequency components.

Original: Due to the small initial[Figure] parameters, as an example in Fig.
Revised: Due to the small initial parameters, as an example in Fig.

Original: 6( low-frequency components, high frequencies stay relatively small.
Revised: 6(d), when the DNN is ﬁtting low-frequency components, high frequencies stay relatively small.

Original: As the relative error shown in Fig. 6(e), the ﬁrst ﬁve frequency peaks converge from low to high in order.
Revised: As the relative error shown in Fig. 6(e), the ﬁrst ﬁve frequency peaks converge from low to high in order.

Original: Next, we initialize DNN parameters by a Gaussian distribution with mean 0 and standard deviation(initialization with large parameters).
Revised: Next, we initialize DNN parameters by a Gaussian distribution with mean 0 and standard deviation(initialization with large parameters).

Original: After training, the DNN can well capture the training data, as shown in the left in Fig. 6(f).
Revised: After training, the DNN can well capture the training data, as shown in the left in Fig. 6(f).

Original: However, the DNN output at the test pixels are very noisy, as shown in the right in Fig. 6(f).
Revised: However, the DNN output at the test pixels are very noisy, as shown in the right in Fig. 6(f).

Original: For the pixels at the red dashed lines in Fig. 6(a), as shown in Fig. 6(g), the DNN output ﬂuctuates a lot.
Revised: For the pixels at the red dashed lines in Fig. 6(a), as shown in Fig. 6(g), the DNN output ﬂuctuates a lot.

Original: Compared with the case of small initial parameters, as shown in
Revised: Compared with the case of small initial parameters, as shown in

Original: Fig. 6(h), the convergence order of the ﬁrst ﬁve frequency peaks do not have a clear order.
Revised: Fig. 6(h), the convergence order of the ﬁrst ﬁve frequency peaks do not have a clear order.