From: ICLR.cc/2023/Conference/Paper4638/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper4638/Authors
title: We have submitted our revised manuscript.
comment: We would like to thank the reviewers for the suggestions that have a great improvement of the presentation of our results. Now, our revised manuscript and supplementary materials are available in OpenReview.
Major corrections done in the manuscript are indicated by blue text. In addition to our responses to the reviewers' valuable comments, we have corrected the following to improve the presentation of our results.

* We read our manuscript carefully and corrected several typos and grammatical errors.
* To enhance the readability of our manuscript, we changed several notations to more accessible ones. For instance, we avoided the usage of the restriction operator $R_i$ introduced in the previous manuscript. Moreover, we replaced the term "$r_i$-minimization problem" with "$(w_i, b_i)$-minimization problem", which is easier to follow.

From: ICLR.cc/2023/Conference/Paper4638/Authors
title: Responses to Reviewer kjxn (Part 2/2)
comment: Comment 4. The numerical results are not sufficient to support that the proposed method works better than existing methods. Most experiments are done on simple problems.

Response 4. In the revised manuscript, we added additional numerical results in order to support that NPSC works better than the other methods:

    - Example 3 is replaced by a more difficult one: a highly oscillatory elliptic PDE when $f(x) = \cos 3\pi x + \cos 11\pi x$.

    - Decay of the energy error is shown for $10^3$ epochs in Figures 2(a,d) and 3(a,c) (20 epochs in the previous manuscript).

    - Numerical solutions obtained by all algorithms are plotted in Figure 2(b,e).

    - $L^2$-errors with respect to the number of neurons are presented for all training algorithms in Figures 2(c,f) and 3(b,d). 

    - $H^1$-errors in the previous manuscript are removed because they show almost the same behavior as $L^2$-errors so that no additional discussions are needed.


According to the additional results, we revised discussions in Section 4. Now, the following features of NPSC are highlighted in Section 4.

    - Only the $L^2$-errors with respect to the number of neurons generated by NPSC are comparable to the theoretical optimal rates.

    - NPSC captures both low- and high-frequency parts of the target function well, while the other algorithms captures the low-frequency part only. The fact that conventional training algorithms captures the low-frequency only was explained rigorously in [XZLXZ:2020].

    - NPSC achieves a superior level of accuracy such that the other algorithms do not reach even with a large number of epochs.

    - NPSC works well for high-dimensional problems as well.


[HSTX:2022] Hong, Tan, Siegel, and Xu. On the activation function dependence of the spectral bias of neural networks. arXiv preprint arXiv:2208.04924, 2022.

[XZLXZ:2020] Xu, Zhang, Luo,  Xiao, and Ma. Frequency principle: Fourier analysis sheds light on deep neural networks. Commun. Comput. Phys., 28(5):1746-1767, 2020.

From: ICLR.cc/2023/Conference/Paper4638/Authors
title: Responses to Reviewer kjxn (Part 1/2)
comment: We would like to thank the reviewer for valuable comments that led great improvement on our manuscript. The following are comment-by-comment responses to your review report.

Comment 1. The application of the proposed algorithm is restricted. It is designed only for two-layer networks, which is not used in practice. A useful algorithm should be able to be applied on more general network architectures.

Response 1. Our main motivation for this manuscript is that, even a simple two-layer neural network for a simple task hardly achieves the theoretically guaranteed approximation properties in practice with conventional training algorithms. Based on a theoretical understanding of ill-conditioning of layers [HTSX:2022], we presented how to resolve this issue. Our method is nontrivial and, to the best of our knowledge, it is the first result that resolves this issue. We believe that a complete understanding of a simple environment must precede that of a complex environment, that is, efficient and accurate training of more complicated neural network architecture would not be possible if we cannot make the simplest model (given enough approximation power) work. Therefore, our algorithm and results play an important role in designing relevant training algorithms for more complicated networks.

To highlight our claim, we added the following paragraph at the end of Section 5: ``Reducing the gap between the theoretical approximation properties and training results is a challenging issue even for simple neural networks. To the best of our knowledge, NPSC is the first result that deals with the ill-conditioning of layers and successfully reduces the gap. We believe that this paper can play a role of a good preliminary work for efficient and accurate training of general network architectures.''

Comment 2. The LM method for neurons become inefficient when the dimension of the input becomes large.

Response 2. As the reviewer indicated, the Levenberg--Marquardt algorithm becomes inefficient when the input dimension $m$ becomes large because it has to solve an $m \times m$ linear system at each iteration. Nevertheless, we would like to emphasize that the Levenberg--Marquardt algorithm is a reasonable strategy since we aim to solve PDEs; PDEs that we are mostly interested usually arise in physics and the dimensions are not very big in general. We tested various algorithms including the Levenberg--Marquardt algorithm as local solvers, and observed that the Levenberg-Marquardt algorithm is most efficient; the problem size is small enough and the convergence rate is much faster that that of gradient-based methods. 

In the last part of Section 3.3, we added the following statement: "Various optimization algorithms including first- and second-order methods can be used to solve (8). Among them, a good option is the Levenberg--Marquardt algorithm; PDEs usually arise from physics, so that the dimension $d+1$ of $r_i$ is not very big in general."

Comment 3. The comparison with SGD and Adam in the numerical experiments is not fair. The NPSC proposed in the paper consists inner problems in each iteration. Hence, in each iteration NPSC does much more computation than SGD and Adam. The comparison should be made on the same cpu time, or matching the inner iterations of NPSC with the iterations of SGD.

Response 3. It does not make sense to compare the computation time of NPSC with the others directly because the others never reach the same level of accuracy as NPSC. In other words, if we use either GD or Adam to obtain the NPSC level of accuracy, then it requires almost infinitely many epochs. To support our claim, we added new numerical results for $10^3$ epochs of various training algorithms in Figures 2(a,d) and 3(a,c). In these figures, we can observe that all algorithms other than NPSC never achieves the NPSC level even with a large number of epochs. Furthermore, the required additional computational time can be largely reduced because all the single neuron problems are solved simultaneously in parallel computers.

From: ICLR.cc/2023/Conference/Paper4638/Authors
title: Responses to Reviewer Csdy (Part 2/2)
comment: Comment 5. In the paragraph below Theorem 1, missing $)$ after $b_i$

Response 5. We corrected the typo indicated by the reviewer. In addition, we carefully read our manuscript and corrected several other typos.

Comment 6. In Theorem 1 as well as the paragraph below it, it seems not clear to me what the linear dependence of a set of functions. It would be great if authors could clearly define it.

Response 6. At the first appearance of "linear dependence'' in the main text, we added the following definition of linear dependence of the functions $\sigma (w_i \cdot x + b_i ) $'s: there exist $a_i \in R$, $1 \leq i \leq n$, not all zero, such that $\sum_{i=1}^n a_i \sigma (w_i \cdot x + b_i) = 0$

Comment 7. In the last part of Section 3.3, the I believe the convergence guarantee of Levenberg-Marquardt algorithm is likely not a global convergence, but a local convergence or under certain assumptions. The current writing seems to be misleading.

Response 7. We thank to the reviewer for indicating our misleading statement in the previous manuscript. In the last sentence of Section~3.3, we clarified that the Levenberg--Marquardt algorithm converges to a local minimum as follows: "It converges to a local minimum with the superlinear convergence rate."

[HSTX:2022] Hong, Tan, Siegel, and Xu. On the activation function dependence of the spectral bias of neural networks. arXiv preprint arXiv:2208.04924, 2022.

[VYS:2021] Vardi, Yehudai, and Shamir. Learning a single neuron with bias using gradient descent. NeurIPS 2021.

[LSSK:2020] Lu, Shin, Su, and Karniadakis. Dying ReLU and initialization: Theory and numerical examples. Commun. Comput. Phys., 28(5):1671:1706, 2020.

From: ICLR.cc/2023/Conference/Paper4638/Authors
title: Responses to Reviewer Csdy (Part 1/2)
comment: We would like to thank the reviewer for valuable comments that led great improvement on our manuscript. The following are comment-by-comment responses to your review report.

Comment 1. The proposed algorithm only works for two-layer ReLU networks. It is not clear if such a method could be used in deep networks and other activations.

Response 1. Our main motivation for this manuscript is that, even a simple two-layer neural network for a simple task hardly achieves the theoretically guaranteed approximation properties in practice with conventional training algorithms. Based on a theoretical understanding of ill-conditioning of layers [HSTX:2022], we presented how to resolve this issue. Our method is nontrivial and, to the best of our knowledge, it is the first result that resolves this issue. We believe that a complete understanding of a simple environment must precede that of a complex environment, that is, efficient and accurate training of more complicated neural network architecture would not be possible if we cannot make the simplest model (given enough approximation power) work. Therefore, our algorithm and results play an important role in designing relevant training algorithms for more complicated networks.

To highlight our claim, we added the following paragraph at the end of Section 5: "Reducing the gap between the theoretical approximation properties and training results is a challenging issue even for simple neural networks. To the best of our knowledge, NPSC is the first result that deals with the ill-conditioning of layers and successfully reduces the gap. We believe that this paper can play a role of a good preliminary work for efficient and accurate training of general network architectures.''

Comment 2. The precondition step (Theorem 2) seems only work on 1-dimensional setting, which greatly limits the possibility of using the proposed algorithm in the general settings.

Response 2. As shown in our manuscript, designing an optimal preconditioner is not a trivial task even for one-dimensional problems.
Generalizing our preconditioner to multidimensional problems should be a separate task that is worth more effort and attention, given its importance in designing optimal linear solvers. Nevertheless, a great news for our proposed algorithm is that efficient iterative solvers such as the conjugate gradient method can be always used in multidimensional problems as well under our novel subspace correction framework for neural networks.  Indeed, we provided a numerical example that solves a two-dimensional PDE and verified that our method still works best among the existing training algorithms.

Comment 3. Theorem 3 does not seem to be strong in the sense that it only shows if at initialization $E_i(w_i^0,b_i^0)$ is smaller than $E_i(0)$, then the value will always be smaller than $E_i(0)$. However, it does not give any convergence guarantee and the algorithm may converge to some local minima instead of global minima.

Response 3. Theorem 3 implies that training each neuron in the nonlinear layer separately has an advantage that flat regions can be easily avoided [VYS:2021]. Avoiding the flat regions is very important because the flat regions make the neuron dead, i.e., the function vanishes on the domain [LSSK:2020]. In this viewpoint, we believe that Theorem~3 is meaningful since it shows that neuron-wise parallel training prevents dead neurons.
We corrected our writing of Section 3.3 to further highlight the meaning of Theorem 3 and the importance of avoiding the flat regions.
In particular, we modified the first paragraph of Section 3.3 as follows: "Training the nonlinear layer is challenging because of its nonconvexity. Moreover, it was shown in [LSSK:2020] that training  of ReLU networks usually suffers from dead neurons. In the following, we claim that training each neuron in the nonlinear layer separately has an advantage that some "good'' local minima that avoid dead neurons can be found."

Comment 4. In the experiments, it seems that the algorithms only run for 20 iterations (from Figure 2 and 3). I was wondering if the results of running the algorithms for longer iterations could be provided, as some of the algorithms do not seem to converge (e.g., Figure 3(a)(d)).

Response 4. We replaced all the numerical results for 20 iterations in Figures 2 and 3 by the ones for $10^3$ iterations; see Figures 2(a,d) and 3(a,c) in the revised manuscript. In these figures, we can observe that NPSC always outperforms the other algorithms. We modified the discussions related to Figures 2 and 3 accordingly to further clarify our results.

From: ICLR.cc/2023/Conference/Paper4638/Authors
title: Responses to Reviewer buJi
comment: We would like to thank the reviewer for valuable comments that led great improvement on our manuscript. The following are comment-by-comment responses to your review report.

Comment 1. The choice of the architecture made by the authors is very restrictive. The authors consider a two layer neural network in their work, while in practice the architectures considered are very complicated. It is hard to say if the method proposed will scale (for example, how easy will it be to maintain linear independence in a network with multiple hidden layers, or handle preconditioning etc)

Response 1. Our main motivation for this manuscript is that, even a simple two-layer neural network for a simple task hardly achieves the theoretically guaranteed approximation properties in practice with conventional training algorithms. Based on a theoretical understanding of ill-conditioning of layers [HSTX:2022], we presented how to resolve this issue. Our method is nontrivial and, to the best of our knowledge, it is the first result that resolves this issue. We believe that a complete understanding of a simple environment must precede that of a complex environment, that is, efficient and accurate training of more complicated neural network architecture would not be possible if we cannot make the simplest model (given enough approximation power) work. Therefore, our algorithm and results play an important role in designing relevant training algorithms for more complicated networks.
To highlight our claim, we added the following paragraph at the end of Section 5:
"Reducing the gap between the theoretical approximation properties and training results is a challenging issue even for simple neural networks. To the best of our knowledge, NPSC is the first result that deals with the ill-conditioning of layers and successfully reduces the gap. We believe that this paper can play a role of a good preliminary work for efficient and accurate training of general network architectures.''

Comment 2. It also seems that the algorithm proposed will become quite compute intense as the number of parameters in the network increase.

Response 2. The preconditioner presented in Theorem~2 ensures that the number of iterations to solve $c$-minimization problems do not increase even if the number of neurons $n$ becomes very big. Moreover, $r_i$-minimization problems are solved independently to each other, so that they can be efficiently processed on distributed memory computers. Hence, the computational cost of the proposed algorithm do not increase dramatically when $n$ becomes big.
In addition, we would like to mention that our framework allows us to use various algorithms to solve local problems. We tested various algorithms including the Levenberg--Marquardt algorithm as local solvers, and observed that the Levenberg-Marquardt algorithm is most efficient; the problem size is small enough and the convergence rate is much faster that that of gradient-based methods.

Comment 3. The empirical analysis is not very extensive. The authors only test their methodology in simple functions in 1 dimensions, and for simple PDEs like the Elliptic equation. Can they show how their method works for other equations (like Burgers?).

Response 3. Most numerical algorithms are not universal in the sense that each problem has its own most efficient algorithm. Given the different nature of elliptic and hyperbolic PDEs, it is more appropriate to design a separate algorithm to solve hyperbolic PDEs such as the Burgers' equation efficiently. Our algorithm is designed to solve elliptic PDEs, which already has numerous applications in mathematics, physics, etc. All the analysis, problem settings,  given in our manuscript are for elliptic PDEs.

Comment 4. it says that $\tilde{w}_i$ is equal to $\tilde{w}_i/|w_i|$. Is that correct? or is that a potential typo?

Response 4. This is correct; by substituting $\tilde{w}_i = (w_i, b_i)$ with $\tilde{w}_i / |w_i|$, $w_i$ is normalized and $b_i$ is scaled accordingly. To avoid confusion, we replaced "$\tilde{w}_i \leftarrow \tilde{w}_i / |w_i|$'' in Algorithm 2 by "$w_i \leftarrow w_i / |w_i|$ and $b_i \leftarrow b_i / |w_i|$."

[HSTX:2022] Hong, Tan, Siegel, and Xu. On the activation function dependence of the spectral bias of neural networks. arXiv preprint arXiv:2208.04924, 2022.

From: ICLR.cc/2023/Conference/Paper4638/Reviewer_buJi
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper introduces a methodology called Neuron-wise Parallel Subspace correction method (NPSC), which disentangles the optimization of the outer parameters from the inner parameters in a two-layer ReLU network. The primary motivation behind this is that for certain activations, neural networks fail to converge fast even for simple functions, and therefore, the convergence can be slow for applying NNs for approximating solutions to PDEs.

The algorithm optimizes the outer parameters c (jointly) and then for each weight parameters w_i in the hidden layer, it learns what residual term needs to be added to each weight parameter. In order to achieve this, they propose a transformation to ensure that each neural is “active”, i.e, the input to ReLU is not negative, which ensure the functions are linearly independent. 

In terms for experiments, the authors show for simple functions in 1 dimension that NPSE converges much faster (as opposed to usual training techniques like ADAM and GD) and show results solution to Elliptic PDE in 1 dimension.
strength_and_weaknesses: [Strengths]

The fact that gradient descent is slow for simple functions is a very known phenomena and the methodology proposed by the authors seems to be working quite well (in terms of the convergence rate). 

This is useful for PDEs since the solutions to a few PDEs can comprise of some of  these simple functions. 

The theory seems correct. 

[Weaknesses]

- The choice of the architecture made by the authors is very restrictive. The authors consider a two layer neural network in their work, while in practice the architectures considered are very complicated. It is hard to say if the method proposed will scale (for example, how easy will it be to maintain linear independence in a network with multiple hidden layers, or handle preconditioning etc)
- It also seems that the algorithm proposed will become quite compute intense as the number of parameters in the network increase.
- The empirical analysis is not very extensive. The authors only test their methodology in simple functions in 1 dimensions, and for simple PDEs like the Elliptic equation. Can they show how their method works for other equations (like Burgers?).

A small question to the authors,

it says that \tilde{w}_i is equal to \tilde{w}_i/|w_i|. Is that correct? or is that a potential typo?
clarity,_quality,_novelty_and_reproducibility: The paper is clear and easy to read. 
summary_of_the_review: The authors propose NPSC an algorithm to get faster convergence than GD/ADAM for simpler functions, and potentially helping faster PDE solvers. 
The architecture considered is too restrictive and he main contribution in my opinion would be the empirical analysis is not very extensive.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4638/Reviewer_Csdy
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: In this paper, the authors proposed a new algorithm called Neuron-wise Parallel Subspace Correction Method (NPSC) for solving partial differential equations (PDEs) using two-layer ReLU networks. The proposed algorithm is based on a classical method called subspace correction method, which optimizes the first layer and second layer separately. To deal with the ill-condition issue when optimizing the second layer, a precondition matrix is proposed to solve this issue. Numerical experiments are provided to show the performance of the proposed method, which is better than other methods in these examples.
strength_and_weaknesses: Strength:
- The paper is overall clearly written and east to follow.
- Using neural networks to solve PDEs is an important research direction, and this paper proposed a new algorithm for this problem.
- The proposed algorithm is simple and clearly stated. Experiments seem to show that it performs well against other methods.

Weaknesses:
- The proposed algorithm only works for two-layer ReLU networks. It is not clear if such a method could be used in deep networks and other activations. Moreover, the precondition step (Theorem 2) seems only work on 1-dimensional setting, which greatly limits the possibility of using the proposed algorithm in the general settings.
- Theorem 3 does not seem to be strong in the sense that it only shows if at initialization $E_i(w_i^0, b_i^0)$ is smaller than $E_i(0)$, then the value will always be smaller than $E_i(0)$. However, it does not give any convergence guarantee and the algorithm may converge to some local minima instead of global minima.
- In the experiments, it seems that the algorithms only run for 20 iterations (from Figure 2 and 3). I was wondering if the results of running the algorithms for longer iterations could be provided, as some of the algorithms do not seem to converge (e.g., Figure 3(a)(d)).

Minor:
- In the paragraph below Theorem 1, missing ) after $b_i$

clarity,_quality,_novelty_and_reproducibility: Clarity:
- The paper is overall easy to follow and well-written.
- In Theorem 1 as well as the paragraph below it, it seems not clear to me what the linear dependence of a set of functions. It would be great if authors could clearly define it.
- In the last part of Section 3.3, the I believe the convergence guarantee of Levenberg-Marquardt algorithm is likely not a global convergence, but a local convergence or under certain assumptions. The current writing seems to be misleading.

Quality:
- The quality of the paper is overall good. The proposed algorithm is clearly stated. Experiments are provided to show its performance.

Novelty:
- The proposed algorithm seems to be new, and it aims to tackle an important problem of using neural networks to solve PDEs.

Reproducibility:
- The full proofs of the theorems are provided in the appendix. The codes are provided in the supplementary material. I didn’t check/run them.

summary_of_the_review: In summary, in this paper the authors proposed a new algorithm NPSC to use two-layer ReLU network to solve PDEs. While the ideas are simple and empirical results seem to be good, as mentioned in the weakness part, it seems the current method is quite limited to two-layer ReLU network with 1-dimensional input, which makes the reviewer to worry about the possibility of using it in the general settings. Therefore, I'm currently leaning towards rejection.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4638/Reviewer_kjxn
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposed a new training method, called Neuron-wise Parallel Subspace Correction Method, to train two-layer neural networks to learn function approximation or PDE problems. The method decomposes the update of the output layer parameters with the update of the input layer parameters. For the output layer parameters, a conjugate gradient method with preconditioner is used to solve the least squares problem. For the input layer problems, a Levenberg-Marquardt algorithm is applied separately on each neural, which can theoretically achieve super linear convergence. The algorithm is tested on l2 function approximation and elliptic PDE problems, and shows faster and better convergence compared with other commonly used optimization algorithms (with the same number of iterations).
strength_and_weaknesses: Strength:

The paper studies the optimization of neural networks for scientific computing applications, which is not focused on before. 

Weaknesses:

1. The application of the proposed algorithm is restricted. It is designed only for two-layer networks, which is not used in practice. Also, the LM method for neurons become inefficient when the dimension of the input becomes large. A useful algorithm should be able to be applied on more general network architectures.
2. The comparison with SGD and Adam in the numerical experiments is not fair. The NPSC proposed in the paper consists inner problems in each iteration. Hence, in each iteration NPSC does much more computation than SGD and Adam. The comparison should be made on the same cpu time, or matching the inner iterations of NPSC with the iterations of SGD. 
clarity,_quality,_novelty_and_reproducibility: Clarity: The writing is clear.

Quality: The numerical results are not sufficient to support that the proposed method works better than existing methods. 

Novelty: The proposed method is new for training neural networks. 

Reproducibility: Most experiments are done on simple problems. I am not concerned about the reproducibility of the experiment results. 
summary_of_the_review: This paper proposes a new method to train two-layer neural networks on PDE problems. It's range of application is limited, and the numerical results showed in the paper are not sufficient. 
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: Not applicable
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough