Paper: An operator preconditioning perspective on training in physics-informed machine learning (ICLR-2024)

Reviewer comment (Experiments):
The experimental section of the paper is quite weak. It solely provides a method for reducing the condition number in linear PDEs using PINNs based on the theoretical analysis presented, which considerably narrows the applicability of the proposed method.

Author response:
1. (W1) Regarding the reviewer's concern that the **The experimental section of the paper is quite weak**, we would like to, at length, frame the contributions of our paper. To this end, we take the liberty to repeat the relevant portion of our reply to all the reviewers above here: At the outset, it is amply clear in the literature that a major issue with physics-informed machine learning models such as PINNs is the difficulty of training them. This has been widely recognised and several ad-hoc strategies have been proposed to improve training. However, it is also clear that a suitable theoretical framework that precisely formulates the problem of optimizing physics-informed loss functions was missing. We believe that a lack of precision in defining what the problem actually is also impedes finding suitable measures for ameliorating the problem itself. 

    With this background in mind, our main aim in the paper was precisely to propose a suitable theoretical framework in which the issues affecting training of physics-informed machine learning models can be formulated and analyzed. To this end, we have rigorously connected the rate of convergence of gradient descent algorithms for physics-informed loss functions to the condition number of an operator, which involves the Hermitian-square of the underlying differential operator. This contribution is novel to the best of our knowledge and has been acknowledged as such by most of the reviewers. A corollary of this observation is the fact that preconditioning (now defined rigorously as reducing the condition number of the underlying operator) is necessary to accelerate training of physics-informed machine learning models. Again, this provides a mathematically precise definition of preconditioning for these models and stands in stark contrast to the existing literature where the words *preconditioning* and *condition numbers* are used very loosely and imprecisely. These two aforementioned aspects constitute our main contribution and has been emphasised as such repeatedly in our paper. 

    Given the precise notion of preconditioning that we propose, it was incumbent upon us to also analyse several existing strategies for improving training in physics-informed machine learning through the perspective of our framework of operator preconditioning. To this end, we have in Section 3 and in the **SM**, analyzed via a combination of theoretical analysis and empirical evaluation, several such strategies (Choice of balance parameter $\lambda$, Hard Boundary conditions, Second-order optimizers, Domain decomposition and Casual learning, we will also include *Sobolev Training*) through the lens of operator preconditioning, providing a novel interpretation of these strategies and identifying conditions under which they might or might not work. We had gone further and analyzed an additional strategy that essentially boils down to *rescaling Fourier features*, whether with respect to linear basis functions or nonlinear neural networks. These are secondary contributions and we feel that it is not fair to judge the article based on secondary, rather than primary contributions. 

    Moreover, the fact that **only linear PDEs was considered** in the empirical evaluation was based on the observation that if physics-informed machine learning models do not work for simple linear PDEs, there is little hope of them working for complicated nonlinear PDEs. Furthermore, we follow widely cited papers such as Krishnapriyan et al, 2021, which flagged the problems of training PINNs by focussing considerably on simple linear PDEs. Finally, the page limit also automatically limits the scope of an article. Given these considerations, we believe that linear PDEs already provide a reasonably representative arena for analyzing the issue of training PINNs and related models and lessons learnt there can be applied to nonlinear PDEs in future works. In particular, at some level, nonlinear PDEs have to be **linearized** and the lessons learnt from linear PDEs will be germane. Finally in this context, we would like to mention that there is a tradition of analyzing training in machine learning through linear models (see ArXiv:1810.02281 and references therein) as analyzing gradient descent even with linear models is highly non-trivial and representative of the difficulties faced in training non-linear models.

Reviewer comment (Presentation):
The logic in the latter experimental and discussion sections is somewhat jumbled. For instance, the last two sections, in my opinion, should be consolidated into one section as related work.

Author response:
2. (W2+W3) *Some parts of the paper are not clearly written* and *the last two sections, in my opinion, should be consolidated into one section as related work*. We thank the reviewer for this excellent suggestion. In the process of incorporating all the 6 reviewers' multiple suggestions, we will consider a restructuring of these sections with your suggestion in mind in a **CRV**, if accepted.

Reviewer comment (Novelty):
The reviewer’s first concern is that this paper doesn't discuss its relation to previous work on preconditioning. Specifically, one line of work that researchers within the ML for PDE community have studied is "Sobolev Training", where the main idea is to incorporate the Sobolev gradients of the training objective into the loss function. Examples of theoretical and empirical work in this direction include [1] (theory) and [2, 3, 4] (empirical studies). In fact, Sobolev training can be interpreted as a form of operator preconditioning. For example, consider the task of solving the PDE $\mathcal{D}u = f$ over the domain $\Omega$ with zero boundary condition via PINN, where $\mathcal{D}$ is some differential operator. Let $E(u) :=\frac{1}{2} \|\mathcal{D}u - f\|_{L^2(\Omega)}^2$ be the standard $L^2$ loss function, whose associated gradient flow is $u_t =  -\frac{\delta}{\delta u}E(u) = -\mathcal{D}^\ast(\mathcal{D}u-f)$. In contrast, if we consider solving the same problem under the $H^2$ loss $E_{S}(u) := \frac{1}{2} \|(I-\Delta)(\mathcal{D}u - f)\|_{L^2(\Omega)}^2$  (Sobolev training), we have that the new gradient flow is given by $u_t =  -\frac{\delta}{\delta u}E_S(u) = -\mathcal{D}^\ast\mathcal{P}^\ast\mathcal{P}(\mathcal{D}u-f)$, where the differential operator $\mathcal{P} = I - \Delta$. This implies that training under a loss function that includes Sobolev gradient is equivalent to preconditioning the Hermitian operator $\mathcal{D}^\ast\mathcal{D}$ via the differential operator $\mathcal{P}$. Another approach for reducing the condition number is to reformulate the problem of minimizing the $L^2$ loss as a min-max optimization problem, which was proposed in recent work [5]. Moreover, the idea of using preconditioning/Sobolev training to speed up optimization/training of a prespecified model in machine learning and scientific computing is not new either. A few examples of previous work include using preconditioning in neural network training [6, 7], application of Sobolev preconditioned GD in image processing [8, 9, 10] and graphics [11, 12], etc. Hence, the authors are encouraged to have a subsection discussing how this paper is related to previous work on the usage of preconditioning/Sobolev training in ML for PDE and related fields.

Author response:
1. (W1): The reviewer has pointed to **Missing related work**. We would like to thank the reviewer for identifying very interesting related work, especially on Sobolev training and competitive PINNs that we will certainly include in a **CRV**, if accepted. Although we have taken care to mention previous work on preconditioning in the field of numerical analysis, we apologize having missed the stated work in ML and will include them in a **CRV**, if accepted.

Reviewer comment (Presentation):
The presentation of this paper can possibly be improved in a few aspects. Firstly, the review of ML-based/Physics-Informed PDE solvers in the second paragraph of Section 1 seems to be incomplete, as a lot of recent work in this field is missing. The authors are encouraged to take a look at the first part of Subsection 1.1 in [1] for a more complete review of related work in this field. Secondly, it is probably better to move the subsection of related work from Section 4 to Section 1 (Introduction), as this might help the readers understand the main contributions presented at the end of the introduction in a better way. It would also be helpful to include a subsection on the paper's organization and a subsection discussing the mathematical notations used in this paper within Section 1 (Introduction).

Author response:
4. (W3) We thank the reviewer for their careful inspection of our paper and have edited all typos pointed out by you. In addition, we will expand the bibliography following their suggestions in the **CRV**, if accepted.

Reviewer comment (Theory):
On p.3, the authors mention that their aim is to analyze the convergence of gradient descent. However, the analysis in section 2 only considers a simplified gradient step where they neglect all higher order terms in an error term epsilon_k, which they assume to be small. Is that equivalent to performing a local analysis close to a local minima and if so, how does the current analysis relate to the NTK analysis in the context of PINNs (Wang et al., 2022)? It is not clear whether this explains the poor performance of PINNs in practice as the main issue is to understand the pre-asymptotic behavior of gradient descent.

Author response:
1. (W1) Regarding the reviewer's point that **they neglect all higher order terms in an error term $\epsilon_k$, which they assume to be small**, we want to highlight that we do not just assume that this error term is small, but that rather we show that it is identically zero for linear models and that we prove rigorously in Lemma 2.2 that this error term is small in the NTK regime for neural networks. Furthermore, the reviewer asks **Is that equivalent to performing a local analysis close to a local minima**: we stress that this is not the setting of our analysis, but rather our analysis requires the smallness of the error term in Lemma 2.1. This error term is exactly zero for linear models and can be made small (Lemma 2.2) in the NTK regime for neural networks. There are no further assumptions. Hence, our analysis also covers the pre-asymptotic behavior of gradient descent as we explicitly consider the discrete time setting with finitely many parameters. This is precisely the setting where condition numbers make sense.

Reviewer comment (Theory):
Thm. 2.4 shows that the condition number of the Gram matrix is greater than the condition number of a Hermitian square operator A\circ TT^*. However, to perform preconditioning, one should instead derive an upper bound on the condition number of the Gram matrix. I am aware that Thm 2.4 gives an equality case but it assumes that the Gram matrix is invertible.

Author response:
2. (W2) Regarding the reviewer's question on **However, to perform preconditioning, one should instead derive an upper bound on the condition number of the Gram matrix**, we fully agree with the reviewer that this general upper bound does not prove that a well-conditioned matrix implies a well-conditioned operator $\mathcal{A}\circ TT^*$. Fortunately, the equality holds for many practical cases. For linear basis functions the Gram matrix is trivially invertible, and also for randomly initialized neural networks (e.g. using normal distribution) it will be invertible as the $\phi_i$ will be independent functions with probability 1, and hence equality will also hold. Finally, in practice it is the conditioning of the matrix $\mathbb{A}$ that determines the convergence of gradient descent, and not directly that of the operator $\mathcal{A}\circ TT^*$, so the condition number of the matrix $\mathbb{A}$ is much more important in preconditioning in practice.

Reviewer comment (Novelty):
The practical section 3 is the main weakness of the paper. It seems that the authors approach might only be useful when one already know a really good approximant to the eigenvectors and eigenvalues of the PDE, when it is linear (e.g. the Poisson equation example in Fig. 1). In this case, why would one use a neural network to solve the PDE instead of a spectral method? The authors states that they employ a linear combination of eigenvectors of the Poisson equation as the machine learning model, but this is exactly a spectral method and we know that the performance of an iterative solver (e.g. conjugate gradient, GMRES) for solving the system is related to the condition number of the matrix. The authors give very little details on how one might use their approach in a real context with a complex neural networks to solve a nonlinear PDE.

Author response:
3. (W3) Regarding the reviewer's comment that **The practical section 3 is the main weakness of the paper**, we would like to explicitly point out the context and contributions of the paper. To this end, we take the liberty to repeat the relevant portion of our reply to all the reviewers above here: At the outset, it is amply clear in the literature that a major issue with physics-informed machine learning models such as PINNs is the difficulty of training them. This has been widely recognised and several ad-hoc strategies have been proposed to improve training. However, it is also clear that a suitable theoretical framework that precisely formulates the problem of optimizing physics-informed loss functions was missing. We believe that a lack of precision in defining what the problem actually is also impedes finding suitable measures for ameliorating the problem itself. 

   With this background in mind, our main aim in the paper was precisely to propose a suitable theoretical framework in which the issues affecting training of physics-informed machine learning models can be formulated and analyzed. To this end, we have rigorously connected the rate of convergence of gradient descent algorithms for physics-informed loss functions to the condition number of an operator, which involves the Hermitian-square of the underlying differential operator. This contribution is novel to the best of our knowledge and has been acknowledged as such by most of the reviewers. A corollary of this observation is the fact that preconditioning (now defined rigorously as reducing the condition number of the underlying operator) is necessary to accelerate training of physics-informed machine learning models. Again, this provides a mathematically precise definition of preconditioning for these models and stands in stark contrast to the existing literature where the words *preconditioning* and *condition numbers* are used very loosely and imprecisely. These two aforementioned aspects constitute our main contribution and has been emphasised as such repeatedly in our paper.  

2. (continued) Given the precise notion of preconditioning that we propose, it was incumbent upon us to also analyse several existing strategies for improving training in physics-informed machine learning through the perspective of our framework of operator preconditioning. To this end, we have in Section 3 and in the **SM**, analyzed via a combination of theoretical analysis and empirical evaluation, several such strategies (Choice of balance parameter $\lambda$, Hard Boundary conditions, Second-order optimizers, Domain decomposition and Casual learning) through the lens of operator preconditioning, providing a novel interpretation of these strategies and identifying conditions under which they might or might not work. We had gone further and analyzed an additional strategy that essentially boils down to *rescaling Fourier features*, whether with respect to linear basis functions or nonlinear neural networks. These are secondary contributions and we feel that it is not fair to judge the article based on secondary, rather than primary contributions. 

   Regarding the reviewer's specific point about **Spectral methods for the Poisson Equation**, we would like to point that preconditioning for spectral methods is indeed well-established. However, there are essential differences between our approach and using a spectral method. Because, we minimize the loss in an $L^2$-sense, we are confronted with having to analyze the spectrum of the *Hermitian-square* of the underlying differential operator, where as in a conventional spectral method, only the differential operator itself has to be pre-conditioned. Following the reviewer's excellent suggestion, we will explicitly mention the differences with a spectral method in a **CRV**, if accepted. 

   In response to the reviewer's points about **lack of general preconditioning strategies**, we highlight Sec 3 where many existing strategies are examined through the lens of operator preconditioning. Moreover, in **SM* Section C.3, we have additionally explored another general simple preconditioning strategy which consists of using an approximate inverse of the matrix $\mathbb{A}$ (Eqn. (2.6)) as a preconditioner for the gradient descent step, i.e., pre-multiplying $(\mathbb{A} + \varepsilon I)^{-1}$ to the gradient descent update. This strategy is also applicable to any PDE and any linear or nonlinear model.

Reviewer comment (Theory):
Lemma 2.2, is there a way of controlling the error epsilon_k with respect to m for large m?

Author response:
4. (Q3) Regarding the reviewer's question on **Lemma 2.2, is there a way of controlling the error $\epsilon_k$ with respect to $m$ for large $m$?** Our analysis for particular Lemma builds on the results from Wang et. al, which do not control the error in terms of $m$. However, a careful examination of the proof reveals that this precise control is likely to be possible, yet very tedious. We will attempt to perform this analysis and include it in a **CRV**, if accepted and thank the reviewer for pointing it to us.

Reviewer comment (Experiments):
It is less convincing if the proposed method can be applicable to a wide variety of neural networks, which is also pointed out in the manuscript, “preconditioning A for nonlinear models such as neural networks can, in general, be hard” on page 7 and in the limitation paragraph. In the manuscript, preconditioning strategies are showcased only with the simple cases, approximating solutions of Poisson equations via linear and nonlinear models with Fourier feature mapping. It does not seem that the paper provides some guidelines or insights for preconditioning strategies for general cases.

Author response:
1. Regarding the reviewer's comment about us explicitly mentioning **preconditioning A for nonlinear models such as neural networks can, in general, be hard**, we start by clarifying what me meant in this context, namely that, in the neural network setting the operator $TT^*$ appearing in the main theorem 2.4 does not reduce to a simple operator like it does in the case of Fourier features or other linear basis functions. This implies that the condition number of the operator $\mathcal{A} \circ TT^*$ is  more difficult to study, but by no means is it impossible
as we consider in Section 3 as well as in **SM** Section C. 
2. Regarding the reviewer's concern that **It does not seem that the paper provides some guidelines or insights for preconditioning strategies for general cases** and the question **how to build preconditioner for more general cases**, we start by repeating our detailed reply to all the reviewers providing the context of our contributions.  At the outset, it is amply clear in the literature that a major issue with physics-informed machine learning models such as PINNs is the difficulty of training them. This has been widely recognised and several ad-hoc strategies have been proposed to improve training. However, it is also clear that a suitable theoretical framework that precisely formulates the problem of optimizing physics-informed loss functions was missing. We believe that a lack of precision in defining what the problem actually is also impedes finding suitable measures for ameliorating the problem itself. 

   With this background in mind, our main aim in the paper was precisely to propose a suitable theoretical framework in which the issues affecting training of physics-informed machine learning models can be formulated and analyzed. To this end, we have rigorously connected the rate of convergence of gradient descent algorithms for physics-informed loss functions to the condition number of an operator, which involves the Hermitian-square of the underlying differential operator. This contribution is novel to the best of our knowledge and has been acknowledged as such by most of the reviewers. A corollary of this observation is the fact that preconditioning (now defined rigorously as reducing the condition number of the underlying operator) is necessary to accelerate training of physics-informed machine learning models. Again, this provides a mathematically precise definition of preconditioning for these models and stands in stark contrast to the existing literature where the words *preconditioning* and *condition numbers* are used very loosely and imprecisely. These two aforementioned aspects constitute our main contribution and has been emphasised as such repeatedly in our paper. 

   Given the precise notion of preconditioning that we propose, it was incumbent upon us to also analyse several existing strategies for improving training in physics-informed machine learning through the perspective of our framework of operator preconditioning. To this end, we have in Section 3 and in the **SM**, analyzed via a combination of theoretical analysis and empirical evaluation, several such strategies (Choice of balance parameter $\lambda$, Hard Boundary conditions, Second-order optimizers, Domain decomposition and Causual learning) through the lens of operator preconditioning, providing a novel interpretation of these strategies and identifying conditions under which they might or might not work. We had gone further and analyzed an additional strategy that essentially boils down to *rescaling Fourier features*, whether with respect to linear basis functions or nonlinear neural networks. These are secondary contributions and we feel that it is not fair to judge the article based on secondary, rather than primary contributions.

Reviewer comment (Experiments):
Computational aspects are weak. In particular, comparisons with other works improving optimizers of PINNs in more general classes of PDEs, which also include the assessment on computational wall times, would be more appreciated and informative for readers/users to decide which method to use. Even if it’s not computational comparisons, it would be better to have some descriptions on differences from other baselines [2,3] (as shown in page 8, where the connections to [1] is presented in the Choice of lambda paragraph.) [1] Wang, et al, 2022, JCP (the reference in the paper) [2] Basir and Senocak, Physics and equality constrained artificial neural networks: Application to forward and inverse problems with multi-fidelity data fusion, 2022, JCP [3] Kim et al, DPM: A novel training method for physics-informed neural networks in extrapolation, 2021, AAAI.

Author response:
3. Following the reviewer's excellent suggestion, we have now included **wall-clock times for the methods in **SM** Table 2. The reviewer's suggestion to discuss Refs. [2] and [3] are well-taken and would be included in a **CRV**, if accepted. In this context, we would like to comment that the contributions of these papers can be viewed through a preconditioning perspective as done for the contributions of your reference [1] as described in **SM** section B.2.

Reviewer comment (Novelty):
Although the problem that is investigated in the paper is an important problem and the authors provide good contributions in the theoretical side, it is less convincing that the paper makes significant contributions in ML or DL perspectives.

Author response:
4. Regarding the reviewer's concern about **it is less convincing that the paper makes significant contributions in ML or DL perspectives**, please refer to our detailed framing of the contributions as we have outlined them to you in point 2. above.

Reviewer comment (Reproducibility):
Could the authors provide more information/insight on how to build preconditioner for more general cases? For complex PDEs where the PINN does not employ Fourier feature transform?

Author response:
2. Regarding the reviewer's concern that **It does not seem that the paper provides some guidelines or insights for preconditioning strategies for general cases** and the question **how to build preconditioner for more general cases**, we start by repeating our detailed reply to all the reviewers providing the context of our contributions.  At the outset, it is amply clear in the literature that a major issue with physics-informed machine learning models such as PINNs is the difficulty of training them. This has been widely recognised and several ad-hoc strategies have been proposed to improve training. However, it is also clear that a suitable theoretical framework that precisely formulates the problem of optimizing physics-informed loss functions was missing. We believe that a lack of precision in defining what the problem actually is also impedes finding suitable measures for ameliorating the problem itself. 

   With this background in mind, our main aim in the paper was precisely to propose a suitable theoretical framework in which the issues affecting training of physics-informed machine learning models can be formulated and analyzed. To this end, we have rigorously connected the rate of convergence of gradient descent algorithms for physics-informed loss functions to the condition number of an operator, which involves the Hermitian-square of the underlying differential operator. This contribution is novel to the best of our knowledge and has been acknowledged as such by most of the reviewers. A corollary of this observation is the fact that preconditioning (now defined rigorously as reducing the condition number of the underlying operator) is necessary to accelerate training of physics-informed machine learning models. Again, this provides a mathematically precise definition of preconditioning for these models and stands in stark contrast to the existing literature where the words *preconditioning* and *condition numbers* are used very loosely and imprecisely. These two aforementioned aspects constitute our main contribution and has been emphasised as such repeatedly in our paper. 

   Given the precise notion of preconditioning that we propose, it was incumbent upon us to also analyse several existing strategies for improving training in physics-informed machine learning through the perspective of our framework of operator preconditioning. To this end, we have in Section 3 and in the **SM**, analyzed via a combination of theoretical analysis and empirical evaluation, several such strategies (Choice of balance parameter $\lambda$, Hard Boundary conditions, Second-order optimizers, Domain decomposition and Causual learning) through the lens of operator preconditioning, providing a novel interpretation of these strategies and identifying conditions under which they might or might not work. We had gone further and analyzed an additional strategy that essentially boils down to *rescaling Fourier features*, whether with respect to linear basis functions or nonlinear neural networks. These are secondary contributions and we feel that it is not fair to judge the article based on secondary, rather than primary contributions.

Reviewer comment (Reproducibility):
In the implementation, the gradient descent rule has been reimplemented following Eq. (3.2)?

Author response:
5. Regarding the reviewer's question on **how the gradient descent rule has been reimplemented following Eq. (3.2)**, we would like to point out that we have shown that rescaling the model parameters is equivalent to preconditioning the gradient directly. Hence, in our experiments with linear models or MLP with Fourier features, we rescaled our model parameters and therefore did not use (3.2) directly. On the other hand, with the matrix inversion strategy of **SM** Sec C.3, we use a form of Eqn (3.2) for the gradient update.

Reviewer comment (Reproducibility):
I assume that there are models for which kernel integral operator cannot be chosen manually. What you propose to do if expressing precodnitioned operator explicitly is not possible?

Author response:
Regarding the reviewer's question about **What to do if the kernel integral operator cannot be chosen manually**: we agree that it is indeed often not possible to chose the kernel integral operator $TT^*$ explicitly. In this case, one can consider the various preconditioning techniques listed and analysed in the last paragraphs of Section 3 as well those in **SM**, Section C.3, which can always be applied.

Reviewer comment (Experiments):
What do you think about applying additional optimisation procedure for training this kernel operator?

Author response:
The reviewer's sugggestion to **add another optimization layer to learn the kernel integral operator** is excellent. However, given the limited time for rebuttal, we will consider it as an interesting avenue for future research.

Reviewer comment (Theory):
The paper assumes that the solution can be expressed as a linear combination of basis functions (such as Fourier features, which was shown in the experiments). Such an approximation can be quite limiting, and the paper does not show how the preconditioning can be extended for non-linear models.

Author response:
1. (W1) Regarding the point that **the paper assumes that the solution can be expressed as a linear combination of basis functions** and **the paper does not show how the preconditioning can be extended for non-linear models**: we start by pointing out that for our theoretical analysis, we only assume that the tangent kernel of the model is (approximately) constant along the optimization path, which can also hold for nonlinear models as neural networks. Infact, in Lemma 2.1, a precise charecterization of this error has been provided. Moreover, in the last paragraphs of Section 3 we have discussed and analysed many different  preconditioning approaches that also hold for nonlinear models. We have also added another simple preconditioning strategy which consists of using an approximate inverse of the matrix $\mathbb{A}$ (Eqn. (2.6)) as a preconditioner for the gradient descent step, i.e., pre-multiplying $(\mathbb{A} + \varepsilon I)^{-1}$ to the gradient descent update (see Section C.3 of **SM**) which is applicable to any nonlinear model.

Reviewer comment (Evaluation):
The evaluation of the proposed preconditioning approach is poor. Firstly, the paper only covers linear PDEs with linear PINN models. Second, only the training loss curves were shown for the two PDEs. It is well-known that the PINNs can learn trivial solutions and achieving very low training losses does not guarantee convergence. See paper [1].

Author response:
2. (W2) Regarding the reviewer's concern that **the paper only covers linear PDEs with linear PINN models**,  we refer the reviewer to our discussion and experiments using neural networks in Section 3 as well as **SM** section C, and hence we do consider nonlinear models. Moreover, the fact that only linear PDEs was considered in the empirical evaluation was based on the observation that if physics-informed machine learning models do not work for simple linear PDEs, there is little hope of them working for general nonlinear PDEs. Furthermore, we burrowed from widely cited papers such as Krishnapriyan et al, 2021, which flagged the problems of training PINNs and focus considerably on simple linear PDEs. Finally, the page limit also automatically limits the scope of an article. Given these considerations, we believe that linear PDEs already provide a reasonably representative arena for analyzing the issue of training PINNs and related models and lessons learnt there can be applied to nonlinear PDEs in future works. In particular, at some level, nonlinear PDEs have to be **linearized** and the lessons learnt from linear PDEs will be germane. Finally in this context, we would like to mention that there is a long tradition of analyzing training in machine learning through linear models (see ArXiv:1810.02281) and references therein) as analyzing gradient descent even with linear models is highly non-trivial and representative of the difficulties faced in training non-linear models.  
3. Regarding the reviewer's point that **only the training loss curves were shown for the two PDEs**, we follow the reviewer's excellent suggestion and plot the resulting $L^2$-test errors in the **SM** Figures 17-19, where the tight coupling between the residual losses and the test errors are demonstrated.

Reviewer comment (Theory):
For a typical Taylor series expansion, the Hessian would be computed at the point $\theta_0$. Can the authors provide some justification on why the Hessian was computed at an interpolated $\theta$ value between 0 and k?

Author response:
4. Regarding reviewer's (Q1+MC2) about **infinite Taylor series and ignoring higher-order terms**, we wish to clarify that we did not consider an infinite Taylor series and then ignored the higher-order terms. Instead, we considered a first-order Taylor polynomial with the corresponding *exact* remainder term (this is where the Hessian comes in and why it is evaluated at an interpolated value due to the mean-value theorem). We again stress that this formula is exact and no terms were ignored herein. We later show that the remainder term is small under suitable conditions (Lemma 2.2).

Reviewer comment (Theory):
Can the authors provide some discussions on the connections and differences of their approach and NTK theory, especially considering that the latter also offers a convergence rate analysis for PINNs?

Author response:
5. Regarding reviewer's (Q2) on the relation of our analysis with **NTK theory**, we would like to emphasize that NTK theory was originally formulated for the supervised MSE ($L^2$)-loss and for infinite-width neural networks. In contrast, we consider a physics-informed loss and finite-width neural networks, resulting in significant departures from the conventional NTK framework. Compared to other works that use NTK theory to analyse PINNs e.g. (Wang et al, 2022), we want to note that our analysis is valid in the practical case of *discrete time* rather than continuous time (as in Wang et. al.) and also we consider a finite number of parameters (corresponding to reality), along with the fact that we rigorously keep track of all error terms. Discrete time and finite parameters are absolutely crucial to our analysis as the condition number only appears in this discrete setting. Moreover, to the best of our knowledge, Wang et. al. do not identify an underlying operator and its conditioning as the key issues in training physics-informed machine learning models.

Reviewer comment (Experiments):
For the Advection Equation with Fourier Features without preconditioning (Figure 2), it seems that the loss does not converge at all (the value of the loss is $10^3$). Can the authors provide some justification behind this? Is it because the model was chosen to be a simple linear one with Fourier Features?

Author response:
6. Regarding the reviewer's Q3 about **non-convergence of unpreconditioned Fourier features for advection**, we start by pointing out that this cannot only be attributed to the *linear* form of the underlying model as the MLP also barely converges (see **SM** Figure 11). Rather, the ill-conditioning of the model is responsible and only we precondition the linear model, it converges very fast, as acknowledged by the reviewer. The non-convergence of MLPs in this case was also observed in Krishnapriyan et. al., 2021.

Reviewer comment (Theory):
The Taylor expansion in Equation 2.5 should contain higher order terms or the paper should mention that the 3rd order and higher terms are ignored. Similarly, the proof in appendix A.1 should mention that higher order multiplicative terms by substituting 2.5 in 2.2 are ignored.

Author response:
4. Regarding reviewer's (Q1+MC2) about **infinite Taylor series and ignoring higher-order terms**, we wish to clarify that we did not consider an infinite Taylor series and then ignored the higher-order terms. Instead, we considered a first-order Taylor polynomial with the corresponding *exact* remainder term (this is where the Hessian comes in and why it is evaluated at an interpolated value due to the mean-value theorem). We again stress that this formula is exact and no terms were ignored herein. We later show that the remainder term is small under suitable conditions (Lemma 2.2).