Paper: PINNACLE: PINN Adaptive ColLocation and Experimental points selection (ICLR-2024)

Reviewer comment (Experiments):
Not necessarily a weakness, but I am quite interested in the selection procedure of $Z_{pool}$ from $Z$ as introduced in Algorithm 1 and whether the authors have considered some structured designs that may give it more advantage than random sampling.

Author response:
We thank the reviewer for the interesting suggestions. Indeed, one possibility is to incorporate low-discrepancy sequence sampling (e.g. Hammersley sequence sampler) to potentially construct a more diverse candidate pool $Z_{pool}$, which may lead to better point diversity for both the PINNACLE-S and PINNACLE-K variants, or serve as a foundation for other convergence degree-based heuristics.

In this work, while we have proposed two variants of PINNACLE based on different point selection heuristics (SAMPLING and K-MEANS++), we see the PINNACLE algorithm as being centered around the proposed convergence degree criterion and evolving eNTK eigenfunctions. Hence, we believe that our framework could be readily extended to other point selection heuristics that also select points with high convergence degrees (as per our proposed criterion) while also achieving some level of points diversity. This includes potentially weighted space-filling designs, as you have kindly suggested.

We hope that our current framework will inspire future works on joint collocation and experimental points selection for PINNs.

Reviewer comment (Experiments):
Similar question regarding the selection procedure from subset $Z$ of $Z_{pool}$. I see two different design approaches are used here, but I wonder whether the authors have considered additional options, say weighted space-filling designs. Just a thought.

Author response:
We thank the reviewer for the interesting suggestions. Indeed, one possibility is to incorporate low-discrepancy sequence sampling (e.g. Hammersley sequence sampler) to potentially construct a more diverse candidate pool $Z_{pool}$, which may lead to better point diversity for both the PINNACLE-S and PINNACLE-K variants, or serve as a foundation for other convergence degree-based heuristics.

In this work, while we have proposed two variants of PINNACLE based on different point selection heuristics (SAMPLING and K-MEANS++), we see the PINNACLE algorithm as being centered around the proposed convergence degree criterion and evolving eNTK eigenfunctions. Hence, we believe that our framework could be readily extended to other point selection heuristics that also select points with high convergence degrees (as per our proposed criterion) while also achieving some level of points diversity. This includes potentially weighted space-filling designs, as you have kindly suggested.

We hope that our current framework will inspire future works on joint collocation and experimental points selection for PINNs.

Reviewer comment (Theory):
My understanding is that the constraints by initial conditions or boundary conditions are only considered as "soft" constraints here due to how the objective function is set up. I guess for most situations its impact on the results would not be significant, or in other words, the approximation is going to be quite good. However, I wonder whether there could be cases where even a minor violation of the limiting conditions will cause significant distruptions to the system, and if so, how the authors propose to tackle that.

Author response:
We would like to highlight the **transfer learning setting** that we presented (Sec. 6.3 in the main paper, Fig. 6), which **empirically demonstrates how PINNACLE may be also able to take into account the situation that you have described and still optimize its point selection**.

- In this setting, we consider having access to a pre-trained PINN that had been trained for a PDE with a given initial condition (IC), but instead want a solution for the same PDE but with a perturbed IC. In the case of the Advection problem, as seen in Fig. 6, **a small perturbation in the IC results in relatively large global changes in the true solution** which can be qualitatively observed in the plots.

- From Fig. 6, we can see that PINNACLE is able to **use a small number of points to fine-tune the pre-trained PINN to the perturbed IC conditions, and outperform other benchmarks.**

-  In particular, we would like to draw your attention to Fig. 18 in Appendix K.4, which plots out the point selections of the two PINNACLE variants as well as that of the benchmarks. Notice how **both PINNACLE-S and PINNACLE-K were able to focus on the perturbed IC locations early on in training**, and as a result were able to more quickly learn the new solution with just very limited number of points.

Hence, the results above provide some preliminary evidence that **PINNACLE is able to identify situations where minor changes in the IC/BC limiting conditions can have large effects on the final solution, and select training points** accordingly to learn the right solutions. We thank the reviewer for the insightful question, which could be investigated further in future works.

Reviewer comment (Experiments):
This follows my previous question. I see the limiting conditions are introduced as regularization components into the objective function, with the corresponding regularizer coefficients described as having a certain value. I wonder whether the authors have considered how to best place values on these regularizers according to the physical system, and whether that would have a substantial influence on the final result. I presume different limiting conditions will have varying impact on the system, so I am curious on whether the selection of coefficients may be incorporated into the framework.

Author response:
While we did not focus on varying loss term coefficients for the different training point types in this work to simplify the presentation, we would like to highlight that our PINNACLE algorithm could readily do so and **incorporate any prior domain knowledge on the relative importance of the loss terms without any change to the algorithm design**. The different loss term coefficients set by the domain experts would be directly incorporated in the computation of the empirical Neural Tangent Kernel (eNTK) and effectively reweigh the residual terms, which will then impact the convergence degree computation and subsequently the choice of points. Training point types with larger loss term coefficients will therefore then likely be prioritized during the point selection process.

Reviewer comment (Novelty):
The strength of this paper seems to be that collocation points for initial boundary values and those for PDEs can be treated in a unified way, but a method of training networks without collocation points for initial boundary values is also proposed. When such a method is employed, the proposed method may lose a certain extent of significance.

Author response:
While one of PINNACLE's strength lies in its ability to consider PDE and all initial/boundary conditions (IC/BC) collocation points in a unified way as you had kindly pointed out, we would like to emphasize that PINNACLE's strength also lies in:

1. **Better individually optimize each point type (e.g., PDE collocation points even when there are no IC/BC points)** based on the PINNs training dynamics as described in Sec. 4.2. To further demonstrate this, **we ran a simple example based on the method by Lagaris et al. that you have kindly raised, and showed that PINNACLE still outperform benchmarks even when considering just one training point type.**

2. **Its ability to jointly optimize experimental and just PDE collocation training points** as well in the absence of IC/BC points, which can be seen from the inverse experiment in our paper and also an additional experiment that we have run.
    
For (1), this means that **even if there are only PDE collocation points to be considered, i.e., PDE forward problems with hard IC/BC constraints, PINNACLE's convergence degree criterion still applies** which could take into account the neural network's training dynamics as characterized by its Neural Tangent Kernel (NTK). The choice of PDE collocation points could still be optimized based on the criterion to focus on points to potentially increase training convergence and reduce generalization loss.

- **We could see some signs of this optimized selection of PDE collocation points**, by observing the selected PDE collocation points by PINNACLE for the 1D Advection and 1D Burgers' problems (Fig. 3 of main paper). Apart from the shift in budget allocation of between IC/BC and PDE collocation points, **notice that the choice of PDE collocation points clusters around more informative regions**, such as the stripes in the Advection equation problem and areas with richer features in the Burgers' equation problem.

- In addition, we have also performed a simple **demonstration involving only PDE collocation points, where we used the method by Lagaris et al.** that you have kindly raised, and showed that PINNACLE still outperforms benchmarks. We consider a simple 1D Diffusion equation forward problem where we use a PINN with hard constraints, with results in Fig. 23 of the updated Appendix. We can see that **even with just one type of training point (PDE collocation point), PINNACLE still provides a substantial training performance gain over other training point selection methods**. While the results are still preliminary, this shows that PINNACLE is likely useful even when we do not have to consider a joint training dynamics of multiple training point types.

- We would like to also point out that while such methods that incorporate hard constraints are useful for certain settings, **it is still in general challenging to achieve such hard constraints for a wide range of other boundary conditions**. In those cases, PINNACLE would not only contribute in better PDE collocation point selection, but also joint optimization of all training point types.

---

&#8595; **Continued below** &#8595;

## Rebuttal 2 (ID: Bxi4zBCXlx)
---

&#8593; **Continued from above** &#8593;

---

For (2), this means that **even in situations where no IC/BCs were provided but experimental and PDE collocation points need to be selected, PINNACLE would still be able to provide significant value and outperform benchmarks**.

- For example, we can see this in the 2D Navier-Stokes inverse problem setting (details in Appendix J.1, empirical results in Fig. 5 of our main paper), where no IC/BCs are imposed. In that example, we have two coupled PDEs involving 3 outputs each with 2+1 (time) dimensions, and our task is to estimate two unknown PDE parameters given experimental data on 2 out of the 3 outputs and no IC/BC conditions. We found that **PINNACLE was still able to outperform benchmarks in estimating the two unknown parameters, given its optimized joint selection of experimental and PDE collocation points alone**.

- In addition, we have also performed an additional inverse experiment (added in Fig. 19 of the updated paper Appendix), **the 3D Eikonal inverse problem, where we tackle the challenging task of learning unknown inverse quantity that is a 3D field instead of just a few scalar PDE parameter values**. The Eikonal equation links wave propagation dynamics with the wave speed at different locations,and our task is to reconstruct the wave speed at each location which can be interpreted as representing the composition of the medium.  In this inverse problem, there are no IC/BCs, and we are able to query for experimental data regarding the time for wave to propagate to a certain 3D location. In this case, we see that **PINNACLE is also able to more efficiently select both experimental points and collocation points to achieve a substantial improvement in the inverse field reconstruction**.

We thank the reviewer for suggesting this interesting PINN training setting which we can investigate further in future works.

---

We hope that we have sufficiently addressed your queries, and that our response will help strengthen your support for our paper.

Reviewer comment (Novelty):
As mentioned above, for the initial boundary values, a method of designing neural networks to satisfy them has been proposed, e.g. in the following paper: Lagaris, I. E., Likas, A., and Fotiadis, D. I. (1998). Artificial neural networks for solving ordinary and partial differential equations. IEEE Transactions on Neural Networks, 9(5):987–1000. What is the significance of the proposed method when such a method is employed?

Author response:
While one of PINNACLE's strength lies in its ability to consider PDE and all initial/boundary conditions (IC/BC) collocation points in a unified way as you had kindly pointed out, we would like to emphasize that PINNACLE's strength also lies in:

1. **Better individually optimize each point type (e.g., PDE collocation points even when there are no IC/BC points)** based on the PINNs training dynamics as described in Sec. 4.2. To further demonstrate this, **we ran a simple example based on the method by Lagaris et al. that you have kindly raised, and showed that PINNACLE still outperform benchmarks even when considering just one training point type.**

2. **Its ability to jointly optimize experimental and just PDE collocation training points** as well in the absence of IC/BC points, which can be seen from the inverse experiment in our paper and also an additional experiment that we have run.
    
For (1), this means that **even if there are only PDE collocation points to be considered, i.e., PDE forward problems with hard IC/BC constraints, PINNACLE's convergence degree criterion still applies** which could take into account the neural network's training dynamics as characterized by its Neural Tangent Kernel (NTK). The choice of PDE collocation points could still be optimized based on the criterion to focus on points to potentially increase training convergence and reduce generalization loss.

- **We could see some signs of this optimized selection of PDE collocation points**, by observing the selected PDE collocation points by PINNACLE for the 1D Advection and 1D Burgers' problems (Fig. 3 of main paper). Apart from the shift in budget allocation of between IC/BC and PDE collocation points, **notice that the choice of PDE collocation points clusters around more informative regions**, such as the stripes in the Advection equation problem and areas with richer features in the Burgers' equation problem.

- In addition, we have also performed a simple **demonstration involving only PDE collocation points, where we used the method by Lagaris et al.** that you have kindly raised, and showed that PINNACLE still outperforms benchmarks. We consider a simple 1D Diffusion equation forward problem where we use a PINN with hard constraints, with results in Fig. 23 of the updated Appendix. We can see that **even with just one type of training point (PDE collocation point), PINNACLE still provides a substantial training performance gain over other training point selection methods**. While the results are still preliminary, this shows that PINNACLE is likely useful even when we do not have to consider a joint training dynamics of multiple training point types.

- We would like to also point out that while such methods that incorporate hard constraints are useful for certain settings, **it is still in general challenging to achieve such hard constraints for a wide range of other boundary conditions**. In those cases, PINNACLE would not only contribute in better PDE collocation point selection, but also joint optimization of all training point types.

---

&#8595; **Continued below** &#8595;

## Rebuttal 2 (ID: Bxi4zBCXlx)
---

&#8593; **Continued from above** &#8593;

---

For (2), this means that **even in situations where no IC/BCs were provided but experimental and PDE collocation points need to be selected, PINNACLE would still be able to provide significant value and outperform benchmarks**.

- For example, we can see this in the 2D Navier-Stokes inverse problem setting (details in Appendix J.1, empirical results in Fig. 5 of our main paper), where no IC/BCs are imposed. In that example, we have two coupled PDEs involving 3 outputs each with 2+1 (time) dimensions, and our task is to estimate two unknown PDE parameters given experimental data on 2 out of the 3 outputs and no IC/BC conditions. We found that **PINNACLE was still able to outperform benchmarks in estimating the two unknown parameters, given its optimized joint selection of experimental and PDE collocation points alone**.

- In addition, we have also performed an additional inverse experiment (added in Fig. 19 of the updated paper Appendix), **the 3D Eikonal inverse problem, where we tackle the challenging task of learning unknown inverse quantity that is a 3D field instead of just a few scalar PDE parameter values**. The Eikonal equation links wave propagation dynamics with the wave speed at different locations,and our task is to reconstruct the wave speed at each location which can be interpreted as representing the composition of the medium.  In this inverse problem, there are no IC/BCs, and we are able to query for experimental data regarding the time for wave to propagate to a certain 3D location. In this case, we see that **PINNACLE is also able to more efficiently select both experimental points and collocation points to achieve a substantial improvement in the inverse field reconstruction**.

We thank the reviewer for suggesting this interesting PINN training setting which we can investigate further in future works.

---

We hope that we have sufficiently addressed your queries, and that our response will help strengthen your support for our paper.

Reviewer comment (Experiments):
For the K-MEANS++ method, the authors are encouraged to provide the time consumption comparison with other baselines. I am not sure if K-MENAS++ will introduce much extra computation cost.

Author response:
From our experiments, we find that **PINNACLE-K (the K-MEANS++ variant) achieves lower overall runtime compared to benchmarks even though the K-MEANS++ mechanism may incur computation costs for point selection**. This is because the better training points selected using this mechanism allows the model to converge faster and incur lower training time.

- In Sec. 6.3. of the main paper, we mentioned that **PINNACLE is able to reach the target loss faster and achieve shorter computation time compared to other algorithms in experiments of forward and inverse settings**, though due to space constraints we only provided the supporting empirical results in the Appendix (Fig. 20 of the updated Appendix), which shows runtime results of the 1D Advection forward problem (Fig. 20a) and the 2D Navier-Stokes inverse problem (Fig. 20c). The runtime in these empirical results captures all computational costs of the algorithms, which consist of both training set selection time (including K-MEANS++ computational overheads) and also the model training time.
- To better address your concern, we also ran additional timing experiments for a more challenging problem setting, the **3D Eikonal inverse problem where the unknown PDE parameter to be estimated is a 3D field rather than just a scalar quantity** (added to Fig. 20b). We demonstrate that even for this challenging problem setting, PINNACLE-K is also able to achieve much lower loss with a shorter amount of runtime. Hence, even from a runtime perspective there is strong motivation to use PINNACLE-K over other point selection mechanism even though the K-MEANS++ mechanism may incur additional computation costs.

Additionally, we would like to stress that **the benefits from using PINNACLE significantly extends beyond reduced training time, especially in problems involving experimental data that is costly to obtain**. More specifically, in inverse problems where the goal is to learn unknown PDE parameters from experimental data (e.g. the 2D Navier-Stokes and 3D Eikonal settings mentioned above), the cost of acquiring these experimental data (e.g. from expensive experimental set-ups and measurements) is often much larger than PINN training costs. **PINNACLE is able to more efficiently select experimental data that is most beneficial to tackling the inverse problem, by taking into account overall PINNs training dynamics with optimized collocation point selection**.

As other PINNs work on collocation points selection mainly report test error with respect to the number of training iterations, we have similarly done so in order to be consistent with them [1, 2].  However, we agree with the reviewer that the running time is also an important consideration in some problem setups, and will look to more prominently highlight some of our runtime results as space permits.

---

#### References

[1] Wu et al. A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks. Computational Methods in Applied Mechanics and Engineering, 2023.

[2] Krishnapriyan et al. Characterizing possible failure modes in physics-informed neural networks. NeurIPS 2021.

Reviewer comment (Theory):
For the K-MEANS++ method, the author claims that "this method select points with high convergence degrees". The authors are expected to provide more explanation why this method increase the convergence degree. It is the same for SAMPLING method. The authors are expected to explain how to "select a point which is proportional to $\hat{\alpha}(z)$ and how it improves the convergence degree" explicitly.

Author response:
The SAMPLING and K-MEANS++ approaches that we proposed are heuristics meant to select points with high convergence degrees while also promoting some diversity of points selected, which is useful in the earlier stages of training as the eNTK would still be evolving (as we have discussed in Sec. 4.2).

We first explain why the K-MEANS++ method selects points with high convergence degrees as claimed. This method, as noted in Sec. 5.2 of our paper, has also been used by active learning works to select points with large magnitudes in the embedding space while encouraging diversity of points [3]. To aid our explanation of the point selection mechanism, we have added additional plots in Fig. 9 in the updated Appendix.

- For our K-MEANS++ method, we first represent each point with a vector embedding, where **its Euclidean norm will correspond to the convergence degree $\hat{\alpha}(z)$**, and hence points that have the largest Euclidean norm will have the largest individual convergence degree. We then perform the standard K-Means++ initialization on these embedded vectors to select a batch of points, which basically involves sequential sampling of $k$ points with the probability proportional to each point's Euclidean distance to the closest point that had already been selected.

- For illustration, we can consider the empirical point distribution from one of our experiments (1D Burger's), visualized in a few embedding dimensions as blue points in Fig. 9a. We can make two observations: (a) **the points with low convergence degree (i.e., lower norm in embedded space) will all be "clustered" together close to the origin** and hence will be less likely to be selected by K-MEANS++ which preferentially samples points that are further spread out; (b) **the dimensions in the embedding space that correspond to the dominant eNTK eigenfunctions (higher $\hat{\lambda}$) will be stretched out**, leading to coefficients that are more spread out and cause points with high convergence degree to be further away from other points.

- Fig. 9a highlights points that are eventually selected by PINNACLE-K in orange. Notice that (a) and (b) described above can be observed in the plot: **points with high convergence degrees are more widely spread out and are therefore preferentially selected**. Fig. 9b explicitly sorts the candidate points based on their convergence degrees, and further shows that PINNACLE-K mainly selects points that have much higher convergence degrees.

For our SAMPLING method, we basically sample points with probability proportional to their convergence degrees $\hat{\alpha}(z)$. Hence, by design, points with larger convergence degree $\hat{\alpha}(z)$ will have a larger chance of being chosen by PINNACLE-S.

---

#### References

[3] Ash et al., Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds. ICLR 2020.

Reviewer comment (Experiments):
For the experiments, the authors are encouraged to provide the results mean error with running time rather than only steps. Furthermore, since PINNACLE achieves great results in 1D Advection and 1D Burger settings, it will be interesting to explore with more settings, e.g., 1D Wave equation and 1D KdV equation.

Author response:
From our experiments, we find that **PINNACLE-K (the K-MEANS++ variant) achieves lower overall runtime compared to benchmarks even though the K-MEANS++ mechanism may incur computation costs for point selection**. This is because the better training points selected using this mechanism allows the model to converge faster and incur lower training time.

- In Sec. 6.3. of the main paper, we mentioned that **PINNACLE is able to reach the target loss faster and achieve shorter computation time compared to other algorithms in experiments of forward and inverse settings**, though due to space constraints we only provided the supporting empirical results in the Appendix (Fig. 20 of the updated Appendix), which shows runtime results of the 1D Advection forward problem (Fig. 20a) and the 2D Navier-Stokes inverse problem (Fig. 20c). The runtime in these empirical results captures all computational costs of the algorithms, which consist of both training set selection time (including K-MEANS++ computational overheads) and also the model training time.
- To better address your concern, we also ran additional timing experiments for a more challenging problem setting, the **3D Eikonal inverse problem where the unknown PDE parameter to be estimated is a 3D field rather than just a scalar quantity** (added to Fig. 20b). We demonstrate that even for this challenging problem setting, PINNACLE-K is also able to achieve much lower loss with a shorter amount of runtime. Hence, even from a runtime perspective there is strong motivation to use PINNACLE-K over other point selection mechanism even though the K-MEANS++ mechanism may incur additional computation costs.

Additionally, we would like to stress that **the benefits from using PINNACLE significantly extends beyond reduced training time, especially in problems involving experimental data that is costly to obtain**. More specifically, in inverse problems where the goal is to learn unknown PDE parameters from experimental data (e.g. the 2D Navier-Stokes and 3D Eikonal settings mentioned above), the cost of acquiring these experimental data (e.g. from expensive experimental set-ups and measurements) is often much larger than PINN training costs. **PINNACLE is able to more efficiently select experimental data that is most beneficial to tackling the inverse problem, by taking into account overall PINNs training dynamics with optimized collocation point selection**.

As other PINNs work on collocation points selection mainly report test error with respect to the number of training iterations, we have similarly done so in order to be consistent with them [1, 2].  However, we agree with the reviewer that the running time is also an important consideration in some problem setups, and will look to more prominently highlight some of our runtime results as space permits.

---

#### References

[1] Wu et al. A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks. Computational Methods in Applied Mechanics and Engineering, 2023.

[2] Krishnapriyan et al. Characterizing possible failure modes in physics-informed neural networks. NeurIPS 2021.

Reviewer comment (Novelty):
The algorithm seems straightforward, and the presentation can be more concise.

Author response:
We would like to emphasize that while our novel algorithm (PINNACLE) may seem straightforward at first glance, **the analysis and insights leading to the algorithm are challenging and non-trivial, and may themselves also be of broader interest** to the PINNs community. These all build up to PINNACLE, which we aimed for it to be **simple in form for straightforward implementation, but also theoretically-motivated and with strong empirical performance**, as you have kindly pointed out.

Our paper could have been made more concise if we were to focus primarily on the algorithm and empirical results, but we decided to **include additional insights and elaboration where relevant to help provide a better understanding of phenomena and design choices underlying our algorithm** for readers who may find them useful.
Here, we highlight key challenges and some of the contributions we have addressed in our work:

- *How could selecting points intelligently improve PINNs' training?* Past works have predominantly focused on optimizing just the selection of PDE collocation points to try improve PINNs performance, which may naturally seem to be the only consequential point type for learning PDE solutions. It was non-trivial to go beyond the prevailing norm and realize that the various terms in the PINN composite loss function (Eq. 2 in the paper) **corresponding to different point types (experimental points, PDE and IC/BC collocation points) are actually tightly coupled** with each other, leading to training dynamics that is complex but **if understood could be exploited to significantly boost training performance**.
    - We are the first to adopt this perspective and **provide a unifying treatment for all types of training collocation and experimental points through an augmented space** (Sec. 4.1) that allows them to be considered jointly.

- *How can we analyze the PINN training dynamics involving all experimental and collocation point types?* The cross interactions among the various point types are complex and trying to exploit or even analyze them may seem an intractable challenge.
    - Based on the augmented space view, we managed to **both theoretically and empirically analyze the PINN training dynamics** using the combined PINN empirical Neural Tangent Kernel (eNTK) and its eigenfunctions (Sec. 4.2).
    - We also explicitly considered the **PINN eNTK evolution during training**, unlike many past NTK works that assumes a fixed NTK, and demonstrated how the eigenfunctions would evolve such that the dominant ones would be able to reconstruct the final PINN solution.
    - Our analysis of PINNs training dynamics using the combined eNTK eigenfunctions, to the best of our knowledge, have not been considered in previous PINN works, and **may be of broader interest** beyond its use in the development of the PINNACLE algorithm.

- *How can we design a criterion that leads to faster training convergence and lower generalization error loss?* Even if there were exploitable cross-information among point types to achieve faster training convergence, it would be non-trivial to design a criterion for point selection that would also lead to lower generalization error loss.
    - We proposed a **novel criterion, the convergence degree**, based on our analysis of the training dynamics (Sec. 4.3), that is also **theoretically shown to be related to lower generalization error loss**.

- *How do we come up with an effective and efficient algorithm that uses our theoretically-motivated criterion to achieve significant performance improvement empirically?* It is often challenging to convert a theoretically-motivated approach to a practical algorithm that is straightforward to effectively implement.
    - To achieve this, we proposed a method of **approximating the proposed criterion using the Nystrom approximation** (Sec. 5.1). We find that the method of approximating the true convergence degree criterion through approximation of eNTK eigenfunctions involves kernel methods which has not been previously used in training point selection for PINNs.
    - Along with additional design choices, the resulting PINNACLE algorithm is theoretically-motivated while still being relatively **straightforward and elegant, and is able to provide strong empirical performances** for a wide range of problem settings, including forward, inverse and transfer learning problems.

Reviewer comment (Experiments):
In Algorithm 1, how to select SAMPLING or K-Means sampling strategy?

Author response:
In general, we suggest first trying PINNACLE-K (K-MEANS++) over PINNACLE-S (SAMPLING). Empirically, we found that **both PINNACLE-K and PINNACLE-S methods tend to produce similar results, though in some cases PINNACLE-K may be able to produce lower loss faster at early stages of training**. This may be because PINNACLE-K places a greater emphasis on point diversity during selection, which could be more important earlier in training when the eNTK tends to evolve more (Sec. 4.2), before it stabilizes nearer the end of training. PINNACLE-S, on the other hand, is better motivated theoretically as it involves sampling from an implicit distribution (whose p.d.f. is proportional to $\hat{\alpha}$), and therefore is more directly compatible with assumptions required in Theorem 1 on generalization error.

Reviewer comment (Reproducibility):
Is the calculation of NTK used to compute the convergence degree?

Author response:
The short answer is yes. The convergence degree $\alpha$  (Eq. 10) is defined based on the eigenfunctions and eigenvalues of the PINN eNTK. The eigenfunctions and eigenvalues are empirically estimated using the Nystrom approximation, which requires the computation of the eNTK of a small reference set $Z_{ref}$, as shown in Eq. 14. We also show, in Proposition 1, that our use of Nystrom approximation with the eNTK provides a good estimation of the true convergence degree criterion.

Reviewer comment (Evaluation):
The runtime comparison is not provided.

Author response:
In Sec. 6.3. of the main paper, we mentioned that **PINNACLE is able to reach the target loss faster and achieve shorter computation time compared to other algorithms in experiments of forward and inverse settings**, though due to space constraints we only provided the supporting empirical results in the Appendix (Fig. 20 of the updated Appendix), which shows **runtime results of the 1D Advection forward problem and the 2D Navier-Stokes inverse problem**.

The runtime in these empirical results captures all computational costs of the algorithms, which consist of both training set selection time and also the model training time. The results showed that **the benefits gained from faster training convergence and better generalization error due to better point selection outweighs the additional overheads from PINNACLE's point selection**.  

However, to better address concerns about running time, we also ran **additional timing experiments for a more challenging problem setting, the 3D Eikonal inverse problem where the unknown PDE parameter to be estimated is a 3D field rather than just a scalar quantity**. The additional plot can be found in Fig. 20b in the Appendix. We find that even for this challenging problem setting, PINNACLE is able to achieve much lower loss in a shorter amount of runtime.

Overall, we can see that PINNACLE **select points that can result in faster training convergence, and hence lower overall runtime despite incurring some computation time to select the points**. We find that this gives a strong motivation to use PINNACLE over other point selection mechanism even though the point selection process itself may incur additional computational costs.

Additionally, we would like to highlight that **PINNACLE is also able to provide significant advantages beyond reducing training time, particularly in problems of selecting costly experimental data**. For example, in inverse problems (such as the 2D Navier-Stokes and 3D Eikonal examples) where the goal is to learn unknown PDE parameters from experimental data, the cost of acquiring these experimental data is often much larger than PINN training costs, due to the need of performing physical experiments and measurements to obtain these data. PINNACLE is able to more efficiently select experimental data that is most beneficial to tackling the inverse problem, by taking into account overall PINNs training dynamics with optimized collocation point selection.

We will work towards more prominently highlighting these discussions/additional results as space permits.