Paper: PINNacle: A Comprehensive Benchmark of Physics-Informed Neural Networks for Solving PDEs (ICLR-2024)

Reviewer comment (Reproducibility):
The authors did not give much detail on what computer resources are needed for PINN methods. They did not say if the methods work fast or slow. It would be helpful to know what computer tools or power is needed, as people who want to use these methods would find that information useful.

Author response:
**A2.** All our experiments were conducted using a Linux server with 20 Intel(R) Xeon(R) Silver 4210 CPUs @ 2.20GHz and eight NVIDIA GeForce RTX 2080 Ti each with 12 GB GPU memory. We have included this information in **Appendix E.1**.

Reviewer comment (Experiments):
The authors worked with a set of 20 PDE problems, but they might have missed some other important problems. In future studies, it would be good to add more problems to their list so that we can learn even more.

Author response:
**A3.** The selection of PDEs for our benchmark was derived from a range of representative literature, encompassing typical equations categorized by their mathematical properties. Specifically, we chose 7 major categories of PDEs, relevant to various fields like fluid dynamics and electromagnetism. The primary challenge lay in the multitude of PDE types and the complexity of categorization. Our solution involved classifying PDEs by mathematical form, then grouping practical problems under these different types, and designing varying levels of difficulty.

Reviewer comment (Novelty):
The paper lacks technical novelty to be considered for the main track. In my opinion, the paper is more suitable for an application/dataset track, for e.g., NeurIPS Dataset/Benchmark Track.

Author response:
We appreciate your perspective, yet it's pertinent to note that at the ICLR Conference, numerous benchmark and dataset articles are accepted each year, as exemplified by references [1,2,3,4]. While benchmark articles may not introduce new techniques, they play a crucial role in highlighting existing issues in the field and guiding future research directions. Therefore, we respectfully request that you reconsider the contribution of our paper in this context.

Reviewer comment (Experiments):
Table 3 shows that all of the selected SOTA methods fail on the KS Equation. However, some PINN methods can solve KS Equations such as Causal PINNs [1].

Author response:
CausualPINNs represent a combination of various methods and techniques, and the training parameters and iteration numbers in their original work significantly differ from those in our study. Although their results suggest the resolution of the KS equation, it does not entirely apply under our problem settings. Given the plethora of improvements in PINN methods, we promise to gradually include more widely recognized, published, and effective works in our future research.

Reviewer comment (Evaluation):
When comparing the effect of the parametric PDEs on different PINN variants (shown in Table 4), using the Average L2RE is not a good choice. It would be more informative to show the mean and the standard deviations for the different parameter choices. The average L2RE can be skewed if one (or few) of the parameter settings fails (i.e., have L2RE of 100%) while others have very low errors (such as ~1e-4).

Author response:
Due to the limited space in the main text, we only presented averaged L2RE, but comprehensive statistics, including mean and standard deviation, are provided in the appendix's **Table 27** for reference.

Reviewer comment (Presentation):
This paper may be more suited for a journal format, as the page limit constraint is negatively impacting the work. By the time the authors present the data and experiments, there is less than half a page left to interpret the results and provide discussions and conclusions.

Author response:
We acknowledge your concern regarding the format constraints. Due to space limitations, a considerable portion of our results and discussions had to be relegated to the appendix, which might inconvenience readers. However, it's worth noting that at the ICLR conference, many benchmark articles are accepted annually [1, 2, 3, 4]. Furthermore, the main conclusions of our article are presented in the main text, with the appendix serving primarily to supplement and corroborate these findings.

Reviewer comment (Theory):
The work utilizes 10 different methods for training PINNs, but a brief description of these methods in a single mathematical framework is missing. Adding such a description and correlating the numerical findings to the theoretical properties of each method is probably the most important, yet underperformed, part of the work. While space constraints are understood, including such a section, even briefly in the appendix, would be valuable.

Author response:
Thank you for your valuable suggestion. It is important to note that different methods contribute to the improvement of PINNs in varied ways, hence we categorized these methods and described each category within a unified framework. Due to constraints in the main text's length, we have included these descriptions in an additional section denoted as **Appendix B.4**.

Reviewer comment (Presentation):
It would be great to have a table or list (in the appendix) detailing a comparison of the provided data sets to those in PDEarena (and PDEbench).

Author response:
**Q6.** Comparing datasets and PDEs with those used in PDEBench/PDEArena.

**A6.** The PDEs we selected are not significantly related to those chosen in PDEBench or PDEArena. PDEBench and PDEArena primarily focus on time-dependent PDEs like compressible Naiver-Stokes equations, Diffusion Reaction equations, etc. Moreover, their complexity is such that PINNs often fail to solve them correctly or yield substantial errors. Our selection of PDEs, derived from various PINN publications, encompasses a diverse range of types and complexities. For common PDEs, we selected the incompressible Naiver-Stokes equation and the Poisson equation (Darcy flow) due to their representativeness and widespread applicability in numerous fields.

Reviewer comment (Experiments):
The relative error values in the results tables are for the most part shockingly bad and simply not useful for many numerical analysis contexts. Given that PINNs seem to be mostly providing different function spaces for PDE solutions, one original base PINN should be included in the benchmark, which is to give each hat function on a finite element mesh one parameter, and hence include finite element methods. Because some of the data sets were created using FEM, the original mesh would yield 0 error, but different meshes may not, and in particular coarser meshes would accumulate error. Analyzing a curve of remeshing from same resolution to coarse would provide a baseline for the performances of the other PINNs.

Author response:
**Q1.** The relative error values in the results tables are for the most part shockingly bad and simply not useful for many numerical analysis contexts.

**A1.** In the experiments presented in this paper, a variety of metrics such as L2RE, L1RE, MSE, and Fourier error at different frequencies were employed to demonstrate the discrepancies between the predicted solutions and the true solutions. Importantly, the poor performance of L2RE in certain tasks is attributed to the deliberate selection of PDEs with varying degrees of difficulty. For instance, NS2d-LT and Heat2d-LT highlight the current challenges in mitigating error accumulation over long durations in PINNs, thereby drawing attention to this issue. Moreover, most PINN-related studies compare L2RE in their results; hence, our utilization of L2RE as a primary criterion facilitates a more effective comparison with other research findings.

**Q2.** Analyzing a curve of remeshing from the same resolution to coarse.

**A2.** Firstly, PINNs are not purely data-driven machine learning methods; they do not require a training set. Thus, even using original FEM grid points does not reduce the error to zero at these points. Secondly, we present the trend of L2RE variation with increasing grid points (corresponding to batch size) in **Table 15** and **Figure 2**, accompanied by an analysis in the main text.

**Q3.** On using PINNs to predict FEM's hat function.

**A3.** We believe that combining the strengths of PINNs and FEM in this manner could be an interesting approach worthy of in-depth investigation. However, as a benchmark, comparing this currently nascent method falls outside the scope of this paper. We anticipate considering its inclusion in our comparisons once more detailed related research is published in the future.

Reviewer comment (Evaluation):
It also becomes important to quantify flop counts. It appears that most PINNs need to be fitted for each PDE solution, incurring the typically high flop count of solving an optimization problem (compared to one forward pass), and only some of them can learn solutions conditional on hyperparameters given as input and require only forward passes to solve e.g. from different initial conditions. For all cases, there should be 3 different flop counts provided: 1) The number of flops required to create the training set 2) The number of flops required for any general training of the method 3) the number of flops required to evaluate/fit the method on a particular example. Many PINNs, and the FEM baseline would only have nonzero counts in point 3, and it would be good to compare them. Having flop counts or even wall time counts would allow answering questions like "at equal error rate, does fitting a PINN or fitting FEM cost more computational power?" and "At equal computing power, can FEM beat the error rates of the listed PINNs?"

Author response:
**Q4.** Calculation of FLOPS.

**A4.** We have listed the training and inference FLOPs of different methods across various PDEs in the **Table 13, Table 16, Appendix E.1**. As for the flops for data generation in main experiments, we only generate FEM solution for evaluating different approaches on a single instance per PDE. Thus the cost is negligible as we do not need parametric data like PDEBench.

Reviewer comment (Experiments):
Continuing the discussion about flop counts, methods learning from multiple data sets/examples should be included in order to compare flop counts and provide additional reference error values. In particular for the time propagating PDEs, solutions using U-nets or FNOs from e.g. PDE bench should be included as reference values, in terms of performance, flops required for training, flops required to generate the required training data, and flops required to run a forward pass to obtain a solution. Then one can assess how many PINNs or FEM solutions one can compute for the same budget as a certain number of forward passes of the propagator network. There should be a break-even point at some number of forward passes justifying the training effort.

Author response:
**Q5.** Comparison with methods like U-Net, FNO, etc.

**A5.** In all our experimental settings for positive PDE problems, no data was provided; we sought to solve using PINNs solely based on the PDE itself. In contrast, neural operator methods such as FNO and U-Net are data-driven and, thus, are not applicable to our problem settings.