Paper: Neural Evolutionary Kernel Method: A Knowledge-Based Learning Architechture for Evolutionary PDEs (ICLR-2024)

Reviewer comment (Theory):
It seems the method heavily relies on the closed form formula of the fundamental solution $G_0$. The numerical error of the integration involving $G_0$ seems troublesome.

Author response:
For a linear partial differential operator, its fundamental solution exists, and the expressions for many fundamental solutions are known. Even if the fundamental solution is currently unknown, NeuralNet approximations of the fundamental solutions are still available (through radial basis function representations) and under our investigation. Regarding the accurate numerical integration, we have already employed asymptotic expansions and integration by parts, as mentioned in the text, to handle singular integrals. In mathematical terms, this does not introduce errors, so there is no need to worry about significant errors arising from the integration.

Reviewer comment (Experiments):
The experimental results of Allen-Cahn equation is not compared with the exact one or any other method.

Author response:
We have already compared the examples of the Allen-Cahn equation we worked on in the rectangular region (Section 3.2) and on the hemisphere (Section 3.3) with the solutions of classical numerical methods or the true solutions. In Appendix D.2, we explain the feasibility of the NEKM method compared to PINNs in this case. We invite the reviewer to go through these details.

Reviewer comment (Presentation):
Some minor issues: Figure 12 is too small.

Author response:
We appreciate the reviewer’s suggestion. The size of Figure 12 has been adjusted.

Reviewer comment (Evaluation):
Now that the Green function $G$ is computed by pre-training a neural network, the error of this step may propagate to solving the time evolutionary PDE. Was this problem an issue in the experiments? How accurate should the numerically approximated Green function be so as not to affect the performance?

Author response:
As introduced in the first paragraph, the training of the Green's function is a one-time effort, allowing us to take sufficient sampling points and computational efforts to approximate it as accurately as possible. This is definitely not a problem in the numerical experiments as can be observed in the presented results. By the numerical stability of the scheme we applied in time discretization, the errors cannot be amplified during the evolution. Nevertheless, one can apply the feedback control by sampling more data from the true trajectory of the PDE to reduce the approximation and enhance training and inference of the solution process in every time marching block.

Reviewer comment (Theory):
As mentioned in the paper, the possible singularity of $G_0$ may demand special handling. But the form of $G$ is generally unknown. How can the singularity appearing in $G$ be dealt with?

Author response:
In Appendix A.2, we elaborate on the acquisition of the Green's function G, whose singularity essentially arises from the fundamental solution G_0, and therefore, it can be estimated.

Reviewer comment (Theory):
Energy stsability is claimed as one of the contributions. Is this only empirically observed or grounded with some particular design?

Author response:
Energy stability is achieved through the time discretization of numerical schemes, and this can be rigorously proven mathematically, rather than being a baseless claim or anecdotal evidence. Please refer to Reference given in section 2.2 for more details.

Reviewer comment (Experiments):
While NEKM is claimed to work in complex domains, the paper primarily provides examples in small and relatively simple domains. It would be beneficial to demonstrate its performance in more complex and realistic domains, similar to the level in the referenced paper (https://arxiv.org/pdf/2309.00583), including real-world scientific and engineering geometries.

Author response:
Applying our method to more complex domains is theoretically feasible, but in practice, it may pose many challenges. For instance, the accuracy of Green's function training may decrease due to the complexity of the domain. This would be a long-term aspect of our work that needs further exploration.

Reviewer comment (Novelty):
The paper lacks references to related work that adopts neural networks only at the spatial level while using time discretizations to evolve spatial fields over time. Including references to papers like "Evolutional deep neural network (Physical Review E 2021)," "Implicit Neural Spatial Representations for Time-dependent PDEs (ICML 2023)," and "Neural Galerkin Scheme with Active Learning for High-Dimensional Evolution Equations" could help provide context and comparisons.

Author response:
We believe that our approach differs from the methods for solving time-dependent PDEs mentioned in the literature, although both involve time discretization. We emphasize an evolutionary pattern that allows us to minimize reliance on neural network training in the steps following the initial stage. The papers referred by the reviewer are all based on the L^2 residual loss minimization, which is essentially different from ours. As explained in the opening paragraph above, we aim to develop an inference-type NeuralNet architecture which preserves the physical principle such as energy stability. This is the novel point of our work.

Reviewer comment (Evaluation):
The paper does not provide information about the computational cost and scalability of NEKM compared to classical numerical methods, especially for larger 3D problems. It would be valuable to include performance comparisons in terms of computational efficiency.

Author response:
Due to its inference nature, the computational efficiency can be good even in comparison with classical numerical methods, as the proposed method avoids the solution of linear solver but instead saves the solution operator via a well-trained NeuralNet. The comparison of computational efficiency with PINNs was also provided in the appendix.

Reviewer comment (Novelty):
My biggest confusion and concern is the relationship between this paper (Lin et al., 2023a) as well as (Lin et al., 2023b). Those papers also use a convolution representation of the solutions using Green's functions. What exactly is the author's contribution except working with time-dependent problems?

Author response:
Our main contribution is proposing an evolution law for time-dependent equations using a time discretization scheme, operator splitting, and boundary integral equations. In the solving process, we utilized the knowledge from the papers (Lin et al., 2023a and Lin et al., 2023b). As stated in our article, BINet and BI-GreenNet are the two essential tools we require. Additionally, we provided a detailed description of how to handle singularities introduced by the fundamental solution in boundary integrals.

Reviewer comment (Theory):
The paper focuses on semi-linear PDEs, but it would be interesting to know if NEKM can be extended to handle nonlinear PDEs. Clarification on the limitations and potential extensions of the method for nonlinear problems would be beneficial.

Author response:
We focus on semi-linear PDEs because they allow us to derive the evolutionary pattern of the desired ideal solution, something that is not achievable with nonlinear PDEs (which may not even have a corresponding fundamental solution for the operator). This is discussed in the opening paragraph above. In fact, we claim to be able to solve semi-linear PDEs, which already covers a broad range of time-evolving PDEs. We only require the spatial part of the differential operator to be linear, meaning terms like (u_x)^2 are excluded. We allow terms such as f(u) to appear, encompassing a significant class of PDE equations.

Reviewer comment (Writing):
The authors should define earlier what they mean by evolutionary PDEs.

Author response:
Thank the reviewer for his/her suggestion, but we proposed it at the beginning of the method section, which should be considered a relatively conventional approach.

Reviewer comment (Novelty):
Connection to other kernel operator methods such as the Fourier Neural Operator (FNO) should be considered. It is only briefly discussed in one sentence of related work with a majority on the PINNs literature. In particular, in the related works, the authors discuss in detail how boundary conditions are incorporated into Physics-Informed Neural Networks (PINNs). The related work in the Neural Operator community should be discussed, such as how to incorporate boundary conditions into Neural Operators in Saad et al., "Guiding continuous operator learning through Physics-based boundary constraints", ICLR 2023.

Author response:
As suggested, we incorporated content related to FNO in the related work section in the revised paper.

Reviewer comment (Theory):
The method only works on semi-linear PDEs. This is actually a very strong assumption and limitation. The authors should discuss the extension to nonlinear PDEs.

Author response:
We focus on semi-linear PDEs because they allow us to derive the evolutionary pattern of the desired ideal solution, something that is not achievable with nonlinear PDEs (which may not even have a corresponding fundamental solution for the operator). In fact, we claim to be able to solve semi-linear PDEs, covering a broad range of time-evolving PDEs. We only require the spatial part of the differential operator to be linear, meaning terms like (u_x)^2 are excluded. We allow terms such as f(u) to appear, encompassing a significant class of PDE equations.

Reviewer comment (Experiments):
Evaluation: the method is only tested on the simple linear heat/diffusion equation and Allen-Cahn equations. The heat equation is smooth and parabolic and very easy for numerical methods to solve. It would be nice to test hyperbolic problems with shocks, e.g., in the GPME benchmarking framework in Hansen, et al., "Learning physical models that can respect conservation laws", ICML 2023 (https://arxiv.org/abs/2302.11002).

Author response:
Yes, the heat equation is a fundamental example, and we have added examples of the Allen-Cahn equation. Traditional numerical methods for solving the Allen-Cahn equation are not straightforward, often requiring considerations of numerical stability and knowledge related to energy stability. Additionally, our goal is to solve the equation in a general domain, posing new challenges for traditional methods. Examples of other equations will be explored in future numerical experiments.

Reviewer comment (Theory):
The method seems to have strong limitations if the first step requires an analytical or numerical solution to the ODE.

Author response:
Regarding the concerns about the difficulty of analytically solving ODEs, there is no need to worry. The ODEs we solve have the form u_t = f(u), excluding spatially related differential operators, making them amenable to analytical solutions through integration. In Appendix D.1, we provide a specific example of implementation.

Reviewer comment (Theory):
The authors should clarify in the last paragraph of the introduction where the nonlinear ODE is coming from in step 1 and then how there is "numerically integration" for the related linear PDE. Typically, in numerical methods, a (non)linear PDE is first discretized in space and then the resulting semi-discrete form of the ODE is discretized in time. The authors should clarify what they mean here.

Author response:
Nonlinear ODEs are given by operator splitting, as detailed in case 2 of Section 2.1. In essence, our method alternates between solving the nonlinear ODE part, solving the linear PDE, and solving the ODE—a standard Strang splitting process. For a better understanding, refer to the detailed examples in case 2 of Section 2.1 and Appendix D.1. Due to space limitations, we were unable to showcase the specific implementation of the method in the numerical experiments section but included it in the appendix.

Reviewer comment (Presentation):
Some of the equation details of BINet in the related work should be moved to an appendix or background section.

Author response:
As suggested, we have reduced the introduction section's discussion of BINet.

Reviewer comment (Theory):
Care should be taken with the discretization because this adds a first order error into the scheme. For example, the first equation should not be discretized with the 1st order accurate Forward Euler without even citing the method. This is an explicit method and there are necessary bounds on $\Delta t$/$\tau$ to ensure numerical stability. See Krishnapriyan et al., "Learning continuous models for continuous physics", 2023 (https://arxiv.org/pdf/2202.08494.pdf) on how the time discretization matters in NeuralODE and the 4th order RK4 is advantageous but even that scheme without being careful about the numerics can lead to convergence issues.

Author response:
The reviewer pointed out the discrete formulation in our article: our time discretization uses a first-order implicit scheme. However, in principle, we can adopt higher-order time discretization schemes without increasing difficulty. Traditional numerical methods need to balance spatial and temporal step sizes to ensure stability, but our method, not relying on spatial discretization, eliminates stability concerns. If one is concerned about energy stability and other properties, our method can still be combined with the corresponding time discretization formats to achieve the desired results.

Reviewer comment (Presentation):
Ideally the method and presentation wouldn't need to be separated into separate cases for linear equations or not.

Author response:
Our method indeed relies on BINet, but our primary contribution lies in proposing an evolution law for time-dependent equations using time discretization, operator splitting, and boundary integral equations. It can be combined with the appropriate time discretization format to maintain the desired properties. Additionally, we provided a detailed description of how to handle singularities introduced by the fundamental solution in boundary integrals.

Reviewer comment (Novelty):
It seems like the method depends too strongly on the BINet method and the authors should better differentiate the novelty between the two.

Author response:
This is corrected.

Reviewer comment (Writing):
The exposition of the method in Section 2 isn't too clear and some of the details can be moved to an appendix.

Author response:
In Appendix D.1, we compared the accuracy of the NEKM method with the PINN method; in Appendix D.2, we explained situations where the PINN method is challenging to train, while our method remains effective. We apologize for any confusion caused.

Reviewer comment (Writing):
"heat equation" shouldn't be plural in the last bullet point of the contributions.

Author response:
This is corrected.

Reviewer comment (Writing):
Add a comma after "In this section" at the beginning of Section 2 Method.

Author response:
Minor points: We followed the reviewer’s instructions to revise the paper and correct those items.

Reviewer comment (Writing):
Comma missing after Equation 7.

Author response:
Minor points: We followed the reviewer’s instructions to revise the paper and correct those items.

Reviewer comment (Theory):
Does the method only work on semi-linear PDEs? If so, this is a bit limiting and the authors should discuss the extension to nonlinear PDEs.

Author response:
We focus on semi-linear PDEs because they allow us to derive the evolutionary pattern of the desired ideal solution, something that is not achievable with nonlinear PDEs (which may not even have a corresponding fundamental solution for the operator). In fact, we claim to be able to solve semi-linear PDEs, covering a broad range of time-evolving PDEs. We only require the spatial part of the differential operator to be linear, meaning terms like (u_x)^2 are excluded. We allow terms such as f(u) to appear, encompassing a significant class of PDE equations.

Reviewer comment (Reproducibility):
**Complexity**: The proposed NEKM method, while innovative, is complex in its approach, involving the integration of operator splitting, boundary integral techniques, and Deep Neural Networks. This complexity might make it challenging for practitioners who are not well-versed in all of these areas to implement and understand.

Author response:
The technique we employed in NEKM is quite standard in the community of mathematics. Though challenging for practitioners, the process is formally interpreted using machine learning terms. The solution process for semi-linear PDEs involves two stages: 1) training—pretraining Green functions and neural network representations of evolutionary kernels; 2) inference—evolving PDE dynamics using the trained kernel. This decomposition results from observing the PDEs' mathematical structure. Linear parts have a convolutional representation through BINet, allowing time-independent training in advance. Nonlinear parts resemble CNN activation processes. A well-designed time discretization integrates structures into a NeuralNet architecture, resembling a CNN mechanism. Green's function training is a one-time effort for inference, evolution, and solving similar equations. Energy-stable property and a designed time discretization maintain energy, ensuring physical reliability in the proposed NeuralNet framework.

Reviewer comment (Reproducibility):
**Computational Resources**: The paper does not extensively discuss the computational resources required for training and applying the NEKM method. Deep learning methods often demand significant computational power, which could be a limitation for some users, particularly those without access to high-performance computing resources.

Author response:
Specifically, in the first step, we train the kernel function and the Green's function (if necessary), and in subsequent steps, all that is required is network inference and numerical integration calculations. The training of our Green's function is a one-time effort and can be used for the inference, evolution, and even the solution of other similar equations after the fact.

Training the neural network in Equation 4 was very fast, sometimes taking less than a minute. Training the Green's function could take longer. This can be further improved if more sophisticated machine could be used (our computation is based on TITAN XP GPU). After training, calculations can be performed on a CPU, which usually does not take much time.

Reviewer comment (Experiments):
**Limited Real-World Use Cases**: While the paper demonstrates NEKM's effectiveness in solving specific mathematical problems, it remains largely theoretical. More real-world use cases and practical applications in various domains would strengthen the paper's relevance and utility.

Author response:
Applying our method to more complex domains is theoretically feasible, but in practice, it may pose many challenges. This would be a long-term aspect of our work that needs further exploration.

Please see 3 in ‘Weakness’ part.

We believe that the tasks we can prioritize mainly involve solving problems in more complex domains, including how to improve the accuracy of the method and reduce computation time. In fact, we already have collaborators who have proposed using neural networks to approximate G_0, which will reduce the time required for directly computing G_0 using explicit expressions.

It's evident that the reviewer values the practical utility of a method and its potential integration with engineering problems. This is an aspect we will pay attention to, and some of our collaborators are from commercial companies, so we will pay attention to their needs and continuously modify and improve our method.

Reviewer comment (Evaluation):
**Interpretability**: The paper discusses the use of neural networks, which are often seen as "black-box" models. While the paper addresses some interpretability challenges, it might not provide a complete solution to the interpretability issues associated with deep learning approaches.

Author response:
Regarding the concern about interpretability, we integrated mathematical knowledge into our method precisely because of the lack of interpretability in some neural network methods, aiming to enhance the interpretability of the equation solutions. Nevertheless, achieving complete interpretability in deep learning is a challenging area for us, as well.

Please see 4 in ‘Weakness’ part.

Reviewer comment (Reproducibility):
**Algorithm Complexity**: The proposed method involves a combination of different techniques, such as boundary integral representation and neural networks. This may make the implementation and understanding of NEKM challenging for some users, potentially limiting its widespread adoption.

Author response:
Yes, our method is more difficult than some other methods such as PINNs in coding, but our advantages has shown in 1.

The proposed method mainly rely on numerical integration, which we believe is not quite sophisticated for practitioners. In fact, given the advantages we elaborate in our response to Weakness 1, the CNN-like structure of our method makes it easily generalized.

When handling singular boundary integrals, it is true that our method introduces some complexity at the code level. But it contributes to improved accuracy.

Reviewer comment (Experiments):
**Empirical Validation Scope**: While the paper includes empirical validation on heat and Allen-Cahn equations, the scope of the empirical validation might be limited. A more extensive range of test cases across different scientific and engineering domains would strengthen the method's generalizability.

Author response:
Please see 3 in ‘Weakness’ part.

Reviewer comment (Experiments):
**Scalability**: The paper does not explicitly address the scalability of the NEKM method. As the complexity of problems increases, it remains to be seen whether NEKM can efficiently scale to handle more complex and larger-scale scenarios.

Author response:
This may require extensive study including both sufficient computational experiments and deep theoretical thinking; this is indeed one of the long-term tasks in our future planning.

Yes, it is a long-term challenge, and we need to invest more effort to address it.

Reviewer comment (Experiments):
**Comparison to Existing Methods**: The paper lacks a comprehensive comparison of the NEKM method with existing approaches for solving similar problems. Such comparisons would help to better assess the relative strengths and weaknesses of NEKM.

Author response:
In Appendix D.1, we compared the accuracy of the NEKM method with the PINN method; in Appendix D.2, we explained situations where the PINN method is challenging to train, while our method remains effective.

Please see 8 in ‘Weakness’ part.