Paper: Solving High Frequency and Multi-Scale PDEs with Gaussian Processes (ICLR-2024)

Reviewer comment (Novelty):
The manuscript misses a range of closely related work about solving partial differential equations with Gaussian processes: Howhadi [1] discusses Bayesian numerical homogenisation (solving multi-scale PDEs with GPs), a central piece of related work that this paper should address prominently. Accessing the derivatives of the Gaussian process as a function of $\vec{\mathcal{U}}$ (the trick suggested in Equation (13)) is discussed in Section 2 and Section 3 in the paper by Krämer et al. [2]. The related work section (Section 5) mentions the work on solving PDEs with GPs by Chen et al. (2021) and Long et al. (2022b) but completely misses all work that belongs to the field of probabilistic numerical algorithms (see, for instance, [1-3] below and the references therein). While large parts of the ICLR community may be more familiar with physics-informed neural networks than with Gaussian-process-based differential-equation solvers, the works mentioned above are closely related and must be discussed appropriately.

Author response:
Thanks much for providing these excellent references. We do agree these works are related and important. We will cite and discuss them in our paper to appreciate their contributions.

Reviewer comment (Novelty):
The submission never mentions traditional approaches to solving multi-scale problems, such as those based on numerical homogenisation (a starting point could be [4]). While the paper targets the ICLR community, and large parts of this community may be more interested in machine-learning-based PDE solvers than traditional PDE solvers, existing approaches should be credited by at least mentioning their existence. This relates to point 4 below.

Author response:
We appreciate the reviewer suggesting and providing references for traditional approaches.  We do agree. We will add references and credit those methods (including [4]) in our paper.

Reviewer comment (Evaluation):
The experiments only discuss the precision of the approximations rather than the work required to achieve this precision. The experiments in Appendix C suggest that the per-iteration runtime of the proposed algorithm is comparable to that of a physics-informed neural network. However, Figure 9 indicates that tens of thousands of iterations are needed. What is each algorithm's overall runtime (training time) to achieve the precisions in Tables 1 and 2?

Author response:
Great observation and questions. Actually, the number of iterations needed to reach a decent accuracy depends on the problem. That is why we report the average per-iteration time. In most cases, our method needs  <100K iterations to achieve the accuracy reported in the tables, e.g., 10K for $u_1$ and 20K for $u_3$ for 1D Poisson. In the most challenging cases, including $u_5$ for 1D Poisson and $u_7$ for 2D Poisson, it needs more iterations (>=200K). This is also comparable to the number of iteration required in PINN based methods, e.g., in [1], the PINN is trained with 10K ADAM iterations and 50K L-BFGS iterations.  
Thanks for the great suggestion. We will supplement the detailed total running time for each test case in our paper.  

[1] Shibo Li, Michael Penwarden, Yiming Xu, Conor Tillinghast, Akil Narayan, Robert Kirby, and Shandian Zhe, “Meta Learning of Interface Conditions for Multi-Domain Physics-Informed Neural Networks”, The 40th International Conference on Machine Learning (ICML), 2023.

Reviewer comment (Novelty):
One of the weaknesses of the paper is that the difference between prior GP-based works in the literature is unclear. The authors should clearly state the differences between their work and the existing ones, in particular the papers by Chen et al (JCP, 2021) and Raissi et al (JCP, 2017), cited in this work. I believe that the two differences are: (1) The use of student-t distribution and (2) using a Kronecker product grid to speed-up computations. In the current version, the title and abstract of the paper might be misleading to readers in the sense that this paper is not the first one to consider solving a PDE with GP.

Author response:
In fact, we did explicitly emphasize the key difference with Chen et al (JCP, 2021). Please see the bottom of page 4 and top of page 5. In Section Related Work, we have also discussed the challenge of the SE/Matern kernel used by Chen et al (JCP, 2021) in solving the high-frequency/multi-scale PDEs (4th line, 2nd paragraph), which motivates our design of mixture kernel (a 3rd critical difference).  We agree your summary about the differences as well. We believe the key difference of Raissi et al (JCP, 2017) has also been covered by our paper --- see Section Releated Work, the 1st sentence of 2nd paragraph --- it only solves **linear PDEs with noisy measurement of source terms**. Our work (as well as Chen et al (JCP, 2021)) is more general in that it applies to both linear and nonlinear PDEs. We will highlight the difference in a stronger manner.

Second, we **never** claim/imply that our paper is "the first one to consider solving a PDE with GP". In fact, In Introduction Section (2nd paragraph, Page 2), we explicitly claim that **"we resort to an alternative arising ML solver framework, Gaussian processes (GP)" with (Chen et al., 2021) cited**. We will emphasize more on this point.

Reviewer comment (Experiments):
I am very surprised by the Spectral method experiment performed by the authors. A quick experiment with Chebfun (https://www.chebfun.org/) allows me to solve the 1D Poisson equation to 10 digits accuracy in a fraction of a second.

Author response:
Our spectral method was implemented based on Fourier bases to be consistent with kernel construction in our method for a fair comparison. The performance is indeed inferior (as shown in the tables). This is to show that introducing a mixture model in the frequency domain (rather than the temporal domain), with which to construct the GP, is much more effective.

Second, we do agree there can be more effective traditional solvers, such as Chebfun  (but not all the numerical methods work better for this problem; please see R4 to Reviewer J8Ew below). In many forward problems, machine-learning (ML) based PDE solvers, like GP or PINN, have not been able to perform equally well (in accuracy/efficiency) as traditional solvers. However, as explained in the 1st paragraph of Introduction, ML-based solvers "are simple to implement, and can solve inverse problems efficiently and conveniently". In addition, GP models can easily, robustly handle noisy measurements and/or source terms, offering a principled and convenient framework to quantify and reason under uncertainty. We therefore view **developing ML-based solvers as a promising direction, like the concurrent blooming efforts in this direction, even in several aspects the ML-based solvers are now still behind traditional numerical solvers that have been developed for decades**.

Reviewer comment (Novelty):
The authors state at the end of Section 5 that their work is "the first to [...] use the Kronecker product structure to efficiently solve PDEs". I believe this sentence is erroneous. It is completely natural when using spectral methods to exploit the Kronecker product structure of the domain (see e.g. Fortunato & Townsend, 2019).

Author response:
Thanks for the comment. We do agree that our statement is not precise. We mean to say GP plus Kronecker product structure (rather than Kronekcer product structure only). We will revise our statement to be accurate and clear. Please see R2 to Reviewer GK3t above for the detailed clarification.

Reviewer comment (Novelty):
The authors offer very little conclusion and the main limitation section from the Appendix should appear in the main text. One of the key limitations of the method is that it is limited to simple geometry, where one can use a Kronecker product structure. However, this severely limits the applicability of the method and questions its advantages over spectral methods.

Author response:
Due to the space limit, we leave the limitation section in Appendix. We will re-organize the paper to merge the limitation and conclusion section, and put it in the main paper. 

We believe our method is **NOT** limited to simple geometry. Even if the domain is not a grid, we can create a grid that covers the domain, and use the GP conditional mean (see Eq4 and Eq13) to predict the boundary values and fit the boundary conditions via the GP likelihood $\text{log} \mathcal{N}(\mathbf{g}|\mathbf{u}_b, \tau_1^{-1}\mathbf{I})$ (see Eq14). We then maximize the log joint probability to solve the problem. Note that the computation efficiency (via Kronecker product plus tensor algebra) remains the same.

Reviewer comment (Evaluation):
Is there any concrete advantage of using the Student t distribution over the Gaussian distribution? The authors state that the Student t distribution is beneficial for high frequency but the performance seems very similar in the experiments of Table 1.

Author response:
Great question. As explained in Section 3 (see the text under Eq5), the student $t$ distribution has a fat-tailed density, and hence can more robustly capture "the (potentially many) minor frequencies". By contrast, the Gaussian distribution density has a very thin tail, and can be much more sensitive to minor frequencies, especially when they are not ignorable.

However, we agree that our most test cases comprise only one or just a few principle frequencies, and hence the results do not exhibit a strong contrast of the two choices. But two examples worth attention. One is the 1D Poisson with solution $u_4 = x\cdot sin(200 x)$, and the other is 1D Poisson with solution $u_5 = sin(500x) - 2(x-0.5)^2$. In both cases, the solution contains many minor frequencies (due to non-trigonometric terms). We can see that our method with student $t$ mixture (GP-HM-StM) achieves much smaller error (one magnitude smaller) than with Gaussian mixture (GP-HM-GM).

We will add more such test cases to highlight their difference.

Reviewer comment (Reproducibility):
The experiments in Section 6 should provide the source terms (at least in Appendix) to allow for reproducibility.

Author response:
Thanks for the suggestion. As explained in Section 6, specifically, the text under Eq17, Eq18 and Eq19, we follow existing literature to first craft the solution, e.g., $u_1 = \sin(100x)$ and $u_2=\sin(x) + 0.1\sin(20x) + 0.05\cos(100x)$, and then apply the differentiation operator to obtain the source term $f$. We will supplement differential results (e.g., $f$) in the paper.

Reviewer comment (Novelty):
The paper claims novelty by stating: "By contrast, for PDE solving, it is natural to estimate the solution values on a grid, which opens the possibility of using Kronecker products for efficient computation. To our knowledge, our work is the first to realize this benefit and use the Kronecker product structure to efficiently solve PDEs." This claim is disputed because Kronecker product structure on regular grids has been used before for solving PDEs (e.g., via preconditioners or tensor numerical methods such as Gavrilyuk et al. 2019). Additionally, the computational efficiency of product kernels on regular grids is known in Gaussian Processes (Saatci, 2012), and PDE solvers based on GPs that use tensor product kernels (specifically Matérn) have been proposed previously (e.g., Wang et al. 2021).

Author response:
Thanks for the valuable comments.  Our statement is indeed **not** precise, and we will remove the claim. From the context, we mean to say, we are using GP + Kronecker product structure --- **taking advantage of the corresponding computational advantage** --- to solve PDEs (rather the Kronecker product itself). The computational advantage is fulfilled by combining the properties of Kronecker product of kernel matrices and tensor algebra, which we believe is rarely used in the domain of PDE solving. In fact, we have acknowledged tensor-product structures are widely used in PDE solving; see the text under Eq11.

We never claimed that our work invents/discovers the computation advantage of GP+Kronecker product. As emphasized in related work, this advantage has been known in the GP community; our contribution is to **enable this computational advantage** in PDE solving.

We do agree (Wang et al. 2021) have also proposed the product kernel, but the paper is mainly about theoretical analysis (e.g., the sample path property) and does not fulfill efficient computation. Thanks this nice paper --- it provides additional strong justification of using a product kernel; we will cite and discuss about it. Thus we believe, the contributions of our work are still orthogonal to existing work in a large degree. We will make a more detailed, careful discussion and clarification.

Reviewer comment (Presentation):
The plots in the paper, while informative, are small; larger plots would improve legibility.

Author response:
Great suggestion. We will enlarge the figures to improve the legibility.

Reviewer comment (Experiments):
The experimental evaluation raises concerns about the fairness of the baseline comparisons. Specifically, the vanilla GP baseline with a Matérn kernel may not use a product kernel, and it is unclear why Kronecker product plus Toeplitz structure (as discussed in Wilson et al. 2015) could not be applied to improve efficiency for the vanilla GP as well. If such techniques were not used, the presented baseline might be rather weak, and although a scale mixture may still outperform a Matérn kernel, the gap in time and memory performance should likely be smaller.

Author response:
Great question. We did use product kernels for vanilla GP baselines. We use the same computational approach (Kronecker product + tensor algebra) to fulfill efficient training and prediction. We will clarify it in the paper.

Reviewer comment (Novelty):
The related work section omits relevant prior work on Gaussian-process-based PDE solvers. Examples include:
- Cockayne et al., "Probabilistic numerical methods for PDE-constrained Bayesian inverse problems" (2017)
- Wang et al., "Bayesian numerical methods for nonlinear partial differential equations" (2021)
- Pförtner et al., "Physics-informed Gaussian process regression generalizes linear PDE solvers" (2022)
- Chen et al., "Sparse Cholesky factorization for solving nonlinear PDEs via Gaussian processes" (2023)

Author response:
We appreciate the reviewer for providing these great references. We will cite and discuss them in our work.

Reviewer comment (Experiments):
The paper does not investigate the impact of the number of components in the mixture kernel on the approximated solution. An ablation study plotting the number of components versus error would be informative. Additionally, it is not discussed how the number of components affects optimization difficulty, with the concern that too many components could make optimization significantly harder in practice.

Author response:
Great question! We have followed your suggestion to give the result on 1D Poisson (with solution $u_1$ and $u_3$). We varied the component number from {3, 10, 30, 50, 80}. See [Here](https://github.com/wctghlbgtat/Solving-High-Frequency-and-Multi-Scale-PDEs-with-Gaussian-Processes/blob/main/components.png). As we can see, using two few or too many components can both increase optimization challenges (too few can make it harder to move to the right frequency). The best choice is in between. We will supplement the results and discussion in our paper.