Paper: Understanding and Mitigating Extrapolation Failures in Physics-Informed Neural Networks (ICLR-2024)

Reviewer comment (Theory):
Leveraging the concept of spectral bias for extrapolation as a hypothesis is problematic. The spectral bias states that NN tends to learn low-frequency components more easily and faster than high-dimensional components during training [1]. While extrapolation is a validation/testing process. Thus, one should not explain a phenomenon in testing with a theory for training, or use it as a hypothesis (despite that the paper beats the hypothesis to the end). The hypothesis reference paper [2] does not make any statement on extrapolation with spectral bias either, they only assert PINNs fail to train when the training time window becomes large, possibly due to spectral bias.

Author response:
We thank the reviewer for voicing their concern. The issue with the citation pointed out by the reviewer has been fixed. By spectral bias, we indeed mean PINNs tendency to learn lower frequencies better and faster in interpolation. However, prior work has also studied the connections between NTK theory, spectral bias, and extrapolation, for example [1]. Following this, we would expect most of the extrapolation error to come from the higher frequencies (the predicted function might become smooth or flat, as in [2]). Our Fourier-based results in section 3 show that this is not the case: a large share of the error seems to come from the lower frequencies.

Reviewer comment (Experiments):
For the proposed transfer learning method, what is the benefit of transfer learning rather than directly learning new PINNs over the full domain? As shown by massive previous works, PINNs can easily learn accurate solutions for Burger's equation, as showcased in this work.

Author response:
We agree that it is generally possible to train a new model on a larger domain (assuming that domain isn’t too large) to get accurate solutions. This is, however, based on the fact that PINNs can perform well in interpolation.

As recent works have argued [3], models that can extrapolate well are generally desirable, for example because extrapolation failures indicate that the model hasn’t correctly learned the underlying physical laws [4], which are the reason for using PINNs in the first place.
Simply training a new model (on a larger domain) does not address this issue as the new model will still suffer from poor extrapolation outside of its training domain.

Transfer learning is interesting in this regard in that it seems to encourage inductive biases in the model that improve extrapolation performance (see our discussion in section 4) without training on a larger domain with the target PDE.

Reviewer comment (Evaluation):
Could the author explain why (or what does it mean) for a constant WWF distance of the true solutions for diffusion/reaction-diffusion equation in Figures 15 & 16?

Author response:
For numerical stability reasons, the WWF distance was clipped at $10^{-3}$ – the true WWF distance is in fact zero for both the diffusion and diffusion-reaction equations.

This is because the support of the Fourier spectrum of the diffusion and diffusion-reaction equation does not change over time, only the amplitudes decrease. These changes in amplitude are not captured by the WWF, since we normalize the Fourier spectra before computing the Wasserstein Distance.

Apologies for any confusion this may have caused – we have clarified this in our revision.

Reviewer comment (Evaluation):
Could the author explain why diffusion and reaction-diffusion (Figures 15 & 16) show similar WWF distance differences between true solutions and predictions as Burger's and Allen-Cahn (Figures 13 & 14), but the first two do not show significant extrapolation failure, while the latter two show significant extrapolation failures (Figure 7)?

Author response:
Note that we do not claim that the WWF between true solutions and predictions is indicative of the size of the extrapolation error. We only claim this (and experimentally show it) for the WWF between the true solution in the interpolation region and in the extrapolation region.

We believe that what the reviewer has observed here relates back to the previous comment, i.e. that the WWF is insensitive to differences in amplitude. The WWF (and WF more generally) only measures the change in the underlying distribution of different frequencies, neglecting any shift in overall amplitude. For example, multiplying the spectrum by a constant would lead to large L2 error, but the WWF distance would still be zero.

As can be seen in section A.3 of the appendix, both Burger’s and Allen-Cahn exhibit significantly larger magnitude differences between predictions and true solutions in the Fourier spectra than both the diffusion and diffusion-reaction questions, only some of which is captured in the WWF distance.

Reviewer comment (Theory):
Although the paper provides some empirical evidence of correlations of spectral shifts and poor extrapolation performance, the paper lacks a theoretical understanding of why the hypothesized Spectral Shift is the root cause of extrapolation error.

Author response:
We agree that a theoretical analysis of the role of spectral shifts in the extrapolation performance of PINNs is a natural next step, and we included this as a promising future direction in our discussion section.

In this paper, we focused primarily on putting forth evidence supporting the novel hypothesis that spectral shifts could be responsible for the poor extrapolation behavior of PINNs. We certainly welcome a more thorough theoretical investigation of this phenomenon, but we chose to leave that to future work.

Reviewer comment (Evaluation):
The empirical results for improved extrapolation performance are not convincing. The L2 Extrapolation Error on the Schrodinger Equation (imag) for the proposed method is still quite poor (290%), which is still quite unusable from a practical stand-point.

Author response:
We agree with the reviewer that the relative L2 error in extrapolation is still too high for practical usage, even after transfer learning. The goal of this paper was not necessarily to produce results fit for practical usage but rather to put forth the idea that transfer learning may help mitigate the poor extrapolation performance induced by spectral shifts. All our analysis is conducted with standard PINNs, which naturally produce worse results than current state-of-the-art methods. We hope that applying our insights to more complex architectures/models should yield similar benefits, while proving to be more practically useful.

Reviewer comment (Presentation):
The placement of Figure 1 can be improved. It is referenced in Page 6 while the Figure is present in page 1. Changing the location of the figure to a more appropriate location can improve the readability of the paper.

Author response:
We had placed Figure 1 on page 1 to highlight our main findings, but we agree that it may be better placed later in the paper. We have modified the placement of Figure 1 (now Figure 4) accordingly in our revision, as suggested by the reviewer.

Reviewer comment (Experiments):
In my opinion, transfer Learning on the full domain is an unfair comparison, as the PDEs with a similar set of coefficients were already trained on the entire domain.

Author response:
Note that due to this shared concern, we do include results for transfer learning on the same temporal domain as the baseline for comparison; these are the Transfer (half) results in Tables 1 and 2 respectively. In both cases, we observe significant decreases in the relative L2 error of the predictions compared to baseline, though these decreases are indeed smaller than those we see when we train on the full domain.

There are also instances in which our results for transfer learning from the full domain are applicable. For example, one might want to train a model on several PDEs with a vast array of different spatiotemporal domains, some of which may overlap with future target PDEs. This would of course come at some computational cost, but as seen in our transfer learning results, it could lead to improved extrapolation performance.