Paper: Efficient Certification of Physics-Informed Neural Networks (ICLR-2024)

Reviewer comment (Experiments):
The major concern that reviewer has is that the certification procedure is the on PINN residual error, $f$, but not on the estimation to the true PDE solution. While the authors have provided Exp 2 and Appendix C to demonstrate that there is a strong relationship between the residual error and the PDE solution, it is only conducted on PDEs with a clear PDE solution. More experiments should be conducted with various PDE equations (with multiple solutions, with trivial solutions, etc.) to find out how bounding $f$ is effective. Especially for the recent papers that discuss the failure modes of PINN[1], how would bounding the residual errors help, both theoretically and empirically?

Author response:
**(W1) On the relation of residual and solution error.** This is an important point. We believe the analysis in Section 6.2 and Appendix C indicates a strong correlation between the two types of errors, but there is more to be done in this direction in future work.

Reviewer comment (Presentation):
Figure 2 looks meaningless. The Figure 2(a)-2(d) only shows the visualization of the time evolution of $u$, versus the residual errors for three different PDE equations. However, it is hard to observe any clear patterns of the residual errors, and the authors only briefly mention the residual errors without explaining any scientific insights from the Figure. The reviewer thinks it is better to put the Figure into Appendix, and instead briefly mention Appendix A and B in the main text.

Author response:
**(W2) On Figure 2.** The main purpose of Figure 2 is to give an intuition to readers regarding the landscape of the solution and residual error of each of the PDEs studied. However, we agree with the reviewer that the figure would likely be better suited for the appendix, and will move it there in the final version of the paper.

Reviewer comment (Experiments):
In Section 6.3, the authors should consider also including the ablation study on $N_b$ into the main text.

Author response:
**(W5) Including N_b ablation in the main text.** Given the plan to move Figure 2 to the Appendix as per (W2) of this rebuttal, we will move the ablation on N_b to the main text in the final version of the manuscript.

Reviewer comment (Writing):
In Section 5.3, the authors mention that "exploring the areas... via sampling (SAMPLE, line 3)", is there a typo and should be line 5 instead of line 3?

Author response:
**(W6) Typo in Algorithm 1.** We thank the reviewer for pointing out that typo which will be corrected in the final version of the paper.

Reviewer comment (Experiments):
In the appendix A, the authors have mentioned to use Physics-informed Adversarial Training to reduce the errors of PINN and demonstrate that $\partial$- CROWN can also bound the residual errors of PIAT. However, there are been various ways of reduce the errors of PINN, for example, scheduled training [1] or building a more complex architecture with transformer [2]. The reviewer is wondering if the $\partial$-CROWN can adapt to NN training that involves more complex loss functions, or more complex architectures?

Author response:
**(Q1) On verifying different training regimes and architectures.** $\partial$-CROWN is a post-training certification framework, i.e., it’s completely agnostic to the training method. This is why it can be applied to PIAT, and why it can naturally be applied to the curriculum learning scheme proposed by [1], which is similar to the one used for training Takamoto et a., (2022). When it comes to a different architecture such as the transformer one considered in [2], we believe the extension of $\partial$-CROWN to this setting would be doable using a hybrid scheme and by deriving theorems similar to Theorem 1 and 2.

Reviewer comment (Theory):
Have the authors study the relation between $\partial$-CROWN's time complexity to the size of the PINN's neural network?

Author response:
**(Q3) On the relation between $\partial$-CROWN’s time complexity and solution network size.** From a theoretical perspective, the complexity of computing the linear bounds in Theorem 1 and 2 are of the order of $\mathcal{O}(L)$ where $L$ is the number of layers, if we assume a constant number of neurons per layer (otherwise it would be a bilinear dependency on the number of layers and layer dimensionality). From an experimental point of view, we have not run extensive experiments on the scalability on arbitrary networks, but in the table below we show the runtime of bounding the first and second derivatives using $\partial$-CROWN on the full global domain of each PINN as a function of the number of layers and neurons per layer:

|  | # layers | # neurons per layer | Total parameters | Runtime [s] |
| :--- | :---: | :---: | :---: | :---: |
| Burgers’ $\partial_{x} u_\theta$ | 8 | 20 | $1,740$ | $3.7\times 10^{-2}$ |
| Allen-Cahn’s $\partial_{x} u_\theta$ | 6 | 40 | $8,320$ | $3.6\times 10^{-2}$ |
| Diffusion-Sorption $\partial_{x} u_\theta$ | 7 | 40 | $9,960$ | $5.6\times 10^{-2}$ |
| Schrödinger’s $\partial_{x} u_\theta$ | 5 | 100 | $40,800$ | $1.7\times 10^{-1}$ |
| | | | |
| Burgers’ $\partial_{xx} u_\theta$ | 8 | 20 | $1,740$ | $6.6\times 10^{-2}$ |
| Allen-Cahn’s $\partial_{xx} u_\theta$ | 6 | 40 | $8,320$ | $6.2\times 10^{-2}$ |
| Diffusion-Sorption $\partial_{x} u_\theta$ | 7 | 40 | $9,960$ | $5.3\times 10^{-2}$ |
| Schrödinger’s $\partial_{xx} u_\theta$ | 5 | 100 | $40,800$ | $2.8\times 10^{-1}$ |

In practice to achieve the values of Table 1 we used $N_b = 2M$, i.e., we ran bound computations on 2 million subdomains of each PINN’s global domain. Interestingly, we observed that smaller domains induced lower bounding times than the more global ones, mostly due to the nonlinearity relaxations for tanh activations required in Theorem 1 and 2, which we detail in Appendix F. We will include this analysis in the appendix of the final version of the paper.

Reviewer comment (Theory):
First of all, I am not sure why the authors naturally assume that the ground truth can be represented by a FNN. For instance, even a single electron wave function decays exponentially, which an FNN cannot represent. Since the goal here is to bound the global error, I do not think the gap between the best FNN versus the actual ground truth can be ignored. Similarly, assumption 1 limits the applicability of your theorems - what happens if I add some exponential layers, or what if I use other architecture?

Author response:
**(W1) On the assumptions and applicability of $\partial$-CROWN.** Regarding the reviewer’s first point on the representation capacities of feedforward networks, we note that these networks are, in theory, universal approximators in the infinite width limit, so they should be able to represent arbitrary solution functions given an appropriate choice of activation. However, we agree with the reviewer in that it might not be an optimal architecture choice for some PINNs (which is likely the reason behind at least some training failures). Despite this, it is widely used in the previous literature, and hence why we chose to focus on these types of networks.

Note that some previous works providing guarantees for PINNs such as Shin et al., (2020), Wang et al., (2022b), or Mishra and Molinaro (2022) only provide them for specific families of PINNs, whereas our work is more general since it can be applied to all PINNs represented by FNNs.

Reviewer comment (Theory):
Your theorems only prove the existence of bounds, while your algorithm is still greedy. If I understand correctly, this actually means that: the approximation error (say the numbers in Table 1) is not *guaranteed* to be an upper bound of the global error, right? If so, despite the acceleration you achieve, how can you demonstrate that in a large system, your error approximation is *accurate*?

Author response:
**(W2) On the greedy nature of our method.** Our method provides a ***guaranteed upper bound*** on the global errors over a domain – so the upper bounds provided in Table 1 are guaranteed bounds. We introduce greedy input branching in Section 5.3. to improve the tightness of the bounds obtained using $\partial$-CROWN in the full PDE domain by splitting it into subdomains. Since the union of all the subdomains corresponds to the full PDE domain, taking the worst-case bound of each of the subdomains gives us a bound on the error in the full domain. As such, the bounds obtained via greedy input branching are **guaranteed** to be globally satisfied over the domain - there is no "approximation" that breaks the bound. More details on the procedure of greedy input branching is available in Appendix H.

Reviewer comment (Experiments):
I do think the paper should include more complex tasks, as PINN is known to be less accurate in large systems (or long simulation time intervals). Current experiments, while sufficient as a proof of concept, are insufficient to demonstrate the proposed algorithm can indeed provide a better approximation. For instance, you can try multi-body Schrodinger equations, like [1], or HJB equations [2]. These are of large scales and non-linear, making it challenging (IMO) to your method.

Author response:
**(W3) On further experiments.** The PINNs in the experiments section were chosen primarily because they are well established in previous literature in the field, from the classic Burgers’ and Schrödinger’s equation in Raissi et al. (2019a) to more recent and difficult ones to learn in the Allen-Cahn’s equation from Monaco and Apiletti (2023) and the Diffusion-Sorption from Takamoto et al. (2022). The HJB equation from [2] is modeled as an MLP, so our framework could be applied to this problem, but as the reviewer mentions the scale of the network would be challenging for $\partial$-CROWN as it is over $1,500\times$ bigger than the networks considered. This drawback of our method is explicitly mentioned in the Section 7 of the paper, and we hope future work will help scale verification methods to those larger networks.

Reviewer comment (Experiments):
From my understanding, speed is one of your strengths. Can you more comprehensively compare with previous methods to demonstrate how fast your method is? A table or figure will be ideal.

Author response:
**(Q1) On the efficiency of our method.** To the best of our knowledge, $\partial$-CROWN is the first framework designed to bound the errors of general PINNs. However, for completeness of analysis, in Appendix B we extend Interval Bound Propagation (IBP) (Gowal et al., 2018; Mirman et al., 2018) – known for its simplicity and trading-off bound tightness for speed – to this setting. The results are presented in Table 3 of our paper (Appendix B). It presents the performance of IBP and $\partial$-CROWN on the initial, boundary and residual errors for a fixed runtime limit in Burgers’ equation. This is a fair comparison which takes into account the runtime/tightness trade-off of the two methods. We observe that $\partial$-CROWN is significantly more efficient than IBP, achieving bounds that are $165 - 1,566\times$ tighter than the baseline in the same total runtime.

Reviewer comment (Theory):
I think the paper could be benefited by including more takeaways, especially for the theorems. I can sort of imagine how this theorem is proved, but it would be great if the authors could give a proof sketch and discuss how each assumption and condition works.

Author response:
**(Q2) On takeaways for Theorem 1 and 2.** While the expressions for the linear lower and upper bounds are quite efficient to compute in practice, we found it difficult to include them in the main text without excessive notation. However, we agree that the dependencies of the linear functions are not explicit in the theorem, and that a proof sketch should be added in the following paragraph. We will make these modifications in the final version of the paper.

Reviewer comment (Writing):
Prima facie the paper is very badly written – to the point of being very much devoid of motivations. Till one gets to the experiments, it's entirely unclear what is the “certification” question that is even being solved. In fact even after reading everything in the main paper, it's not clear at all as to why the random point sampling way is not enough to implement a check of the inequalities as required in Definition 1. What exactly is being gained by this method (as demonstrated in the third column of Table 1) as opposed to the results in the first two columns?

Author response:
**(W1) On the motivation of PINN certification and efficiency.** We stress in the introduction that the main goal of this paper is to provide a formal guarantee for *every* input within the feasible domain of the PINN. While most previous works report computed mean and/or max $\ell_2$ solution error computed over a finite number of points, this cannot be relied on in practice as they might not sample domain points where the error is large. We showcase this in the Diffusion-Sorption example in Table 1, where the residual bound estimate using $10^4$ MC samples is $1.1\times 10^{-3}$, while the certified upper bound is obtained at $21.34$. **Through sampling we are only able to obtain lower bound estimates on the error** which, as we can see from Table 1, can be quite distant from the actual error at specific domain points (visible on the same equation when we increase the number of samples from $10^4$ to $10^6$). This highlights the need for **certified upper bounds, which we provide with $\partial$-CROWN**.

As far as we are aware, our framework is the first one that can be used to certify arbitrary PINNs with solutions modeled by fully connected neural networks. And while runtime is still a drawback of our framework, in Appendix B we perform an efficiency comparison with an adaptation of Interval Bound Propagation (IBP) (Gowal et al., 2018; Mirman et al., 2018) – known for its simplicity and trading-off bound tightness for speed – to this setting. We compare the performance of IBP and $\partial$-CROWN on the initial, boundary and residual errors for a fixed runtime limit in Burgers’ equation. This is a fair comparison which takes into account the runtime/tightness trade-off of the two methods. We observe that $\partial$-CROWN is significantly more efficient than IBP, achieving bounds that are $165 - 1,566\times$ tighter than the baseline in the same total runtime.