Paper: Investigating the Ability of PINNs To Solve Burgers' PDE Near Finite-Time BlowUp (ICLR-2024)

Reviewer comment (Theory):
I believe that inequalities estimating numerical errors should show that the error bound converges to zero in some sense. If I understand the result correctly, the error bound in the first theorem does not converge to zero because $C_1$ and $C_2$ include the terms given by the norm of the solutions. So, the inequality (5) does not appear to make sense as an error analysis.

Author response:
Firstly, we would like to draw your attention to the newly added Appendix E in the draft where we have reviewed multiple classical results in numerical analysis for Burgers' PDE to establish that to the best of our knowledge even for this smallest non-trivial case of finite-time blow-up, classical error bounds do not exist which can be invoked on PINN experiments with Burgers' PDE.

Thus, our bounds and experiments in this work can be seen to be a first-of-its-kind result to address a challenging interface between deep-learning and non-linear PDEs. 

>Does the error bound (5) converge to zero in certain situations?

Thanks for raising this interesting question! As you would note we have specified this exact question as an open question in our conclusion in the language of "stability" - which we have briefly defined in the footnote on page 6.

For PDEs for which the ML method of solving it has a "stability" guarantee in this precise sense are the very cases where we can get a data-driven sufficient condition for the error of the approximation to be $0$. 

As we have pointed out in our discussion below Theorem 3.2, that this property of stability is not immediate for many of the ML losses used for PDE solving and one of our key observations (in Theorem 3.2) is that for Burger's PDE in one dimensions such a stable bound is possible despite allowing for the difficult solution of a finite-time blow-up.

Reviewer comment (Evaluation):
As for the results of the numerical experiments, although it is interesting that certain correlations between RHS and LHS of the inequalities are observed, the magnitudes of them are very different. So, I am not sure whether these results are meaningful or not.

Author response:
>Although it is interesting that certain correlations between RHS and LHS of the inequalities are observed, the magnitudes of them are very different. So, I am not sure whether these results are meaningful or not.

As we have emphasized in the literature review done at the beginning of Section $4$, getting non-vacuous generalization bounds for deep-learning is an extremely open problem. The objective of our research here is not to try to solve this question which is open for even much more basic setups with neural nets. 

The crux of our work - in particular our experiments - is to establish that the theoretical bounds we prove are not only applicable to this involved setup of nets solving PDEs but that they also evaluate to values that are highly correlated to the functional distance of the found surrogate from the true PDE solution. 

We posit that it is somewhat of a surprise that this correlation between the bound and the truth is strong (approaching nearly $1$ at times) despite the vacuity of the bound and despite the proximity to singularities - which is a unique feature of our PDE under consideration.

Reviewer comment (Theory):
The claim by the authors 
>Most importantly, Theorem 3.2 shows that despite the setting here being of proximity to finite-time
blow-up, the naturally motivated PINN risk in this case 3
is “(L2, L2, L2, L2)-stable”4 in the precise sense as defined in Wang et al. (2022a). This stability property being true implies that if the PINN
risk of the solution obtained is measured to be O(ϵ) then it would directly imply that the L2-risk
with respect to the true solution (10) is also O(ϵ). And this would be determinable without having
to know the true solution at test time.

is misleading. If the exact solution is unknown, neither is the $L^\infty$ value of its gradient at a given time, preventing the bounding of the error vs the true solution.

Author response:
>[The $(L_2,L_2,L_2,L_2)$-stability claim] is misleading

We would like to draw your attention to how stability has been defined in the footnote of page $6$. This definition is inspired from an earlier work, https://doi.org/10.48550/arXiv.2206.02016. As you would note that this definition is about an order estimate - that if the population risk of the PINN loss is ${\cal O}(\epsilon)$ then it should be a sufficient condition to ensure that the distance of the surrogate from the true PDE solution is also ${\cal O}(\epsilon)$. 

We note that in general for arbitrary PDE, it is not guaranteed (and may even be provably impossible) for the PINN loss to have this property of stability.

And this is the special phenomenon that happens in our Theorem $3.2$. 

Thus even when the true solution is unknown, a PINN loss could be proven to be stable if the bound on the true risk can be shown to depend on the true solution such that the dependencies can be absorbed into the constants kept implicit in the ${\cal O}$ estimate.

Reviewer comment (Novelty):
I do find the expression "generalization bound" for Theorem 3.1 somewhat misleading. These type of stability estimates (of the operator mapping right hand side and initial condition to the solution) are standard tools in the theory of partial differential equations, making this seem more like a rebranding. It would strengthen the paper if the authors would discuss related results in the PDE literature.

Author response:
>"generalization error" term for Theorem 3.1 is misleading. These type of stability estimates (of the operator mapping right hand side and initial condition to the solution) are standard tools in the theory of partial differential equations, making this seem more like a re-branding. It would strengthen the paper if the authors would discuss related results in the PDE literature.

We would like to emphasize that the key message of the theorems like what we have proven here is to able to estimate the proximity of a surrogate solution from the true PDE solution in terms of errors made by the solution in satisfying the PDE in the interior of the domain considered and the errors made in satisfying the boundary and the initial conditions. 

To the best of our knowledge, there are scant results in conventional PDE literature which give such bounds. Kindly see the newly added Appendix E where we have reviewed various classical literature to establish that to the best of our knowledge existing results in numerical analysis cannot be deployed to understand PINN training - as is the target here. In particular, specifically for the $0$ viscosity Burgers' PDE we can only find results like this, https://doi.org/10.1090/S0025-5718-1987-0906180-5, where the theory does not give a bound on the distance from the true solution of the PDE to that found by the finite element method. 

More generally, in works such as Corollary 3.5 in https://doi.org/10.1137/0728048 the authors consider a weak solution of the $\varepsilon$-viscosity regularized Burgers' PDE and derive bounds on the local $L^p$-distance between the weak solution and the actual solution. There is no obvious way to apply these bounds for a PINN solution since the trained net has no guarantee to be satisfying the conditions required of the surrogate here. 

For results like Theorem 2.1 in https://doi.org/10.1137/0729087, we observe that these too don't have an obvious way for the bounds to be applied for PINN experiments because they need stringent conditions (like satisfying the conservativeness property) to be true for the approximant, for the bounds to apply and there is no natural way to know if the neural surrogate satisfies these conditions. Also neither of the above cited classical bounds are tailored to any compact domain and hence there is no boundary condition error that is getting tracked there as in our Theorem $3.2$.


In contrast to such literature, even in the context of existing PINN guarantees, we posit that our theorems are a step towards novel and useful guarantees.

Reviewer comment (Novelty):
The literature review on operator learning approaches misses the works on both neural operators and BCR-NET (the latter predates both neural operators and DeepONet).

Author response:
>The literature review on operator learning approaches misses the works on both neural operators and BCR-NET

Thanks for the suggestion. In the 3rd paragraph of the Introduction we have added these references.

Reviewer comment (Novelty):
The referral to the works on the euler singularity of Wang should make more clear the differences between the two works. To my understanding, the work of Wang et al uses a rescaled coordinate system and therefore does not actually solve a PDE with singular solution. The blow-up studied by this community is also specific to incompressible problems as the blowup of the compressible Euler equation (of which the Burgers equation is the zero sound speed limit) arises from a different phenomenon.

Author response:
> Wang et al uses a re-scaled coordinate system and therefore does not actually solve a PDE with singular solution. The blow-up studied by this community is also specific to incompressible problems as the blowup of the compressible Euler equation (of which the Burgers equation is the zero sound speed limit) arises from a different phenomenon. This difference is not clear from the writing.

We would like to emphasize that we do not claim any technical connection or dependency on the results and observations made in https://arxiv.org/abs/2201.06780. We cite that paper to make the limited point that recently there has risen evidence that PINNs might be able to get reasonable answers even in the proximity of singularity formation happening.