Paper: Neural functional a posteriori error estimates (ICLR-2024)

Reviewer comment (Writing):
The introduction provides minimum information about astra -- only that it is a loss function with certain benefits like more robustness over residuals and variational losses. There should be a high-level description of (a) what motivates this specific loss function, and (b) what exactly is this loss function, along with reasoning behind it at a high level. Without this, it is difficult to grasp the concept and generate interest.

Author response:
There is a part of the paper that provides high-level details on the Astral loss. We tried to convey this information in 2.1, where the rationale behind using the functional error estimate as a loss function is given (certain favorable properties are listed).

Reviewer comment (Theory):
Section 2.3 is difficult to parse because the paper states several equations without sufficient explanatory depth. For example, in equation (9), the origin of the definition of astral is not explained. The paper says "It is possible to derive" but does not provide these derivations, even in the appendix.

Author response:
We agree that the presentation is sketchy here. The main reason why is the limit on the length of the paper. We provided references in the text sufficient for the reproduction of equations but we agree that it is better to make paper as self-sufficient as possible. Shortly we will provide a revised version of the manuscript with two additional appendices: the first one with the derivation of upper bound (9), and the second one with the explanation of how to arrive at (12) from (9).

Reviewer comment (Theory):
In equation 12, the derivation from equation 9 to equation 12 is unclear. There should also be sufficient reasoning for why equation 12 is a good measure of the predicted solution.

Author response:
We agree that this is an important question and thank the reviewer for bringing this up. Indeed, the manuscript does not explain properly why the energy norm (and its upper bound) is a good measure of accuracy. In some sense this material is standard, but we will review it properly. For details we refer to the answer https://openreview.net/forum?id=z62Xc88jgF&noteId=SQ0cDlqod2, where we address this question in some detail.

Reviewer comment (Theory):
Although this is a viable approach in a guaranteed way to produce reliable neural PDE solvers, the application could be limited, e.g., in the case for the problems where there is no theoretical posteriori estimates available.

Author response:
We want to thank the reviewer for taking part in the discussion of the manuscript.

We agree that the approach is limited to the PDEs where the functional upper bound can be derived, and this is the main downside. However, for many practically relevant equations, the functional error estimate exists. A non-exhaustive list of these equations is given at the end of 2.3 of the present manuscript.

Here, we would like to mention that error majorants are available for the Maxwell equation and a wide class of variational problems with convex functions.

Besides that, we provide additional examples of how to apply functional error estimate: $D=2$ elliptic equation in the L-shaped, $D=1+1$ convection-diffusion equation. More details are available in https://openreview.net/forum?id=z62Xc88jgF&noteId=Wy4TDErd0T.

Reviewer comment (Reproducibility):
Not sure why the full GitHub repository is not made available for review.

Author response:
As for the code, we decided not to share the GitHub repository because direct sharing is prohibited by rules. On the other hand, anonymization of the repository is tedious. The Google Colab notebooks that we share are anonymous and can be immediately used to reproduce the results since they run in the browser. We will make the code available after the review stage.

Reviewer comment (Theory):
It seems like this approach will have limited applicability. The first limitation is that the U function must be specified by the practitioner, but this is addressed in the paper, seems reasonable, and is a way to incorporate domain knowledge.

Author response:
> Constructing the the upper bound on the solution error is non-trivial and requires expert knowledge, but this provides a way of incorporating domain knowledge into the model.

Indeed, we agree that the construction of the upper bound is not straightforward and requires expert knowledge. We recognize this as a weakness of the approach and provide some discussion in these lines in https://openreview.net/forum?id=z62Xc88jgF&noteId=L8Rln3shGY. This discussion includes the list of equations where the upper bound is known and the additional example of the convection-diffusion equation that we introduce to showcase that not only elliptic equations can be treated with functional a posteriori error estimate (see https://openreview.net/forum?id=z62Xc88jgF&noteId=Wy4TDErd0T).

Reviewer comment (Theory):
My intuition is that the error certificate w_U can be thought of as an *explanation* of how the solution emerges. For example, the certificate could explain the predictions of a day-ahead weather forecast model by providing the evolution of the weather variables over the intermediate time steps. When a good certificate can be generated, the practitioner can feel confident that the solution is correct (with hard error bounds!), but otherwise the error bounds will be loose and the practitioner will not have much confidence in the solution. This all seems desirable, but I wonder if there exists a large set of problems for which the surrogate model can produce good solutions but not produce good certificates?

Author response:
As for the question. Indeed, the solution may be good be the certificate produced by the surrogate model is not good, i.e., they lead to an upper bound that is much larger than the energy norm. In our experiments, however, we did not observe this behavior when input data is in distribution. Out of distribution, this may happen, but the situation is hard to analyze. In this unfortunate case when certificates are useless, the practitioner has two choices:

1. Recompute the solution with a more reliable method.
2. Optimize the loss with a fixed solution to find better certificates.

So the certificates are useful only in the situation when the predicted upper bound is smaller than the desired tolerance. The situation in machine learning and statistics when the method provides a measure of certainty for the obtained result (e.g., vote counting, document sorting). When the measure is good enough, no intervention is needed. When the measure is bad, the sample is processed with a more reliable method (e.g., by a human being).

Reviewer comment (Writing):
Section 2.1 "deep learning is to"

Author response:
- “deep learning is to” --> “deep learning to”

Reviewer comment (Evaluation):
Experiment is a very important component in this paper, however, the evaluation metric of the solution is unusual compared to prior work. Specifically, let \( u \) be the output of neural networks and \( u^* \) be the exact solution. The test error is usually computed using the relative \( L^2 \) norm (see for example [1][2]), i.e.
\[
|| u - u^*||_2^2 / ||u^*||_2^2 = \int |u - u^*|^2 dx / \int |u^*|^2 dx.
\]
However, in Figure 4, the mean error is computed using equation (15), the energy norm.

Author response:
We would like to thank the reviewer for bringing up an important question. We agree that the relation between errors computed in energy and the $L_2$ norm is an important topic not covered in the article. There are two parts to consider: the theoretical one — how the losses are related, and the practical one — what are typical magnitudes of errors in each norm when networks are trained with Astral and residual losses.

The theoretical part is as follows. Suppose the error is $e$, the diffusion coefficient is $a$, and the reaction term is $b^2$. In this case, we can find

$$|||e|||^2 = \int_{\Gamma} dx \sum_{ij} \frac{\partial e}{\partial x_{i}}\frac{\partial e}{\partial x_{j}}a_{ij} + ||be||^2 \geq \lambda_{\min}^2 ||e||^2 + ||be^2|| \geq \lambda_{\min}^2 ||e||^2 + \inf_{x} b(x)^2||e^2||,$$

where $\lambda_{\min}$ is a minimal eigenvalue of the elliptic problem $-\frac{\partial}{\partial x_{i}} a_{ij} \frac{\partial}{\partial x_{j}}u(x) = \lambda u(x)$ defined on the same domain and with the same boundary conditions as the original elliptic problem. From the expression above, we obtain the bound

$$||e||\_{2}^{2} \\leq \\frac{1}{\\lambda_{\\min}^2 + \\inf_{x} \\left(b(x)\right)^2} |||e|||.$$

This bound can be simplified and made more practical (e.g., by removing minimal eigenvalue that may be hard to compute) by various means but the bound above is already sufficient to claim that when error is sufficiently small in the energy norm, this is also the case for the standard $L_2$ norm.

To answer the practical part, we conducted the set of additional experiments described in the reply https://openreview.net/forum?id=z62Xc88jgF&noteId=Wy4TDErd0T. There, we compared relative errors and energy norms for two additional test cases ($D=2$ elliptic equation in L-shaped domain, $D=1+1$ convection-diffusion equation) for Astral and residual losses.

We would like to also add that, indeed, solutions of PDE can be considered in various spaces. Most typically, however, not a space $L_2$ is used but Sobolev spaces to properly define weak solutions (see [E] Part II). Errors in natural Sobolev norms are sometimes used in deep learning literature. For example, a recent paper with FNO extension uses this norm [K]. Other metrics besides relative $L_2$ norms are also used, e.g., $L_1$ norm divided by $L_{\infty}$ [B], relative RMSE of energy spectrum (turbulence) [C], etc

The main reason we use energy norms is because it is a natural norm for elliptic problems. Namely, the energy norm will weigh the error according to the importance of spaces spanned by eigenvalues of the elliptic operator in question (eigenspaces with larger eigenvalues receive more weight). The energy norm is a de facto standard for elliptic PDEs. For example, FEM error estimate is done with error energy norm [SB, for example, 1.5.2 and 1.5.3]; the convergence theory for Conjugate Gradient method (Krylov subspace method used to solve a linear system with symmetric positive-definite matrices including the ones originated from elliptic equations) is naturally formulated with energy norm [S].

Finally, we would like to point out that we do not claim that the solution obtained with Astral loss is better. Our main claim is that Astral loss provides a natural way to construct an error majorant that provides a tight bound on the error. Note that the residual does not have this property. It is easy to see that the residual can be arbitrarily bad for the solution that is arbitrarily close to the exact one. Namely, we can consider a small high-frequency perturbation $\\delta(x) = \\epsilon\\sin(2\\pi L x)$ to the exact solution. This perturbation will introduce an error of the order $\epsilon$ and residual of order $\left(2 \\pi L\right)^2 \epsilon$. Now one can take $L$ large enough to have an arbitrarily large residual.

We hope that this discussion clarifies the relation between $L_2$ and energy norms and explains our choice of metric. We will incorporate this material in the corrected version of the paper that will appear shortly.

[E] — Evans LC. Partial differential equations. American Mathematical Society; 2022 Mar 22.

[K] — https://arxiv.org/abs/2310.00120v1

[B] — https://arxiv.org/abs/2202.03376

[C]— https://arxiv.org/abs/2112.01527

[SB] — Szabó B, Babuška I. Finite Element Analysis: Method, Verification and Validation.

[S] — Shewchuk JR. An introduction to the conjugate gradient method without the agonizing pain.

Reviewer comment (Evaluation):
Why not use the relative \( L^2 \) norm for evaluation? How does Astral loss perform if the evaluation is done in \( L^2 \)?

Author response:
We would like to thank the reviewer for bringing up an important question. We agree that the relation between errors computed in energy and the $L_2$ norm is an important topic not covered in the article. There are two parts to consider: the theoretical one — how the losses are related, and the practical one — what are typical magnitudes of errors in each norm when networks are trained with Astral and residual losses.

The theoretical part is as follows. Suppose the error is $e$, the diffusion coefficient is $a$, and the reaction term is $b^2$. In this case, we can find

$$|||e|||^2 = \int_{\Gamma} dx \sum_{ij} \frac{\partial e}{\partial x_{i}}\frac{\partial e}{\partial x_{j}}a_{ij} + ||be||^2 \geq \lambda_{\min}^2 ||e||^2 + ||be^2|| \geq \lambda_{\min}^2 ||e||^2 + \inf_{x} b(x)^2||e^2||,$$

where $\lambda_{\min}$ is a minimal eigenvalue of the elliptic problem $-\frac{\partial}{\partial x_{i}} a_{ij} \frac{\partial}{\partial x_{j}}u(x) = \lambda u(x)$ defined on the same domain and with the same boundary conditions as the original elliptic problem. From the expression above, we obtain the bound

$$||e||\_{2}^{2} \\leq \\frac{1}{\\lambda_{\\min}^2 + \\inf_{x} \\left(b(x)\right)^2} |||e|||.$$

This bound can be simplified and made more practical (e.g., by removing minimal eigenvalue that may be hard to compute) by various means but the bound above is already sufficient to claim that when error is sufficiently small in the energy norm, this is also the case for the standard $L_2$ norm.

To answer the practical part, we conducted the set of additional experiments described in the reply https://openreview.net/forum?id=z62Xc88jgF&noteId=Wy4TDErd0T. There, we compared relative errors and energy norms for two additional test cases ($D=2$ elliptic equation in L-shaped domain, $D=1+1$ convection-diffusion equation) for Astral and residual losses.

We would like to also add that, indeed, solutions of PDE can be considered in various spaces. Most typically, however, not a space $L_2$ is used but Sobolev spaces to properly define weak solutions (see [E] Part II). Errors in natural Sobolev norms are sometimes used in deep learning literature. For example, a recent paper with FNO extension uses this norm [K]. Other metrics besides relative $L_2$ norms are also used, e.g., $L_1$ norm divided by $L_{\infty}$ [B], relative RMSE of energy spectrum (turbulence) [C], etc

The main reason we use energy norms is because it is a natural norm for elliptic problems. Namely, the energy norm will weigh the error according to the importance of spaces spanned by eigenvalues of the elliptic operator in question (eigenspaces with larger eigenvalues receive more weight). The energy norm is a de facto standard for elliptic PDEs. For example, FEM error estimate is done with error energy norm [SB, for example, 1.5.2 and 1.5.3]; the convergence theory for Conjugate Gradient method (Krylov subspace method used to solve a linear system with symmetric positive-definite matrices including the ones originated from elliptic equations) is naturally formulated with energy norm [S].

Finally, we would like to point out that we do not claim that the solution obtained with Astral loss is better. Our main claim is that Astral loss provides a natural way to construct an error majorant that provides a tight bound on the error. Note that the residual does not have this property. It is easy to see that the residual can be arbitrarily bad for the solution that is arbitrarily close to the exact one. Namely, we can consider a small high-frequency perturbation $\\delta(x) = \\epsilon\\sin(2\\pi L x)$ to the exact solution. This perturbation will introduce an error of the order $\epsilon$ and residual of order $\left(2 \\pi L\right)^2 \epsilon$. Now one can take $L$ large enough to have an arbitrarily large residual.

We hope that this discussion clarifies the relation between $L_2$ and energy norms and explains our choice of metric. We will incorporate this material in the corrected version of the paper that will appear shortly.

[E] — Evans LC. Partial differential equations. American Mathematical Society; 2022 Mar 22.

[K] — https://arxiv.org/abs/2310.00120v1

[B] — https://arxiv.org/abs/2202.03376

[C]— https://arxiv.org/abs/2112.01527

[SB] — Szabó B, Babuška I. Finite Element Analysis: Method, Verification and Validation.

[S] — Shewchuk JR. An introduction to the conjugate gradient method without the agonizing pain.

Reviewer comment (Theory):
The a posteriori error bound is in the energy norm, i.e.
\[
L(u, w_L) \leq |||u-u^*||| \leq U(u, w_U).
\]
This suggests Astral loss would achieve fairly small error in this energy norm, but that does not necessarily imply the solution is "better". Equations can be solved in different spaces, and in fact the space \( L^2 \) is more commonly used when studying existence and uniqueness of PDE solutions.

Author response:
We would like to thank the reviewer for bringing up an important question. We agree that the relation between errors computed in energy and the $L_2$ norm is an important topic not covered in the article. There are two parts to consider: the theoretical one — how the losses are related, and the practical one — what are typical magnitudes of errors in each norm when networks are trained with Astral and residual losses.

The theoretical part is as follows. Suppose the error is $e$, the diffusion coefficient is $a$, and the reaction term is $b^2$. In this case, we can find

$$|||e|||^2 = \int_{\Gamma} dx \sum_{ij} \frac{\partial e}{\partial x_{i}}\frac{\partial e}{\partial x_{j}}a_{ij} + ||be||^2 \geq \lambda_{\min}^2 ||e||^2 + ||be^2|| \geq \lambda_{\min}^2 ||e||^2 + \inf_{x} b(x)^2||e^2||,$$

where $\lambda_{\min}$ is a minimal eigenvalue of the elliptic problem $-\frac{\partial}{\partial x_{i}} a_{ij} \frac{\partial}{\partial x_{j}}u(x) = \lambda u(x)$ defined on the same domain and with the same boundary conditions as the original elliptic problem. From the expression above, we obtain the bound

$$||e||\_{2}^{2} \\leq \\frac{1}{\\lambda_{\\min}^2 + \\inf_{x} \\left(b(x)\right)^2} |||e|||.$$

This bound can be simplified and made more practical (e.g., by removing minimal eigenvalue that may be hard to compute) by various means but the bound above is already sufficient to claim that when error is sufficiently small in the energy norm, this is also the case for the standard $L_2$ norm.

To answer the practical part, we conducted the set of additional experiments described in the reply https://openreview.net/forum?id=z62Xc88jgF&noteId=Wy4TDErd0T. There, we compared relative errors and energy norms for two additional test cases ($D=2$ elliptic equation in L-shaped domain, $D=1+1$ convection-diffusion equation) for Astral and residual losses.

We would like to also add that, indeed, solutions of PDE can be considered in various spaces. Most typically, however, not a space $L_2$ is used but Sobolev spaces to properly define weak solutions (see [E] Part II). Errors in natural Sobolev norms are sometimes used in deep learning literature. For example, a recent paper with FNO extension uses this norm [K]. Other metrics besides relative $L_2$ norms are also used, e.g., $L_1$ norm divided by $L_{\infty}$ [B], relative RMSE of energy spectrum (turbulence) [C], etc

The main reason we use energy norms is because it is a natural norm for elliptic problems. Namely, the energy norm will weigh the error according to the importance of spaces spanned by eigenvalues of the elliptic operator in question (eigenspaces with larger eigenvalues receive more weight). The energy norm is a de facto standard for elliptic PDEs. For example, FEM error estimate is done with error energy norm [SB, for example, 1.5.2 and 1.5.3]; the convergence theory for Conjugate Gradient method (Krylov subspace method used to solve a linear system with symmetric positive-definite matrices including the ones originated from elliptic equations) is naturally formulated with energy norm [S].

Finally, we would like to point out that we do not claim that the solution obtained with Astral loss is better. Our main claim is that Astral loss provides a natural way to construct an error majorant that provides a tight bound on the error. Note that the residual does not have this property. It is easy to see that the residual can be arbitrarily bad for the solution that is arbitrarily close to the exact one. Namely, we can consider a small high-frequency perturbation $\\delta(x) = \\epsilon\\sin(2\\pi L x)$ to the exact solution. This perturbation will introduce an error of the order $\epsilon$ and residual of order $\left(2 \\pi L\right)^2 \epsilon$. Now one can take $L$ large enough to have an arbitrarily large residual.

We hope that this discussion clarifies the relation between $L_2$ and energy norms and explains our choice of metric. We will incorporate this material in the corrected version of the paper that will appear shortly.

[E] — Evans LC. Partial differential equations. American Mathematical Society; 2022 Mar 22.

[K] — https://arxiv.org/abs/2310.00120v1

[B] — https://arxiv.org/abs/2202.03376

[C]— https://arxiv.org/abs/2112.01527

[SB] — Szabó B, Babuška I. Finite Element Analysis: Method, Verification and Validation.

[S] — Shewchuk JR. An introduction to the conjugate gradient method without the agonizing pain.

Reviewer comment (Evaluation):
There could be a relation between the energy norm and \( L^2 \) norm. More explanation is needed for the specific choice of the evaluation metric since it differs from the previous literature.

Author response:
We would like to thank the reviewer for bringing up an important question. We agree that the relation between errors computed in energy and the $L_2$ norm is an important topic not covered in the article. There are two parts to consider: the theoretical one — how the losses are related, and the practical one — what are typical magnitudes of errors in each norm when networks are trained with Astral and residual losses.

The theoretical part is as follows. Suppose the error is $e$, the diffusion coefficient is $a$, and the reaction term is $b^2$. In this case, we can find

$$|||e|||^2 = \int_{\Gamma} dx \sum_{ij} \frac{\partial e}{\partial x_{i}}\frac{\partial e}{\partial x_{j}}a_{ij} + ||be||^2 \geq \lambda_{\min}^2 ||e||^2 + ||be^2|| \geq \lambda_{\min}^2 ||e||^2 + \inf_{x} b(x)^2||e^2||,$$

where $\lambda_{\min}$ is a minimal eigenvalue of the elliptic problem $-\frac{\partial}{\partial x_{i}} a_{ij} \frac{\partial}{\partial x_{j}}u(x) = \lambda u(x)$ defined on the same domain and with the same boundary conditions as the original elliptic problem. From the expression above, we obtain the bound

$$||e||\_{2}^{2} \\leq \\frac{1}{\\lambda_{\\min}^2 + \\inf_{x} \\left(b(x)\right)^2} |||e|||.$$

This bound can be simplified and made more practical (e.g., by removing minimal eigenvalue that may be hard to compute) by various means but the bound above is already sufficient to claim that when error is sufficiently small in the energy norm, this is also the case for the standard $L_2$ norm.

To answer the practical part, we conducted the set of additional experiments described in the reply https://openreview.net/forum?id=z62Xc88jgF&noteId=Wy4TDErd0T. There, we compared relative errors and energy norms for two additional test cases ($D=2$ elliptic equation in L-shaped domain, $D=1+1$ convection-diffusion equation) for Astral and residual losses.

We would like to also add that, indeed, solutions of PDE can be considered in various spaces. Most typically, however, not a space $L_2$ is used but Sobolev spaces to properly define weak solutions (see [E] Part II). Errors in natural Sobolev norms are sometimes used in deep learning literature. For example, a recent paper with FNO extension uses this norm [K]. Other metrics besides relative $L_2$ norms are also used, e.g., $L_1$ norm divided by $L_{\infty}$ [B], relative RMSE of energy spectrum (turbulence) [C], etc

The main reason we use energy norms is because it is a natural norm for elliptic problems. Namely, the energy norm will weigh the error according to the importance of spaces spanned by eigenvalues of the elliptic operator in question (eigenspaces with larger eigenvalues receive more weight). The energy norm is a de facto standard for elliptic PDEs. For example, FEM error estimate is done with error energy norm [SB, for example, 1.5.2 and 1.5.3]; the convergence theory for Conjugate Gradient method (Krylov subspace method used to solve a linear system with symmetric positive-definite matrices including the ones originated from elliptic equations) is naturally formulated with energy norm [S].

Finally, we would like to point out that we do not claim that the solution obtained with Astral loss is better. Our main claim is that Astral loss provides a natural way to construct an error majorant that provides a tight bound on the error. Note that the residual does not have this property. It is easy to see that the residual can be arbitrarily bad for the solution that is arbitrarily close to the exact one. Namely, we can consider a small high-frequency perturbation $\\delta(x) = \\epsilon\\sin(2\\pi L x)$ to the exact solution. This perturbation will introduce an error of the order $\epsilon$ and residual of order $\left(2 \\pi L\right)^2 \epsilon$. Now one can take $L$ large enough to have an arbitrarily large residual.

We hope that this discussion clarifies the relation between $L_2$ and energy norms and explains our choice of metric. We will incorporate this material in the corrected version of the paper that will appear shortly.

[E] — Evans LC. Partial differential equations. American Mathematical Society; 2022 Mar 22.

[K] — https://arxiv.org/abs/2310.00120v1

[B] — https://arxiv.org/abs/2202.03376

[C]— https://arxiv.org/abs/2112.01527

[SB] — Szabó B, Babuška I. Finite Element Analysis: Method, Verification and Validation.

[S] — Shewchuk JR. An introduction to the conjugate gradient method without the agonizing pain.