Paper: BENO: Boundary-embedded Neural Operators for Elliptic PDEs (ICLR-2024)

Reviewer comment (Experiments):
It remains unclear to what extent the separation of the model into two terms is necessary without further empirical proof. Notably, both Branch 1 and Branch 2 share an identical architecture, differing only in their input, which comprises the forcing term and the boundary condition.

Author response:
Thanks for raising the question about different branches. Branch 1, with the boundary input set to zero, is posited to approximate the impact emanating from the interior, while Branch 2, nullifying the interior inputs, is conjectured to capture the boundary's influence on the interior. Specifically, we visualize the outputs of two branches which can fully validate our idea. The visualization is shown in the **Appendix** **M** in the updated manuscript, which indicates a discernible delineation of roles between the two branches.

Besides, you can also refer to **BENO w/o. D in Table 6 Sec. 4.4** of the updated manuscript to see the effectiveness of our dual branch architecture

Reviewer comment (Experiments):
As illustrated in Figure 1, it seems that the encoding of boundary conditions is achieved using a shared transformer, while the forcing term employs two distinct MPNNs. Is this particular choice of architecture a necessary one?

Author response:
Thanks for highlighting the importance of examining the architecture of Transformers. In response, we have conducted an ablation study where we employed the MPNN architecture to encode boundary information, as an alternative to the Transformer model. The results of this study are presented below. They demonstrate that the MPNN architecture is less effective than the Transformer in encoding global boundary information, underscoring the superiority of the latter in this context.

| Test set | 4-Corners L2 | 4-Corners MAE | 3-Corners L2 | 3-Corners MAE | 2-Corners L2 | 2-Corners MAE | 1-Corner L2 | 1-Corner MAE | No-Corner L2 | No-Corner MAE |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| BENNO w/o dx,dy | 0.5985±0.1111 | 1.8584±0.6242 | 0.6175±0.1105 | 2.0114±0.6173 | 0.6255±0.1122 | 2.0424±0.6686 | 0.6145±0.1172 | 1.8982±0.6453 | 0.6445±0.1103 | 2.3310±0.7341 |
| BENO (ours) | 0.3523±0.1245 | 0.9650±0.3131 | 0.4308±0.1994 | 1.2206±0.4978 | 0.4910±0.1888 | 1.4388±0.5227 | 0.5416±0.2133 | 1.4529±0.4626 | 0.5542±0.1952 | 1.7481±0.5394 |

Reviewer comment (Experiments):
Despite the acknowledged disadvantages of concatenating the boundary conditions alongside the other inputs, it is surprising to see the catastrophic failure of recent advances, such as MP-PDE, as illustrated, especially in Figure 5. The authors have mentioned that the learning rate and the weight decay factor are fixed for all experiments through grid search, but it remains unclear with which model these hyperparameters are searched for. Furthermore, aligning optimization hyperparameters across all architectures may benefit certain models but harm the performance of others, and this aspect should be considered. Therefore, more justification should be issued to prove that other baselines do not work at all to support more concretely the claim of a performance gain.

Author response:
Thanks for the comment. In pursuit of a more equitable comparison, we have adopted additional strategies to enhance the baseline's performance. Firstly, we augmented the original MP-PDE with a virtual node linked to all boundaries (which has demonstrated to improve the performance, as is also supported by our experiments). Subsequently, we conducted an optimal hyper-parameter search to fine-tune the hyper-parameters. The ensuing results are presented below. Despite these efforts, a notable performance disparity remains when contrasted with our BENO approach. The baseline, even with these enhancements, does not successfully resolve the issue at hand.

| Learning rate | Relative L2 | MAE |
| --- | --- | --- |
| 1e-5 | 0.9324±0.1521 | 2.9755±1.0400 |
| 5e-5 | 0.8684±0.1471 | 2.7438±1.0008 |
| 1e-4 | 0.8560±0.1610 | 2.6336±1.0063 |
| 5e-4 | 1.0036±0.0121 | 3.1454±0.8793 |

| MLP layers | Relative L2 | MAE |
| --- | --- | --- |
| 1 | 0.9007±0.1381 | 2.8799±0.9726 |
| 2 | 0.8915±0.1561 | 2.8696±1.0253 |
| 3 | 0.8684±0.1471 | 2.7438±1.0008 |
| 4 | 0.8879±0.1312 | 2.8139±0.9726 |

Reviewer comment (Experiments):
From the visualization, it appears that there are visible artifacts aligned with boundaries. These artifacts may be introduced by including $(dx_i, dy_i)$ in the node embedding. This raises questions about the necessity of incorporating $(dx_i, dy_i)$, as the Transform already appears to serve in learning a representation of the boundary shape.

Author response:
Thanks for the comment. We have added ablation study that removes the dx, dy to validate its effectiveness, which is shown as follows. It is obvious that this feature is helpful for our PDE solving.

| Test set | 4-Corners L2 | 4-Corners MAE | 3-Corners L2 | 3-Corners MAE | 2-Corners L2 | 2-Corners MAE | 1-Corner L2 | 1-Corner MAE | No-Corner L2 | No-Corner MAE |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| BENNO w/o dx,dy | 0.3654±0.1591 | 0.9989±0.3752 | 0.4535±0.2261 | 1.2912±0.5961 | 0.5549±0.2044 | 1.5689±0.5708 | 0.5797±0.2219 | 1.535±0.5039 | 0.5818±0.2219 | 1.7535±0.5547 |
| BENO (ours) | 0.3523±0.1245 | 0.9650±0.3131 | 0.4308±0.1994 | 1.2206±0.4978 | 0.4910±0.1888 | 1.4388±0.5227 | 0.5416±0.2133 | 1.4529±0.4626 | 0.5542±0.1952 | 1.7481±0.5394 |

Reviewer comment (Experiments):
What motivates the use of different implementations for encoding the boundary condition (with Transformer) and the forcing term (with MPNNs)? Even if the goal is to learn a global representation for the boundary condition, couldn't MPNN achieve the same objective?

Author response:
Thanks for highlighting the importance of examining the architecture of Transformers. In response, we have conducted an ablation study where we employed the MPNN architecture to encode boundary information, as an alternative to the Transformer model. The results of this study are presented below. They demonstrate that the MPNN architecture is less effective than the Transformer in encoding global boundary information, underscoring the superiority of the latter in this context.

| Test set | 4-Corners L2 | 4-Corners MAE | 3-Corners L2 | 3-Corners MAE | 2-Corners L2 | 2-Corners MAE | 1-Corner L2 | 1-Corner MAE | No-Corner L2 | No-Corner MAE |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| BENNO w/o dx,dy | 0.5985±0.1111 | 1.8584±0.6242 | 0.6175±0.1105 | 2.0114±0.6173 | 0.6255±0.1122 | 2.0424±0.6686 | 0.6145±0.1172 | 1.8982±0.6453 | 0.6445±0.1103 | 2.3310±0.7341 |
| BENO (ours) | 0.3523±0.1245 | 0.9650±0.3131 | 0.4308±0.1994 | 1.2206±0.4978 | 0.4910±0.1888 | 1.4388±0.5227 | 0.5416±0.2133 | 1.4529±0.4626 | 0.5542±0.1952 | 1.7481±0.5394 |

Reviewer comment (Writing):
How are the 'distances to boundary $(dx_i, dy_i)$' defined? It appears to be more like an offset with respect to the closest point on the boundary. If there are multiple closest points, how is the choice made?

Author response:
The calculation is based on the nearest point along the boundary. Given that the mesh points are arranged in a regular pattern, there is only one closest point for each calculation.

Reviewer comment (Experiments):
The paper only deal with Poisson equations with Dirichlet boundary conditions, which limits the effect on convincing readers that BENO is superior than baselines.

Author response:
A1:  Thanks for your comment. We have generated new datasets on different boundary condition (i.e., **Neumann BCs**) under our existing setting with different corner cases. Typically boundary conditions can be categorized into Dirichlet and Neumann BCs, or the combination of the two. We believe that the addition of Neumann BCs to the manuscript makes the paper to be applicable to a much wider range of systems. We represent the supplementary experiment results as follows: 

- Performances of our BENO and the compared baseline under **Neumann boundary condition** are shown below, which are **trained on 900 4-corners** samples and tested on 5 datasets under relative L2 norm and MAE separately. The whole comparison table with all the baselines is shown in **Appendix L**.

| Test set | 4-Corners L2 | 4-Corners MAE | 3-Corners L2 | 3-Corners MAE | 2-Corners L2 | 2-Corners MAE | 1-Corner L2 | 1-Corner MAE | No-Corner L2 | No-Corner MAE |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MP-PDE | 1.0070±0.0813 | 2.3595±0.6941 | 0.9895±0.0973 | 2.5480±0.8955 | 1.0134±0.1120 | 2.7345±0.8393 | 0.9782±0.1240 | 2.9679±0.7958 | 0.9670±0.1164 | 3.4807±1.1143 |
| BENO (ours) | 0.3568±0.0988 | 0.8311±0.2864 | 0.4201±0.1170 | 1.0814±0.3938 | 0.5020±0.1648 | 1.3918±0.5454 | 0.5074±0.1422 | 1.5676±0.4815 | 0.5221±0.1474 | 1.8649±0.5472 |
- Performances of our BENO and the compared baseline under **Neumann boundary condition** are shown below, which are **trained on 900 mixed** samples (180 samples each from 5 datasets) and tested on 5 datasets under relative L2 norm and MAE separately. The whole comparison table with all the baselines is shown in **Appendix L**.

| Test set | 4-Corners L2 | 4-Corners MAE | 3-Corners L2 | 3-Corners MAE | 2-Corners L2 | 2-Corners MAE | 1-Corner L2 | 1-Corner MAE | No-Corner L2 | No-Corner MAE |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MP-PDE | 1.0245±0.1048 | 2.3973±0.7015 | 0.9989±0.1277 | 2.5510±0.8717 | 1.0277±0.1399 | 2.7722±0.8091 | 0.9940±0.1543 | 2.9998±0.7781 | 0.9731±0.1414 | 3.4930±1.0867 |
| BENO (ours) | 0.4237±0.1237 | 1.0114±0.4165 | 0.3970±0.1277 | 1.0378±0.4221 | 0.3931±0.1347 | 1.0881±0.3993 | 0.3387±0.1279 | 1.0520±0.4253 | 0.3344±0.1171 | 1.2261±0.4467 |

Reviewer comment (Theory):
How about more general elliptic equations?

Author response:
A2: Thanks for the comment. We have generated new datasets on **Darcy flow** (mapping from coefficient to solution) under our existing settings. Performances of our BENO and the compared baseline on **Darcy flow** are shown below**,** which are **trained on 900 4-corner** samples (180 samples each from 5 datasets) and tested on 5 datasets under relative L2 norm and MAE separately. The whole comparison table with all the baselines is shown in **Appendix M**.

| Test set | 4-Corners L2 | 4-Corners MAE | 3-Corners L2 | 3-Corners MAE | 2-Corners L2 | 2-Corners MAE | 1-Corner L2 | 1-Corner MAE | No-Corner L2 | No-Corner MAE |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MP-PDE | 0.4802±0.1840 | 0.3269±0.2085 | 0.5332±0.1742 | 0.4652±0.2999 | 0.6197±0.1709 | 0.6307±0.3282 | 0.6906±0.1432 | 0.8469±0.4087 | 0.7406±0.1271 | 1.0906±0.3949 |
| BENO (ours) | 0.2431±0.0895 | 0.1664±0.0773 | 0.2542±0.1252 | 0.2150±0.1270 | 0.2672±0.1497 | 0.2585±0.1313 | 0.2466±0.1405 | 0.3091±0.2350 | 0.2366±0.1104 | 0.3591±0.2116 |

Reviewer comment (Novelty):
The paper's focus on elliptic PDEs seems somewhat restrictive, especially considering that it primarily deals with the simplest case of the Poisson equation with Dirichlet boundary conditions. This might give the impression that the novelty of the paper is somewhat limited.

Author response:
Despite the focus on elliptic PDEs, we believe this is an important and critical research that can be applicable to a wide range of natural phenomena and engineering applications.  

- **Importance of boundary conditions for any PDE systems.** It is known that any PDEs can be characterized as parabolic, hyperbolic, and elliptic PDEs, which further can be categorized as initial-boundary value problem (IBVP), and initial value problem (IVP) and boundary value problem (BVP), respectively. For any of these PDEs, the treatment of boundary conditions plays a critical role in accurately modeling natural phenomena and engineering applications. While we focus on the application of BENO on elliptic PDEs, we believe that the boundary condition treatment and incorporation into the overall solver is applicable to other PDE systems.
- **Applicability of elliptic PDEs to the real-world phenomena.** The utility of elliptic PDEs is not limited to steady problems, e.g., we can obtain unsteady solution of nonlinear PDEs by repeated solution of elliptic PDEs. For instance, the pressure-Poisson equation (which is an elliptic PDE) for incompressible flows can be applied to unsteady phenomena. Various plasma simulations uses Poisson solver that is time dependent. Helmholtz equation is used in a variety of phenomena, including electromagnetic radiation, seismology, and acoustics. These systems can have complex geometries, e.g., airfoil, propulsion devices, plasma sources for semiconductor manufacturing, with different types of materials (e.g., dielectric, conducting, catalytic, chemically reactive) and inhomogeneous boundary values (e.g., dependent on space and time). Thus, we believe that demonstration of BENO on elliptic PDEs has immense impact to a large number of applications. This introduction is updated to emphasize this point. Besides, we have revised the manuscript to delete the statements that elliptic PDEs are only applicable for steady-state phenomena.

Furthermore, to validate the generalization ability of our BENO, we **also supplement more experiments** on other scenarios in the following part.

Besides, to validate the generalization ability of our BENO, we **also supplement more experiments** on other scenarios in the following part.

Reviewer comment (Theory):
The proposed BENO method appears to be tailored to Dirichlet boundary conditions, which might be quite restrictive. It is unclear whether the method can be applied to Poisson equations with Neumann boundary conditions or mixed Robin boundary conditions.

Author response:
Thanks for your comment. We have generated new datasets on different boundary condition (i.e., **Neumann BCs**) under our existing setting with different corner cases. Typically boundary conditions can be categorized into Dirichlet and Neumann BCs, or the combination of the two. We believe that the addition of Neumann BCs to the manuscript makes the paper to be applicable to a much wider range of systems. We represent the supplementary experiment results as follows: 

- Performances of our BENO and the compared baseline under **Neumann boundary condition** are shown below, which are **trained on 900 4-corners** samples and tested on 5 datasets under relative L2 norm and MAE separately. The whole comparison table with all the baselines is shown in **Appendix L**.

| Test set | 4-Corners L2 | 4-Corners MAE | 3-Corners L2 | 3-Corners MAE | 2-Corners L2 | 2-Corners MAE | 1-Corner L2 | 1-Corner MAE | No-Corner L2 | No-Corner MAE |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MP-PDE | 1.0070±0.0813 | 2.3595±0.6941 | 0.9895±0.0973 | 2.5480±0.8955 | 1.0134±0.1120 | 2.7345±0.8393 | 0.9782±0.1240 | 2.9679±0.7958 | 0.9670±0.1164 | 3.4807±1.1143 |
| BENO (ours) | 0.3568±0.0988 | 0.8311±0.2864 | 0.4201±0.1170 | 1.0814±0.3938 | 0.5020±0.1648 | 1.3918±0.5454 | 0.5074±0.1422 | 1.5676±0.4815 | 0.5221±0.1474 | 1.8649±0.5472 |
- Performances of our BENO and the compared baseline under **Neumann boundary condition** are shown below, which are **trained on 900 mixed** samples (180 samples each from 5 datasets) and tested on 5 datasets under relative L2 norm and MAE separately. The whole comparison table with all the baselines is shown in **Appendix L**.

| Test set | 4-Corners L2 | 4-Corners MAE | 3-Corners L2 | 3-Corners MAE | 2-Corners L2 | 2-Corners MAE | 1-Corner L2 | 1-Corner MAE | No-Corner L2 | No-Corner MAE |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MP-PDE | 1.0245±0.1048 | 2.3973±0.7015 | 0.9989±0.1277 | 2.5510±0.8717 | 1.0277±0.1399 | 2.7722±0.8091 | 0.9940±0.1543 | 2.9998±0.7781 | 0.9731±0.1414 | 3.4930±1.0867 |
| BENO (ours) | 0.4237±0.1237 | 1.0114±0.4165 | 0.3970±0.1277 | 1.0378±0.4221 | 0.3931±0.1347 | 1.0881±0.3993 | 0.3387±0.1279 | 1.0520±0.4253 | 0.3344±0.1171 | 1.2261±0.4467 |

Reviewer comment (Experiments):
To better highlight the advantages of the BENO approach, it would be beneficial to demonstrate its applicability to a wider range of PDEs, including parametric PDEs such as convection-diffusion equations, nonlinear Burgers' equations, and time-dependent parabolic PDEs, thereby leading to a more generalized operator learning model.

Author response:
Thanks for the comment. We have generated new datasets on **Darcy flow** (mapping from coefficient to solution) under our existing settings with different corner casess. Performances of our BENO and the compared baseline on **Darcy flow** are shown below**,** which are **trained on 900 4-corner** samples (180 samples each from 5 datasets) and tested on 5 datasets under relative L2 norm and MAE separately. The whole comparison table with all the baselines is shown in **Appendix M**.

| Test set | 4-Corners L2 | 4-Corners MAE | 3-Corners L2 | 3-Corners MAE | 2-Corners L2 | 2-Corners MAE | 1-Corner L2 | 1-Corner MAE | No-Corner L2 | No-Corner MAE |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MP-PDE | 0.4802±0.1840 | 0.3269±0.2085 | 0.5332±0.1742 | 0.4652±0.2999 | 0.6197±0.1709 | 0.6307±0.3282 | 0.6906±0.1432 | 0.8469±0.4087 | 0.7406±0.1271 | 1.0906±0.3949 |
| BENO (ours) | 0.2431±0.0895 | 0.1664±0.0773 | 0.2542±0.1252 | 0.2150±0.1270 | 0.2672±0.1497 | 0.2585±0.1313 | 0.2466±0.1405 | 0.3091±0.2350 | 0.2366±0.1104 | 0.3591±0.2116 |

Reviewer comment (Experiments):
The paper experimentally investigates only a single type of operator, specifically the one between (f, g) and the solution u. It would be interesting to explore whether other types of solution operators could also be learned, such as in the FNO paper with Darcy flow, where the coefficient could be an input to the operator, or where the initial condition of a PDE can be an input.

Author response:
Thanks for the comment. We have generated new datasets on **Darcy flow** (mapping from coefficient to solution) under our existing settings with different corner casess. Performances of our BENO and the compared baseline on **Darcy flow** are shown below**,** which are **trained on 900 4-corner** samples (180 samples each from 5 datasets) and tested on 5 datasets under relative L2 norm and MAE separately. The whole comparison table with all the baselines is shown in **Appendix M**.

| Test set | 4-Corners L2 | 4-Corners MAE | 3-Corners L2 | 3-Corners MAE | 2-Corners L2 | 2-Corners MAE | 1-Corner L2 | 1-Corner MAE | No-Corner L2 | No-Corner MAE |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MP-PDE | 0.4802±0.1840 | 0.3269±0.2085 | 0.5332±0.1742 | 0.4652±0.2999 | 0.6197±0.1709 | 0.6307±0.3282 | 0.6906±0.1432 | 0.8469±0.4087 | 0.7406±0.1271 | 1.0906±0.3949 |
| BENO (ours) | 0.2431±0.0895 | 0.1664±0.0773 | 0.2542±0.1252 | 0.2150±0.1270 | 0.2672±0.1497 | 0.2585±0.1313 | 0.2466±0.1405 | 0.3091±0.2350 | 0.2366±0.1104 | 0.3591±0.2116 |

Reviewer comment (Experiments):
Experiments in complex domains appear to be limited to similar, complex domains, such as the "5 different corner elliptic dataset." It would be valuable to explore whether the BENO method can be extended to more diverse scenarios, such as domains with circular holes in the center of a square, corners at non-right angles, or a circle domain as in Lotzsch et al. (2022).

Author response:
Thanks for the positive feedback. Previous work i.e., GNN-PDE, utilizes methodology on circular holes but may have neglected the treatment of complex physical boundary conditions (including combination of Dirichlet and Neumann boundaries, which can be inhomogeneous), which we believe is the key innovativeness of our work. Generalization of the method (e.g., circular, spherical, unstructured meshes) is reserved for future work.

Reviewer comment (Experiments):
While FNO suggests using Fourier continuation for different boundary conditions and domains ('Non-uniform and Non-periodic Geometry' in [4]), it is unclear whether the paper’s chosen approach is the most optimal way to address complex scenarios.

Author response:
Thank you for your comment. In pursuit of a more equitable comparison, we have adopted additional strategies to enhance the baseline's performance. Firstly, we augmented the original MP-PDE with a virtual node linked to all boundaries. Subsequently, we conducted an optimal hyper-parameter search to fine-tune the hyper-parameters. The ensuing results are presented below. Despite these efforts, a notable performance disparity remains when contrasted with our BENO approach. The baseline, even with these enhancements, does not successfully resolve the issue at hand.

| Learning rate | Relative L2 | MAE |
| --- | --- | --- |
| 1e-5 | 0.9324±0.1521 | 2.9755±1.0400 |
| 5e-5 | 0.8684±0.1471 | 2.7438±1.0008 |
| 1e-4 | 0.8560±0.1610 | 2.6336±1.0063 |
| 5e-4 | 1.0036±0.0121 | 3.1454±0.8793 |

| MLP layers | Relative L2 | MAE |
| --- | --- | --- |
| 1 | 0.9007±0.1381 | 2.8799±0.9726 |
| 2 | 0.8915±0.1561 | 2.8696±1.0253 |
| 3 | 0.8684±0.1471 | 2.7438±1.0008 |
| 4 | 0.8879±0.1312 | 2.8139±0.9726 |

And it should be noted that the boundary conditions utilized in the related work are all homogeneous. We have also added the ****DeepONet**** as the baseline, and implement experiments under our main setting, which is trained on 900 4-corners samples and tested on 5 datasets. The results are shown as below and it is obvious that our BENO surpass the baseline by a large margin.

| Test set | 4-Corners L2 | 4-Corners MAE | 3-Corners L2 | 3-Corners MAE | 2-Corners L2 | 2-Corners MAE | 1-Corner L2 | 1-Corner MAE | No-Corner L2 | No-Corner MAE |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| DeepONet | 1.0002±0.1029 | 2.4815±0.3216 | 1.0002±0.1277 | 2.4768±0.3519 | 1.0005±0.1487 | 2.5216±0.3078 | 1.0117±0.1762 | 2.7018±0.4219 | 1.0075±0.1923 | 2.5529±0.3116 |
| BENO (ours) | 0.3523±0.1245 | 0.9650±0.3131 | 0.4308±0.1994 | 1.2206±0.4978 | 0.4910±0.1888 | 1.4388±0.5227 | 0.5416±0.2133 | 1.4529±0.4626 | 0.5542±0.1952 | 1.7481±0.5394 |

Reviewer comment (Experiments):
There exists another model for operator learning, DeepONet [5], which can be applied directly to challenging boundary conditions and complex domains. It is unclear whether the paper conducted a comparison with this model.

Author response:
Thank you for your comment. In pursuit of a more equitable comparison, we have adopted additional strategies to enhance the baseline's performance. Firstly, we augmented the original MP-PDE with a virtual node linked to all boundaries. Subsequently, we conducted an optimal hyper-parameter search to fine-tune the hyper-parameters. The ensuing results are presented below. Despite these efforts, a notable performance disparity remains when contrasted with our BENO approach. The baseline, even with these enhancements, does not successfully resolve the issue at hand.

| Learning rate | Relative L2 | MAE |
| --- | --- | --- |
| 1e-5 | 0.9324±0.1521 | 2.9755±1.0400 |
| 5e-5 | 0.8684±0.1471 | 2.7438±1.0008 |
| 1e-4 | 0.8560±0.1610 | 2.6336±1.0063 |
| 5e-4 | 1.0036±0.0121 | 3.1454±0.8793 |

| MLP layers | Relative L2 | MAE |
| --- | --- | --- |
| 1 | 0.9007±0.1381 | 2.8799±0.9726 |
| 2 | 0.8915±0.1561 | 2.8696±1.0253 |
| 3 | 0.8684±0.1471 | 2.7438±1.0008 |
| 4 | 0.8879±0.1312 | 2.8139±0.9726 |

And it should be noted that the boundary conditions utilized in the related work are all homogeneous. We have also added the ****DeepONet**** as the baseline, and implement experiments under our main setting, which is trained on 900 4-corners samples and tested on 5 datasets. The results are shown as below and it is obvious that our BENO surpass the baseline by a large margin.

| Test set | 4-Corners L2 | 4-Corners MAE | 3-Corners L2 | 3-Corners MAE | 2-Corners L2 | 2-Corners MAE | 1-Corner L2 | 1-Corner MAE | No-Corner L2 | No-Corner MAE |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| DeepONet | 1.0002±0.1029 | 2.4815±0.3216 | 1.0002±0.1277 | 2.4768±0.3519 | 1.0005±0.1487 | 2.5216±0.3078 | 1.0117±0.1762 | 2.7018±0.4219 | 1.0075±0.1923 | 2.5529±0.3116 |
| BENO (ours) | 0.3523±0.1245 | 0.9650±0.3131 | 0.4308±0.1994 | 1.2206±0.4978 | 0.4910±0.1888 | 1.4388±0.5227 | 0.5416±0.2133 | 1.4529±0.4626 | 0.5542±0.1952 | 1.7481±0.5394 |

Reviewer comment (Experiments):
The grid sizes for the domain are quite small (32x32, 64x64) when compared with FNO. I would like to see if the approach generalizes to larger and more practical grid sizes as well. This is especially important given the sequence length constraints that come with using transformers.

Author response:
A1: Thanks for the comment. We have conducted additional super-resolution experiment, where the models are trained on 64x64 resolution, and are tested on irregular domains at a high **128x128 resolution**, pushing the boundaries of our BENO's applicability. The results are shown as below. Specifically, BENO consistently shows superior performance by a large margin.

| Test set | 4-Corners L2 | 4-Corners MAE | 3-Corners L2 | 3-Corners MAE | 2-Corners L2 | 2-Corners MAE | 1-Corner L2 | 1-Corner MAE | No-Corner L2 | No-Corner MAE |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MP-PDE | 1.0543±0.2839 | 0.0997±0.0426 | 1.0475±0.2629 | 0.0982±0.0394 | 1.0206±0.1826 | 0.1291±0.0420 | 1.0144±0.1416 | 0.1645±0.0543 | 1.0063±0.1008 | 0.2257±0.0587 |
| BENO (ours) | 0.5815±0.1276 | 0.0372±0.0236 | 0.6599±0.2923 | 0.0369±0.0197 | 0.6796±0.2039 | 0.0625±0.0280 | 0.6852±0.1869 | 0.0954±0.0452 | 0.7429±0.1594 | 0.1600±0.0508 |

We see that our model still outperforms the MP-PDE baseline by a large margin, at this high resolution.

Reviewer comment (Evaluation):
It would also be useful to get some additional information about training and inference times for BENO and baselines.

Author response:
A2: Thank you for your comment. The average time taken for BENO and the graph-based baseline methods during a single epoch, computed over five iterations, is listed below. We can see that the modest increase in computational time of BENO is justified by the substantial performance improvements offered by the integration of the Transformer and dual-branch GNN structures. The added complexity of these components contributes to the enhanced capability of the BENO model, making it a feasible and efficient choice in scenarios where advanced performance is crucial.

| Method | GNN-PDE | MP-PDE | BENO (ours) |
| --- | --- | --- | --- |
| Time | 0.0054 | 0.0092 | 0.0210 |

Reviewer comment (Experiments):
Lack of comparison with more SOTA works like multiwavelet neural operator.

Author response:
A1: Thanks for your insightful feedback. We have added the ****Multiwavelet-based Operator**** as the baseline, and implement experiments under our main setting, which is trained on 900 4-corners samples and tested on 5 datasets. The results are shown as below and it is obvious that our BENO surpass the baseline by a large margin.

| Test set | 4-Corners L2 | 4-Corners MAE | 3-Corners L2 | 3-Corners MAE | 2-Corners L2 | 2-Corners MAE | 1-Corner L2 | 1-Corner MAE | No-Corner L2 | No-Corner MAE |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MWTNO | 1.0264±0.2678 | 2.5639±0.3871 | 1.0465±0.2741 | 2.7752±0.3216 | 1.0433±0.3301 | 2.7399±0.3186 | 1.0428±0.2973 | 2.7350±0.3119 | 1.0397±0.2711 | 2.7015±0.3228 |
| BENO (ours) | 0.3523±0.1245 | 0.9650±0.3131 | 0.4308±0.1994 | 1.2206±0.4978 | 0.4910±0.1888 | 1.4388±0.5227 | 0.5416±0.2133 | 1.4529±0.4626 | 0.5542±0.1952 | 1.7481±0.5394 |

Reviewer comment (Novelty):
Focusing on solving elliptic PDEs with complex geometries and inhomogeneous boundary values may limit the applications.

Author response:
A2:  Despite the focus on elliptic PDEs, we believe this is an important and critical research that can be applicable to a wide range of natural phenomena and engineering applications.  

- **Importance of boundary conditions for any PDE systems.** It is known that any PDEs can be characterized as parabolic, hyperbolic, and elliptic PDEs, which further can be categorized as initial-boundary value problem (IBVP), and initial value problem (IVP) and boundary value problem (BVP), respectively. For any of these PDEs, the treatment of boundary conditions plays a critical role in accurately modeling natural phenomena and engineering applications. While we focus on the application of BENO on elliptic PDEs, we believe that the boundary condition treatment and incorporation into the overall solver is applicable to other PDE systems.
- **Applicability of elliptic PDEs to the real-world phenomena.** The utility of elliptic PDEs is not limited to steady problems, e.g., we can obtain unsteady solution of nonlinear PDEs by repeated solution of elliptic PDEs. For instance, the pressure-Poisson equation (which is an elliptic PDE) for incompressible flows can be applied to unsteady phenomena. Various plasma simulations uses Poisson solver that is time dependent. Helmholtz equation is used in a variety of phenomena, including electromagnetic radiation, seismology, and acoustics. These systems can have complex geometries, e.g., airfoil, propulsion devices, plasma sources for semiconductor manufacturing, with different types of materials (e.g., dielectric, conducting, catalytic, chemically reactive) and inhomogeneous boundary values (e.g., dependent on space and time). Thus, we believe that demonstration of BENO on elliptic PDEs has immense impact to a large number of applications. This introduction is updated to emphasize this point. Besides, we have revised the manuscript to delete the statements that elliptic PDEs are only applicable for steady-state phenomena.

Furthermore, to validate the generalization ability of our BENO, we **also supplement more experiments** on other scenarios in the following part.

Reviewer comment (Experiments):
How can the dataset used to evaluate the BENO model be expanded to include more diverse scenarios and improve the generalization performance of the model?

Author response:
A3: Thanks for your comment. We have generated new datasets on different boundary condition (i.e., **Neumann BCs**) type and different types of solution operators (i.e., **Darcy flow**) under our existing setting with different corner cases. Typically boundary conditions can be categorized into Dirichlet and Neumann BCs, or the combination of the two. We believe that the addition of Neumann BCs to the manuscript makes the paper to be applicable to a much wider range of systems. We represent the supplementary experiment results as follows: 

- Performances of our BENO and the compared baseline under **Neumann boundary condition** are shown below, which are **trained on 900 4-corners** samples and tested on 5 datasets under relative L2 norm and MAE separately. The whole comparison table with all the baselines is shown in **Appendix L**. We see that our method significantly outperforms the MP-PDE baseline in this setting.

| Test set | 4-Corners L2 | 4-Corners MAE | 3-Corners L2 | 3-Corners MAE | 2-Corners L2 | 2-Corners MAE | 1-Corner L2 | 1-Corner MAE | No-Corner L2 | No-Corner MAE |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MP-PDE | 1.0070±0.0813 | 2.3595±0.6941 | 0.9895±0.0973 | 2.5480±0.8955 | 1.0134±0.1120 | 2.7345±0.8393 | 0.9782±0.1240 | 2.9679±0.7958 | 0.9670±0.1164 | 3.4807±1.1143 |
| BENO (ours) | 0.3568±0.0988 | 0.8311±0.2864 | 0.4201±0.1170 | 1.0814±0.3938 | 0.5020±0.1648 | 1.3918±0.5454 | 0.5074±0.1422 | 1.5676±0.4815 | 0.5221±0.1474 | 1.8649±0.5472 |
- Performances of our BENO and the compared baseline under **Neumann boundary condition** are shown below, which are **trained on 900 mixed** samples (180 samples each from 5 datasets) and tested on 5 datasets under relative L2 norm and MAE separately. The whole comparison table with all the baselines is shown in **Appendix L**. Again, we see that our method significantly outperforms the MP-PDE baseline in this setting

| Test set | 4-Corners L2 | 4-Corners MAE | 3-Corners L2 | 3-Corners MAE | 2-Corners L2 | 2-Corners MAE | 1-Corner L2 | 1-Corner MAE | No-Corner L2 | No-Corner MAE |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MP-PDE | 1.0245±0.1048 | 2.3973±0.7015 | 0.9989±0.1277 | 2.5510±0.8717 | 1.0277±0.1399 | 2.7722±0.8091 | 0.9940±0.1543 | 2.9998±0.7781 | 0.9731±0.1414 | 3.4930±1.0867 |
| BENO (ours) | 0.4237±0.1237 | 1.0114±0.4165 | 0.3970±0.1277 | 1.0378±0.4221 | 0.3931±0.1347 | 1.0881±0.3993 | 0.3387±0.1279 | 1.0520±0.4253 | 0.3344±0.1171 | 1.2261±0.4467 |
- Performances of our BENO and the compared baseline on **Darcy flow** are shown below**,** which are **trained on 900 4-corner** samples and tested on 5 datasets under relative L2 norm and MAE separately. The whole comparison table with all the baselines is shown in **Appendix M**.

| Test set | 4-Corners L2 | 4-Corners MAE | 3-Corners L2 | 3-Corners MAE | 2-Corners L2 | 2-Corners MAE | 1-Corner L2 | 1-Corner MAE | No-Corner L2 | No-Corner MAE |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MP-PDE | 0.4802±0.1840 | 0.3269±0.2085 | 0.5332±0.1742 | 0.4652±0.2999 | 0.6197±0.1709 | 0.6307±0.3282 | 0.6906±0.1432 | 0.8469±0.4087 | 0.7406±0.1271 | 1.0906±0.3949 |
| BENO (ours) | 0.2431±0.0895 | 0.1664±0.0773 | 0.2542±0.1252 | 0.2150±0.1270 | 0.2672±0.1497 | 0.2585±0.1313 | 0.2466±0.1405 | 0.3091±0.2350 | 0.2366±0.1104 | 0.3591±0.2116 |

Reviewer comment (Reproducibility):
Would the authors be able to provide the number of parameters to quantify each model's complexity?

Author response:
A4: We would like to present the parameter count for our BENO model. Specifically, the Transformer component, dedicated to encoding the boundary, comprises 0.8M parameters, while each branch of the GNN-based network encompasses 1.14M parameters. This indicates that the complexity of our approach is entirely manageable and within an acceptable range.

| Method | GKN | FNO | GNN-PDE | MP-PDE | BENO(ours) |
| --- | --- | --- | --- | --- | --- | --- |
| Parameters | 1.09M | 0.44M | 0.84M | 0.63M | 3.08M |

Reviewer comment (Theory):
How can the BENO model be adapted to handle time-dependent PDEs or other types of dynamic systems?

Author response:
A5: As we responded in A2, even some elliptic PDEs are designed to handle time-dependent solutions and the importance of boundary condition treatment remains same for other dynamic systems. We have modified our introduction to highlight this point.

Reviewer comment (Evaluation):
Can the BENO model be used to generate physically meaningful insights and interpretability?

Author response:
A6: The innovativeness of BENO is in accounting for the boundary condition treatments in the PDE solutions obtained from boundary-embedded GNN. Handling of the boundary conditions is a requirement for any PDE equations. Elliptic PDEs are good test problems to test such boundary condition effects (cf. boundary value problem). In addition, without accurately capturing the boundary conditions, the solutions for the PDEs obtained from a solver are not physically meaningful and cannot be applied to real systems. Hence, we believe that BENO’s strengths are in producing an acceptable tolerance with much faster computational speed, which generate physically meaningful results and enhance the interpretability of the solutions.

Reviewer comment (Novelty):
There are some latest related operator works that should be referred to for a boarder audience: [Xiao, Xiongye, et al. "Coupled Multiwavelet Neural Operator Learning for Coupled Partial Differential Equations."] [Gupta, Gaurav, et al. "Multiwavelet-based operator learning for differential equations."]; [Gupta, Gaurav, et al. "Non-linear operator approximations for initial value problems."].

Author response:
A7: We have added [Xiao, Xiongye, et al. "Coupled Multiwavelet Neural Operator Learning for Coupled Partial Differential Equations."], [Gupta, Gaurav, et al. "Multiwavelet-based operator learning for differential equations."], and [Gupta, Gaurav, et al. "Non-linear operator approximations for initial value problems."] in the revised version of **Related Work** for a boarder audience.