Physics-Embedded Neural Networks: Graph Neural PDE Solvers with Mixed Boundary Conditions
Abstract
Graph neural network (GNN) is a promising approach to learning and predicting physical phenomena described in boundary value problems, such as partial differential equations (PDEs) with boundary conditions.
However, existing models inadequately treat boundary conditions essential for the reliable prediction of such problems.
In addition, because of the locally connected nature of GNNs, it is difﬁcult to accurately predict the state after a long time, where interaction between vertices tends to be global.
We present our approach termed physics-embedded neural networks that considers boundary conditions and predicts the state after a long time using an implicit method.
It is built based on an E( n ) -equivariant
GNN, resulting in high generalization performance on various shapes.
We demonstrate that our model learns ﬂow phenomena in complex shapes and outperforms a well-optimized classical solver and a state-of-the-art machine learning model in speed-accuracy trade-off.
Therefore, our model can be a useful standard for realizing reliable, fast, and accurate GNN-based PDE solvers.
The code is available at https://github.com/yellowshippo/penn-neurips2022 .
Introduction
Partial differential equations (PDEs) are of interest to many scientists because of their application in various ﬁelds such as mathematics, physics, and engineering.
Numerical analysis is used to solve
PDEs because most PDE problems in real life cannot be solved analytically.
For example, predicting ﬂuid behavior in complex shapes is an essential topic because it is helpful for product design, disaster reduction, weather forecasting and many others; however, it is a difﬁcult problem and takes time to solve using classical solvers.
Machine learning is a promising approach to predicting such phenomena because it can utilize data similar to the state to be predicted, while classical solvers cannot.
However, the main challenge in dealing with complex phenomena such as ﬂuids is to guarantee generalization performance because possible states in complex systems can be huge and may not be covered using a purely data-driven approach.
Therefore, we must apply appropriate inductive biases to machine learning models.
Many approaches successfully introduced various inductive biases such as local connectedness using graph neural networks (GNNs) and symmetry under coordinate transformations using equivariance.
While these methods have made great progress in solving PDEs using machine learning, there is still room for improvement.
First, there is need for an efﬁcient and provable way to respect boundary conditions like Dirichlet and Neumann, i.e., mixed boundary conditions.
Rigorous fulﬁllment of Dirichlet boundary conditions is indispensable because they are hard constraints and different
Dirichlet conditions correspond to different problems users would like to solve.
Second, there is need to reinforce the treatment of global interaction to predict the state after a long time, where interactions
[Figure] tend to be global. GNNs have excellent generalization properties because of their locally-connected nature; however, they may miss global interaction due to their localness.
We propose physics-embedded neural networks (PENNs), a machine learning framework to address these issues by embedding physics in the models.
We build our model based on IsoGCN (Horie et al., 2021), a lightweight E( n ) -equivariant GNN to reﬂect physical symmetry and realize fast prediction.
Furthermore, we construct a method to consider mixed boundary conditions.
Finally, we reconsider a way to stack GNNs based on a nonlinear solver, which naturally introduces the global pooling to GNNs as the global interaction with high interpretability.
In experiments, we demonstrate that our treatment of Neumann boundary conditions improves the predictive performance of the model, and our method can fulﬁll Dirichlet boundary conditions with no error.
Our method also achieves state-of-the-art performance compared to a classical, well-optimized numerical solver and a baseline machine learning model in speed-accuracy trade-off.
Figure 1 shows the overview of the proposed model.
Our main contributions are summarized as follows:
• We construct models to satisfy mixed boundary conditions: the boundary encoder , Dirichlet layer , pseudoinverse decoder and NeumannIsoGCN (NIsoGCN).
The considered models show provable fulﬁllment of boundary conditions, while existing models cannot.
• We propose neural nonlinear solvers , which realize global connections to stably predict the state after a long time.
• We demonstrate that the proposed model shows state-of-the-art performance in speedaccuracy trade-off, and all the proposed components are compatible with E( n ) -equivariance.
Background and related work
In this section, we review the foundation of PDEs to clarify the problems we solve and introduce related works where machine learning models are used to solve PDEs.
2.1 Partial differential equations (PDEs) with boundary conditions
A general form of the d -dimensional temporal PDEs that we consider can be expressed as follows:
[Equation]
where Ω is the domain, ∂ Ω is the boundary of Ω , and ∂ Ω Dirichlet and ∂ Ω Neumann are boundaries with Dirichlet and Neumann (mixed) boundary conditions.
ˆ · is a known function, and D is a known nonlinear differential operator, which can be nonlinear and contain spatial differential operators (see equation 18 for an example of D ).
n ( x ) is the normal vector at x ∈ ∂ Ω .
Equation 3 is called the Dirichlet boundary condition, where the value on ∂ Ω Dirichlet is set as a constraint.
Equationcorresponds to the Neumann boundary condition, where the value of the derivative u in the direction of n is set on ∂ Ω Neumann rather than the value of u .
When u : (0 , T ) × Ω → R f satisﬁes
Equations 1 – 4, it is called the solution of the (initial-) boundary value problem.
2.1.1 Discretization
PDEs are deﬁned in a continuous space to make differentials meaningful.
Discretization can be applied in space and time so that computers can solve PDEs easily.
In numerical analysis of complexshaped domains, we commonly use meshes (discretized data of shapes), which can be regarded as graphs.
We denote the position of the i th vertex as x i and the value of a function f at the x i as f i .
The simplest method to discretize time is the explicit Euler method formulated as:
[Equation]
) which updates u ( t, x i ) with a small increment D ( u )( t, x i )∆ t .
Another way to have time discretization is the implicit Euler method formulated as:
[Equation]
which solves Equation 6 rather than simply updating variables to ensure the original PDE is satisﬁed numerically.
The equation can be viewed as a nonlinear optimization problem by formulating it as:
[Equation]
where R ( v ) is the residual vector of the discretized PDE.
The solution of Equation 8 corresponds to u ( t +∆ t, x ) .
By letting ∇ φ = R for an appropriate φ , solving Equation 8 corresponds to optimizing φ in an ( f × n ) -dimensional space, where n is the number of vertices in the considered mesh.
A simple way to solve such an optimization problem is to apply gradient descent formulated as:
[Equation]
where α ( i ) ∈
R is determined using line search.
However, due to the high computational cost of the search, α can be ﬁxed to a small value, which corresponds to the explicit Euler method with the time step size α ∆ t .
Barzilai & Borwein (1988) suggested another simple yet effective way to determine the step size using a two-point approximation to the secant equation underlying quasi-Newton methods.
2.2 Neural PDE solvers
We review machine learning models used to solve PDEs called neural PDE solvers, typically formulated as u ( t n +1 , x i )
≈ F NN ( u )( t n , x i ) for ( t n , x i ) ∈ { t 0 , t 1 , . . .
× Ω , where F NN is a machine learning model.
2.2.1 Physics-informed neural networks (PINNs)
Raissi et al.
(2019) made a pioneering work combining PDE information and neural networks, called
PINNs, by adding loss to monitor how much the output satisﬁes the equations.
PINNs can be used to solve forward and inverse problems and extract physical states from measurements (Pang et al., 2019; Mao et al., 2020;
Cai et al., 2021).
However, PINNs’ outputs should be functions of space because PINNs rely on automatic differentiation to obtain loss regarding PDEs.
This design constraint signiﬁcantly limits the model’s generalization ability because the solution of a PDE could be entirely different when the shape of the domain or boundary condition changes.
Besides, the loss reﬂecting
PDEs helps models learn physics at training time; however, prediction by PINN models can be out of physics because of lacking PDE information inside the model.
Therefore, these methods are not applicable in building models that are generalizable over shape and boundary condition variations.
As seen in Section 3, our model contains PDE information inside and does not take absolute positions of vertices, thus resulting in high generalizability (See Figure 3).
2.2.2 Graph neural network based PDE solvers
As discussed in Section 2.1.1, one can regard a mesh as a graph.
GNNs can take any graphs as inputs (Gori et al., 2005;
Scarselli et al., 2008; Kipf & Welling, 2017; Gilmer et al., 2017), having the possibility to generalize over various graphs, i.e., meshes.
Therefore, GNNs are strong candidates for learning mesh-structured numerical analysis data, as seen in Alet et al.
; Chang & Cheng (2020); Pfaff et al.
Brandstetter et al.
(2022) advanced these works for efﬁcient and stable prediction.
Their method could also consider boundary conditions by feeding them to the models as inputs.
Here, one could expect the model to learn to satisfy boundary conditions approximately, while there is no guarantee to fulﬁll hard constraints such as Dirichlet conditions.
In contrast, our model ensures the satisfaction of boundary conditions.
Besides, most GNNs use local connections with a ﬁxed number of message passings, which lacks consideration of global interaction.
We suggest an effective way to incorporate a global connection with GNN through the neural nonlinear solver.
2.2.3 Equivariant models
In addition to GNNs, another essential concept to help machine learning models generalize is equivariance.
Equivariance is characterized by using group action as f ( g · x ) =
g · f ( x ) for f : X → Y and g ∈ G acting on X and Y .
In particular, E( n ) -equivariance is essential to predict the solutions of physical PDEs because it describes rigid body motion, i.e., translation, rotation, and reﬂection.
Ling et al.
(2016) and Wang et al.
introduced equivariance to a simple neural network and CNN to predict ﬂow phenomena.
Both works showed that equivariance improved predictive and generalization performance compared to models without equivariance.
Horie et al. (2021) proposed E( n ) -equivariant GNNs based on GCNs (Kipf & Welling, 2017), called IsoGCNs.
A form of their model is formulated as:
[Equation]
where N i is the neighborhood of the i th vertex, ⊗ is the tensor product operator, and W is a trainable matrix acting on feature index.
Here, we denote IsoGCN
→ 1 an IsoGCN layer that converts the input scalar (rank-0 tensor) ﬁeld ψ to the output vector (rank-1 tensor) ﬁeld.
This layer corresponds to the gradient operator, which helps learn PDEs because spatial derivatives such as gradient play an essential role in PDEs.
They applied the model to the heat equation problem, showing high predictive performance and fast prediction, while boundary condition treatment was out of their scope.
Proposed method
We present our model architecture.
We adopt an encode-process-decode architecture, proposed by
Battaglia et al.
(2018), which has been applied successfully in various previous works, e.g., Horie et al. ; Brandstetter et al.
Our key concept is to encode input features, including information on boundary conditions, apply a GNN-based nonlinear solver loop reﬂecting boundary conditions in the encoded space, then decode carefully to satisfy boundary conditions in the output space.
3.1 Dirichlet boundary model
As demonstrated theoretically and experimentally in literature (Hornik, 1991; Cybenko, 1992;
Nakkiran et al., 2021), the expressive power of neural networks comes from encoding in a higherdimensional space, where the corresponding boundary conditions are not trivial.
However, if there are no boundary condition treatments in layers inside the processor, which resides in the encoded space, the trajectory of the solution can be far from the one with boundary conditions.
Therefore, boundary condition treatments in an encoded space are essential for obtaining reliable neural PDE solvers that fulﬁll boundary conditions.
To ensure the same encoded space between variables and boundary conditions, we use the same encoder for variables and the corresponding Dirichlet boundary conditions, which we term the boundary encoder , as follows:
[Equation]
One can easily apply Dirichlet boundary conditions in the aforementioned encoded space using the
Dirichlet layer deﬁned as:
[Equation]
This process is necessary to return to the state respecting the boundary conditions after some operations in the processor, which might disrespect the conditions.
After the processor layers, we decode the hidden features using functions satisfying:
[Equation] [Equation]
This condition ensures that the encoded boundary conditions correspond to the ones in the original physical space.
Demanding that Equation 13 holds for arbitrary ˆ u ; we obtain f decode ◦ f encode = Id u , resulting in f decode = f
+encode , which we call the pseudoinverse decoder .
It is pseudoinverse because f encode , in particular encoding in a higher-dimensional space, may not be invertible.
Therefore, we construct f +encode using pseudoinverse matrices.
For more details, see Appendix A.1.
3.2 Neumann boundary model
Matsunaga et al.
(2020) proposed a wall boundary model to deal with Neumann boundary conditions for the least squares moving particle semi-implicit (LSMPS) method (Tamai & Koshizuka, 2014), a framework to solve PDEs using particles.
The LSMPS method is the origin of the IsoGCN’s gradient operator, so one can imagine that the wall boundary model may introduce a sophisticated treatment of Neumann boundary conditions into IsoGCN.
We modiﬁed the wall boundary model to adapt to the situation where the vertices are on the Neumann boundary, which differs from the situation of particle simulations (see Appendix A.2 for more details).
Our formulation of IsoGCN with Neumann boundary conditions, which is termed NeumannIsoGCN (NIsoGCN), is expressed as:
[Equation]
where ˆ g i is the value of the Neumann boundary condition at x i , W is a trainable matrix, and w i >is an untrainable parameter to control the strength of the Neumann constraint.
As w i → ∞ , the model strictly satisﬁes the given Neumann condition in the direction n i , while the directional derivatives in the direction of ( x j − x i ) tend to be relatively neglected.
Thus, we keep the value of w i moderate to consider derivatives in both n and x directions.
In particular, we set w i = 10 . 0 , assuming that around ten vertices may virtually exist "outside" the boundary on a ﬂat surface in a 3D space.
NIsoGCN is a straightforward generalization of the original IsoGCN by letting n i = [-
-]0 when x i / ∈ ∂
Ω Neumann .
This model can also be generalized to vectors or higher rank tensors, similarly to the original IsoGCN’s construction (see Appendix A.2).
Therefore, NIsoGCN can express any spatial differential operator, constituting D in PDEs.
3.3 Neural nonlinear solver
As reviewed in Section 2.1, one can regard solving PDEs as optimization.
Here, we adopt the
Barzilai–Borwein method (Barzilai & Borwein, 1988) to solve Equation 8 in the encoded space.
In our case, the step size α ( i ) of gradient descent is approximated as:
[Equation]
wherer R ( h ) is the residual vector in the encoded space and (cid:104)
f , g (cid:105) Ω [Equation] =
[Equation]
x i ∈
Ω f
[Equation] x i ) · g ( x i ) denotes the inner product over the mesh.
Because the inner product is taken all over the mesh (graph), computing α ( i )BB corresponds to global pooling.
With that view, one can ﬁnd similarities between
Equation 9 and deep sets (Zaheer et al., 2017), which is a successful method to learn point cloud data and has a strong background regarding permutation equivariance.
For more details, see Appendix A.3.
[Table]
Our aim is to use Equation 16, approximating the nonlinear differential operator D in Equationwith NIsoGCN.
By doing this, we expect the processor to consider both local and global information, which may have an advantage over simply stacking GNNs corresponding to the explicit method as discussed in Section 2.1.1.
Combinations of solvers and neural networks are already suggested in, e.g., NeuralODE
(Chen et al., 2018).
The novelty of our study is the extension of existing methods for solving PDEs with spatial structure and the incorporation of global pooling into the solver, enabling us to capture global interaction, which we refer to as the neural nonlinear solver .
Finally, the update from the state at the i th iteration h ( i ) to the ( i + 1) th in the neural nonlinear solver is expressed as:
[Equation]
where h (0) is the encoded u ( t, · ) reﬂecting Equation 9 and D NIsoGCN is an E( n ) -equivariant GNN reﬂecting the structure of D using differential operators provided by NIsoGCN.
Here, Equationenforces hidden features to satisfy the encoded PDE, including boundary conditions, motivating us to call our model physics-embedded neural networks because it embeds physics (PDEs) in the model rather than in the loss.
Experiments
Using numerical experiments, we demonstrate the proposed model’s validity, expressibility, and computational efﬁciency.
We use two types of datasets: 1) the gradient dataset to verify the correctness of NIsoGCN and 2) the incompressible ﬂow dataset to demonstrate the speed and accuracy of the model.
We also present ablation study results to corroborate the effectiveness of the proposed method.
The implementation of our model is based on the original IsoGCN’s code.
Our implementation is available online.
3 All the details of the experiments and another simple experiment can be found in Appendix B, C, and D.
4.1 Gradient dataset
As done in Horie et al.
(2021), we conducted experiments to predict the gradient ﬁeld from a given scalar ﬁeld to verify the expressive power of NIsoGCN.
We generated cuboid-shaped meshes
[Table]
[Figure]
[Table]
[Figure]
[Table]
PENN prediction is consistent under rotation and translation due to the E( n ) -equivariance nature of the model, while MP-PDE’s predictive performance degrades under transformations.
randomly with 10 to 20 cells in the X, Y, and Z directions.
We then generated random scalar ﬁelds over these meshes using polynomials of degree 10 and computed their gradient ﬁelds analytically.
Our training, validation, and test datasets consisted of 100 samples.
Table 1 and Figure 2 show that the proposed NIsoGCN improves gradient prediction, especially near the boundary, showing that our model successfully considers Neumann boundary conditions.
4.2 Incompressible ﬂow dataset
We tested the expressive power our model by learning incompressible ﬂow in complex shapes.
The corresponding nonlinear differential operator is denoted as:
[Equation]
with the incompressible condition ∇ · u = 0 , where, in the present case, u is the ﬂow velocity ﬁeld, p is the pressure ﬁeld, and Re is the Reynolds number.
4.2.1 Data
To generate the dataset, we ﬁrst generated pseudo-2D shapes, with one cell in the Z direction, by changing design parameters, starting from three template shapes.
Thereafter, we performed numerical analysis using a classical solver, OpenFOAM, 4 with ∆ t = 10 − 3 , and the initial conditions were the solutions of potential ﬂow, which can be computed quickly and stably using the classical solver.
The
Reynolds number Re was around 10 3 .
The linear solvers used were generalized geometric-algebraic multi-grid for p and the smooth solver with the Gauss–Siedel smoother for u .
Template shapes, design parameters, and boundary conditions used can be found in Appendix C.2.
To conﬁrm the expressive power of the proposed model, we used coarse input meshes for machine learning models.
We generated these coarse meshes by setting cell sizes roughly four times larger than the original numerical analysis.
We obtained ground truth variables using interpolation.
The task was to predict ﬂow velocity and pressure ﬁelds at t = 4 .
using information available before numerical analysis, e.g., initial conditions and the geometries of the meshes.
Training, validation, and test datasets consisted of 203, 25, and 25 samples, respectively.
We generated the dataset by randomly rotating and translating test samples to monitor the generalization ability of machine learning models.
4.2.2 Machine learning models
We constructed the PENN model corresponding to the incompressible Navier–Stokes equation.
In particular, we adopted the fractional step method, where the pressure ﬁeld was also obtained as a PDE solution along with the velocity ﬁeld.
We encoded each feature in a 4, 8, or 16-dimensional space.
After features were encoded, we applied a neural nonlinear solver containing NeumanIsoGCNs
and
Dirichlet layers, reﬂecting the fractional step method (See Equations 51 and 52).
Inside the nonlinear solver’s loop, we had a subloop that solved the Poisson equation for pressure, which also reﬂected the considered PDE (See Equation 50).
We looped the solver for pressure ﬁve times and four or eight times for velocity.
After these loops stopped, we decoded the hidden features to obtain predictions for velocity and pressure, using the corresponding pseudoinverse decoders.
For the state-of-the-art baseline model, we selected MP-PDE (Brandstetter et al., 2022) as it also provides a way to deal with boundary conditions.
We used the authors’ code 5 with minimum modiﬁcation to adapt to the task.
We tested various time window sizes such as 2, 4, 10, and 20, where one step corresponds to time step size ∆ t = 0 .
With changes in time window size, we changed the number of hops considered in one operation of the GNN of the baseline to have almost the same number of hops visible from the model when predicting the state at t = 4 . 0 .
The numbers of hidden features, 32, 64, and 128, were tested.
All models were trained for up to 24 hours using one GPU (NVIDIA A100 for NVLink 40GiB HBM2).
4.2.3 Results
Table 2 and Figure 3 show the comparison between MP-PDE and PENN.
The predictive performances of both models are at almost the same level when evaluated on the original test dataset.
The results show the great expressive power of the MP-PDE model because we kept most settings at default as much as possible and applied no task-speciﬁc tuning.
However, when evaluating them on the transformed dataset, the predictive performance of MP-PDE signiﬁcantly degrades.
Nevertheless,
PENN shows the same loss value up to the numerical error, conﬁrming our proposed components are compatible with E( n ) -equivariance.
In addition, PENN exhibits no error on the Dirichlet boundaries, showing that our treatment of Dirichlet boundary conditions is rigorous.
Figure 4 shows the speed-accuracy trade-off for OpenFOAM, MP-PDE, and PENN.
We varied mesh cell size, the time step size, linear sover settings for OpenFOAM to have different computation speeds and accuracy.
The proposed model achieved the best performance in speed-accuracy trade-off between all the tested methods under fair comparison conditions.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Figure]
MP-PDE, and PENN.
The error bar represents the standard error of the mean.
All computation was done using one core of Intel Xeon CPU E5v2@2.40GHz.
Data used to plot this ﬁgure are shown in Tables 5, 6, and 7.
Table 3 presents the results of the ablation study.
Comparison between models with and without the proposed components shows that the proposed components, i.e., the boundary encoder, Dirichlet layer, pseudoinverse decoder, and neural nonlinear solver, signiﬁcantly improve the models.
The neural nonlinear solver in the encoded space turned out to have the biggest impact on the performance, while the Dirichlet layer ensured reliable models that strictly respect
Dirichlet boundary conditions.
Conclusion
We have presented an E( n ) -equivariant, GNN-based neural PDE solver, PENN, which can fulﬁll boundary conditions required for reliable predictions.
The model has superiority in embedding the information of PDEs (physics) in the model and speed-accuracy trade-off.
Therefore, our model can be a useful standard for realizing reliable, fast, and accurate GNNbased PDE solvers.
Although the property of our model is preferable, it also limits the applicable domain of the model because we need to be familiar with the concrete form of the PDE of interest to construct the effective PENN model.
For instance, the proposed model cannot exploit its potential to solve inverse problems where explicit forms of the governing PDE are not available for such tasks.
Therefore, combining PINNs and PENNs could be the next direction of the research community.
Potential negative societal impacts
We have built a foundation to learn PDEs in a steerable manner rather than focusing on a speciﬁc application.
Because of that, we envisage minimal risk of direct abusing our present work.
However, as mentioned in the introduction, solving PDEs has many impacts on various domains, from both positive and negative aspects.
Thus, our work and possible successive ones may be abused, aiming to harm lives and the environment.
Therefore, the research community, including us, must be careful in using them and control the research direction to prevent abusing these technologies.
Acknowledgments and Disclosure of Funding
This work was supported by PRESTO, Japan Science and Technology Agency, Grant Number
JPMJPR21O9, Japan and by the New Energy and Industrial Technology Development Organization,
Grant Number JPNP14012, Japan.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contribu- tions and scope?
[Yes] See the abstract and Section 1.
(b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] Our paper, in particular Section 6, is written in accordance with the guideline.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Footnote 3.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix B, C, D, and Footnote 3.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section 4, Table 1, 2, 3, and Figure 4.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.
If you are using existing assets (e.g., code, data, models)
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Footnote(b)
Did you mention the license of the assets?
[Yes] See Footnote
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] See Section 4, Table 1, 2, 3, and Figure 4.
(d) Did you discuss whether and how consent was obtained from people whose data you’re us- ing/curating?
[N/A]
The dataset we used in the paper is generated by ourselves. (e) Did you discuss whether the data you are using/curating contains personally identiﬁable infor- mation or offensive content? [N/A] The dataset is based on numerical analysis, so there is no concern about personally identiﬁable information nor offensive content.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Details of the proposed method A.1 Construction of pseudoinverse decoder
We can construct the pseudoinverse decoders for a wide range of neural network architectures.
For in- stance, the pseudoinverse decoder for an multilayer perceptron (MLP) with one hidden layer f ( x ) = σ 2 ( W 2 σ 1 ( W 1 x + b 1 ) + b 2 ) can be constructed as:
[Equation]
where W + is the pseudoinverse matrix of W and σ is an invertible activation function whose Dom( σ )[-= 
-]
= Im( σ ) =
R .
We chose LeakyReLU
[Equation]
where set a = 0 .
5 because an extreme value of a (e.g., 0.01) could lead to an extreme value of gradient for the inverse function.
In addition, one may choose activation functions whose Im( σ ) (cid:54) =
R , such as tanh .
However, in that case, we must ensure that the input value to the pseudoinverse decoder is in Im( σ ) (in case of tanh , it is ( − 1 , 1) ); otherwise, the computation would be invalid.
Besides, similar to the Dirichlet encoder and pseudoinverse decoder, we could deﬁne the speciﬁc encoder and decoder for the Neumann boundary condition.
However, this is not included in the contributions of our work because it does not improve the performance of our model, which may be because the Neumann boundary condition is a soft constraint in contrast to the Dirichlet one and expressive power seems more important than that inductive bias.
A.2 Derivation of NIsoGCN
Matsunaga et al.
(2020) derived a gradient model that can treat the Neumann boundary condition with an arbitrary convergence rate with regard to spatial resolution.
Here, we derive our gradient model, i.e., NIsoGCN, in a different way to simplify the discussion because we only need the ﬁrst-order approximation for fast computation.
Before deriving NIsoGCN, we review introductory linear algebra using simple normation.
Using a unit basis { e i ∈ R d : (cid:107) e i (cid:107) = 1 } di =1 , one can decompose a vector v ∈ R d using: v =
[Equation]
Now, consider replacing the basis { e i ∈ R d } di =1 with a set of vectors B = { b i ∈ R d } d (cid:48) i
=1 , called a frame , that spans the space but is not necessarily independent (thus, d (cid:48) ≥ d ).
Using the frame, one can assume v is decomposed as:
[Equation]
where A is a matrix that corrects the "overcount" that may occur using the frame (for instance, consider expanding (1 , 0) (cid:62) with the frame { (1 , 0) (cid:62) , ( − 1 , 0) (cid:62) , (0 , 1) (cid:62) } ).
A set { Ab i } d (cid:48) i =0 is called a dual frame for B .
We can ﬁnd the concrete form of A considering:
[Equation]
[Equation] Requiring that Equation 24 holds for any v ∈ R d , one can conclude A = (cid:80) i ( b i ⊗ b i ) − 1 .
Finally, we obtain
[Equation]
[Equation]
b
[Equation]
[Equation]
For more details on frames, see, e.g., Han et al.
Then, we can derive NIsoGCN at the i th vertex on the Neumann boundary, by letting
[Equation]
where { j 1 , j 2 , . . .
, j m } are indices of neighboring vertices to the i th vertex.
In addition [Equation] we assume the approximated gradient of a scalar ﬁeld ψ at the i th vertex, (cid:104)∇ ψ (cid:105) i , satisﬁes the following conditions:
[Equation]
Equation 27 is a natural assumption because we expect the directional derivative in the direction of ( x j k − x i ) / (cid:107) x
j k − x i (cid:107) should correspond to the slope of ψ in the same direction.
Equation 28 is the Neumann boundary condition, which we want to satisfy.
Finally, by substituting Equations 26, 27, and 28, we obtain
NIsoGCN, i.e., Equation 14.
To apply NIsoGCN to t , the rank k tensors ( k ≥ 1 ), one can recursively deﬁne the operation as:
[Equation]
where t i is the i th component of t , resulting in the rank ( k − 1) tensor.
In case of the rank 1 tensor v , it can be formulated as:
[Equation]
Please note that each component v i has multiple features in the encoded space, e.g., 16 or 64, resulting in NIsoGCN 1 → 2 ( v ) represents multiple rank 2 tensors for each vertex (see Figure 1 of Horie et al.
As discussed in Horie et al.
(2021), IsoGCNs (NIsoGCNs) correspond to spatial differential operators as:
• NIsoGCN 0 → 1 ( ψ ) :
Gradient ∇ ψ (rank 0 tensor to rank 1 tensor) • NIsoGCN 1 → 0 ( v ) :
Divergence ∇ · v (rank 1 tensor to rank 0 tensor) • NIsoGCN 0 → 1 → 0 ( ψ ) :
= NIsoGCN 1 → 0 ◦ NIsoGCN 0 → 1 ( ψ ) :
Laplacian ∇ · ∇ ψ (rank 0 tensor to rank 1 tensor to rank 0 tensor) • NIsoGCN 1 → 2 ( v ) :
Jacobian ∇ v (rank 1 tensor to rank 2 tensor) • NIsoGCN 0 → 1 → 2 ( ψ ) :
= NIsoGCN l → 2 ◦ NIsoGCN 0 → 1 ( ψ ) :
Hessian ∇∇ ψ (rank 0 tensor to rank 1 tensor to rank 2 tensor)
Because NIsoGCN contains a learnable weight matrix (see Equation 14), the component learns to predict the derivative of the corresponding tensor rank in an encoded space.
This feature of NIsoGCNs enables us to construct machine learning models corresponding to PDE in the encoded space.
A.3 Derivation of the step size in the Barzilai–Borwein method
We derive Equation 16 by applying the Barzilai–Borwein method to our case.
We start with Equation 8, which corresponds to a nonlinear problem:
[Equation]
Solve v R ( v
[Equation]
( x
[Equation]
We consider solving it by applying the linear iterative method using the Taylor expansion, assuming the update ∆ v ( i ) := v ( i +1) − v ( i ) is small enough.
The iterative method is expressed as:
[Equation] [Equation]
[Equation]
35) where ∇ v R ( v ( i ) ) denotes the Jacobian matrix with the shape of n × n ( n roughly corresponds to the number of vertices of the mesh).
To optain update, we may solve Equation 35 as:
[Equation]
corresponding to the Newton–Raphson method.
However, it may take enormous computation resources because ∇ v R ( v ( i ) ) is usually a huge matrix.
Instead, we can approximate:
[Equation]
which corresponds to gradient descent:
[Equation]
Substituting Equation 37 into Equation 35, we obtain:
[Equation]
[Table]
[Figure]
r
We want to ﬁnd a good α ( i ) satisfying Equation 39 the best.
Thus, we obtain α ( i )BB through:
[Equation]
[Equation]
2 , where ∆ R ( [Equation] =
R ( v ( [Equation] +1)
[Equation]
− R ( v ( i ) ) .
[Equation]
[Equation]
Because of the convexity of the problem, it is enough to ﬁnd alpha satisfying:
[Equation]
where < · , · > denotes the inner product in the corresponding space.
Using the linearity of the inner product, we obtain:
[Equation]
Equation 45 is equivalent to Equation 16.
As seen from the derivation, α ( i )BB is determined to satisfy Equation 39 as much as possible for all vertices and all feature components.
That means α ( i )BB has global information because it considers all vertices, making the global interaction possible.
In addition, α ( i )BB is equivariant because it is scalar, which does not depend on coordinate.
Therefore, α ( i )BB is suitable for realizing efﬁcient PDE solvers with E( n ) -equivariance.
B Experiment details: gradient dataset
Figure 5 shows the architectures we used for the gradient dataset.
The dataset is uploaded online.
6 We followed the instruction of Horie et al.
(2021) (in particular, Appendix D.1 of their paper) to make the features and models equivariant.
To facilitate a fair comparison, we made input information for both models equivalent, except for
M − 1 in Equation Equation 15, which is a part of our novelty.
For both models, we used Adam (Kingma & Ba, 2014) as an optimizer with the default setting.
Training for both models took around ten minutes using one GPU (NVIDIA A100 for NVLink 40GiB HBM2).
Figure 5 shows model architectures used for the experiment.
[Table]
[Figure]
template shapes used to generate the dataset.
a 1 , b 1 , b 2 , c 1 , and c 2 are the design parameters.
C Experiment details: incompressible ﬂow dataset C.1 Governing equation
The incompressible Navier–Stokes equations, the governing equations of incompressible ﬂow, are expressed as:
[Equation]
We also consider the following incompressible condition:
[Equation]
which may be problematic when solving these equations numerically.
Therefore, it is common to divide the equations into two: one to obtain pressure and one to compute velocity.
There are many methods to make such a division; for instance, the fractional step method derives the Poisson equation for pressure as follows:
[Equation]
50) where
[Equation]
51) is called the intermediate velocity.
Once we solve the equation, we can compute the time evolution of velocity as follows:
[Equation]
[Equation]
Because the fractional step method requires solving the Poisson equation for pressure, we also need the boundary conditions for pressure as well:
[Equation]
Our machine learning task is also based on the same assumption: motivating pressure prediction in addition to
[Table]
[Figure] [Figure]
[Table]
[Figure]
C.2 Dataset
We generated numerical analysis results using various shapes of the computational domain, starting from three template shapes and changing their design parameters as shown in Figure 6.
For each design parameter, we varied from 0 to 1.0 with a step size of 0.1, yielding 11 shapes for type A and 121 shapes for type B and C.[-
-]
The boundary conditions were set as shown in Figures 7 and 8.
These design and boundary conditions were chosen to have the characteristic length of 1.0 and ﬂow speed of 1.0.
The viscosity was set to 10 − 3 , resulting in
Reynolds number Re ∼ 10 3 .
The linear solvers used were generalized geometric-algebraic multi-grid for p and the smooth solver for u analysis up to
Xeon v
MLP 1, 8,tanh, Identity
MLP 1,
Identity
[Table]
[Table] [Table]
+
[Table]
iterations
Pseudoinversedecoder (Weight
[Table]
with *1)
Pseudoinversedecoder (Weight share with *2) *
<latexit sha1_base64="bkj3uSGdg5liys8Voy2YrCNRmLQ
[Table] [Table]
C.3 Model architectures
The input features of the model are:
• u ( t = 0 . 0) :
The initial velocity ﬁeld, the solulsion of potential ﬂow • ˆ u : The Dirichlet boundary condition for velocity • p ( t = 0 . 0) :
The initial pressure ﬁeld • ˆ p : The Dirichlet boundary condition for pressure • e − 0 .
5 d , e − 1 .
d , e − 2 .
d : Features computed from d , the distance from the wall boundary condition
• u ( t = 4 . 0) :
The velocity ﬁeld at t = 4 .•
p ( t = 4 . 0) :
The pressure ﬁeld at t = 4 .[-
-]
The strategy to construct PENN for the incompressible ﬂow dataset is the following:
• Consider the encoded version of the governing equation • Apply the neural nonlinear solver containing the Dirichlet layer and the NIsoGCN to the encoded equation • Decode the hidden feature using the pseudoinverse decoder.
[Table]
Concatenation
[Table] [Equation]
i
Reﬂecting the fractional step method, we build PENN using spatial differential operators provided by NIsoGCN.
We use a simple linear encoder for the velocity and the associated Dirichlet boundary conditions.
For pressure and its Dirichlet constraint, we use a simple MLP with one hidden layer.
We encode each feature in a 16-dimensional space.
After features are encoded, we apply a neural nonlinear solver containing NeumanIsoGCNs
and Dirichlet layers, reﬂecting the fractional step method (Equations 51 and 52).
The encoded equations are expressed as:
[Equation] [Equation]
[Equation]
(57) where h u is the encoded u and h p is the encoded p .
Note that these equations correspond to Equations 50, 51, and 52, by regarding IsoGCNs as spatial derivative operators.
The corresponding neural nonlinear solvers are
[Table] [Table]
Dirichlet Layer
NIsoGCN 0 → 1 →16, 16,
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
expressed as:
[Equation]
, (59) for h u and h
[Equation]
i
[Equation]
+1) p
[Equation] [Equation]
(61) for h p , where h (0) u
= h u ( t, · ) , h (0) p = h p ( t, · ) , and h ( i ;0) p =
h ( i ) p .
For notation regarding NIsoGCNs, please see Appendix A.2.
Figures 9, 10, and 11 present the PENN model architecture used for the incompressible ﬂow dataset.
As seen in Figure 10, we have a subloop that solves the Poisson equation for pressure in the nonlinear solver’s loop for velocity.
We looped the solver for pressure ﬁve times and eight times for velocity.
After these loops stopped, we decoded the hidden features to obtain predictions for velocity and pressure, using the corresponding pseudoinverse decoders.
C.4 Implementation details
As discussed in Horie et al.
(2021), nonlinearity can be applied to the scalar but cannot be applied to the tensors with a rank equal to or greater than one.
For such a tensor, nonlinearity can be applied to its norm as:
[Equation]
This strategy to apply nonlinearity is used not only in the MLP blocks but also NIsoGCN blocks.
To facilitate the smoothness of pressure and velocity ﬁelds, we apply GCN layers corresponding to numerical viscosity in the standard numerical analysis method.
Here, please note that the PENN model consists of components that accept arbitrary input lengths, e.g., pointwise MLPs, deep sets, and NIsoGCNs.
Thanks to the model’s ﬂexibility, we can apply the same model to arbitrary meshes similar to other GNNs.
C.5 Training details
Because the neural nonlinear solver applies the same layers many times during the loop, the model behaved somehow similar to recurrent neural networks during training, which could cause instability.
To avoid such unwanted behavior, we simply retried training by reducing the learning rate of the Adam optimizer by a factor of 0.5.
We found our way of training useful compared to using the learning rate schedule because sometimes the loss value of PENN can be extremely high, resulting in difﬁculty to reach convergence with a lower learning rate after such an explosion.
Therefore, we applied early stopping and restarted training using a lower learning rate from the epoch with the best validation loss.
Our initial learning rate was 5 .
× 10 − 4 , and we restarted the training twice, which was done automatically, within the 24-hour training period of PENN.
For the ablation study, we used the same setting for all models.
For PENN and ablation models, we used Adam (Kingma & Ba, 2014) as an optimizer.
For MP-PDE solvers, we used the default setting written in the paper and the code.
C.6 Result details
Table 4 presents the detailed results of the comparison between MP-PDE and PENN.
Interestingly, the perfor- mance of MP-PDE gets better as the time window size increases.
Therefore, our future direction may be to incorporate MP-PDE’s temporal bundling and pushforward trick into PENN to enable us to predict the state after a far longer time than we do in the present work.
Tables 5 and 6 show the speed and accuracy of the machine learning models tested.
PENN models show excellent performance with a lot smaller number of parameters compared to MP-PDE models.
It is achieved due to efﬁcient parameter sharing in the proposed model, e.g., the same weights are used repeatedly in the neural nonlinear encoder.
Also, as pointed out in Ravanbakhsh et al.
(2017), there is a strong connection between parameter sharing and equivariance.
PENN has equivariance in, e.g., permutation, time translation, and E( n ) through parameter sharing, which is in line with them.
We tested three conﬁgurations of linear solvers:
• Generalized geometric-algebraic multi-grid (GAMG) for p and the smooth solver for u • Generalized geometric-algebraic multi-grid (GAMG) for both p and u • The smooth solver for p and u
In addition, we tested different resolutions for space and time by changing:
• The number of divisions per unit length: 22.5, 45.0, 90.
• Time step size: 0.001, 0.005, 0.010, 0. Ground truth is computed using the number of divisions per unit length of and time step size of 0.001; thus, this combination is eliminated from the comparison because the MSE error is underestimated (in particular,
zero
C.7 Ablation study details
To validate the effectiveness of our model through an ablation study on the following settings:
(A) Without encoded boundary:
In the nonlinear loop, we decode features to apply boundary conditions to fulﬁll Dirichlet conditions in the original physical space (B) Without boundary condition in the neural nonlinear solver: We removed the Dirichlet layer in the nonlinear loop.
Instead, we added the Dirichlet layer after the (non-pseudoinverse) decoder.
(C) Without neural nonlinear solver: We removed the nonlinear solver from the model and used the explicit time-stepping instead (D) Without boundary condition input: We removed the boundary condition from input features (E) Without Dirichlet layer: We removed the Dirichlet layer.
Instead, we let the model learn to satisfy boundary conditions during training.
[Figure]
We again put the results of the ablation study in Table 8, which is already presented in Table 3, for the convenience of the readers.
Comparison with Model (A) shows that the nonlinear loop in the encoded space is inevitable for machine learning.
This result is quite convincing because if the loop is made in the original space, the advantage of the expressive power of the neural networks cannot be leveraged.
Comparison with Model (C) conﬁrms that the concept of the solver is effective compared to simply stacking GNNs, corresponding to the explicit method.
If the boundary condition input is excluded (Model (D)), the performance degrades in line with Brandstetter et al.
That model also has an error on the Dirichlet boundaries.
Model (E) shows a similar result, improving performance using the information of the boundary conditions.
If the pseudoinverse decoder is excluded (Model (F)), the output may not satisfy the Dirichlet boundary conditions as well.
Besides, the decoder has more effect than expected because PENN is better than Model (G).
Both models satisfy the Dirichlet boundary condition, while PENN has signiﬁcant improvement.
This may be because the pseudoinverse decoder facilitates the spatial continuity of the outputs in addition to the fulﬁllment of the Dirichlet boundary condition.
In other words, using a simple decoder and the Dirichlet layer after that may cause spatial discontinuity of outputs. Visual comparison of part of the ablation study is shown in Figure 12.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
D Experiment details: advection-diffusion dataset
To test the generalization ability of PENNs regarding PDE’s parameters and time series, we run an experiment with the advection-diffusion dataset.
The governing equation regarding the temperature ﬁeld T used for the experiment is expressed as:
[Equation]
where c ∈ R is the magnitude of a known velocity ﬁeld, and D ∈ R is the diffusion coefﬁcient.
We set Ω = { x ∈ R 3 | 0 < x 1 < 1 ∧ 0 < x 2 < 1 ∧ 0 < x 3 < 0 .
01 } , ∂ Ω Dirichlet = { x ∈ ∂ Ω | x 1 = 0 } and ∂ Ω Neumann =
∂ Ω \ ∂ Ω Dirichlet .
D.1 Dataset
We varied c and D from 0.0 to 1.0, eliminating the condition c
= D = 0 .
because nothing drives the phenomena, and and varied ˆ T from 0.1 to 1.0.
Like the incompressible ﬂow dataset, we generated ﬁne meshes, ran computation with OpenFOAM, and interpolated the obtained temperature ﬁelds onto coarser meshes.
We split the generated data into training, validation, and test dataset containing 960, 120, and 120 samples.
The dataset is uploaded online.
D.2 Model architecture
The strategy to construct PENN for the advection-diffusion dataset is consistent with one for the incompressible ﬂow dataset (see Appendix C.3).
The input features of the model are:
• T ( t = 0 . 0) :
The initial temperature ﬁeld • ˆ T : The Dirichlet boundary condition for the temperature ﬁeld • ( c, 0 , 0) (cid:62) :
The velocity ﬁeld • c : The magnitude of the velocity • D : The diffusion coefﬁcient
• e − 0 .
5 d , e − 1 .
d , e − 2 .
d : Features computed from d , the distance from the Dirichlet boundary and the output features are:
• T ( t = 0 . 25) :
The temperature ﬁeld at t = 0 .•
T ( t = 0 . 50) :
The temperature ﬁeld at t = 0 .•
T ( t = 0 . 75) :
The temperature ﬁeld at t = 0 .•
T ( t = 1 . 00) :
The temperature ﬁeld at t = 1 .
The encoded governing equation is expressed as:
[Equation]
The corresponding neural nonlinear solver is:
[Equation] [Equation]
Because the task is to predict time series data, we adopt autoregressive architecture for the nonlinear neural solver, i.e., input the output of the solver of the previous step (which is in the encoded space) to predict the encoded feature of the next step (see Figure 13).
Figures 14 and 15 present the detailed architecture of the PENN model for the advection-diffusion dataset experiment.
To conﬁrm the PENN’s effectiveness, we ran the ablation study similar to that in the incompressible ﬂow dataset.
The training is performed for up to ten hours using the Adam optimizer for each setting.
[Table]
/+59F9L//32F2/X3r55+zvT9Oc/a/r8esn5efvv/ws/o6fV</latexit> T ( t = 1 . 00)
Decoding Decoding
Decoding Decoding
[Table]
boundary
[Table]
Without boundary condition
[Table]
solver
[Table]
Without neural nonlinear solver
[Table]
decoder with Dirichlet [Table] decoding
[Table]
D.3 Results
Table 9 presents the results of the ablation study.
As well as the incompressible ﬂow dataset, we found that the PENN model with all the proposed components achieved the best performance.
Because the boundary condition applied is relatively simple compared to the incompressible ﬂow dataset, the conﬁguration without the Dirichlet layer (Model (E)) showed the second best performance; however, the fulﬁllment of the Dirichlet condition of that model is not rigorous.
Figures 16, 17, and 18 show the visual comparison of the prediction with the PENN model against the ground truth.
As seen in the ﬁgures, one can see that our model is capable of predicting time series under various boundary conditions and PDE parameters, e.g., pure advection (Figure 16), pure diffusion (Figure 17), and mixed advection and diffusion (Figure 18).
[Table] [Table]
[Figure]
[Table]
[Figure]
tnCvy78auG7hbWF3y
[Figure]
8YWFv4WThcmH47JtnK
[Figure]
z3b
[Figure]
785r
[Figure]
f
[Figure]
Ks7vMvC97Pm
[Figure] [Figure]
[Equation]
[Figure]
[Equation]
[Figure] [Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]