L EARNING
T IME - DEPENDENT
PDE
S OLVER
USING (R ECURRENT )
M ESSAGE
P ASSING
G RAPH
N EURAL
N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
One of the main challenges in solving time-dependent partial differential equations is to develop computationally efﬁcient solvers that are accurate and stable.
Here, we introduce a graph neural network approach to ﬁnding efﬁcient PDE solvers through learning using message-passing models.
We ﬁrst introduce domain invariant features for PDE-data inspired by classical PDE solvers for an efﬁcient physical representation.
Next, we use graphs to represent PDE-data on an unstructured mesh and show that message passing graph neural networks (MPGNN) can parameterize governing equations, and as a result, efﬁciently learn accurate solver schemes for linear/nonlinear PDEs.
We further show that the solvers are independent of the initial trained geometry, i.e. the trained solver can ﬁnd PDE solution on different complex domains.
Lastly, we show that a recurrent graph neural network approach can ﬁnd a temporal sequence of solutions to a PDE.
I NTRODUCTION
Physical phenomena are generally modeled through partial differential equations (PDEs) that govern the dynamic evolution or static solution of a physical system.
Numerically solving partial differential equations is an important aspect of scientiﬁc and mathematical modeling in a broad range of ﬁelds including physics, biology, material science, and ﬁnance.
There have been many efforts to develop efﬁcient and accurate numerical solvers for PDEs using different techniques including ﬁnite difference (LeVeque, 2007; Shashkov & Steinberg, 2018), ﬁnite volume (Eymard et al., 2000; Brenner et al., 2008), and ﬁnite element schemes (Reddy, 2014; Wriggers, 2008).
While these methods have been successful in producing accurate solutions, major challenges for accelerating and reducing computational cost when the governing PDE is known, and also determining the governing PDE when the physical system is unknown, remains to be addressed, problems such as those in climate modeling, turbulent ﬂow, contact problems, or plastic material deformation.
With the recent developments in deep learning, faster algorithms have been proposed to evaluate the response of a physical system, using only observational data.
While deep learning approaches, such as multi-layer perceptron (MLP) or convolutional neural networks (CNNs), are powerful in learning PDE solutions, they are, however, restricted to a speciﬁc discretization of the physical domain in which they are trained.
As a result, the learned model is limited to a speciﬁc domain and can not be generalized to solve on different domains or for different discretizations, although the underlying physics remains to be the same.
New training is required for any change in the physical domain or discretization.
Here we propose a discretization and domain invariant neural network time-dependent PDE solver based on message passing graph neural nets (MPGNN) which is trained on a sample domain with different discretizations.
The trained MPGNN can then be used to solve for different discretization or even on other domains as long as the underlying physics remains the same.
We further show that a recurrent version of MPGNN can be used to ﬁnd a temporal sequence of solutions to a PDE.
R ELATED W ORKS
One class of neural-net-based PDE solvers focuses on using neural networks as proxies of PDEs and aims at ﬁnding the solution by minimizing a loss that corresponds to the solution satisfying the governing equations and the boundary conditions (Raissi et al., 2017a;b; Lagaris et al., 1998; Weinan
Yu, 2018; Sirignano & Spiliopoulos, 2018; Khoo et al., 2021).
Although such an approach helps to ﬁnd the one-time solution of a PDE with an instance of parameters, a slight modiﬁcation to the PDE parameters, boundary conditions, or the domain requires re-training of the network.
Another approach to solving PDEs is to use convolutional neural nets and snapshots of observations over the discretized input domain and to learn the dynamic evolution of a PDE (Long et al., 2018; Shi et al., 2020).
Further modiﬁcations such as using residual connections (Ruthotto & Haber, 2020), or autoregressive dense encoder-decoder (Geneva & Zabaras, 2020), or symbolic multi-layer neural network (Long et al., 2019) in addition to the CNN can be used to improve the results.
While these models do not require prior knowledge of the PDE, they are limited to domain discretization (as a result cannot be generalized to arbitrary domains) and are limited to certain time discretization (as a result unable to handle temporally and spatially sparse or non-uniform observations).
Inspired by the discretization techniques in solving PDEs, a class of methods uses observational data to learn the discretization approximation required for the updates in classical computational PDE solver methods (Bar-Sinai et al., 2019; Kochkov et al., 2021; Zhuang et al., 2021; Han et al., 2018).
In this approach, a neural network is used for better interpolation at coarse scale to be used in the framework of traditional numerical discretization.
These methods are used in conjunction with classical numerical methods and can improve the accuracy and accelerate the solutions of the traditional numerical schemes (Kochkov et al., 2021).
Although these methods have been shown to generalize to new parameter regimes of a PDE, they are still bounded to the initially trained discretization and can not be used for arbitrary domains without re-training.
Lastly, a class of neural PDE solvers focus on graph representation of the discretized mesh datastructure to approximate the PDE solution (Li et al., 2020a;b; Iakovlev et al., 2020; Belbute-Peres et al., 2020).
The numerical solution of a PDE is an approximation of the solution on discrete locations comprising a discretized mesh of continuous space.
Each node represents a region in the continuous space and the approximate solution of the PDE in that region is assigned to the representative node.
The discretized mesh forms a graph where each node is used to model the state of the system and forms a connectivity graph connecting to the neighboring nodes.
This method has successfully been used to solve time-independent PDEs with different mesh sizes on the same physical domain (Li et al., 2020a).
The connectivity and the location of the nodes can further be optimized to learn the solution with different levels of precision (Alet et al., 2019).
If the PDE includes long-range interactions, which happens mostly in time-independent PDEs, a multi-level graph neural network framework to encapsulate long-range interactions can be used to improve the results (Li et al., 2020b).
In contrast to time-independent PDEs, in the case of time-dependent PDEs, it has been shown that a continuous-time model similar to physics informed neural nets but with a graph neural network can be used to recover system’s dynamics with sparse observational data recorded at irregular times (Iakovlev et al., 2020; Poli et al., 2019). Recently, it have been shown that message passing graph neural networks can be used to implement powerful physical simulation engines (Pfaff et al., 2020; Sanchez-Gonzalez et al., 2020). The state of a physical system can be expressed using a particle-based method as a reduced order model. The particles are then expressed as nodes in a graph and the message passing neural network learns to compute the dynamics of the particles(Sanchez-Gonzalez et al., 2020).
In addition to particle-based methods, mesh-based methods have been shown to be successful in physical simulations (Pfaff et al., 2020).
Such graph-based models, ﬁrst encodes the input data into a latent space and then process it in the latent space (reduced model), and to obtain the physical results decode the data back to the physical space.
Here, we ﬁrst show why graph neural networks can generalize to learn fast PDE solvers inspired by ﬁnite difference schemes. We introduce domain invariant features and boundary conditions inspired by classical PDE solvers to improve the generalization of the learned PDE solver operator.
With the introduced features, we show that message passing graph neural network architecture efﬁciently ﬁts the classical PDE solvers and can learn time-stepping/solver operators for linear and nonlinear PDEs with different boundary conditions.
We further demonstrate that our trained graph neural network solver can be generalized to solve PDEs on physical domains different from the domain that it is trained on.
This is beneﬁcial to train GNN on a sample of small domains for even with unknown dynamics, and further, explore the dynamic behavior on different larger physical domains.
Lastly, we show that a recurrent version of our MPGNN can be used to predict the temporal sequence of solutions to a PDE.
T IME - DEPENDENT PDE S
We consider continuous dynamical system u ( x , t ) ∈ R evolving over time t ∈ R + and spatial coordinate x ∈ Ω ⊂ R d where Ω is a bounded d -dimensional domain.
We assume the system is governed by a partial differential equation of the form
[Equation]
where N [ · ; λ ] denotes the linear/nonlinear differential operator(s) parameterized by the vector λ .
In the above general form, the temporal evolution of the current state u t depends on the differential operator N [ · ; λ ] which may include various spatial derivatives of the state including ∇ u, ∇ 2 u , etc.
Depending on the differential operator N , appropriate boundary conditions on ∂ Ω is required for a well-posed PDE with a unique solution.
Such PDE model is the cornerstone of mathematical models and is widely used to model various systems, from ﬂuid dynamics, thermal sciences, to acoustics, and quantum mechanics.
As an example, u t = N [ u ; λ ] = λ 1 ∇ 2 u
+ λ 2 · ∇ u constitutes a convection diffusion equation for u ∈ R as variable of interest, where λ
= λ 1 , λ 2 are the diffusitivity and the velocity ﬁeld vector with which the quantity is moving with.
The state of the system at each time can be obtained using its initial state and time integration as u ( x , t ) =
u ( x , 0)
+ (cid:82) t 0 N ( u ; λ ) dt .
Numerous numerical techniques such as Finite Elements,
Spectral Methods, Finite Difference, or Finite Volume techniques have been developed to efﬁciently approximate the differential operator N ( · ; λ ) and solve for a dynamical system over time.
In all numerical schemes, the domain is ﬁrst discretized with a mesh, the differential operator is approximated locally using neighboring points, and the solution is calculated over small time steps using a time integrator such as Euler’s scheme, i.e.,
[Equation]
x i )) (2) where the superscript n shows the solution over discretised time t n , and the differential operator
N ( u ; λ ) =
F ( u, ∇ u, ∇ 2 u, · · · ; λ ) shows that it contains information about local spatial derivatives. As an example, consider solving heat equation, u t =
D ∇ 2 u , where D is the diffusion constant and ∇ 2 u = ∂ 2 u/∂x 2 + ∂ 2 u/∂y 2 , on an structured grid shown in Fig.
1a.
Let u ni,j be the discretized solution at time t = nδt and spatial location x = iδx and y = jδy where δt, δx , and δy are the time, horizontal, and vertical spatial discretization respectively. The time and spatial derivatives in the {+heat equation can be expanded using Taylor series at each discretized point, where ∂u ni,j /∂t = ( u n +
+}
{+i,j − u ni,j ) /δt , ∂ 2 u ni,j /∂x 
+}
= (cid:0) u ni +1 ,j − 2 u ni,j + u ni
+1 ,j (cid:1) /δx 2 , and etc.
Re-writing the equation for an arbitrary discretized point, we ﬁnd u n +1 i,j = u ni,j + δt F where F = α ( u i,j +1 − u i,j ) + α ( u i,j − 1 − u i,j ) + β ( u i +1 ,j − u i,j ) + β ( u i − 1 ,j − u i,j ) , where α = Dδt/δx 2 and β =
Dδt/δy 2 .
Solving for this equation for all the points along with the boundary conditions the updates for the discretized points can be achieved.
Note that here the update rule can be seen as the summation of updates that only depend on neighboring points.
Although in this example with a simple linear equation and a structured grid it was easy to ﬁnd the update rule
F , given an arbitrary domain that requires a triangular mesh for discretization (see Fig.
1b) and a nonlinear governing equation the update rule is not straight forward to be worked out.
Our objective here is to learn the approximation of the differential operator using a graph representation of the domain with message passing neural networks.
Since in a general PDE the differential operator F ( u, ∇ u, ∇ 2 u, · · · ; λ ) contains local spatial derivatives and only local neighboring values at a point are relevant to approximate the differential operator at that point.
As a result, a graph neural network is a promising framework to approximate the above right-hand side for the next value predictions.
C ONTRIBUTIONS
In this paper, we propose a graph-based model to learn domain-invariant and free-form solvers for a PDE on arbitrary spatial discretization using message passing neural networks.
Our method here is inspired by the ﬁnite difference method and the possibility of approximating a partial differential equation using discrete point stencils. Here, we use graph neural networks as a nonlinear function approximator and use the simulated data to learn the required stable stencils. In order to show that that the graph neural network has learned the correct PDE solvers, we test our learned graph network to ﬁnd PDE solutions in a domain with a different geometry and mesh discretization. We ﬁnd that the trained model on a sample domain can be used to predict the PDE solution on other physical domains and mesh discretization.
This is only possible by creating relevant features inspired by classical PDE solver techniques to represent the differential operator.
Our contributions are: (i) introducing locally invariant feature representation inspired by classical PDE solvers to efﬁciently learn a differential operator solver; (ii) showing graph representation with message passing neural networks that can parametrize PDE solvers with a domain-invariant representation; (iii) obtaining robust high-performance models for various linear/nonlinear PDEs with arbitrary spatial discretization and showing that it can generalize to arbitrary physical domains, (iv) proposing a recurrent message passing graph neural network approach to predict a temporal sequence of PDE solution over time.
G RAPH N EURAL N ETWORKS FOR PDE S
Let G =
( V , E ) be graph representation of a mesh with nodes V =
{ x i } Ni =1 where x i denotes the positions, and edges E = { e ij } where e ij represents the connecting neighboring points at x i and x j .
Given a physical domain, we use a uniform random distribution of points for the nodes, and use Delaunay triangulation to ﬁnd the neighboring points.
We denote neighbors of node i with
M ( i ) =
{ j | e ij ∈ E} .
We further assign the node and edge attributes with u i and e ij respectively (see Fig. 2).
In a message-passing neural network (Gilmer et al., 2017), we propagate the latent state of nodes u i of for K layers, where at each layer k , we have
[Equation]
where φ ( k ) and γ ( k ) are differentiable deep neural networks.
Note that N = |M ( i ) | represents the number of neighbors for node i , and furthermore instead of the average used in equation 3, other permutation invariant aggregation function such as sum or max can also be used.
Since equation (3) is an approximation of equation (2) where N ( u ; λ ) includes various spatial differentials, we take our edge features to include x j − x i for j ∈ M
( i ) , and also λ ( x ij ) which is the PDE parameters at the midpoint of the edge x ij =
( x i + x j ) / 2 .
We further can include, higher derivatives of the PDE parameters such as ∇ λ ( x ij ) , ∇ 2 λ ( x ij ) for a better approximation representation.
At each point in time, we set u i =
[ u ( x i , t ) , u ( x i , t − δt ) , · · · , u ( x i , t − nδt )] as the last n snapshots for the initial latent space and use it to create the desired u ( K )
i
[Equation]
as the desired last m frames of the solution.
The main assumption here is that the derivatives can be approximated using the graph nodes, which is possible in most physical simulations where the solutions are smooth.
Additionally, note that the predicted values here include all inside and boundary nodes. We use multilayer perceptron for the γ ( k ) and φ ( k ) with three hidden layers.
Note that three hidden layers, allows each point to connect to the third order neighbors (i.e., neighbors of neighbors of neighbors), which potentially only allows for a maximum 6 th order derivative estimation.
In general, the node features consist of the solution in previous time steps and edge features, motivated by the logic of solving PDEs, are made up of the distance between connecting neighboring nodes, along with the PDE parameters calculated at the center of the edge.
We also add an extra feature to the node attributes showing that if the node lies on the boundary or not.
This extra feature is 1 if the node lies on the boundary and 0 otherwise.
Our decision on the node and edge features might slightly differ for different equations, and we point out if there is any change in the features set.
T EST CASES
In this section, we go through different linear/nonlinear PDEs for physical systems and evaluate the performance of our modeling.
First, we start with the time-dependent heat equation and show how our model can learn to predict the future state(s) (see section 6.1), and more importantly, the learned model can be used for predictions in new physical domains different from initial learned domain (see ﬁgure 4).
To predict the sequence of temporal data of a PDE, we use a recurrent message passing graph neural network approach and show that our model is able to predict the sequence of
PDE temporal data (see ﬁgure 6).
Next, to show that the model is able to learn the nonlinearities, we focus on the Navier-Stokes equation to predict future solutions with an arbitrary discretization (see section 6.2).
Lastly, to show that the model is able to include PDE parameters, we focus on the advection-diffusion equation and by including the PDE parameters as edge attributes, we show that the model is able to learn to predict for various PDE parameters (see section 6.3).
6. H EAT E QUATION
Time-dependent heat equation in two dimensions can be written as
[Equation]
which can be solved with an initial condition u ( x , t = 0) and the Dirichlet boundary condition, i.e., u ( x ∈ ∂ Ω , t ) = u 0 , where ∂ Ω denotes domain’s boundary.
We chose a square grid [0 , 1] × [0 , 1] , and use Firedrake (Rathgeber et al., 2016) with a characteristic length of δx = 0 . 0625 for a [Figure] triangular mesh and time-stepping of δt = 8e-4. Note that the mesh in the simulations are generated using built-in mesh generator in Firedrake .
In order to sample the data for training the MPGNN, we construct a Delaunay triangulation of uniformly distributed nodes (Poisson point process) in the domain.
We chose different Dirichlet boundary conditions where we set top, left, right, and bottom boundary conditions to different constant values of u 0 ∈ [0 , 200] . Note that u for different simulation also remains in the same range as u i ∈ [0 , 200] .
For each simulation, we use a newly generated mesh and set of boundary conditions.
We set record the data every 20 δt , and use four subsequent observations for the input data (i.e., n = 4 ), and predict the next frame (i.e., m = 1 ).
As a result, we input frames, 0 , 20 δt, 40 δt, 60 δt and predict the frame 80 δt .
Since per each simulation, we can further use the initial frame to be any of 0 to 20 δt frames, we can generate 20 data inputs per simulation.
We generate 1000 simulations with different meshes and boundary conditions, and with that, we create 20,000 input data.
We deﬁne a MSE loss between the network output and the true values as (cid:80) i (cid:107) ˆ u i − u
i (cid:107) 2 /N , where ˆ u i , u i are the network prediction and ground truth for the node i value and N is the number of nodes.
We further use three layers for the message passing graph neural network with 64 nodes, three message passing layers K = 3 , and choose two three-layer neural networks for γ ( k ) and φ ( k ) where the hidden layers are of the size 6 × 128 , 128 × 128 and 128 ×respectively.
We set the learning rate to 0 .
001 using ADAM optimizer and a step learning scheduler of 0 .
2 after every 5 epochs.
Note that the network architecture and hyperparameters remains the same for different simulations, unless mentioned otherwise.
We ﬁnd that the MSE error starts at 179 .and reduces to 7 × 10 − 7 after 20 epochs.
The relative L 2 on the test data after training falls below 6 × 10 − 6 .
The results for the prediction and ground truth for different boundary conditions on test simulations that the network has never seen are shown in ﬁgure 3.
We further use the same generated data, however with more number of nodes (changing from 64 to 128) in the graph neural network and ﬁnd that the error on the MPGNN on the test data does not change (mean of relative L 2 error on the test data changes from 5 .
2 × 10 − 6 to 5 .
1 × 10 − 6 ).
[Figure] [Figure]
Next, to show that the learned MPGNN can generalize to different physical domains, we create a distorted physical domain with a curved boundary, two inclined edges, and a vertical wall.
The newly generated geometry is different from the initial square geometry and the network has never seen such domains.
We introduce a random mesh on this new physical domain with a similar characteristic mesh size and use the MPGNN (that is only trained on the square domain) to predict the results.
We ﬁnd that the MPGNN indeed predicts the output with high accuracy (average MSE loss of 2 .
8 × 10 − 4 ).
Examples of different simulations with different boundary conditions are shown in ﬁgure 4.
This shows that the graph neural network here learns to predict the future values independent of the initial physical geometry and can be extended to predict a PDE results on new unseen physical domains. Here we have used four previous snapshots as the input to our network.
In order to show the effect of number of input frames on the prediction power of our network, we run tests by changing the number of input frames and calculating the ﬁnal average MSE error on the test data set on both similar and different geometry.
In all the cases we predict the frame at 80 δt after the ﬁnal frame.
When we change the number of input frames, the input frames are respectively 40 δt , 25 δt , 20 δt , 15 δt , and 10 δt apart for two, three, four, ﬁve, and eight input frames respectively.
Interestingly, we ﬁnd that two input frames are enough for prediction on similar geometries, however to have a reliable MSE on a different geometry we need at least three to four frames.
Lastly, we show that the MPGNN is capable of predicting sequence of temporal data, we use our
MPGNN in a recurrent format (see ﬁgure 6a).
We ﬁrst train our MPGNN with a sequence of three past results (i.e, n = 3 ) to predict the next frame (i.e., m = 1 ).
As a result, in the trained network, given input data of initial three frames of a simulation ( t 1 , t 2 , t 3 ) , it can predict the next time step solution t 4 .
Next, we concatenate the last three obtained results, i.e. ( t 2 , t 3 , t 4 ) to predict the next time frame and so on.
The results for a sample test case is shown in ﬁgure 6b.
It is to be noted that we did not use a recurrent loss function to train our network, and as a result, the average MSE loss keeps growing with the number of predicted frames.
The average MSE error for 1000 different simulations along with the range of error are shown in ﬁgure 6c where the error slowly increases with the number of predicted frames.
It is expected that training the MPGNN with a recurrent loss function would improve the overall result. [Figure] [Figure] over time.
6. N AVIER -S TOKES E QUATION
In order to ﬁnd MPGNN’s performance on learning nonlinear PDEs, we use a two-dimensional incompressible Navier-Stokes equation as
[Equation]
where v = ( u, v, 0) is the velocity ﬁeld, p is pressure, ρ is the ﬂuid’s density, and ν is the diffusion constant.
The second equation is known as the incompressibility equation which assures a divergencefree velocity ﬁeld (i.e., ∇ .
v = 0 ).
We use spectral methods to solve the above equation on a square domain with periodic boundary conditions (see supplementary material for the details).
We chose a square domain with [0 , 2 π ] × [0 , 2 π ] and input a random initial condition to generate the initial conditions we sample random numbers uniformly in the range [0 , 5] for the nodes in the graph and then we then remove the high frequency patterns by taking a Fourier transformation and discarding the top 1/3 of high frequency wave numbers.
Furthermore, in this setting our ρ = 1 and ν = 3e-4. and run our numerical PDE solver for T = 50 and record frames with δt = 0 . 002 . Note that u i for different simulation also remains in the same range as u i ∈ [0 , 4] .
We chose δt such that the results become visually different.
We ﬁrst input ﬁve different frames (i.e., n = 5 ) and predict the next frame (i.e., m = 1 ).
We generate 20000 data points from 25 simulations similar to the heat equation data generation.
In this case, we use the same network geometry for the message passing neural network as the one used for the heat equation.
We ﬁnd the relative L 2 error for the test data after 25 epochs drops from 10 . 1 to below 5 . 4 × 10 − 5 .
The results for three different test cases are shown in ﬁgure 7.
Note that in here in the plots, we are presenting the vorticity ﬁeld ζ = ∂ x v − ∂ u u instead of velocity ﬁelds separately.
The result here shows that the nonlinear PDE structure can also be learned using the MPGNN architecture.
[Figure]
[Figure]
In all the PDEs we tried so far, we kept the PDE parameters as constant.
In this section, we focus on the advection-diffusion equation where the PDE includes different free parameters that can be used as edge features in the MPGNN.
The advection-diffusion equation is
[Equation]
where λ 1 and λ 2 are the advection velocity and diffusion constant respectively.
We chose our domain to be [0 , 1] × [0 , 1] and use characteristic mesh size of δx = 0 . 0625 and time stepping of δt = 10 − 4 .
We record the data every 200 δt and use four subsequent input frames (i.e., m = 4 ) and predict the next frame (i.e., n = 1 ).
We use Firedrake (Rathgeber et al., 2016) to simulate our PDE with random initial meshes and initial conditions.
We use periodic boundary conditions at x = 0 , 1 and Neumann boundary conditions at y = 0 , 1 .
The initial condition is created using u 0 = a a sin( x ) + a 2 sin(2 x ) + a 3 cos( x ) + a 4 cos(2 x ) where all a i , i = 1 , · · · , 4 are drawn from a uniform distribution U [ − 1 , 1] .
Note that u for different simulation also remains in the same range as i ∈ [ − 3 . We select λ {+
+} and λ 2 randomly from uniform distributions U [0 . 5 , 1 . 5] .
In total we generate 300 simulations with random initial conditions and random PDE parameters and generate 300 k input data for our network.
We ﬁnd that in training our network the average L 2 error for the test data (new simulations that network has never seen) drops from 3 . 4 before training to 1 .
3 × 10 −after training.
Three different test results are shown here in ﬁgure 8.  B ENCHMARK TESTS
In this section we compare our approach with other existing relevant studies.
We have identiﬁed two relevant neural network approaches closely related to our work: (1) Iakovlev et al.
(2020) and (2)
Li et al.
(2020b) . Since we are interested in generalization of the network to other domains and mesh sizes, we perform two different tests on all of the networks and compare their performance by measuring average MSE for prediction on the test dataset. We select the heat equation as our basis (see Sec. 6.1) to test different approaches on a square geometry and test it on the unseen domain discussed in Sec. 6.1. We perform two set of tests: training on a square domain with a small characteristic mesh length (average edge size is 0 .
1 which we call high res), and testing the performance on the same geometry, and on a different geometry (shown in Fig.
4) with similar resolution as well as a lower resolution (average edge distance of 0 . 2 which we call low res).
Next, we repeat the same experiment but with training on low resolution data.
The results are summarized in
Table 1.
In repeating the work by Li et al.
(2020b), we used their provided code at the corresponding git repository and used their suggested hyper-parameters and made use of u i as node features, and the position of the nodes as edge features.
As seen in table 1, the proposed network has a lower MSE best when it is applied to the same geometry, however, in generalization it performs poorly which is due to the positional feature of the nodes instead of the invariant edge distance.
In following Iakovlev et al.
(2020), similar to the previous case, we followed the proposed structure by the authors (i.e., three hidden layers with 60 neurons along with tanh activation function and one message passing layer).
The main difference here is that the edge features contain relative positions as the edge features instead of absolute positions.
This is indeed the main reason for a better generalization of this approach compared to the other method.
Our method with a different activation function (ReLU instead of tanh ) as well as three layers of message passing, along with extra feature ﬂagging the boundary nodes shows an improved result.
Trained on high res
Trained on low res
MSE on test data
Our work (1) (2)
Our work (1) (2) same geometry (same res) 4.01e3.33e1.
[Table] We train all networks using different mesh resolution and test it on similar resolution networks, moreover to a different geometry with two different resolution.  C ONCLUSION In this paper, we showed how message passing neural networks can parametrize time-dependent partial differential equations and their discrete classical PDE solvers.
Inspired by numerical PDE solvers, we introduce domain invariant features for an efﬁcient representation of PDE data.
Using graphs to represent an unstructured mesh, we trained message passing graph neural networks (MPGNN) to efﬁciently learn accurate solver schemes for linear/nonlinear PDEs independent of the initial trained geometry.
We showed that MPGNN can predict single/multiple frame(s) of the future data and furthermore a trained model can predict results on unseen geometries.
We further proposed a recurrent version of MPGNN to march in time and ﬁnd a temporal sequence of solutions to a PDE.
In summary, the main important features of a message passing graph neural network that make them suitable platforms for learning the time-dependent PDEs are: (i) MPGNNs similar to time-dependent
PDEs are spatially locally dependent where each point is only affected by the neighboring points for a small time-step, (ii) the permutation invariant combination of the messages signiﬁes the rotational and translational symmetries in a given physics-based PDE and helps the network to learn more efﬁciently, (iii) the neural networks used in creating the messages and the updates are general learners that can potentially learn the nonlinear update rules, and lastly (iv) the possibility of running several passes in message-passing neural networks helps the update of a node to see features of neighbors of neighbors and as a result MPGNN can ﬁnd the update rule for larger time-steps.
The possible future extension of our current MPGNN solver are: 1) including random long range connections in the initial mesh to enable long-time predictions with less number of message-passing layers; 2) training on a recurrent loss function and addressing signiﬁcant memory required in backpropagation in order to improve long time predictions, ;3) using more accurate temporal schemes rather than Euler discretization to improve predictions, 4) including symbolic regression analysis to uncover the discretized kernel learned by the MPGNN and exploring the possibility of uncovering new PDE solver schemes.
A S PECTRAL METHODS SOLUTION TO N AVIER -S TOKES E QUATION
Spectral methods solution to the Navier-Stokes equation.
In this section, we review the spectral methods used to solve the 2d incompressible Navier-Stokes (NS) equation (5)
In a 2D-ﬂuid motion with v = ( u, v, 0) the NS equations (equation equation 6.2) in the expanded form are
[Equation]
[Equation] [Equation]
We deﬁne a stream function as u = − ∂ y ψ , v = ∂ x ψ .
The incomprehensibility condition, equation 9, is immediately satisﬁed.
In order to remove the pressure from the equations, we ∂ x (Eq.
8) - ∂ y
(Eq.
7) to ﬁnd
[Equation]
Note that this ∇ 2 ψ = ∂ x v − ∂ y u
= ( ∇ × v ) · ˆ z .
We can simplify the last vorticity equation, equation 10, in terms of ζ = ∇ 2 ψ to ﬁnd
[Equation]
Next, we discretize the equation,
[Equation]
[Equation]
where we used the implicit Crank-Nicolson for the linear part and explicit part for the linear part.
Assuming periodic boundary conditions in both x and y direction [Equation] and taking the Fourier transform of the above equation
[Equation]
we ﬁnd
[Equation]
where ˆ ζ = F ( ζ ) is the Fourier transform of the vorticity ﬁeld, and P = F ( g ( ζ n ))
where g ( ζ n ) = (cid:0) ∂ y ∇ − 2 ζ n (cid:1) ∂ x ζ n − (cid:0) ∂ x ∇ − 2 ζ n (cid:1) ∂ y ζ n .
We use equation 16 to propagate in time and to solve for the vorticity ﬁeld.
Given the vorticity ﬁeld, we can further calculate the horizontal and vertical velocity with u = − ∂ y ∇ − 2 ζ, v = ∂ x ∇ − 2 ζ . B P ERIODIC BOUNDARY CONDITION
In order to impose the periodic boundary condition, we add connections to the boundary nodes that would result in a periodic boundary condition.
Particularly, we consider similar copies of the nodes and by repeating the Delaunay triangulation for the nodes, we ﬁnd how the boundary nodes are connected with the other the inside and boundary nodes (see Fig. 9).