Efficient Continuous Spatio-Temporal Simulation with Graph Spline Networks
Abstract
Complex simulation of physical systems is an invaluable tool for a large number of fields, including engineering and scientific computing.
To overcome the computational requirements of high– accuracy solvers, learned graph neural network simulators have recently been introduced.
However, these methods often require a large number of nodes and edges, which can hinder their performance.
Moreover, they cannot evaluate continuous solutions in space and time due to their inherently discretized structure.
In this paper, we propose G RAPH S PLINE N ETS , a method based on graph neural networks and orthogonal spline collocation (OSC) to accelerate learned simulations of physical systems by interpolating solutions of graph neural networks.
First, we employ an encoder-decoder message passing graph neural network to map the location and value of nodes from the physical domain to hidden space and learn to predict future values.
Then, to realize fully continuous simulations over the domain without dense sampling of nodes, we post– process predictions with OSC.
This strategy allows us to produce a solution at any location in space and time without explicit prior knowledge of underlying differential equations and with a lower computational burden compared to learned graph simulators evaluating more space–time locations.
We evaluate the performance of our approach in heat equation, dam breaking, and flag simulations with different graph neural network baselines.
Our method shows is consistently
Pareto efficient in terms of simulation accuracy and inference time, i.e. 3 × speedup with 10% less error on flag simulation.
1. Introduction
Simulations of partial differential equations (PDEs) describing physical processes are an invaluable tool for an increasing number of disciplines.
As a result, the scientific machine learning community has been focused on crafting computationally inexpensive yet accurate simulation methods to expand the range of applicability of dynamical system simulators.
Traditional simulation methods (Houska et al., 2012), such as the first principle model solver and the generalized
Gauss-Newton methods , can be costly in calculations: in particular, complex physical simulations need substantial computational resources to be performed.
In recent years,
PDEs simulators have been widely used in a variety of applied problems such as game physics engines (Lewin, 2021),
Virtual Reality (VR), (H ¨ oll et al., 2018) and the metaverse (Taheri et al., 2021) where accurate simulators can help users immerse themselves in the virtual environment and make it more appealing.
Thus, the need for a high–precision and low–calculation simulator becomes even more urgent and worthwhile.
Previous research has shown successes in applying deep learning for simulating a variety of PDEs (Raissi et al., 2019).
Graph-based simulation methods (Sanchez-Gonzalez et al., 2020; Pfaff et al., 2021) have used graph neural networks as a natural representation of the simulation underlying discretized dynamics; their discretization is often a fundamental part of making simulations viable.
These paradigms have proven successful in learning generalizable particle and mesh–based simulators thanks to the ability of graphs in capturing local and global phenomena while retaining properties such as spatial equivariance and translational invariance (Bronstein et al., 2021).
Despite their advantages, a significant downside of meshbased graph models is their inherent structural discretization, making it hard to achieve physical space or time– space continuous simulations.
The encoder–decoder strategy has hidden layers representing physical states out of mesh points, enabling a space-continuous representation (Alet et al., 2019).
In addition, message passing neural network (MPNN) of derivatives, combined with numerical accumulation methods, work for time-continuous prediction (Iakovlev et al., 2020).
However, these continuous approaches request more mesh points for higher–accuracy simulations, which brings a heavy calculation workload.
On the other hand, collocation methods have considerable benefits in terms of the computational complexity of the number of collocation points and the ability to yield continuous results, such as smoothest spline collocation, modified spline collocation (Fairweather & Meade, 2020), and the Cubic Spline Orthogonal Spline Collocation (OSC) methods for PDEs (Bialecki & Fairweather, 2001).
In this paper, we introduce G RAPH S PLINE N ETS : by leveraging the OSC method, we can readily obtain space–time continuous, free form simulations by starting from an intrinsically discrete GNN without the need for explicit prior knowledge of the problem to be solved.
We use the method as an efficient post–processing scheme which can be applied to several different GNN simulators.
G RAPH S PLINE N ETS allow us to reduce the number of space and time sample points of the ad–hoc trained underlying graph module that leads to Pareto–efficient simulations in terms of solution accuracy and inference speed.
We demonstrate the method on the heat equation PDE benchmark, on a dam–breaking particle–based meshless simulation, and on a mesh–based cloth simulation.
2. Related Work
We identify related works for this paper in the area of numerical methods and deep learning for PDE simulation and categorize them into three main areas, namely: deep learning for physical simulations, graph neural network simulators, and the relationship between collocation methods and deep learning.
Deep Learning for Physical Simulations Solving differential equations with deep neural networks has been an active research area to solve the issues of traditional PDEs solvers, which often suffer from unsustainable computational requirements and scalability issues that can hinder real–time applications (see Appendix C.1 for further insights).
Deep neural networks have been shown to be a viable alternative to numerical methods to solve the issues of scalability and inference time requirements.
Physics Informed Neural Networks (PINNs) (Raissi et al., 2019), which aid in both the solution and discovery of PDEs by using ad–hoc deep architectures and loss functions enforcing boundary conditions, have received considerable attention due to their flexibility in tackling a wide range of data– driven solutions and discovery of PDE, even though they have been shown suffer from unstable training and convergence issues (Wang et al., 2022).
The problem of solving
PDEs, which we deal with in this paper, has also been explored with convolutional neural networks (Guo et al., 2016;
Bhatnagar et al., 2019) which naturally incorporate the inductive bias of spatial invariance.
Another active area of research concerns the use of neural operators (Lu et al., 2019;
Li et al., 2020a; Kovachki et al., 2021) which map between infinite–dimensional function spaces.
Several software libraries that have been developed to deal with numerical methods for deep learning efficiently include Poli et al.
; Chen et al.
; Lu et al.
Graph Neural Network Simulators
The use of graph neural networks (GNNs) to address the simulation of a system with a finite number of sample points has been investigated to address issues from other deep learning paradigms for simulation.
GNNs extend other models as convolutional neural networks to irregular grids and also capture physical principles deriving from geometric deep learning such as spatial equivariance and permutation (Bronstein et al., 2021) while constraining interactions to local neighborhoods (see Appendix C.2 for further intuitions on the success of GNNs for simulation).
Alet et al.
(2019) introduces the Graph Element Networks architecture to model continuous underlying physical processes with no a–priori graph structure by modeling adaptively sampled points in a graph.
Sanchez-Gonzalez et al.
(2020) develops with the
Graph-based Neural Simulator (GNS) paradigm a model which learns a system dynamic update by creating graph edges on neighbor finite particles and performing message passing: this is shown to faithfully contain rollout errors and generalize well to unseen conditions.
Pfaff et al.
(2021) extends the mesh–free GNS to mesh–based simulations: the resulting model can capture mesh–space interactions by having edges corresponding to the ones of the mesh and obtains cheaper simulations than the baseline numerical solvers.
Other related works include applications to control (Sanchez-Gonzalez et al., 2018), the extension of neural operators to graphs (Li et al., 2020c;b), and hybrid approaches with graph networks with traditional fluid simulation solvers (de Avila Belbute-Peres et al., 2020).
Collocation Methods and Deep Learning While continuous–time graph models have been previously explored (Poli et al., 2019; Xhonneux et al., 2020) that can theoretically capture the system time evolution, space– continuous graph models still suffer from the problem of the inherent graph discretization.
For this reason, representing the space and time–continuous nature of simulations have been mainly dealt by using interpolation methods such as linear interpolation in Alet et al.
However, these methods may not be suitable for simulation due to their lack of differentiability that may be necessary for instance in control problems (Liang et al., 2019) as well as falling short of realistic simulation which veive often continuously differentiable (i.d. C 1 class).
Unlike previous approaches, we employ the Orthogonal Spline Collocation (OSC) (Bialecki & Fairweather, 2001) method to efficiently [Equation] [Figure] [Equation]
[Figure]
Ground
Truth
GNN
Ours obtain C 1 class solutions to differential equations given few partition points and obtain both space and time continuous simulation based on an underlying GNN simulator.
Other deep collocations methods include Guo et al.
(2019), which shows an ad–hoc collocation method for the bending analysis of the Kirchoff plate which cannot be easily tackled with mesh–based methods since it requires C 1 continuity. Brink et al.
(2021) introduces a deep–learning model based on feed–forward networks and collocation method to approximate a variety of strong–form PDEs.
Unlike the paradigms mentioned above that rely on deep learning to obtain collocation weights, we employ the OSC method to obtain the weights which have theoretical guarantees on convergence.
Moreover, the synergy of OSC with GNNs enables our module to tackle diverse problems without the need of crafting over–engineered schemes while efficiently balancing between solution accuracy and inference time. such
3. Methodology 3.1. Problem Set
We consider a continuous dynamic PDE system with state u ( x , t ) ∈
R that evolves over time t ∈ R + and bounded domain
[Equation]
where B ( · ) is the boundary condition and u 0 ( · ) is the initial condition.
Mark X
= { x 0 , x 1 , · · · , x N } ∈ X as a set of physical space locations of sample point.
Y t =
{ y t 0 , y t 1 , · · · , y tN , } ∈ Y is the state of sample points at time t , i.e. y ti = u
( x i , t ) .
A simulator S :
Y → Y maps the current state of sample points to a future state with fixed timestep.
Mark the prediction trajectory from t 0 to t K as { Y t 0 , ˆY t 1 , · · · , ˆY t K }
A simulation model M
( · ; θ ) : Y → Y with learnable parameter θ takes input Y t k and predicts next timestep[-state ˆY t k +
-]
state ˆY t k +1 =
M ( Y t k ; θ ) .
The gap between prediction states and ground truth can be used as the loss function
[Equation]
In the rest of this paper, we will follow the format of notation in which the superscript denotes the time–space index and the subscript denotes the space index.
3.2. Graph OSC Network Architecture
The overall architecture of G RAPH S PLINE N ETS is shown in Figure 2.
Given the initial state of the domain, we firstly employ graph neural networks to obtain discrete predictions.
Then, we apply time–oriented collocation method and space–oriented collocation method on these discrete predictions to get simulation functions, generating time and space–continuous simulations.
We describe our model in three parts: graph neural network, time–oriented orthogonal collocation, and space–oriented orthogonal collocation.
Graph neural network structure.
We employed an
Encoder–Processor–Decoder structure to predict sample point values at the next timestep.
The encoder will represent sample points in the input space as a node in latent graph space, where an adjacency matrix will be created to describe the connection.
Symmetrically, the decoder will map the updated graph to output space, representing hidden features to physical space’s values for each sample point.
A message passing neural network is applied to update node features in hidden graph layers dynamically.
The weights of the encoder, decoder, and message passing layers construct the learning parameters of the graph neural networks, which can be trained by minimizing end–to–end loss of simulation results in physical space and ground truth.
We believe that a graph is an appropriate framework for modeling dynamic physical processes, usually defined by particles interacting across short or long distances.
Encoder E ( X , Y t ) :
X × Y → G encodes observed sample points in physical space to a graph G =
( V, E ) ∈ G as a hidden layer with vertices V
= { v i } Ni =0 connected by undirected edges E = { e ij } , following the strategy of nearest neighbors.
Vertical and edge features are assigned during the encoder process, including position, physical value, vertices distances, etc.
Details of feature are shown in Section 4.1.
Processor P ( G ) : G → G firstly generates messages for every edges based on features of connected vertices, i.e. m ij = ϕ
( v i , v j , e ij ) .
Then update vertices and edges by aggregating messages ˆv i
=
γ ( v i , (cid:76) j ∈N ( i ) m i,j ) , ˆe ij = ψ
( e ij , m ij ) where N ( · ) is connected neighbors, γ and ψ are implemented using MLPs with a residual connection.
Message passing can be multi-steps, depending on graph complexity and physical system scale.
The final output of the processor is an updated graph ˆ G , which can be considered as the predicted graph at the next timestep.
We set the notation of graph at time t k to G t k , then the output of processor is ˆ G t k +
=
P ( G t k ) .
Decoder D ( G ) : G → Y extracts updated values for sam- ple points as prediction at next timestep, i.e. ˆY t k +1 =
D ( ˆ G t k +1 ) .
Decide Partition and Collocation Points
Initial Simulator Polynomial Parameters
Generate an Algebra Problem by Substituting Collocation Points
Solve for Simulator Polynomial Parameters
Simulate Values at the Whole Domain Time-oriented OSC. One sample point’s value changes from 0 to T over time, following an Ordinary Differential
Equation (ODE)
[Equation]
The primary technique for using the OSC approach is
[Figure]
given in Figure 3.
For time–oriented OSC we consider an isometric split of the temporal domain with N partitions π :
=
t 0 < t 1 < · · · < t N = T .
We aim to find one polynomial under order r on each partition and make these
N polynomials C 1 continuous.
These polynomials have the degree of freedom N ( r − 1) .
To decide those parameters, we will select r − 1 collocation points in each partition to decide those parameters.
Note that these collocation points can be isometric or non-isometric: we will leave them isometric for the sake of graph neural network prediction.
Then given an initial value of t 0 = 0 , the graph neural network will rollout predict a series of values at these collocation points.
By substituting locations and values of collocation points to polynomials, we can transfer this ODE problem to algebraic equations.
Notice that the coefficient matrix of this algebraic equation is almost block diagonal (ABD) (De Boor & De Boor, 1978).
This kind of system is commonly easy and cheap to be solved (Amodio et al., 2000).
More details about deriving the degree of freedom, visualization of the ABD coefficient matrix, and detailed technique of applying time–oriented OSC are shown in Appendix A.1.
Space-oriented OSC At one time frame, the state of the domain Ω can be described by
[Equation]
For simplicity and without loss of generality, we consider the unit domain [0 , 1] D = Ω .
Similar with time-oriented
OSC split strategy, for each dimension, we split the domain to N partitions π i : 0 = p i 0 < p i 1 < · · · < p iN = 1 , i = 1 , · · · , D .
Here the partitions can be isometric or non– isometric.
Our target is to find one polynomial under order r on each partition for every dimension and make these N × D polynomials C 1 continuous in the domain.
For example, we can choose piecewise Hermite cubics as the base.
The simulation result is the linear combination of each dimension’s basis, with the degree of freedom N D ( r − 1) D .
For each dimension, we select r − 1 collocation points in each partition by Gauss—Legendre quadrature rule 1 (De Boor & Swartz, 1973).
Then we have in total N D ( r − 1) D collocation points in the space.
We can get prediction values at collocation points by graph neural network and time-oriented OSC.
Then, we substitute locations and values to polynomials to transfer the original problem to an algebraic equation.
Note that the coefficient matrix of this algebraic equation is also ABD.
By solving this algebra equation, we can get the simulation result.
Mode details about applying space–oriented OSC are shown in Appendix A.2.
With the help of a graph neural network, time–oriented OSC, and space-oriented OSC, simulators can cover all the spatio-temporal domains.
To make the model more robust to noisy inputs, we corrupt the input features of the graph with random-walk noise, so the training distribution is similar to the distribution created during rollouts.
3.3. Model Training
After deciding time–oriented OSC time step length and space–oriented OSC collocation points’ locations, we train the graph neural network by supervising on the per-node output features produced by the decoder using a mean square loss between predictions and the corresponding ground truth.
4. Experiments
We evaluate G RAPH S PLINE N ETS on three dynamic systems: heat equation, mesh–free compressible fluids, and deformable mesh–based cloth simulation.
4.1. Experimental Domains
Heat Equation These datasets are generated by FEniCS
Logg et al.
We set the space domain to Ω = [0 , 1] × [0 , 1] and time domain to [0 , 1] with a total of 500 time steps.
Each time step corresponds to ∆ t = 0 .
1 s .
At each time step, the dataset has a fixed mesh with 49 nodes andedges.
This dataset is split into batches of 500 : 100 :of train, validation, and test set.
We initialize these datasets by fixing four boundaries to 0 and setting one or multiple locations within the domain with the initial temperature.
Dam Breaking Simulation
This dataset is generated by Taichi Hu et al.
and simulates a process of 2– dimensional water flowing with an initialized state of water– storing in a rectangular dam.
At t 0 = 0 , one dam wall crashes and lets the water flow down.
This dataset hassample points and 1000 time steps, where each time step corresponds to ∆ t = 0 .
01 s .
This dataset is split into batches of 800 : 100 : 100 for train, validation and test set.
Cloth Simulation We utilize the cloth simulation dataset with no adaptive remeshing 2 in Pfaff et al.
We choose this experimental domain to demonstrate how
G RAPH S PLINE N ETS scale to high–dimensional regimes and deal with chaotic systems such as cloth simulation.
The dataset is generated by the cloth simulator ArcSim (Narain et al., 2012) by having different positions and rotations of a fixed–mesh flag composed of 1579 points. The trajectory in time is simulated in the presence of constant wind and fixed flag handles for a total of 400 timesteps: each timestep corresponds to ∆ t = 0 .
02 s .
The dataset is split into batches of 1000:100:100 of train, validation, and test trajectories.
4.2. Model Training
Heat Equation
In the heat equation simulation, the number of nodes and connections do not change along the process and we hence keep the graph structure fixed.
The input of the graph model is the state value, i.e. the temperature, of each mesh node at the initial time frame and its output is the mesh state at the subsequent.
We train a message passing neural network model by minimizing the squared difference between the target next state and the model prediction (further details are available in Appendix B.2).
Dam Breaking Simulation
The particle–based dam breaking dataset cannot be represented by a fixed mesh due to the chaotic nature of moving particles.
We create the graph neural network iteratively online as in Pfaff et al.
(2021) by searching for the nearest neighbors of each particle in a connectivity radius.
Since the number of particles is fixed, the time-oriented OSC can be applied to each particle.
The graph neural network takes as input the current positions, node types, and their relative positions as well as a history of past states to infer the state update and is thus trained on the next state target update, i.e. the accelerations (more details are available in Appendix B.3).
Cloth Simulation We implement the mesh–based graph simulator in Pfaff et al.
(2021) as a baseline model to efficiently deal with the flag mesh–based simulation and produce stable rollouts.
Similarly to the dam breaking simulation, we encode the absolute and relative positions and velocities of each node and their types as well as the positions in the mesh–space which yields more stable rollouts.
The model is optimized to predict the dynamics state update values.
4.3. Results
OSC and Interpolation Methods
We show the effectiveness of orthogonal by testing spline collocation methods with different basis (orthogonal and non orthogonal) on 2D heat equation simulation and the results is shown in Figure 7 (Left).
With orthogonal basis, the collocation method can get more accurate results with 40% running time.
Moreover, since creating the OSC coefficient matrix involves independent operations, we can use multiprocessing to efficiently create it.
Multiprocessing also helps for solving the generated algebraic equations with almost block diagonal matrices.
We also further our contributions by implementing the algorithm on GPU to increase its efficiency.
We show the results of parallelization and GPU implementation in Figure 7 (Right).
We also compare the OSC with several widely used interpolation methods: linear, bilinear, 1–D cubic, and 2–D cubic interpolation methods.
Note that OSC methods can have customized orders, e.g. the same with cubic interpolation (order 3 ) or higher (order > 4 ).
Higher–order polynomials can better describe non–linear problems.
These methods are applied to four linear and non–linear problems.
Results of running time and accuracy comparisons are shown in Figure 4.
OSC methods have the lowest error while incurring low computational requirements among the compared methods.
The number of sample points affects the performance: however, it is (Toraichi et al., 1987) demonstrated that OSC has lower calculation complexity O ( n 2 log n ) than the cubic interpolation method O ( n 3 / 4) , where n is the number of sample points.
Further insights are shown in Appendix B.1.
Heat Equation Visualization of simulation results for central initialized heat equation is shown in Figure 6.
We can see the GEN has discrete prediction results, while our approach can produce continuous simulations with errors lower than 10 − 7 at any point within the domain.
Finally, we discuss the accuracy and running time with rollout steps for two approaches on heat equation by time region and space region.
With the help of time–oriented OSC, G RAPH S PLINE N ETS have a nearly 90% running time reduction while keeping similar or lower MSELoss compared with GEN as shown in Table 1.
Moreover, Figure 9 shows the comparison of two models’ performance with the increase of collocation point and rollout step.
With the increase of collocation point number, our model can capture more information than
GEN.
Meanwhile, since the G RAPH S PLINE N ETS has a
[Figure]
longer graph neural network prediction step, it would be more stable for long rollout steps results, which is shown in
Figure 9 right.
Dam Breaking We train G RAPH S PLINE N ETS with the time step ∆ t = 0 .
15 s .
Results comparing our model result with the ground truth are shown in Figure 5.
Compared to the baseline GNN, which is trained with the time step ∆ t = 0 .
03 s , our approach takes 47% less time to execute while the difference of average prediction mean square error with GNN is lower than 10 − 3 .
More details about the result are shown in Appendix B.3.
Flag Simulation We train the baseline purely GNN model predicting the state update at each time step of ∆ t = 0 .
02 s and two models predicting the update each 5 and 10 simulation time steps, respectively, corresponding to 0 .
1 and 0 .
2 seconds.
Figure 8 shows rollout errors and our module
[Figure]
I t e r a t i o n R unn i n g T i m e speedup, while Figure 1 illustrates sample time propagation and comparison with the collocation method.
Our method produces stable rollouts even with larger time steps compared to the baseline model while recovering the intermediate time–steps and providing speedups in the range of 1 . 8 × to more than 7 × compared to the baseline model.
5. Conclusion
We introduce G RAPH S PLINE N ETS , a novel method that can be integrated as a post–processing scheme into several learnable GNN modules for improving the solution of a variety of physical processes in terms of both accuracy and inference time.
Our approach integrates the theory of Orthogonal Spline Collocation methods to achieve space and time continuous simulations without heavy burdens on the computational side.
We demonstrate how G RAPH S PLINE N ETS are robust in predicting complex, high–dimensional processes characterized by several different PDE processes, such as the ones arising directly from differential equations, 8 Time 0. R M S E GNN Ours-5 OursGNN
OursOurs-10 0.
1 s e c o n d r o ll o u t
{+R M S E 
+}
C P U R o
ll o
u t
{+T i m e [ s ] 
+}
G P U R o
ll o
{+u t T i m e [ s ] 
 C P U S p ee d u p ( × )+}
G P U S p ee d u p ( × )
GNN
Ours-
[Figure]
[Table]
[Figure]
particle–based fluid dynamics, and mesh–based deformable cloth simulation.
This work represents a significant step forward in learnable simulation approaches and offers key advantages for obtaining solutions to complex systems in science and engineering
[Figure]
A limitation of our current approach is that collocation is introduced as post-processing to obtain continuous solutions; we plan to explore further the possibility of embedding a learnable spline collocation step within the model.
Moreover, our work does not consider evolving collocation points, where the location of collocation points is adapted during the rollout steps to allow for a more efficient (in the number of nodes) inference of the solution.
A. Supplementary Material on Orthogonal Spline Collocation
We further illustrate the OSC method by providing numerical examples in this section.
A.1. 1-D OSC Example
For simplicity and without loss of generality, we consider the function domain as unit domain [0 , 1] and we set N = 3 , r = 2 , which means we will use a three-order three-piece function to simulate the 1-D ODE problem as shown in Equation.
We firstly choose the partition points as x i , i = 0 , · · · , 3 , x 0 = 0 , x 3 = 1 .
The number of partition points is N + 1 = 4 .
Distance between partition points can be fixed or not fixed.
Then, based on Gauss-Legendre quadrature rule, we choose collocation points.
The number of collocation point within one partition is r − 1 = 1 , so we have in total N × ( r − 1) =
collocation points ξ i , i = 0 · · · , 3 .
After getting partition points and collocation points, we will construct the simulator.
Here we have three partitions; in each partition, we assign a 2 order polynomial
[Equation]
Notice that these three polynomials should be C 1 continuous at the connecting points, i.e. partition points within the domain.
For example, Equation S1a and Equation S1b should be continuous at x 1 , then we can get two equations
[Equation]
For boundary condition
[Equation]
we can also get two equations
[Equation]
And sum up the equations we got so far.
Firstly, our undefined polynomials have N × ( r + 1) = 9 parameters.
The Ccontinuous condition will create ( N − 1) × 2 = 4 equations and the boundary condition will create 2 equations.
Then we have N × ( r − 1) collocation points.
For each collocation point, we substitute it to polynomials to get an equation.
For example, if the ODE in Equation 2 is
[Equation]
By substituting collocation point ξ 0 into the equation, we can get
[Equation]
Now we can know that the number of equations can meet with the degree of freedom of polynomials
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
In this example, generated equations will be constructed to an algebra problem Aa = f where the weight matrix is an almost g matrix as shown in Figure S1.
[Figure]
[Equation]
a =   a
[Equation]
a
[Equation]
a [Equation]
a [Equation] a [Equation]
a
[Equation]


[Equation]
 
[Equation]
[Equation]


[Equation]
(S8b)
To solve this problem, we can get the simulation results.
A.2. 2–D OSC Example
For simplicity and without loss of generality, we consider the function domain as unit domain [0 , 1] ×
and we set
N x = N y
= 2 , r = 3 .
Partition points and collocation points selection are similar with 1-D OSC method, we have
N 2 × ( r − 1) 2 = 16 collocation points in total.
For simplicity, we note the partition points at two dimensions to be the same, i.e., x i , i = 0 , 1 , 2 .
Unlike the 1–D OSC method, we choose Hermit bases to describe as the simulator, which keeps
C 1 continuous.
As a case, the base function at point x 1 would be
[Equation]
We separately assign parameters to basis functions, i.e. H 1 ( x ) = a 1 ,i f 1 ( x ) + b 1 ,i g 1 ( x ) for x variable in [ x 0 , x 1 ] ×
[ y i − 1 , y i ] partition.
Then the polynomial in a partition is the multiple combinations of base functions of two dimensions.
For example, the polynomial in the partition [ x 0 , x 1 ] ×
[ y 0 , y 1 ] is
[Equation]
Now we consider the freedom degree of these polynomials.
From definition, we have 2 n ( r − 1)( n + 1) = 24 parameter.
Consider boundary conditions, we have 24 − 4 × N = 16 parameters.
The number is equal with collocation points
N 2 × ( r − 1) 2 , which means we can get an algebra equation by substituting collocation points.
Solving this equation, we can get the simulator parameters.
We can similarly multiple basis functions and set parameters to the simulation result for the higher dimension OSC method.
And then select partition points and collocation points by the same strategy with 2–D OSC method.
The rest algebra equation generating and solving equations parts will not be different.
A.3. Simple Numerical Example
We set N = 3 , r = 3 to simulate the problem
[Equation]
we can get a simulation solution
[Equation]
Visualization of this simulation results is shown in Figure S2.
[Figure]
B. Supplementary Experimental Results B.1. OSC
We compared the OSC with linear, bilinear, 1–D cubic, and 2–D cubic interpolation methods on four types of problems: 1–D linear, 1–D non-linear, 2–D linear, and 2–D non-linear problems.
In these experiments, we tested different simulator orders of the OSC method.
For example, we set the order of the simulator to 4 for 1–D linear problem and 2 for 2–D linear problem.
When the order of the simulator matches the polynomial order of the real solution, OSC can directly find the real solution.
For non-linear problems, increasing the order of the simulator would be an ideal way to get lower loss.
For example, we set the order of the simulator to 4 for 1–D non-linear problem and 5 for 2–D non-linear problem.
Thanks to the efficient calculation of OSC, even though we use higher–order polynomials to simulate, we use less running time to get results.
[Table]
B.2. Heat Equation
We construct the graph by encoding sample points’ location and value to three features.
The connection of nodes is provided by FEniCS dataset generating, which follows the rule of unit mesh connecting.
These connections would not change during the training.
To construct the neural network, we use 3 layers of message passing neural network; each layer contains 3 MPLs by simple layers.
Before training, we apply independent noise ˜ N (0 , 10 − 4 ) normalization to each node’s value.
During training, we load the training trajectories (single frame) randomly to generate target frames nodes value ( ∆ t = 0 .for
GNN and ∆ t = 0 .
05 for G RAPH S PLINE N ETS ).
The loss is the mean square error between the target frames’ node value and the ground-truth one.
We use the Adam optimizer (Kingma & Ba, 2014) to optimize the loss with a batch size of 2 .
We test the performance of G RAPH S PLINE N ETS and GNN on one heat equation which has centrally initialized, and four boundaries are fixed to 0 (Figure S3).
The ground truth is generated with the resolution 256 × 256 .
G RAPH S PLINE N ETS is trained on 8 × 8 collocation points with a time step ∆ t = 0 .
5 s .
After we get the prediction values of collocation points, we apply the space–oriented OSC to get simulation polynomials in the domain.
Then we can get the same resolution predictions with ground truth.
Meanwhile, we train a GNN model on all the ground truth nodes with a time step ∆ t = 0 .
1 s .
We can see from the table that G RAPH S PLINE N ETS uses less than 10% of GNN’s running time while the loss gap is lower than 5 × 10 − 4 .
In some cases, this error gap even becomes closer.
[-4 − 0 .0 . 
-]
[Figure]
We also test G RAPH S PLINE N ETS on different heat equations.
Figure S4shows one heat equation with zero initialization and dynamic boundary system.
We choose 16 sample points within the domain and get the series of values along time, which is shown in Figure S5.
In this experiment, we fixed the order of time–oriented OSC to 4 .
We can see that our model can always have an excellent simulation for different locations’ sample points.
1–D cubic interpolation method works on less dynamic points.
However, for more complex points, the cubic interpolation method collects a considerable amount of error (Figure S5 (0 . 9 , 0 . 9) , while our approach can still stay convergent.
We discussed the influence of collocation point numbers.
By increasing the number of space–oriented OSC collocation
X 0 .0 .0 .0 .0 .1 . Y 0 . 0 0 .0 .
{+Z − 0 . 6 − 0 .− 0 .0 . 
+}
{+X 0 .0 .0 .0 .0 .1 . Y 0 . 0 0 .0 . Z − 0 . 6 − 0 .− 0 .0 . 
+}
[Figure]
[Table]
points, we can see the error of G RAPH S PLINE N ETS has a fast speed to get close to GNN (Figure S6 left) while keeping the running time short.
In the case of (14 × 14) collocation points, G RAPH S PLINE N ETS has a loss gap lower than 3 × 10 −with GNN but takes only 43% time of GNN.
We also illustrate the error curve of our strategy based on the number of collocation sites to demonstrate its strong convergence capabilities.
B.3. Dam Breaking
We define the input ’velocity’ as average velocity between the current and previous timesteps, calculated for three dimensions p t k x = ( v t k x − v t k − 1 x ) / ∆ t .
This property is encoded to the node feature.
For each node, we find all neighbors within a connectivity radius to connect.
We decide the connectivity radius by setting a maximum connectivity k .
For each node, we calculate the average distance of its k nearest neighbors, then keep this distance during training.
We create the graph structure by using 3 message passing neural network layers, and each one contains 3 MLPs.
Before training, we apply independent noise N (0 , 10 − 3 ) to every node’s vector.
We only do one normalization before training.
During the training, the graph takes several previous trajectories as input, and the output would be the velocity at this time step.
Then we accumulate this output with time to get the position of this node at the next time frame.
We calculate the loss by comparing the position and velocity of each node with the ground truth.
Then apply Adam strategy to optimize the loss.
Some results are shown in Table 2.
B.4. Flag Simulation
We retain most of the experimental settings of the F LAG S IMPLE experiment in (Pfaff et al., 2021).
More specifically, we utilize the same latent vector representations of size 128 at each node and edge and the same number of message passing steps (15).
We utilize a batch size of 2 instead of 1 as in (Pfaff et al., 2021), which we found to be less brittle to training for fewer epochs.
We trained the model for 1M training steps with the Adam optimizer subject to an exponential learning rate
[Figure]
decay from 10 − 4 to 10 − 6 .
We empirically found the choice of training noise to be essential for successful model training.
In particular, we used the same training noise of 1e-3 with the noise correction parameter γ set to 0 .
1 in the one–step baseline
GNN.
For models predicting multiple steps, we experimentally found multiplying the step size ratio by the initial training noise to be successful.
For instance, supposing that the baseline model predicts with a step update of 0 .
02 s , the model with step size 5 (i.e. ∆ t = 0 .
1 s ) will be trained with noise of scale 1e-3 × 5 = 5e-3.
In Figure S7 we show differences between linear interpolation and OSC for a sample point in the flag simulation.
B.5. Hardware and Software
Experiments were carried out on a machine equipped with an I NTEL C ORE I9 7900X CPU with 20 threads and a NVIDIA
RTX 2080 T I graphic card with 11 GB of VRAM.
Software–wise, we used FEniCS
(Logg et al., 2012) for Finite Element simulations for the heat equation experiments; Taichi (Hu et al., 2019) for the dam breaking simulation, while the flag dataset was obtained with ArcSim (Narain et al., 2012).
We implemented a parallelizable routine for the OSC method using the multiprocessing libraries in Python and PyTorch for GPU parallelization (Paszke et al., 2019).
The G RAPH S PLINE N ETS code was written in Python, and PyTorch was used for deep learning while the Deep Graph Library (DGL) (Wang et al., 2020) for graph neural networks.
C. Supplementary Material on Physical Simulations and Deep Learning C.1. Classical Simulators
Classical numerical methods for solving PDEs can be broadly divided into mesh–based and mesh–free approaches.
Meshbased methods notably include the Finite Element Method (FEM) (Zienkiewicz et al., 1977), Finite Volume Method, (Eymard et al., 2000) and deformable materials simulators (e.g., cloth) (Baraff & Witkin, 1998;
Narain et al., 2012).
Their mesh–free counterparts include Molecular Dynamics, (MD) (Rapaport & Rapaport, 2004), the Material Point Method (Bardenhagen & Kober, 2004) for finite particles, and the Smoothed Particle Hydrodynamics (Monaghan, 1992) which
[Figure]
[Figure]
specializes in fluid simulation.
The choice between mesh–free and mesh–based methods is usually dictated by computational efficiency.
For example, while turbulent flow simulations may be represented in a particle-by-particle fashion, describing flows at sample points through a mesh may be desirable, thus noticeably reducing inference times.
Similarly, domains with dynamically changing volumes, such as fluid simulation, can benefit more by mesh–free particle simulators rather than forcing an unnatural mesh description.
Major downsides of classical numerical simulators are that they often require extensive knowledge about the simulated domains as well as taking considerable computational resources and time to infer solutions: deep learning has proven to be a valid and faster alternative to existing numerical solvers (Thuerey et al., 2021).
C.2. Further Intuition on Graph Networks Simulators and OSC
Graph neural networks have intrinsically desirable properties for physical simulation.
Since they can be used on unstructured grids, they are able to represent complex systems with changing connectivity.
Moreover, the graph structurally attains equivariance, while translation equivariance can also be easily achieved by considering relativity in nodes’ positions rather than their absolute values.
Like Molecular Dynamics, our module can construct edges based on each particle neighborhood to capture message passing locality (Anonymous, 2022).
Moreover, similarly to how FEM works, we can consider fixed neighbor connections in mesh space, thus constraining our optimization to a specific geometry.
However, unlike the classical
FEM and MeshGraphNets, G RAPH S PLINE N ETS can infer the space continuously without the need to refine the mesh structure.
Moreover, given collocation points in time, our model can also obtain time–continuous simulation without the need to constrain the rollouts to an iterative solver.