Zero or Infinite Data? Knowledge Synchronized Machine Learning Emulation
Abstract
Even when the mathematical model is known in many applications in computational science and engineering, uncertainties are unavoidable. They are caused by initial conditions, boundary conditions, and so on. As a result, repeated evaluations of a costly model governed by partial differential equations (PDEs) are required, making the computation prohibitively expensive. Recently, neural networks have been used as fast alternatives for propagating and quantifying uncertainties.
Notably, a large amount of high-quality training data is required to train a reliable neural networks-based emulator.
Such ground truth data is frequently gathered in advance by running the numerical solvers that these neural emulators are intended to replace.
But, if the underlying PDEs’ form is available, do we really need training data?
In this paper, we present a principled training framework derived from rigorous and trustworthy scientific simulation schemes.
Unlike traditional neural emulator approaches, the proposed emulator does not necessitate the use of a classical numerical solver to collect training data.
Rather than emulating dynamics directly, it emulates how a specific numerical solver solves PDEs.
The numerical case study demonstrates that the proposed emulator performed well in a variety of testing scenarios.
Introduction
Machine learning (ML) is a component of artificial intelligence and a computational technology that can be trained with data to augment or automate human skills.
Across the science community, there has been a surge in interest in developing ML-based frameworks for climate prediction [27, 33], drug discovery [18, 44, 46], renewable energy development [17, 41], etc.
However, if not well grounded, most ML models are purely data-driven.
One impediment to data-driven modeling is that existing methods frequently do not meet the needs of scientific users.
Consider the following:
Data Scarcity
While ML has achieved remarkable success in data-rich applications, such as image recognition and natural language processing, modeling natural phenomena remains challenging as physical observations usually are limited and sparse.
Worse yet, noiseless ground-truth data typically are unknown, and benchmark data sets are not always available for many scientific problems [13, 48].
Discontinuity
Current ML achievements are heavily reliant on discretized data, such as images of varying resolutions in computer vision.
As a result, traditional model development has mainly centered on learning mappings between finite-dimensional Euclidean spaces or finite sets.
On the other hand, most scientific application data are continuous .
For example, air and water temperatures constantly influence atmospheric circulation patterns [11, 32].
Model Interpretability
Human-interpretable models are required for gaining new scientific insights from data.
However, modern ML models, particularly deep-learning-based models, are prone to over-parameterization.
As a result, these models are frequently criticized for being opaque.
Although important for scientific applications, interpreting these black box models remains challenging [34].
Attempts to answer these questions are commonly referred to as physics-informed machine learning [19, 35, 49].
Here, we rigorously ground ML models using physical principles via trustworthy scientific simulation procedures.
This differs fundamentally from the existing body of research that attempts to combine ML and domain sciences, e.g., by using domain-specific knowledge in ML algorithms in simplistic ways, e.g., extended data loss.
Specifically, this paper’s main contributions are centered on answering the following two questions
• "Can we use explicit knowledge exclusively to train the ML model if the governing equation form is known but there are uncertainties in the initial condition, boundary conditions, or others?"
The proposed learning scheme enables the model to be trained without collecting training data beforehand.
The training is derived from off-the-shelf numerical schemes and is performed in an on-the-fly learning mode.
• "How can we determine and validate that the physics of interest is properly captured by the ML model?
" We use Explainable AI tools to examine the trained model, such as comparing the learned convolution filters with numerical integration coefficients.
Background and Related Work
In many scientific applications, it is reasonable to assume that the observed system follows a partial differential equation (PDE).
Here, we assume the underlying PDE form is known but contains uncertainties in the initial condition, boundary conditions, or others.
Without loss of generality, we are interested in PDEs with a single time dimension t =
[0 , T ] and possibly multiple spatial dimensions x = [ x 1 , x 2 , . .
, x N ] ⊤ ∈ X .
These can be written in the form
[Equation]
where u : [0 , T ] × X → R n is the solution, with initial condition u 0 ( x ) at time t = 0 and boundary conditions B [ u ]( t, x ) = 0 where x is on the boundary ∂ x of the domain X .
Specifically, the boundary operator B includes Dirichlet boundary conditions, where B D [ u ] = u − b D for fixed function b D and Neumann boundary conditions, where B N [ u ] = n ⊤ ∂ x u − b N with n denoting an outward facing normal on ∂ X .
The question now is how to solve Eq.
Section 2.1 examines current efforts from a numerical simulation standpoint, while Section 2.2 reviews related work from an ML emulation standpoint.
2.1 Classic numerical simulation
The method of lines (MOL) is a common technique for solving PDEs stated in Eq.
It chooses N nodes in X and uses algebraic approximations to replace the spatial derivatives in F at these nodes.
The discretization causes F to be approximated by ˆ F , resulting in the following system of ordinary differential equations (ODEs):
[Equation]
where N ( i ) is a collection of indices of neighboring nodes other than i that must be evaluated in order to compute ˆ F at x i and x N ( i ) and u N ( i ) are positions and states of nodes N ( i ) .
By doing so, MOL separates space discretization from time evolution.
ˆ F can be formed in a variety of ways by approximating spatial derivatives on the grid and the ODE u ( t ) = u (0)
+ (cid:82) t 0 ˙ u ( τ ) dτ for a fixed location x i can be solved using standard ODE solvers.
The following is a high-level overview of classical techniques for performing spatial differentiation and temporal integration.
Spatial Differentiation
Traditionally, F is approximated by discretizing the space using traditional solvers such as finite difference methods (FDM) [43], finite volume methods (FVM) [22], and finite element methods (FEM) [15].
To represent the domain X by a union of N ele mesh elements, a set of mesh points are introduced separately in the various space directions.
Once the mesh has been established, it is common practice to fit a piecewise polynomial first and then use the derivatives of the fitted polynomial at any new off-mesh point to obtain spatial differentiation information.
In this paper, we consider finite difference methods, where spatial derivative operators (e.g., ∂ x ) are replaced with difference operators known as stencils in FDM.
For example, ∂ x u k (cid:12)(cid:12) x i can be approximated by (cid:0) u ki +1 − u ki (cid:1) /
( x i +1 − x i ) .
In our proposed learning scheme, such numerical approximation will be replaced by its deep learning counterpart, namely convolution filters.
Temporal Integration To numerically integrate Eq.
(2), the domain is typically discretized by a uniformly partitioned time mesh.
Then, to approximate the exact solution at the mesh points, a time integration scheme is chosen.
In general, we have a time step ∆ t and seek approximations, such as the forward Euler method u i =
1 = u i =
+ ∆ t ˆ F ( u i =0 ) .
The computational algorithm is then applied iteratively for i = 1 , 2 , . .
, N K − 1 and solves the ODEs.
We use autoregressive methods in our proposed learning scheme, which share the same recursive philosophy as such a numerical temporal integration scheme.
A learnable neural surrogate function is used to approximate F .
2.2 Deep learning emulation
Deep learning based models have been used to learn dynamical systems and PDEs in recent years, driven by the desire to reduce computational costs and/or learn new physical models.
The variety of problem settings and applications has resulted in a diverse set of neural PDE emulators.
Here, we review the current work on neural PDE emulators for the temporal PDEs stated in Eq.
(1) from two perspectives: Spatial Prediction and Temporal Projection .
Spatial Prediction Inspired by the great success in computer vision, many neural PDE emulators are based on convolutional neural networks (CNN).
Although CNN-based discrete learning can improve training efficiency significantly, CNNs struggle to handle irregular geometries with unstructured meshes and will require modifications and tuning for different resolutions and discretizations [2, 10, 20, 49].
Another approach is to train a fully-connected multilayer perceptron (MLP) to learn continuous functions (e.g., physics-informed neural networks (PINNs)).
However, these neural PDE emulators are often instance-based and have limited generalizability.
When applied to a new instance, a new neural network must be trained.
To address this issue, advanced PINN variants have recently been proposed [19, 26, 31, 45].
In parallel, graph neural networks (GNNs) have shown excellent performance in dealing with unstructured simulation meshes [30, 36, 38].
In this paper, we use a
GNN-based neural architecture to generate continuous spatial predictions [21, 24].
Temporal Projection Mapping from initial conditions to solutions at time t can be treated as an input-output mapping that can be learned through supervised learning (e.g., PINNs)
It is, however, especially difficult if the goal is to generate long-term trajectories.
Most models, on the other hand, are next-step prediction models, which learn the next state u ( t + 1 , x ) in the trajectory from the current one u ( t, x ) [3, 8, 30, 36].
Nonetheless, attempting to fit a standard ML model, such as a recurrent neural network (RNN), directly on the long-time sequences causes the gradient to explode [28, 39].
One treatment method is to use neural ODEs.
The model is trained using a continuous-time adjoint method, which necessitates numerical integration of the neural network used to capture the system dynamics, making training prohibitively expensive [6, 16].
We attempt to reformulate the training of a next-step prediction model in this paper.
We use autoregressive methods with a chosen ML model to predict the next step and then constrain the prediction to satisfy the numerical integration schemes.
Methodology 3.1 Autoregressive forward prediction
In this work, we are focused on developing a ML surrogate model that can efficiently predict the given dynamical system for time-steps u 1 , u 2 , . .
, u N for a given initial state realization u 0 and a set of boundary conditions.
The forward prediction is performed in an autoregressive fashion.
For example, the input-output relationship of the proposed autoregressive model for prediction ˆ u at a given time index t n +1 is written as follows:
[Equation]
ˆ F is our learnable neural emulator.
It is a function of the k + 1 previous states.
Many numerical time integration algorithms, such as Runge-Kutta methods, suggest that a model is more stable when the hyperparameter k, also known as the window size, is greater than 1.
As a result, the forward prediction problem, particularly the first few steps, can be reparameterized as:
[Equation]
Combining Eq.
(3) and Eq.
(4), the state variable prediction ˆ u at time t n can be written as:
[Equation]
with ◦ denoting the composition operation.
In comparison to the standard numerical time integration method, ˆ F is an easy-to-evaluate neural emulator.
Once ˆ F has been trained, we can make inferences much faster.
And the section that follows discusses how to train our neural emulator.
3.2 Random unrolling training
To train the network, we define the loss function as a sum of residuals at a sequence of time instances:
[Equation]
where θ denotes the learnable parameters of ˆ F .
In a departure from traditional data-driven deep learning approaches, the proposed training does not require a classical numerical solver to collect training data.
Instead, we estimate the solution variable u on the fly using a pre-selected numerical time-integrator.
Specifically, we can (1) use ˆ F to perform forward propagation and calculate ˆ u i
−and ˆ u i ; (2) compel ˆ u i and ˆ u i − 1 to follow a time-stepping method and compute u
i using ˆ u i and ˆ u i − 1 ; and (3) minimize the difference between the model’s predictions ˆ u i and a discrete numerical time-integrator’s estimation u i .
To better illustrate the central idea, let’s consider the standard forward Euler time integration scheme.
Given the prediction at time t i − 1 , the goal is to calculate the state variable at time t :
[Equation]
where S ∆ x ( · ) contains the spatial differentiation information.
Substituting Eq.
(7) into the loss function Eq.
(6), we have
[Equation]
From the standpoint of optimization, Eq.
(8) is analogous to the L 2 minimization of the discretized
PDE residual, where ˆ u i is provided by our neural emulator and ˆ u i − 1 , which represents the previous state, is known.
The only calculation left is for the spatial derivatives included in S ∆ x ( · ) .
Because our emulator is a deep learning-based model, the spatial derivatives can be calculated using either auto-differentiation or numerical approximation [14, 29].
In this paper, we will use the second approach to speed the training up.
The emulator is trained by progressively incorporating loss terms estimated from different time steps to stabilize the training and offer reliable long-term predictions.
If the aim is to forecast the dynamics for the next 20 steps u 1 , . .
, u 20 given the initial state u 0 , we will limit the model to gradually explore the system and learn the dynamics by gradually unrolling the number of time steps it predicts as training advances.
We found that adding randomness to the unrolling number improves performance.
The unrolling number, in particular, can be regarded as a uniformly distributed random number.
We can also update the distribution, using a small number at first and allowing a larger number later.
For example, if N t = 20 , a random realization of the unrolling number can be [2, 1, 3, 2, 4, 3, 5], implying that we only back-propagate 2 time-steps in the first step.
Fig.
1 graphically demonstrates how the loss function is calculated.
[Equation]
[Table]
[Equation]
Experiments 4.1 Experiment setup
PDEs
To highlight the benefits of the knowledge synchronized training strategy, we demonstrate its effectiveness on tasks of varying difficulty.
PDEs considered in this paper can be written as
[Equation]
[Equation]
We have the heat equation if α ̸ = 0 , β = 0 , and the Burgers’ equation if α ̸ = 0 , β ̸ = 0 .
Specifically, the heat equation is used to validate the trained neural emulator’s synchronized physics, and the
Burgers’ equation is used to demonstrate its computational and prediction performance.
Appendix A contains detailed information on the set up of the PDEs.
Task
The task is to emulate the underlying dynamics of the PDEs of interest (Eq. 9) using deep neural networks without using any simulation data.
To make the emulation applicable to deep learning models, the continuous solution u is discretized in both the spatial and temporal domains, resulting in U = { u 0 , u 1 , u 2 , . .
. , u N } , where the initial state u 0 is random and the solution at other time instances must be inferred.
As a result, we pose the emulation of a dynamical system as a forecasting problem.
In other words, rather than learning a single solution, we are interested in learning the set of solutions for a distribution of random parameters, in our case, the random initial conditions.
Models
The proposed training strategy is not constrained by the architecture of neural networks.
Therefore, we experimented with two models: the standard convolutional neural network-based model (ResNet) [12] and the state-of-the-art Fourier Neural Operator (FNO) network [24].
After extensive hyperparameter optimization, we configure the two models as follows: (a) ResNet is a 27-layer residual convolutional neural network.
The final dense layer is a convolutional layer with one output channel for the heat equation example and two output channels for the Burgers’ equation example.
(b) FNO has four layers and two fully connected networks (FCN).
The first FCN lifts the dynamics from R d in to R 20 , and the last FCN projects the dynamics from R 20 to R or R 2 .
Also, each
Fourier layer is parameterized with 12 truncated modes for the matrix-vector multiplication involved in the kernel computation.
Training
As discussed in section 3, emulating a numerical solver in solving a set of PDEs requires two key components: spatial derivative computation and time integration.
Numerically, the finite difference can be used to approximate a derivative to an arbitrary order of accuracy for spatial derivatives.
Lagrange interpolation is commonly used to determine the finite difference coefficients involved in the approximation.
Here, we need to compute the first- and second-order derivatives given the heat/Burgers’ equation.
We use the Sobel and Laplace filters to approximate the first- and second-order derivatives with second-order accuracy to emulate the finite difference approximation [5, 42].
Next, we can quickly assemble the S ∆ x defined in Eq.
7 using calculated spatial derivatives.
The loss function can then be crafted using a predetermined time integration method (e.g., the forward
Euler scheme used in 8).
Appendices B and C contain the detailed calculation procedures.
We generate N train =
8192 training instances and another N test = 512 testing instances for all experiments, where an instance denotes an initial condition.
Note each experiment includes two regions, namely, the training [0 , 0 . 5] and the extrapolation (0 . 5 , 1] .
That is, N t in Eq.
8 is 100, and the trained model is used to predict the next 100 time steps.
Overall, the dynamics are projectedsteps in time by the emulator.
An Adam variant called decoupled weight decay regularization [25] is adopted as the optimizer with a weight decay λ = 0 . 0001 .
Both models are trained for 120 epochs with an initial learning rate of 0 .
001 that decays by γ = 0 .
995 every epoch.
4.2 Prediction performance Qualitative assessment
We begin by providing a qualitative assessment of the prediction performance.
Fig.
2 shows the emulation results of a randomly chosen realization from the test dataset.
Obviously, the adopted FNO model outperforms the state-of-the-art ResNet model trained on the same amount of initial condition realizations.
In Fig.
2, it can be observed that both models are able to provide satisfactory predictions in the predefined training time period.
However, ResNet’s performance degrades significantly in the extrapolation region.
ResNet’s predictions deviate from the ground truth, while the prediction results of FNO still can maintain relatively high accuracy.
These results reflect FNO’s superior ability to absorb underlying physical principles, which supports better generalization of unknown scenarios [23, 30, 36].
[Equation]
[Figure] [Figure]
Quantitative assessment
Next, to quantitatively test the present scheme’s precision, the rootmean-square errors (RMSEs) are reported.
We tested 512 different samples of the fluid realizations.
(a) shows the mean and standard deviation of the calculated RMSEs.
It is observed that
FNO outperforms ResNet by an order of magnitude when it comes to controlling error propagation.
Although both models’ accuracy decreases at a similar rate in the extrapolation region, the FNO model has a significantly lower mean value of RMSEs, particularly in the training region.
Furthermore, the
FNO model has a lower standard deviation than ResNet.
This suggests that the FNO model is more reliable and robust in emulating Burgers’ equation.
Computational efficiency
Here, a study of the proposed model’s computational efficiency is carried out.
Two computation platforms are scheduled: (1) an Intel Xeon Processor E5-2698 v with a system memory of 187 GB and (2) a NVIDIA A100 GPU with 40 GB HBM2 memory.
The total computational cost is classified into two broad categories: training and testing.
First, the computational cost of DL-based emulators is compared against the standard numerical simulation.
The computation time for a single simulation/emulation of the 2D coupled Burgers’ equation on a 64 × 64 spatial gridded domain is shown in Fig.3 (b).
Utilizing the same CPU, the speedup of the
FNO model is approximately more than 100 times compared to the numerical solver for a single run.
More significant saving on the computational effort is achieved if inferences are made using the
GPU device.
A speedup of three to four orders of magnitude is accomplished.
These encouraging results show the potential of using a DL-based emulator for problems involving massive queries of the simulation model.
[Equation]
[Figure]
4.3 Generalization performance Out-of-distribution Prediction
Fig.4 (a) shows the mean RMSE of the emulation results.
While both emulators are affected by out-of-distribution samples, the FNO model has lower errors.
Notably, it is observed that FNO outperforms the ResNet in controlling the error propagation approximately by order of magnitude of 10.
This effect gets more pronounced in the extrapolation region, where the mean RMSEs of the ResNet model increase as the system evolves.
This finding is consistent with the intuitive expectation of using a surrogate model to forecast nonlinear dynamics.
However, FNO is able to stabilize, even reduce, the prediction error within the same time region.
In the meantime, it should be noted that FNO out-of-distribution prediction performance outperforms ResNet in-distribution test performance.
These findings suggest that the FNO model possesses superior generalization performance than ResNet.
Super-resolution task
We summarize the performance of the FNO and ResNet on super-resolution prediction tasks to test the model’s ability to emulate a classic numerical solver [7].
The goal is to evaluate the generalization ability of these two emulators on various discretization meshes.
ResNet and FNO are both trained on 64 × 64 meshes and tested at higher resolutions.
Fig.shows the results.
Of note, the error is defined as the pixel-wise error averaged across all spatial grids of a given time instance.
The emulation results show that because convolution kernels are spatial-agnostic and channel-specific, ResNet fails to provide accurate prediction if additional tuning for different resolutions is not available [9].
When evaluated at a higher resolution, the proposed
FNO model, on the other hand, achieves consistent error.
The promising results show that the FNO has greater generalization potential as the model is tailored to learn operators mapping between infinite-dimensional function spaces rather than standard finite-dimensional Euclidean spaces [36].
[Table]
[Figure]
4.4 Diagnosis of emulated physics
In order to diagnose the trained emulator and compare with the known physics, we employ one feature importance approach [47] for post-hoc local explanations of the behaviors of the emulator.
This class of approach aims at highlighting the input features that contribute to the model prediction.
Specifically, DeepLIFT [40] is adopted to depict the heatmaps showing both the spatial and temporal contributions to the prediction of an interested location.
We demonstrate how the explanation works using the example of heat equation with the fixed k = 3 and a focused location in the center.
As illustrated in Fig.5 (a), three heatmaps represent contributing regions at corresponding timesteps t
i − 3 , t i − 2 , and t i − 1 , where blue shows the positive contribution while red shows the negative contribution.
There are two observations about the spatial contributions: 1) the contributions decrease from the focused location to its periphery; and 2) the blue/red colors form a interweaving pattern that is consistent with the wave propagation.
In Fig.5 (b), we aggregate the absolute values of the contribution scores for each temporal input (blue, red, and green representing t i − 3 , t i − 2 , and t i −respectively) and plot for all the test timesteps ( N = 100 ) so as to compare which temporal channel has more impact to the prediction.
We observe that the closer to the current timestep the more impact it brings, which complies with the physical principles.
Conclusion
In this work, we introduce a knowledge synchronized training strategy for building neural emulators for PDEs.
There is no need to collect simulation data via a typical numerical solver.
Instead, we cast the PDEs into a discretized form.
The loss function is derived from variational principles.
The explicit goal is to minimize the difference between the model’s prediction and a pre-chosen numerical scheme.
This is implicitly analogous to the L 2 minimization of the discretized PDE residual.
The numerical case study shows the proposed emulator performed well in various testing scenarios.
A Numerical Examples
Heat equation
In the first example, a time-dependent diffusion equation on a bounded 2D Cartesian domain Ω = [0 , L ] × [0 , L ] is considered,
[Equation]
with boundary conditions ∂ Ω ,
[Equation]
11) where α is the diffusion coefficient which will be held at α = 0 .
0003 and the domain size set to
L = 1 and t ∈ [0 , 1] .
For this problem we will use homogeneous Dirichlet boundary conditions, i.e., u = 0 on ∂ Ω and assume no external source, i.e., f = 0 .
Burgers’ equation
The two-dimensional coupled viscous Burgers’ equation is the second example we’ll look at ∂u
[Equation]
subject to the initial conditions
[Equation]
and the boundary conditions
[Equation]
B Calculation of spatial derivatives
spatial derivatives included in S ∆ x ( · ) can be effectively estimated using either auto differentiation or numerical approximation.
In this paper, we will take the second approach.
In particular, the Sobel and
Laplace filters are utilized to approximate the first- and second-order derivatives with second-order accuracy [5, 42]
[Equation]
It is worth noting that filter size determines the computational efficiency and accuracy.
Large filters, in general, can approximate differential operators with higher approximation orders.
On the other hand, large filters have more memory overhead and a higher computation cost.
In practice, the trade-off must be balanced [4] [Equation]
C Calculation of time integration θ -Rule
For the heat equation, we used the θ -rule for time discretization.
The key feature of θ -rule time discretization scheme is that we can use one formula to generate a family of well-known.
For example, the Forward Euler scheme, Backward Euler scheme, and Crank-Nicolson scheme are denoted by θ = 0 , θ =
[Equation]
and θ = 1 / 2 , respectively.
On the other hand, standard second-order accurate finite differences are used to compute the spatial derivatives.
As a result, Eq.
10 can be written as:
[Equation]
Collecting the unknowns on the left-hand side, we have
[Equation]
where
[Equation]
F x and F y are the Fourier numbers in x and y direction, respectively.
To initialize the problem, we set ∆ x = ∆ y = 1 / 64 and ∆ t = 0 .
Crank–Nicolson method
To construct a numerical solution for the Burgers’ equation, consider
Eq.
12 on a bounded two-dimensional (2D) Cartesian domain Ω = [0 , L x ] × [0 , L y ] , where
[Equation]
Let nodal points u ni,j and v ni,j be the discrete approximation of u and v at the grid point ( i ∆ x, j ∆ y, n ∆ t ) , respectively.
Because the Burgers’ equations are time-dependent, a robust and precise numerical approach for temporal discretization is required.
Compared to the explicit Euler method, the Crank-Nicolson finite-difference method that allows for the relaxation of strict constraints on the number of time steps and provides more accurate solutions is used.
By applying
Crank–Nicolson scheme to Eq.
12, we get
[Equation]
and
[Equation]
The truncation error of the adopted numerical scheme using the Taylor series expansion is of order
O ((∆ x ) 2 + (∆ y ) 2 + (∆ t ) 2 ) .
D Qualitative emulation performance
We begin by applying the heat equation to the diffusion of a Gaussian hill.
The initial value is set to
[Equation]
(22) for a = 1 on the domain [0 , 1] × [0 , 1] .
For this problem we will use homogeneous Dirichlet boundary conditions u D = 0 .
ϵ 1 and ϵ 2 are random variables generated from a normal distribution.
A numerical solver generates the ground truth data.
To solve time-dependent PDEs using the finite element method, first discretize the time derivative using a finite difference approximation, yielding a sequence of stationary problems [Table] and then convert each stationary problem into a variational formulation.
Fig.shows the prediction results from the FNO and ResNet.
ResNet with a hierarchical convolution layer provides better error distribution because the diffusion process can be very local, but FNO performs better in terms of magnitude
[Table]
[Figure]
To investigate the capabilities of FNO and ResNet further, we change the initial condition to a uniform distribution.
Fig.7 shows the emulation results.
The first row displays the predicted dynamics of the emulator, the second row displays the results of numerical simulation, and the third row displays the absolute error between numerical simulation and deep learning emulation.
The results clearly show that FNO outperforms ResNet in simulating the diffusion process.
For FNO, the error is smaller and grows gradually over time, whereas for ResNet, the error jumps to a higher range and stabilizes in most regions.
[Table]
+ukJrz2J1wmqUHJFovCVBATk9nfpM8VMiPGllCmuL2VsCFVlBmbTsGG4C2/vEqaFxXvslK9r5ZqN1kceTiBUyiDB1dQgzuoQwMYDOAZXuHNEc6L8+58LFpzTjZzDH/gfP4AjR6NVA==</latexit> ( b )
Fig.8 shows the FNO’s prediction results of a randomly chosen realization from the Burgers’ equation test dataset.
The target velocity field was obtained by FEniCS [1], an open-source library for finite element simulations.
The emulator was trained on 8192 realizations.
To evaluate the results, the absolute error between the target and prediction is calculated and presented in the third column of two subfigures.
Overall, the FNO model accurately predict the complex fluid structures.
The predictions of FNO match the reference values well in all time instances.
[Equation]
[Figure]
Furthermore, the proposed FNO model outperforms the state-of-the-art ResNet model trained on the same amount of fluid realizations.
In Fig.9, it can be observed that both models are able to provide satisfactory predictions in the predefined training time period, namely, t ∈ [0 , 0 . 5] .
However, the performance of ResNet degrades significantly in the extrapolation region, that is, t ∈ (0 . 5 , 1] .
The
ResNet’s predictions deviate from the ground truth while the prediction results of FNO can still maintain relatively high accuracy.
Such comparison results reflect FNO’s superior ability to absorb underlying physical principles, which supports a better generalization of unknown scenarios.
[Table]
+sZnbym/v7O7tFw4OGzpOFcM6i0WsWgHVKLjEuuFGYCtRSKNAYDMY3k785iMqzWP5YEYJ+hHtSx5yRo2V7kvBWbdQdMvuFGSZeHNShDmq3cJXpxezNEJpmKBatz03MX5GleFM4DjfSTUmlA1pH9uWShqh9rPpqWNyapUeCWNlSxoyVX9PZDTSehQFtjOiZqAXvYn4n9dOTXjjZ1wmqUHJZovCVBATk8nfpMcVMiNGllCmuL2VsAFVlBmbTt6G4C2+vEwa52XvqnxRuyxWak+zOHJwDCdQAg+uoQJ3UIU6MOjDM7zCmyOcF+fd+Zi1rjjzCI/gD5zPH7dPjeI=</latexit> ( b )
E Quantitative emulation performance
Additionally, the quality of the ResNet/FNO predictions are quantitatively evaluated based on the following relative error metric:
[Equation]
and the coefficient of determination, also known as the R 2 score, which is defined as:
[Equation]
where ¯ u true is the mean of u k true .
The 500 test realizations are statistically analyzed using these two metrics.
Table 1 shows the mean and variance of the calculated relative L 2 error and R 2 score.
Again, the FNO model produces a far less relative L 2 error than the ResNet model.
Also, the R 2 score of the
FNO model is closer to 1 than the ResNet model.
All of these findings indicate that the predicted velocity fields from the FNO match the references better than those from the ResNet.
F Computational efficiency
Because the proposed FNO model is mesh-free, greater predictive efficiency is expected when applied to a more refined spatial discretization.
To verify this, we further carried out the quanti- tatively analysis on resolutions of 128 × 128 , 256 × 256 , and 512 × 512 .
Note the ground truth solver (FEniCS) does not fully support GPUs and has varying options of linear solver and pre- conditioner, for example, conjugate gradient method vs. generalized minimal residual method.
Interest here was to provide a hardware-specific performance comparison.
The computation time of both FEM-based and FNO-based simulations is shown in Fig.10 (a).
Compared to the classical FEM-based solver, the proposed FNO-based emulator scales very well.
Evaluating our model on the same CPU hardware can still achieve speedups between 100x-4570x.
Users with access to a GPU could further benefit from an approximate speedup of 60x.
Meanwhile, we investigated FNO and ResNet scaling performance, which is defined as a function of the number of initial conditions generated for training.
Fig.10 (b) depicts the results.
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
+53VSE177Ey6T1KBki0VhKoiJyexv0ucKmRFjSyhT3N5K2JAqyoxNp2BD8JZfXiXNi4p3WaneV0u1myyOPJzAKZTBgyuowR3UoQEMBvAMr/DmCOfFeXc+Fq05J5s5hj9wPn8Ai5mNUw==</latexit>
[Figure]
<latexit sha1_base64="KWh0RLJ0bw8em/x3PU2+HIlN2FQ=">AAAB6nicbVDLSgNBEOyNrxhfUY9eBoMQL2FXgnoMevEY0TwgWcLspDcZMju7zMwKIeQTvHhQxKtf5M2/cZLsQRMLGoqqbrq7gkRwbVz328mtrW9sbuW3Czu7e/sHxcOjpo5TxbDBYhGrdkA1Ci6xYbgR2E4U0igQ2ApGtzO/9YRK81g+mnGCfkQHkoecUWOlh3Jw3iuW3Io7B1klXkZKkKHeK351+zFLI5SGCap1x3MT40+oMpwJnBa6qcaEshEdYMdSSSPU/mR+6pScWaVPwljZkobM1d8TExppPY4C2xlRM9TL3kz8z+ukJrz2J1wmqUHJFovCVBATk9nfpM8VMiPGllCmuL2VsCFVlBmbTsGG4C2/vEqaFxXvslK9r5ZqN1kceTiBUyiDB1dQgzuoQwMYDOAZXuHNEc6L8+58LFpzTjZzDH/gfP4AjR6NVA
[Figure]
</
[Figure]
G Out-of-distribution prediction
For the Burgers’ equation, randomness is introduced to the initial conditions to better model the target system’s stochastic nature, for instance, the presumed chaos in turbulence.
Because Fourier series has been applied to approximate the physical turbulence in many studies, we formalize the random initial states using a truncated Fourier series with random coefficients a , b , and c ∈ R[-
-]
[Equation]
Originally, random coefficients are defined as:
[Equation]
Here, we investigate the emulators’ generalization performance to out-of-distribution initial conditions.
We take the emulators trained on random inputs defined in Eq. 25 and test them on new input distributions:
[Equation]
Fig.11 depicts samples generated from the training distribution as well as the unknown testing distribution.
It can be clearly seen that the newly generated initial conditions are very different from those used in model training.
[Figure]
[Equation] [Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure] [Equation] [Figure] [Equation]
[Figure]
[Equation]
[Figure]
[Table]
+hHtSx5yRo2V7kvBWbdQdMvuFGSZeHNShDmq3cJXpxezNEJpmKBatz03MX5GleFM4DjfSTUmlA1pH9uWShqh9rPpqWNyapUeCWNlSxoyVX9PZDTSehQFtjOiZqAXvYn4n9dOTXjjZ1wmqUHJZovCVBATk8nfpMcVMiNGllCmuL2VsAFVlBmbTt6G4C2+vEwa52XvqnxRuyxWak+zOHJwDCdQAg+uoQJ3UIU6MOjDM7zCmyOcF+fd+Zi1rjjzCI/gD5zPH7dPjeI=</latexit> ( b )
Similar to Fig.4 (a), the error propagation process is presented in Fig.12.
Overall, the FNO model has better generalization than the ResNet model.
It generalizes well to new initial conditions, which is completely different from the training realizations.
Though the accuracy of both models decreases at a very much similar rate in the extrapolation region, the FNO model has a significantly lower prediction error, particularly in the training region.
This indicates that the FNO model has strong flexibility and robustness when emulating high-dimensional advection systems with strongly nonlinear characteristics
[Figure]
[Figure]
Note each prediction experiment includes two regions, namely, the training [0 , 0 . 5] and the extrapolation (0 . 5 , 1] .
Fig.12 shows the prediction performance for a randomly chosen realization with time index specified to 0 .
25 for training and 0 .
75 for extrapolation.
It is found that the predictions of
FNO are almost unaffected by the out-of-distribution samples, while the ResNet model is seriously deteriorated and produces worse predictions, especially in the extrapolation region.
Also, the error contour of the ResNet model shows the bulk of errors are clustered on the leading face of the waves, which is specifically where the extreme values are located.
However, the errors of the FNO predictions are relatively small and evenly distributed [Table] and there is no clear high-error area in the predictions.
Such findings indicate that the FNO model possesses superior robustness to ResNet.
[Table]
[Figure]
H Super resolution prediction
Here [Figure] we summarize the performance of the aforementioned two models on super-resolution prediction tasks.
The purpose of this assessment is to demonstrate the unique ability of the proposed
FNO model, that is, to generate continuous spatiotemporal solutions.
Unlike many convolutional neural networks (CNNs) based approaches, which learn an end-to-end mapping between the low/highresolution images, the FNO model is trained only with low-resolution data.
Due to the inherent properties of convolution kernels, namely, spatial-agnostic and channel-specific, the ResNet fails to provide accurate prediction if additional tuning for different resolutions is not available.
On the other hand, the proposed FNO model achieves consistent error when evaluated at a higher resolution.
It is
[Figure]
worth mentioning that the error is defined as the pixel-wise error, averaging over both the spatial and the temporal domain.
The encouraging results (Fig.14) show great promise of the proposed model for fast emulating high-dimensional spatiotemporal dynamics.
I Notes on data saturation
Because we make direct use of the physics principles, i.e., governing equations in our case, both
ResNet and FNO do not require a classical numerical solver such as a finite element simulator to generate data for training.
Instead, the proposed knowledge synchronized learning framework begins with the initial conditions of the problem, which can be easily generated from a predefined distribution.
Intuitively, one would expect the emulation quality of the model to improve when the number of the initial conditions increases.
We can, in theory, generate infinite initial conditions and push the model to obey the constraints of the problem on all of them.
However, we have to find a suitable number in practice considering constraints posed by the available time and computational resources.
Viewing from the model capacity, another point worth mentioning is that the model may saturate rather than continuing to draw physics from the data.
To verify this concern, the emulator was trained using sample sets with 256, 512, 1024, 2048, 4096, 8192, 16384, and 32768 realizations.
Fig.15 shows the computed results evaluated on a test set of 512 realizations.
For both models, the error decreases as the amount of training data increases, indicating additional data is contributing to training.
The improvement brought by the data is more evident at the beginning, where a small amount of information is used (e.g., 512 vs. 1024).
Though the data size keeps increasing by a factor of 2, the performance enhancement slows down.
In the case in which 32768 samples were utilized for training, the performance is almost the same and partially worse than the case wheresamples were used, implying the model may reach a saturation state.
Also, it should be addressed that the FNO model can achieve the same accuracy using fewer data samples.
After training ofepochs, the prediction performance measured by the mean MSE of different models shows apparent differences.
The FNO model offers a more compelling ability to embedding physics to learn the underlying spatiotemporal dynamics.
[Table] [Figure]