Zero or Infinite Data? Knowledge Synchronized Machine Learning Emulation
Anonymous Author(s)
AffiliationAddress
Abstract
Neural networks have recently been used as fast alternatives for emulating PDEsgoverned systems.
Notably, a large amount of training data is required to traina reliable neural networks-based emulator.
But, if the underlying PDEs’ form isavailable, do we really need training data?
In this paper, we present a principledtraining framework derived from off-the-shelf scientific simulation schemes.
Unliketraditional neural emulator approaches, the proposed emulator does not necessitatethe use of a classical numerical solver to collect training data.
Rather than emulatingdynamics directly, it emulates how a specific numerical solver solves PDEs.
Thenumerical case study demonstrates that the proposed emulator performed well in avariety of testing scenarios.
Introduction
Machine learning (ML) is a component of artificial intelligence and a computational technology thatcan be trained with data to augment or automate human skills.
Across the science community, therehas been a surge in interest in developing ML-based frameworks for climate prediction [27, 33], drugdiscovery [18, 44, 46], renewable energy development [17, 41], etc.
However, if not well grounded,most ML models are purely data-driven.
One impediment to data-driven modeling is that existingmethods frequently do not meet the needs of scientific users.
Consider the following:
Data Scarcity
While ML has achieved remarkable success in data-rich applications, such as imagerecognition and natural language processing, modeling natural phenomena remains challenging asphysical observations usually are limited and sparse.
Worse yet, noiseless ground-truth data typicallyare unknown, and benchmark data sets are not always available for many scientific problems [13, 48].
Discontinuity
Current ML achievements are heavily reliant on discretized data, such as imagesof varying resolutions in computer vision.
As a result, traditional model development has mainlycentered on learning mappings between finite-dimensional Euclidean spaces or finite sets.
On theother hand, most scientific application data are continuous .
For example, air and water temperaturesconstantly influence atmospheric circulation patterns [11, 32].
Model
Interpretability
Human-interpretable models are required for gaining new scientific insights from data.
However, modern ML models, particularly deep-learning-based models, are prone toover-parameterization.
As a result, these models are frequently criticized for being opaque.
Althoughimportant for scientific applications, interpreting these black box models remains challenging [34].
Attempts to answer these questions are commonly referred to as physics-informed machine learning[19, 35, 49].
Here, we rigorously ground ML models using physical principles via trustworthy scientific simulation procedures.
This differs fundamentally from the existing body of research thatattempts to combine ML and domain sciences, e.g., by using domain-specific knowledge in MLalgorithms in simplistic ways, e.g., extended data loss.
Specifically, this paper’s main contributionsare centered on answering the following two questions.
• "Can we use explicit knowledge exclusively to train the ML model if the governing equation
form is known but there are uncertainties in the initial condition, boundary conditions, or
others?"
The proposed learning scheme enables the model to be trained without collecting
training data beforehand.
The training is derived from off-the-shelf numerical schemes andis performed in an on-the-fly learning mode.
• "How can we determine and validate that the physics of interest is properly captured by theML model?
" We use Explainable AI tools to examine the trained model, such as comparing
the learned convolution filters with numerical integration coefficients
Background and Related Work
In many scientific applications, it is reasonable to assume that the observed system follows a partialdifferential equation (PDE).
Here, we assume the underlying PDE form is known but containsuncertainties in the initial condition, boundary conditions, or others.
Without loss of generality,we are interested in PDEs with a single time dimension t =
[0 , T ] and possibly multiple spatialdimensions x = [ x 1 , x 2 , . . .
, x N ] ⊤ ∈ X .
These can be written in the form
[Equation] [Equation]
(1) where u : [0 , T ] × X → R n is the solution, with initial condition u 0 ( x ) at time t = 0 and boundaryconditions B [ u ]( t, x ) = 0 where x is on the boundary ∂ x of the domain X .
Specifically, the boundaryoperator B includes Dirichlet boundary conditions, where B D [ u ] = u − b D for fixed function b Dand Neumann boundary conditions, where B N [ u ] = n ⊤ ∂ x u − b N with n denoting an outward facingnormal on ∂ X .
The question now is how to solve Eq.
Section 2.1 examines current efforts from a numericalsimulation standpoint, while Section 2.2 reviews related work from an ML emulation standpoint.
2.1 Classic numerical simulation
The method of lines (MOL) is a common technique for solving PDEs stated in Eq.
Itchooses N nodes in X and uses algebraic approximations to replace the spatial derivatives in F atthese nodes.
The discretization causes F to be approximated by ˆ F , resulting in the following systemof ordinary differential equations (ODEs):
[Equation]
where N ( i ) is a collection of indices of neighboring nodes other than i that must be evaluated inorder to compute ˆ F at x i and x N ( i ) and u N ( i ) are positions and states of nodes N ( i ) .
By doingso, MOL separates space discretization from time evolution.
ˆ F can be formed in a variety of waysby approximating spatial derivatives on the grid and the ODE u ( t ) = u (0)
+ (cid:82) t 0 ˙ u ( τ ) dτ for a fixedlocation x i can be solved using standard ODE solvers.
The following is a high-level overview ofclassical techniques for performing spatial differentiation and temporal integration.
Spatial Differentiation
Traditionally, F is approximated by discretizing the space using traditionalsolvers such as finite difference methods (FDM) [43], finite volume methods (FVM) [22], and finite element methods (FEM) [15].
To represent the domain X by a union of N ele mesh elements, a setof mesh points are introduced separately in the various space directions.
Once the mesh has beenestablished, it is common practice to fit a piecewise polynomial first and then use the derivatives ofthe fitted polynomial at any new off-mesh point to obtain spatial differentiation information.
In thispaper, we consider finite difference methods, where spatial derivative operators (e.g., ∂ x ) are replacedwith difference operators known as stencils in FDM.
For example, ∂ x u k (cid:12)(cid:12) x i can be approximated by 76 (cid:0) u ki +1 − u ki (cid:1) /
( x i +1 − x i ) .
In our proposed learning scheme, such numerical approximation will bereplaced by its deep learning counterpart, namely convolution filters.
Temporal Integration To numerically integrate Eq.
(2), the domain is typically discretized by auniformly partitioned time mesh.
Then, to approximate the exact solution at the mesh points, a timeintegration scheme is chosen.
In general, we have a time step ∆ t and seek approximations, such asthe forward Euler method u i =
1 = u i =
+ ∆ t ˆ F ( u i =0 ) .
The computational algorithm is then appliediteratively for i = 1 , 2 , . .
, N K − 1 and solves the ODEs.
We use autoregressive methods in ourproposed learning scheme, which share the same recursive philosophy as such a numerical temporalintegration scheme.
A learnable neural surrogate function is used to approximate F .
2.2 Deep learning emulation
Deep learning based models have been used to learn dynamical systems and PDEs in recent years,driven by the desire to reduce computational costs and/or learn new physical models.
The variety ofproblem settings and applications has resulted in a diverse set of neural PDE emulators.
Here, wereview the current work on neural PDE emulators for the temporal PDEs stated in Eq.
(1) from twoperspectives: Spatial Prediction and Temporal Projection .
Spatial Prediction Inspired by the great success in computer vision, many neural PDE emulators arebased on convolutional neural networks (CNN).
Although CNN-based discrete learning can improvetraining efficiency significantly, CNNs struggle to handle irregular geometries with unstructuredmeshes and will require modifications and tuning for different resolutions and discretizations [2,10, 20, 49].
Another approach is to train a fully-connected multilayer perceptron (MLP) to learncontinuous functions (e.g., physics-informed neural networks (PINNs)).
However, these neural PDEemulators are often instance-based and have limited generalizability.
When applied to a new instance,a new neural network must be trained.
To address this issue, advanced PINN variants have recentlybeen proposed [19, 26, 31, 45].
In parallel, graph neural networks (GNNs) have shown excellentperformance in dealing with unstructured simulation meshes [30, 36, 38].
In this paper, we use a
GNN-based neural architecture to generate continuous spatial predictions [21, 24].
Temporal Projection Mapping from initial conditions to solutions at time t can be treated as aninput-output mapping that can be learned through supervised learning (e.g., PINNs)
It is, however, especially difficult if the goal is to generate long-term trajectories.
Most models, on theother hand, are next-step prediction models, which learn the next state u ( t + 1 , x ) in the trajectoryfrom the current one u ( t, x ) [3, 8, 30, 36].
Nonetheless, attempting to fit a standard ML model,such as a recurrent neural network (RNN), directly on the long-time sequences causes the gradientto explode [28, 39].
One treatment method is to use neural ODEs.
The model is trained using acontinuous-time adjoint method, which necessitates numerical integration of the neural network usedto capture the system dynamics, making training prohibitively expensive [6, 16].
We attempt toreformulate the training of a next-step prediction model in this paper.
We use autoregressive methodswith a chosen ML model to predict the next step and then constrain the prediction to satisfy thenumerical integration schemes.
Methodology
3.1 Autoregressive forward prediction
In this work, we are focused on developing a ML surrogate model that can efficiently predict thegiven dynamical system for time-steps u 1 , u 2 , . .
, u N for a given initial state realization u 0 and a set of boundary conditions.
The forward prediction is performed in an autoregressive fashion.
Forexample, the input-output relationship of the proposed autoregressive model for prediction ˆ u at agiven time index t n +1 is written as follows:
[Equation]
) ˆ F is our learnable neural emulator.
It is a function of the k + 1 previous states.
Many numericaltime integration algorithms, such as Runge-Kutta methods, suggest that a model is more stable whenthe hyperparameter k, also known as the window size, is greater than 1.
As a result, the forwardprediction problem, particularly the first few steps, can be reparameterized as:
[Equation]
Combining Eq.
(3) and Eq.
(4), the state variable prediction ˆ u at time t n can be written as:
[Equation]
with ◦ denoting the composition operation.
In comparison to the standard numerical time integrationmethod, ˆ F is an easy-to-evaluate neural emulator.
Once ˆ F has been trained, we can make inferencesmuch faster.
And the section that follows discusses how to train our neural emulator.
3.2 Random unrolling training
To train the network, we define the loss function as a sum of residuals at a sequence of time instances:
[Equation] [Equation]
where θ denotes the learnable parameters of ˆ F .
In a departure from traditional data-driven deeplearning approaches, the proposed training does not require a classical numerical solver to collecttraining data.
Instead, we estimate the solution variable u on the fly using a pre-selected numericaltime-integrator.
Specifically, we can (1) use ˆ F to perform forward propagation and calculate ˆ u i −
1and ˆ u i ; (2) compel ˆ u i and ˆ u i − 1 to follow a time-stepping method and compute u
i using ˆ u i andˆ u i − 1 ; and (3) minimize the difference between the model’s predictions ˆ u i and a discrete numericaltime-integrator’s estimation u i .
To better illustrate the central idea, let’s consider the standard forward Euler time integration scheme.
Given the prediction at time t i − 1 , the goal is to calculate the state variable at time t :
[Equation]
where S ∆ x ( · ) contains the spatial differentiation information.
Substituting Eq.
(7) into the lossfunction Eq.
(6), we have
[Equation]
From the standpoint of optimization, Eq.
(8) is analogous to the L 2 minimization of the discretized
PDE residual, where ˆ u i is provided by our neural emulator and ˆ u i − 1 , which represents the previousstate, is known.
The only calculation left is for the spatial derivatives included in S ∆ x ( · ) .
Becauseour emulator is a deep learning-based model, the spatial derivatives can be calculated using eitherauto-differentiation or numerical approximation [14, 29].
In this paper, we will use the secondapproach to speed the training up.
The emulator is trained by progressively incorporating loss terms estimated from different timesteps to stabilize the training and offer reliable long-term predictions.
If the aim is to forecast thedynamics for the next 20 steps u 1 , . .
, u 20 given the initial state u 0 , we will limit the model togradually explore the system and learn the dynamics by gradually unrolling the number of time stepsit predicts as training advances.
We found that adding randomness to the unrolling number improvesperformance.
The unrolling number, in particular, can be regarded as a uniformly distributed randomnumber.
We can also update the distribution, using a small number at first and allowing a largernumber later.
For example, if N t = 20 , a random realization of the unrolling number can be [2, 1,3, 2, 4, 3, 5], implying that we only back-propagate 2 time-steps in the first step.
Fig.
1 graphicallydemonstrates how the loss function is calculated.
[Equation]
[Table]
[Figure]
Experiments
[Figure]
.1 Experiment setupPDEs
To highlight the benefits of the knowledge synchronized training strategy, we demonstrateits effectiveness on tasks of varying difficulty.
PDEs considered in this paper can be written as
[Equation]
We have the heat equation if α ̸ = 0 , β = 0 , and the Burgers’ equation if α ̸ = 0 , β ̸ = 0 .
Specifically,the heat equation is used to validate the trained neural emulator’s synchronized physics, and the
Burgers’ equation is used to demonstrate its computational and prediction performance.
Appendix Acontains detailed information on the set up of the PDEs.
Task
The task is to emulate the underlying dynamics of the PDEs of interest (Eq. 9) using deepneural networks without using any simulation data.
To make the emulation applicable to deep learningmodels, the continuous solution u is discretized in both the spatial and temporal domains, resultingin U = { u 0 , u 1 , u 2 , . .
. , u N } , where the initial state u 0 is random and the solution at other timeinstances must be inferred.
As a result, we pose the emulation of a dynamical system as a forecastingproblem.
In other words, rather than learning a single solution, we are interested in learning the set ofsolutions for a distribution of random parameters, in our case, the random initial conditions.
Models
The proposed training strategy is not constrained by the architecture of neural networks.
Therefore, we experimented with two models: the standard convolutional neural network-based model (ResNet) [12] and the state-of-the-art Fourier Neural Operator (FNO) network [24].
Afterextensive hyperparameter optimization, we configure the two models as follows: (a) ResNet is a27-layer residual convolutional neural network.
The final dense layer is a convolutional layer withone output channel for the heat equation example and two output channels for the Burgers’ equationexample.
(b) FNO has four layers and two fully connected networks (FCN).
The first FCN lifts thedynamics from R d in to R 20 , and the last FCN projects the dynamics from R 20 to R or R 2 .
Also, each
Fourier layer is parameterized with 12 truncated modes for the matrix-vector multiplication involvedin the kernel computation.
Training
As discussed in section 3, emulating a numerical solver in solving a set of PDEs requirestwo key components: spatial derivative computation and time integration.
Numerically, the finitedifference can be used to approximate a derivative to an arbitrary order of accuracy for spatialderivatives.
Lagrange interpolation is commonly used to determine the finite difference coefficientsinvolved in the approximation.
Here, we need to compute the first- and second-order derivativesgiven the heat/Burgers’ equation.
We use the Sobel and Laplace filters to approximate the first- andsecond-order derivatives with second-order accuracy to emulate the finite difference approximation[5, 42].
Next, we can quickly assemble the S ∆ x defined in Eq.
7 using calculated spatial derivatives.
The loss function can then be crafted using a predetermined time integration method (e.g., the forward
Euler scheme used in 8).
Appendices B and C contain the detailed calculation procedures.
We generate N train =
8192 training instances and another N test = 512 testing instances for allexperiments, where an instance denotes an initial condition.
Note each experiment includes tworegions, namely, the training [0 , 0 . 5] and the extrapolation (0 . 5 , 1] .
That is, N t in Eq.
8 is 100, andthe trained model is used to predict the next 100 time steps.
Overall, the dynamics are projected 200steps in time by the emulator.
An Adam variant called decoupled weight decay regularization [25] isadopted as the optimizer with a weight decay λ = 0 . 0001 .
Both models are trained for 120 epochswith an initial learning rate of 0 .
001 that decays by γ = 0 .
995 every epoch.
4.2 Prediction performance
Qualitative assessment
We begin by providing a qualitative assessment of the prediction performance.
Fig.
2 shows the emulation results of a randomly chosen realization from the test dataset.
Obviously, the adopted FNO model outperforms the state-of-the-art ResNet model trained on thesame amount of initial condition realizations.
In Fig.
2, it can be observed that both models areable to provide satisfactory predictions in the predefined training time period.
However, ResNet’sperformance degrades significantly in the extrapolation region.
ResNet’s predictions deviate from theground truth, while the prediction results of FNO still can maintain relatively high accuracy.
Theseresults reflect FNO’s superior ability to absorb underlying physical principles, which supports bettergeneralization of unknown scenarios [23, 30, 36]. [Equation] [Figure] [Figure]
Quantitative assessment
Next, to quantitatively test the present scheme’s precision, the rootmean-square errors (RMSEs) are reported.
We tested 512 different samples of the fluid realizations
[Figure]
Fig.3 (a) shows the mean and standard deviation of the calculated RMSEs.
It is observed that
FNO outperforms ResNet by an order of magnitude when it comes to controlling error propagation.
Although both models’ accuracy decreases at a similar rate in the extrapolation region, the FNO modelhas a significantly lower mean value of RMSEs, particularly in the training region.
Furthermore, the
FNO model has a lower standard deviation than ResNet.
This suggests that the FNO model is morereliable and robust in emulating Burgers’ equation.
Computational efficiency
Here, a study of the proposed model’s computational efficiency iscarried out.
Two computation platforms are scheduled: (1) an Intel Xeon Processor E5-2698 v3with a system memory of 187 GB and (2) a NVIDIA A100 GPU with 40 GB HBM2 memory.
The total computational cost is classified into two broad categories: training and testing.
First, thecomputational cost of DL-based emulators is compared against the standard numerical simulation.
The computation time for a single simulation/emulation of the 2D coupled Burgers’ equation on a64 × 64 spatial gridded domain is shown in Fig.3 (b).
Utilizing the same CPU, the speedup of the
FNO model is approximately more than 100 times compared to the numerical solver for a singlerun.
More significant saving on the computational effort is achieved if inferences are made using the
GPU device.
A speedup of three to four orders of magnitude is accomplished.
These encouragingresults show the potential of using a DL-based emulator for problems involving massive queries ofthe simulation model.
[Equation]
[Figure]
4.3 Generalization performanceOut-of-distribution Prediction
Fig.4 (a) shows the mean RMSE of the emulation results.
Whileboth emulators are affected by out-of-distribution samples, the FNO model has lower errors.
Notably,it is observed that FNO outperforms the ResNet in controlling the error propagation approximatelyby order of magnitude of 10.
This effect gets more pronounced in the extrapolation region, where themean RMSEs of the ResNet model increase as the system evolves.
This finding is consistent with theintuitive expectation of using a surrogate model to forecast nonlinear dynamics.
However, FNO is ableto stabilize, even reduce, the prediction error within the same time region.
In the meantime, it shouldbe noted that FNO out-of-distribution prediction performance outperforms ResNet in-distributiontest performance.
These findings suggest that the FNO model possesses superior generalizationperformance than ResNet.
Super-resolution task
We summarize the performance of the FNO and ResNet on super-resolutionprediction tasks to test the model’s ability to emulate a classic numerical solver [7].
The goalis to evaluate the generalization ability of these two emulators on various discretization meshes.
ResNet and FNO are both trained on 64 × 64 meshes and tested at higher resolutions.
Fig.
5shows the results.
Of note, the error is defined as the pixel-wise error averaged across all spatialgrids of a given time instance.
The emulation results show that because convolution kernels arespatial-agnostic and channel-specific, ResNet fails to provide accurate prediction if additional tuningfor different resolutions is not available [9].
When evaluated at a higher resolution, the proposed
FNO model, on the other hand, achieves consistent error.
The promising results show that the FNOhas greater generalization potential as the model is tailored to learn operators mapping betweeninfinite-dimensional function spaces rather than standard finite-dimensional Euclidean spaces [36].
[Table]
[Figure]
4.4 Diagnosis of emulated physics
In order to diagnose the trained emulator and compare with the known physics, we employ onefeature importance approach [47] for post-hoc local explanations of the behaviors of the emulator.
This class of approach aims at highlighting the input features that contribute to the model prediction.
Specifically, DeepLIFT [40] is adopted to depict the heatmaps showing both the spatial and temporalcontributions to the prediction of an interested location.
We demonstrate how the explanation worksusing the example of heat equation with the fixed k = 3 and a focused location in the center.
As illustrated in Fig.5 (a), three heatmaps represent contributing regions at corresponding timestepst i − 3 , t i − 2 , and
t i − 1 , where blue shows the positive contribution while red shows the negativecontribution.
There are two observations about the spatial contributions: 1) the contributions decreasefrom the focused location to its periphery; and 2) the blue/red colors form a interweaving patternthat is consistent with the wave propagation.
In Fig.5 (b), we aggregate the absolute values of thecontribution scores for each temporal input (blue, red, and green representing t i − 3 , t i − 2 , and t i − 1respectively) and plot for all the test timesteps ( N = 100 ) so as to compare which temporal channel has more impact to the prediction.
We observe that the closer to the current timestep the more impactit brings, which complies with the physical principles.
Conclusion
In this work, we introduce a knowledge synchronized training strategy for building neural emulatorsfor PDEs.
There is no need to collect simulation data via a typical numerical solver.
Instead, wecast the PDEs into a discretized form.
The loss function is derived from variational principles.
Theexplicit goal is to minimize the difference between the model’s prediction and a pre-chosen numericalscheme.
This is implicitly analogous to the L 2 minimization of the discretized PDE residual.
Thenumerical case study shows the proposed emulator performed well in various testing scenarios.
A Numerical ExamplesHeat equation
In the first example, a time-dependent diffusion equation on a bounded 2D Cartesiandomain Ω = [0 , L ] × [0 , L ] is considered,
[Equation]
with boundary conditions ∂ Ω ,
[Equation]
11) where α is the diffusion coefficient which will be held at α = 0 .
0003 and the domain size set to
L = 1 and t ∈ [0 , 1] .
For this problem we will use homogeneous Dirichlet boundary conditions, i.e.,u = 0 on ∂ Ω and assume no external source, i.e., f = 0 .
Burgers’ equation
The two-dimensional coupled viscous Burgers’ equation is the second examplewe’ll look at
[Equation] [Equation]
subject to the initial conditions
[Equation] [Equation]
and the boundary conditions
[Equation]
B Calculation of spatial derivatives
spatial derivatives included in S ∆ x ( · ) can be effectively estimated using either auto differentiation ornumerical approximation.
In this paper, we will take the second approach.
In particular, the Sobel and
Laplace filters are utilized to approximate the first- and second-order derivatives with second-orderaccuracy [5, 42]
[Equation]
It is worth noting that filter size determines the computational efficiency and accuracy.
Large filters, ingeneral, can approximate differential operators with higher approximation orders.
On the other hand,large filters have more memory overhead and a higher computation cost.
In practice, the trade-offmust be balanced [4].
C Calculation of time integrationθ -Rule
For the heat equation, we used the θ -rule for time discretization.
The key feature of θ -ruletime discretization scheme is that we can use one formula to generate a family of well-known.
Forexample, the Forward Euler scheme, Backward Euler scheme, and Crank-Nicolson scheme aredenoted by θ = 0 , θ =
[Equation]
and θ = 1 / 2 , respectively.
On the other hand, standard second-orderaccurate finite differences are used to compute the spatial derivatives.
As a result, Eq.
10 can bewritten as:
[Equation]
Collecting the unknowns on the left-hand side, we have
[Equation]
where
[Equation] [Equation]
F x and F y are the Fourier numbers in x and y direction, respectively.
To initialize the problem, weset ∆ x = ∆ y
= 1 / 64 and ∆ t = 0 .
Crank–Nicolson method
To construct a numerical solution for the Burgers’ equation, consider
Eq.
12 on a bounded two-dimensional (2D) Cartesian domain Ω = [0 , L x ] × [0 , L y ] , where
[Equation]
[Equation]
Let nodal points u ni,j and v ni,j be the discrete approximation of u and v at the grid point( i ∆ x, j ∆ y, n ∆ t ) , respectively.
Because the Burgers’ equations are time-dependent, a robust andprecise numerical approach for temporal discretization is required.
Compared to the explicit Eulermethod, the Crank-Nicolson finite-difference method that allows for the relaxation of strict constraints on the number of time steps and provides more accurate solutions is used.
By applying
Crank–Nicolson scheme to Eq.
12, we get
[Equation]
and
[Equation]
The truncation error of the adopted numerical scheme using the Taylor series expansion is of order
O ((∆ x ) 2 + (∆ y ) 2 + (∆ t ) 2 ) .
D Qualitative emulation performance
We begin by applying the heat equation to the diffusion of a Gaussian hill.
The initial value is set to
[Equation]
22) for a = 1 on the domain [0 , 1] ×
For this problem we will use homogeneous Dirichlet boundaryconditions u D = 0 .
ϵ 1 and ϵ 2 are random variables generated from a normal distribution.
A numericalsolver generates the ground truth data.
To solve time-dependent PDEs using the finite element method,first discretize the time derivative using a finite difference approximation, yielding a sequence ofstationary problems, and then convert each stationary problem into a variational formulation.
Fig.
6shows the prediction results from the FNO and ResNet.
ResNet with a hierarchical convolution layerprovides better error distribution because the diffusion process can be very local, but FNO performsbetter in terms of magnitude.
[Equation] [Equation]
HIlN2FQ=">AAAB6nicbVDLSgNBEOyNrxhfUY9eBoMQL2FXgnoMevEY0TwgWcLspDcZMju7zMwKIeQTvHhQxKtf5M2/cZLsQRMLGoqqbrq7gkRwbVz328mtrW9sbuW3Czu7e/sHxcOjpo5TxbDBYhGrdkA1Ci6xYbgR2E4U0igQ2ApGtzO/9YRK81g+mnGCfkQHkoecUWOlh3Jw3iuW3Io7B1klXkZKkKHeK351+zFLI5SGCap1x3MT40+oMpwJnBa6qcaEshEdYMdSSSPU/mR+6pScWaVPwljZkobM1d8TExppPY4C2xlRM9TL3kz8z+ukJrz2J1wmqUHJFovCVBATk9nfpM8VMiPGllCmuL2VsCFVlBmbTsGG4C2/vEqaFxXvslK9r5ZqN1kceTiBUyiDB1dQgzuoQwMYDOAZXuHNEc6L8+58LFpzTjZzDH/gfP4AjR6NVA
[Figure]
/latexit
[Figure]
To investigate the capabilities of FNO and ResNet further, we change the initial condition to a uniformdistribution.
Fig.7 shows the emulation results.
The first row displays the predicted dynamics of theemulator, the second row displays the results of numerical simulation, and the third row displays theabsolute error between numerical simulation and deep learning emulation.
The results clearly showthat FNO outperforms ResNet in simulating the diffusion process.
For FNO, the error is smaller andgrows gradually over time, whereas for ResNet, the error jumps to a higher range and stabilizes inmost regions.
[Table]
+ukJrz2J1wmqUHJFovCVBATk9nfpM8VMiPGllCmuL2VsCFVlBmbTsGG4C2/vEqaFxXvslK9r5ZqN1kceTiBUyiDB1dQgzuoQwMYDOAZXuHNEc6L8+58LFpzTjZzDH/gfP4AjR6NVA==</latexit> ( b )
Fig.8 shows the FNO’s prediction results of a randomly chosen realization from the Burgers’ equationtest dataset.
The target velocity field was obtained by FEniCS [1], an open-source library for finiteelement simulations.
The emulator was trained on 8192 realizations.
To evaluate the results, theabsolute error between the target and prediction is calculated and presented in the third column of twosubfigures.
Overall, the FNO model accurately predict the complex fluid structures.
The predictionsof FNO match the reference values well in all time instances.
[Equation]
[Figure]
Furthermore, the proposed FNO model outperforms the state-of-the-art ResNet model trained on thesame amount of fluid realizations.
In Fig.9, it can be observed that both models are able to providesatisfactory predictions in the predefined training time period, namely, t ∈ [0 , 0 . 5] .
However, theperformance of ResNet degrades significantly in the extrapolation region, that is, t ∈ (0 . 5 , 1] .
The
ResNet’s predictions deviate from the ground truth while the prediction results of FNO can stillmaintain relatively high accuracy.
Such comparison results reflect FNO’s superior ability to absorbunderlying physical principles, which supports a better generalization of unknown scenarios.
[Table]
[Figure]
E Quantitative emulation performance
Additionally, the quality of the ResNet/FNO predictions are quantitatively evaluated based on thefollowing relative error metric:
[Equation]
and the coefficient of determination, also known as the R 2 score, which is defined as:
[Equation]
(24) where ¯ u true is the mean of u k true .
The 500 test realizations are statistically analyzed using these twometrics.
Table 1 shows the mean and variance of the calculated relative L 2 error and R 2 score.
Again,the FNO model produces a far less relative L 2 error than the ResNet model.
Also, the R 2 score of the
FNO model is closer to 1 than the ResNet model.
All of these findings indicate that the predictedvelocity fields from the FNO match the references better than those from the ResNet.
F Computational efficiency
Because the proposed FNO model is mesh-free, greater predictive efficiency is expected when appliedto a more refined spatial discretization.
To verify this, we further carried out the quanti- tativelyanalysis on resolutions of 128 × 128 , 256 × 256 , and 512 × 512 .
Note the ground truth solver(FEniCS) does not fully support GPUs and has varying options of linear solver and pre- conditioner,for example, conjugate gradient method vs. generalized minimal residual method.
Interest here wasto provide a hardware-specific performance comparison.
The computation time of both FEM-basedand FNO-based simulations is shown in Fig.10 (a).
Compared to the classical FEM-based solver, theproposed FNO-based emulator scales very well.
Evaluating our model on the same CPU hardware canstill achieve speedups between 100x-4570x.
Users with access to a GPU could further benefit froman approximate speedup of 60x.
Meanwhile, we investigated FNO and ResNet scaling performance,which is defined as a function of the number of initial conditions generated for training.
Fig.10 (b)depicts the results.
[Equation]
[Figure]
G Out-of-distribution prediction
For the Burgers’ equation, randomness is introduced to the initial conditions to better model the targetsystem’s stochastic nature, for instance, the presumed chaos in turbulence.
Because Fourier serieshas been applied to approximate the physical turbulence in many studies, we formalize the randominitial states using a truncated Fourier series with random coefficients a , b , and c ∈ R
[Equation]
Originally, random coefficients are defined as:
[Equation]
Here, we investigate the emulators’ generalization performance to out-of-distribution initial conditions.
We take the emulators trained on random inputs defined in Eq. 25 and test them on new inputdistributions:
[Equation]
Fig.11 depicts samples generated from the training distribution as well as the unknown testingdistribution.
It can be clearly seen that the newly generated initial conditions are very different fromthose used in model training.<latexit
[Equation]
base64=
[Equation] [Equation]
AAAB6nicbVDLSgNBEOz1GeMr6tHLYBDiJeyqqMeAF48JmgckS+
[Equation]
wIsHRbz6Rd78GyePgyYWNBRV3XR3BYng2rjut7Oyura
[Equation]
sZnbym
[Equation]
v7O7tFw4OGzpOFWV1GotYtQLUTHDJ6oYbwVqJYhgFgjWD4e3Ebz4ypXksH8woYX6EfclDTtFY6b6EZ91C0S27U5Bl4s1JEeaodgtfnV5M04hJQwVq3fbcxPgZKsOpYON8J9UsQTrEPmtbKjFi2s
[Equation]
mp47JqVV6JIyVLWnIVP09kWGk9SgKbGeEZqAXvYn4n9dOTXjjZ1wmqWGSzhaFqSAmJpO
[Equation]
SY8rRo0YWYJUcXsroQNUSI1NJ29D8BZfXiaN87J3Vb6oXRYrtadZHDk4hhMogQfXUIE7qEIdKPThGV7hzRHOi
[Equation]
PufMxaV5x5hEfwB87nD7XKjeE=</latexit> ( a ) <latexit sha1_base64="75g5nqES869yctlIFyHJpAB3ONo=">AAAB6nicbVDLSgNBEOz1GeMr6tHLYBDiJeyqqMeAF48JmgckS5id9CZDZmeXmVkhLAF
[Table]
+hHtSx5yRo2V7kvBWbdQdMvuFGSZeHNShDmq3cJXpxezNEJpmKBatz03MX5GleFM4DjfSTUmlA1pH9uWShqh9rPpqWNyapUeCWNlSxoyVX9PZDTSehQFtjOiZqAXvYn4n9dOTXjjZ1wmqUHJZovCVBATk8nfpMcVMiNGllCmuL2VsAFVlBmbTt6G4C2+vEwa52XvqnxRuyxWak+zOHJwDCdQAg+uoQJ3UIU6MOjDM7zCmyOcF+fd+Zi1rjjzCI/gD5zPH7dPjeI
[Figure]
Similar to Fig.4 (a), the error propagation process is presented in Fig.12.
Overall, the FNO model hasbetter generalization than the ResNet model.
It generalizes well to new initial conditions, which iscompletely different from the training realizations.
Though the accuracy of both models decreasesat a very much similar rate in the extrapolation region, the FNO model has a significantly lowerprediction error, particularly in the training region.
This indicates that the FNO model has strongflexibility and robustness when emulating high-dimensional advection systems with strongly nonlinearcharacteristics.
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
<latexit
[Equation]
[Table]
[Figure]
Note each prediction experiment includes two regions, namely, the training [0 , 0 . 5] and the extrapolation (0 . 5 , 1] .
Fig.12 shows the prediction performance for a randomly chosen realization withtime index specified to 0 .
25 for training and 0 .
75 for extrapolation.
It is found that the predictions of
FNO are almost unaffected by the out-of-distribution samples, while the ResNet model is seriously deteriorated and produces worse predictions, especially in the extrapolation region.
Also, the errorcontour of the ResNet model shows the bulk of errors are clustered on the leading face of the waves,which is specifically where the extreme values are located.
However, the errors of the FNO predictionsare relatively small and evenly distributed, and there is no clear high-error area in the predictions.
Such findings indicate that the FNO model possesses superior robustness to ResNet.
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Figure]
H Super resolution prediction
Here, we summarize the performance of the aforementioned two models on super-resolution prediction tasks.
The purpose of this assessment is to demonstrate the unique ability of the proposed
FNO model, that is, to generate continuous spatiotemporal solutions.
Unlike many convolutionalneural networks (CNNs) based approaches, which learn an end-to-end mapping between the low/highresolution images, the FNO model is trained only with low-resolution data.
Due to the inherent
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Table]
[Equation]
[Table]
JzIaaT2MAtsZUdPX895Y
[Table]
M9rpSa88jMuk9SgZNNFYSqIicn4b9LlCpkRQ0soU9zeSlifKsqMTSdvQ
[Table]
[Figure]
properties of convolution kernels, namely, spatial-agnostic and channel-specific, the ResNet fails toprovide accurate prediction if additional tuning for different resolutions is not available.
On the otherhand, the proposed FNO model achieves consistent error when evaluated at a higher resolution.
It isworth mentioning that the error is defined as the pixel-wise error, averaging over both the spatial andthe temporal domain.
The encouraging results (Fig.14) show great promise of the proposed model forfast emulating high-dimensional spatiotemporal dynamics.
I Notes on data saturation
Because we make direct use of the physics principles, i.e., governing equations in our case, both
ResNet and FNO do not require a classical numerical solver such as a finite element simulatorto generate data for training.
Instead, the proposed knowledge synchronized learning frameworkbegins with the initial conditions of the problem, which can be easily generated from a predefineddistribution.
Intuitively, one would expect the emulation quality of the model to improve when thenumber of the initial conditions increases.
We can, in theory, generate infinite initial conditions andpush the model to obey the constraints of the problem on all of them.
However, we have to find asuitable number in practice considering constraints posed by the available time and computationalresources.
Viewing from the model capacity, another point worth mentioning is that the model maysaturate rather than continuing to draw physics from the data.
To verify this concern, the emulatorwas trained using sample sets with 256, 512, 1024, 2048, 4096, 8192, 16384, and 32768 realizations.
Fig.15 shows the computed results evaluated on a test set of 512 realizations.
For both models, theerror decreases as the amount of training data increases, indicating additional data is contributingto training.
The improvement brought by the data is more evident at the beginning, where a smallamount of information is used (e.g., 512 vs. 1024).
Though the data size keeps increasing by a factorof 2, the performance enhancement slows down.
In the case in which 32768 samples were utilizedfor training, the performance is almost the same and partially worse than the case where 16384samples were used, implying the model may reach a saturation state.
Also, it should be addressedthat the FNO model can achieve the same accuracy using fewer data samples.
After training of 120epochs, the prediction performance measured by the mean MSE of different models shows apparentdifferences.
The FNO model offers a more compelling ability to embedding physics to learn theunderlying spatiotemporal dynamics.
[Equation] [Figure]