Neural Basis Functions for Accelerating Solutions to High Mach Euler Equations
Abstract
We propose an approach to solving partial differential equations (PDEs) using a set of neural networks which we call Neural Basis Functions (NBF).
This NBF framework is a novel variation of the POD DeepONet operator learning approach where we regress a set of neural networks onto a reduced order Proper Orthogonal Decomposition (POD) basis.
These networks are then used in combination with a branch network that ingests the parameters of the prescribed PDE to compute a reduced order approximation to the
PDE.
This approach is applied to the steady state
Euler equations for high speed ﬂow conditions (Mach 10-30) where we consider the 2D ﬂow around a cylinder which develops a shock condition.
We then use the NBF predictions as initial conditions to a high ﬁdelity Computational Fluid
Dynamics (CFD) solver (CFD++) to show faster convergence.
Lessons learned for training and implementing this algorithm will be presented as well.
1. Introduction
The past ﬁve years have seen signiﬁcant interest in the use of neural networks to understand the behavior of systems governed by differential equations (Han et al., 2018; Willard et al., 2020;
Karniadakis et al., 2021).
For example, methods like physics-informed neural networks (Raissi et al., 2019; Lu et al., 2019b) can solve a given instance of a differential equation or infer its properties given some amount of data and have been applied in domains like electromagnetism (Khan & Lowther, 2022) and blood cell mechanics (Yazdani et al., 2021).
More recently still, the subﬁeld of operator-learning has attained prominence (Lu et al., 2019a;
Li et al., 2020; Bhattacharya et al., 2021; Mao et al., 2021).
The objective of operator learning is to learn a data-driven approximation to an operator that deﬁnes a differential equation; a trained operator can then produce a solution to this equation at a greatly reduced computational cost.
Despite the ﬁeld’s novelty, applications have already been considered in areas like climate science (Kashinath et al., 2021).
Prior to some of these deep network architectures, efforts to reduce the computational cost often attempted to solve a reduced form of the equations of interest.
Approaches like the Proper Orthogonal Decomposition (POD) (Willcox & Peraire, 2002; Gunzburger et al., 2007; Witman et al., 2017; Quarteroni et al., 2015; Berkooz et al., 1993) use an eigen-decomposition to extract the primary modes of a given solution space.
These modes could then be re-projected in the reduced space to signiﬁcantly reduce the degrees of freedom required to solve the equations.
The approach being proposed here borrows concepts from the more traditional Reduced Order Modeling (ROM) community and fuses it with the newer deep network architectures to build a Neural Basis Function (NBF) framework for solving differential equations.
One of the motivating interests of this work will be the focus on how well this approach can resolve solutions with complicated phenomena.
The speciﬁc phenomenon we will focus on will be solutions with large gradients or discontinuities, as these problems have posed difﬁculties for both traditional techniques and the more recent Neural
Network based approaches.
Existing work that considers similar classes of problems includes (Mao et al., 2020) for physics-informed neural networks and (Mao et al., 2021) for operator-learning.
We ﬁrst propose the network architecture and training methodology that will be used to learn the underlying basis functions and associated unknown vectors.
In this paper, we introduce the formalism of the neural basis function (Section 2.1) and describe how to apply it to data sampled from a given PDE (Section 2.2).
In Section 2.3, we demonstrate how to apply the NBF approach to the steadystate Euler equations.
We evaluate our model in Section 3.
with a comparison to vanilla DeepONet in Section 3.2 and then show, in Section 3.3, that predicted solutions from the NBF can be used to accelerate the process of obtaining high-ﬁdelity data from a CFD solver.
2. Methods 2.1. The Neural Basis Function Formulation
The neural basis function framework at its core takes a traditional reduced basis Finite Element Method (FEM) (Quarteroni et al., 2015) approach and uses a set of approximating neural networks to learn the underlying bases and explicit unknowns of the governing equations.
Our basisapproximation networks are similar to the trunk component of a Deep Operator Net (DeepONet), and our unknownsapproximation networks are similar to the branch component (Lu et al., 2019a), speciﬁcally as found in the recent
POD DeepONet extension (Lu et al., 2022).
We consider a subclass of PDEs deﬁned for a vector ﬁeld w ψ : Ω → R L , where Ω ⊆ R M , that satisﬁes the following equations:
[Equation]
where S Ψ is a (potentially nonlinear) operator involving derivatives with respect to space,
∂ Ω is the boundary of Ω ,
B ψ is a boundary condition operator, and ψ ∈ Ψ is a set of parameters that fully speciﬁes the PDE and determines its solution.
Section 2.3 illustrates this formulation for a speciﬁc problem.
In this paper, we consider steady-state problems with no time-dependence, but, similar to physicsinformed neural networks (Raissi et al., 2019), the NBF approach can be extended to time-dependent problems.
In the operator-learning problem (Li et al., 2020; Lu et al., 2019a; Bhattacharya et al., 2021), we learn an operator F that, for a given novel parameter ψ , produces an approximate solution ˆ w = F ( ψ ) satisfying ||
ˆ w − w ψ
|| under some appropriate norm.
This requires a set of training samples of solutions w ψ for varying values of ψ .
To deﬁne this vector solution ˆ w we introduce a linear combination of reduced basis functions ( φ j ( x ) :
Ω → R M ) and a set of unknowns ( C j ( ψ ) :
Ψ → R M ), where j = 1 , . .
. , n BF indexes the set of bases used to deﬁne ˆ w .
Speciﬁcally, Equation (3) shows how the i th component of ˆ w is calculated as a linear combination of the basis functions φ j ( x ) and the unknowns functions C j ( ψ ) .
The discrete form of this approach has been used successfully as a way to approximate solutions, within the parameters space ψ , of PDEs at a fraction of the computational cost (Willcox & Peraire, 2002; Gunzburger et al.,
[Equation]
Witman et al., 2017).
[Equation]
Basis functions are trained by solving the data-driven minimization problem:
[Equation]
where the ground truth basis data points U i,j ( x l ) , with l = 1 , ..., n representing the index of all the spatial data-points.
We obtain the truth basis data via a process described in
Section 2.2.
Unknowns functions are trained by solving the following physics-informed minimization problem:
[Equation]
where P ψ and P x are probability distributions deﬁned for the PDE parameters ψ and the spatial inputs x , respectively.
Due to the presence of S ψ in Equation (5), evaluating the loss function will require taking derivatives of ˆ w with respect to x .
This can be done using modern automatic differentiation techniques (Baydin et al., 2017).
In practice, we implement our networks in PyTorch (Paszke et al., 2019) and use the functorch (He & Zou, 2021) library to calculate loss function derivatives.
2.2. Data-driven basis construction
Training the basis functions Equation (4) requires data, which we obtain by using a high-ﬁdelity simulator like
CFD++ to solve instantiations of the PDE.
We sweep a subset of our parameter space Ψ and generate ﬁeld values w for each sampled ψ ∈ Ψ .
These samples are divided into training, validation, and test sets as normal in ML.
For training, we will refer to the total number of training instances as D , and we let ψ d refer to the parameters for the d th training instance.
We assume that all of our snapshots { ( w d , ψ d ) : d = 1 , . . .
, D } are evaluated at the same space points.
We index these points as { x l } nl =1 .
Thus, the training set is speciﬁed by { ( { w d ( x l ) } n l =1 , ψ d ) } D d =
Once the snapshot data is collected, we follow the Snapshot
Proper Orthogonal Decomposition method (Sirovich, 1987) and perform singular value decompositions (SVDs) on the data to extract their bases.
For each component w i of w , we construct a matrix W i containing the observed data points, and then we decompose it as W i
=
U i Σ
i V Ti .
[Equation]
The U i matrix captures the row-space of the snapshot set and represents the primary space modes of the set of solutions.
Once we have generated the set of decomposed spatial modes describing the parametric solution space, we can generate a bound on the expected reconstructed error levels.
A well known POD error bound (Witman et al., 2017) can be derived from the singular values in the decomposition via the equation (
cid:80) D d = n BF +1 σ 2 d where D here represents the total number of parameter snapshots available.
This error bound allows use to a-priori choose a suitable error tolerance before initiating the full training process.
For any of the results presented here, we consider the total number of basis functions as equal to the number of training scenarios.
2.3. Euler Equations
We consider the steady-state compressible Euler equations (Certik, 2017), deﬁned for a state vector w ψ =
[ ρ, u, v, E ] T on a domain Ω ⊆ R 2 : [Equation] w [Equation]
+
[Equation]
F
[Equation]
w
[Equation]
) =(7)
where
[Equation]
The components of the state vector w ψ include the ﬂuid density w 0 = ρ , the x and y velocities w 1 = u and w
= v , and the total energy w 3 = E .
The Euler equations Equation (7) also include the system pressure P , which is determined by a closure relation in terms of the other state variables:
[Equation]
where γ is the gas constant and is set to 1.4, using the perfect gas assumption.
Ultimately, we are interested in using solutions to these equations as initial conditions for a high ﬁdelity CFD solvers to accelerate convergence.
In additional term the state variable terms we will also need a relation for the temperature, T , which can be expressed as:
[Equation]
where R represents the ratio of speciﬁc heats and for this problem we set it to a constant 287 . 058 , also under the perfect gas assumption.
The ﬁnal term that will be referenced in this example is the speed of the ﬂow and accounts for the magnitude of the x and y components of velocity ( | u | = √ u 2 + v 2 ).
With these variables we can write the equation for the Mach number:
[Equation]
which will be used in the inlet boundary condition described in Section 3.1.
For further expansion of the Euler equations in terms of our state variables please see Appendix A.
2.4. Related work
The operator-learning problem is frequently viewed as learning mappings between function-spaces (Lu et al., 2019a;
Li et al., 2020; Bhattacharya et al., 2021), where the input function might be an initial condition or forcing function.
We consider a more restricted setting, in that we require our operators S φ Equation (1) and B ψ Equation (2) to be parametric functions of a ﬁnite-dimensional vector ψ .
In both cases, however, the output is a function Equation (3) deﬁned over an entire domain.
Of operator-learning approaches, the NBF is most similar to the DeepONet (Lu et al., 2019a).
Both use multiple types of networks that ingest different types of data (called “branch” and “trunk” networks in (Lu et al., 2019a)).
In particular, the
NBF can be viewed as a extension of the recently-proposed
POD DeepONet (Lu et al., 2022).
The POD DeepONet retains the full discrete set of bases Equation (6), but the
NBF regresses these bases onto a set of neural networks φ i,j .
This has a few beneﬁts: It 1) can interpolate the modes to new regions of space when data are only available on sparse or unstructured grids; 2) is useful for lower memory applications where storing the potentially large full reduced basis is not feasible; and 3) allows for the ability to compute spatial and/or temporal derivatives of modeled quantities like ﬂuid velocity in a batch process via GPU or another distributed architecture.
The NBF formulation, like the DeepONet formulation, does not impose any structure to the domain Ω .
This is in contrast to some methods like the Fourier Neural Operator (Li et al., 2020).
Our use case for the Euler equations in Section 3. relies on this ﬂexibility.
See also (Lu et al., 2022) for a discussion of how different operator-learning approaches deal with non-uniform domains.
3. Results 3.1. Evaluation of NBF for Euler Equation
For this work, we will make use of a rectangular domain with a quarter circle cutout located at x = [0 , 0] with radius 1 (see Figure 3).
There is an inﬂow condition on the left side of the domain which is controlled by the Mach number, which for this study is the sole parameter in ψ .
The domain extends to − 4 in the x -direction and 7 in the y -direction.
This domain was chosen to allow sufﬁcient room for the shock condition based on our predeﬁned parametric Mach number range.
This 2D cylinder in cross-ﬂow problem is often used as a benchmark in the CFD space for testing different algorithms and their ability to capture shockwaves.
We then generated a steady-state dataset of high-ﬁdelity solutions ranging from Mach 10 to Mach 30 in increments of 1.
This data was generated using CFD++ in a timedependent conﬁguration and allowed to evolve until the time derivative portion of Euler equation sufﬁciently decreased.
CFD++ is an unstructured ﬁnite volume ﬂow solver that is maintained by
Metacomp Technologies Inc.
It has the capability of solving the full Navier-Stokes equations with high-order spatial and temporal accuracy.
To resolve the inviscid Euler equations, the solver uses a pseudo timemarching approach where the initial condition is integrated over time until convergence is reached.
The convergence can be accelerated by using spatially varying time steps.
CFD++ version 20.1 was utilized to perform all the simulations.
Once we have the data generated, the next step is to extract the basis from the set of snapshots.
In general, we hope to be able to approximate solutions within the parameter space, so, for analysis of this approach, we hold out a set of Mach cases as our test set.
An ablation study of various number of training data points considered will be presented at the end of this section.
But unless speciﬁcally noted, the results presented will use a dataset of 20 Mach numbers with just a single held-out solution.
This is done to illustrate the best case scenarios for how well this approach applies.
Given the training data considered, we extract the state variables of interest and subtract the mean state from each of the solutions.
Empirically, we have found that this tends to improve the speed of convergence when regressing the basis networks.
The basis networks we consider are general fully-connected networks with ﬁve layers and 40 nodes per layer.
We use leaky rectifed linear units (Maas et al., 2013) as activation functions (with a negative slope of 0.01) on the hidden layers, which tends to help resolve the small-scale features in the basis functions.
For example in Figure 2, the shock is very narrow in space and changes rather rapidly.
We use the Adam optimizer (Kingma & Ba, 2015) andtotal training epochs with an initial learning rate of 10 −while applying an exponential learning rate decay factor (0.9) after 70 training epochs.
For PDEs with complicated phenomena (shockwaves), we have found accurately representing the high-frequency basis functions can be challenging.
To resolve this, we use an importance sampling technique during training that prioritizes diverse data points in the training routine.
We can describe the probability p i,j,l of selecting a single training data point when minimizing Equation (4), ( x l , U i,j ( x l )) , as:
[Equation]
where p ∗ i,j,l = | U i,j ( x l ) − ¯ U
i,j ( x l ) | , and ¯ U is the mean basis value for all spatial points in the domain x .
In practice, we have found that overusing this sampling approach leads to poor generalization.
However, applying this methodology every so often (speciﬁcally every 4 epochs) improves the resolution of nuanced features.
Once the basis networks have been sufﬁciently trained, the next step is to learn the set of unknown networks.
For unknown network conﬁgurations, we use a seven layer feed forward network with 120 nodes per layer with leaky ReLU activation functions, similar to the basis networks.
In practice, we have found that Equation (5) is non-trivial to learn with standard weight initialization techniques.
To aid convergence of this loss term, we use a pre-training approach that attempts to regress the C
i,j terms to known solution data to get the unknown network weights in a sufﬁcient neighborhood before applying the operator loss in Equation (5).
This pre-training minimization loss can be written as:
[Equation]
where ˜ C i,j can be calculated through standard minimization according to the loss function:
[Equation]
this allows the unknown network to start off with a reasonable estimate of the desired states.
It is important to note that, during this learning process, we place no dependencies on the operator portion, S ψ ˆ w ( x , ψ ) , in the loss function.
After pre-training, the ﬁnal step is to optimize the unknown network with respect to the governing equations, boundary conditions and any other constraints.
Minimization of the governing equations and boundary conditions follows the deﬁnition of the Euler equations and the loss function deﬁned in Equation (5).
For this problem we add two additional constraints to minimize the mean square error between predicted and CFD Pressures and Temperatures.
Unfortunately incorporating this constraint restricts our loss function to only use ψ parameters that exist in our training set.
However, we have found that without incorporating this constraint leads to negative values of pressure/temperature which for this problem are non-physical.
Finally with the basis and unknown networks trained, we are able to infer new solutions within the deﬁned parameter space.
We seek to generalize performance across the parameter space Ψ of Mach numbers.
Figure 3 shows the NBF predicted solution for the speed within the domain while considering a Mach 25 inﬂow condition.
Figure 4 shows the log absolute error of this predicted solution with respect to the CFD++ data.
It is worth noting that this particular scenario is held out of our training data set and still shows good predictive performance.
It is also useful to understand how well the NBF approximation works across the parameter space.
Figure 5 shows the relative L 2 error as a function of the Mach number.
The errors appear to be relatively stable across the parameter space with slightly better performance for the higher Mach
[Figure] 3.2. Comparison with Vanilla DeepONet In order to understand how well the NBF approach compares against existing approaches, we will benchmark our solution against the Vanilla DeepONet operator approach. The equation for the vanilla DeepONet is given by (Lu et al., 2022):
[Equation] where v and ξ are the parameter (in our case ψ ) and spatial values (in our case x ). (Lu et al., 2022) focuses on the case where v is a function that parameterizes the PDE – for example, an initial condition. For comparison purposes, we will consider it a parameterization of Ψ .
Additionally, b j ( v ) and t j ( ξ ) represent the branch and trunk networks respectively, with b 0 acting as a bias imposed on the outputs.
Since we are interested in the two dimensional steady state
Euler equations, we require a DeepONet representation of multiple state variables deﬁned in Equation (8).
Making use of recommendations from (Lu et al., 2022), we implement a set of feed-forward DeepONets for each state variable considered.
The network architectures that we found generated the best performance for our use case comprised of: 5 layers of 120 nodes with hyperbolic tangent activation functions for the branch networks and 6 layers of 120 nodes with rectiﬁed linear units for the trunk networks.
These networks were then connected in a shared layer of 64 nodes combined with a bias ( b 0 ) to produce predictions for each state variable.
During training, we used the same Adam optimizer, learning rate decay and number of epochs as was used for the NBF approach.
We added a few components that appeared to aid the training process for our problem speciﬁcally: adding prioritized spatial sampling and zero-mean unit-variance (ZMUV) normalization functions for spatial/parameter inputs as well as the output state variables.
For spatial sampling prioritization we used the Equation (11) to more often select training examples with more deviation from the mean solution.
Additionally, adding the ZMUV normalization to input/output parameters appeared to speed convergence and prevent collapsing the solution to the mean ﬂow.
Figure 6 shows the comparison for the vanilla DeepONet vs the Neural Basis Function approach considering the total number of training solutions as inputs.
We are able to show better performance in the approximation across the number of training examples.
Speciﬁcally we are able to produce signiﬁcantly better results for smaller training data sets, indicating better generalization for sparse data.
We believe we are able to show this better performance due to the preconditioned nature of basis functions included in the NBF estimates as well as application of the governing equation as regularization during training.
It is worth noting, that although the training process lines up reasonably closely between the two approaches, there is some overhead incurred by the NBF approach in: calculating the SVD, then training the unknown network.
Additionally, the DeepONet architecture is signiﬁcantly smaller than the NBF architecture as it is described here.
We believe there are architecture optimizations that could reduce the size and will consider this for future work.
Thus, it is not a completely direct comparison but is as close as we could get to provide a baseline for the NBF approach.
We found that for many of the cases within the ablation study, the Energy ( E ) state variable was the most difﬁcult to get to converge.
This is likely due to the magnitude of values compared to the other state variables ( ρ , u and v ).
For DeepONet, applying the ZMUV transformation on the output state improved the convergence but not signiﬁcantly so.
3.3. Acceleration of CFD solutions
In the conventional CFD approach an initial condition is employed with iterative updates until steady state is reached.
If one was to use an initial condition that is close to the converged solution, fewer iterations would be required to reach the high-ﬁdelity steady-state.
The NBF solution can be leveraged to accelerate the CFD solver by providing this nearly converged initial condition.
In the following section, we will compare the performance of the conventional CFD approach with NBF informed CFD approach.
The two approaches will be evaluated by comparing their average residual history.
The time-dependent form of the Euler equation includes the the term:
[Equation]
and adds to Equation (8).
Additional details on the setup and implementation of the CFD++ conﬁguration can be found in Appendix C
The convergence is determined by the evolution of the timedependent residual.
Once the residual levels off, the simulation is deemed to be converged and the solution has reached steady-state.
In the CFD++ ﬁnite volume approach, the governing equations are integrated over the cell volumes.
For the case of the Euler equations, the right hand side becomes the sum of the scalar product of the inviscid ﬂuxes and the cell face areas.
For a given equation, the residual is deﬁned as the absolute value of the residual divided by the cell volume averaged over all the cells.
Since we are working with a system of equations, we monitor the average residual of our system equations, i.e. the sum of the residual of each equation divided by the number of equations.
Additional detail of all the numerical procedures can be found in the
CFD++ user-manual 1 .
Section 3.3 and Section 3.3 show the time-dependent residual as a function of the number of iterations for Mach 15 and 25 respectively.
As we can see from these plots, the NBF solution provides a noticeable speed up in the convergence of the high ﬁdelity solver.
4. Discussion
This work has shown the potential for using an operator learning based approach, like the Neural Basis Function, as an acceleration mechanism for high-ﬁdelity CFD solvers.
We found that regressing a set of networks to POD basis data, then using an additional unknown network in linear combination can provide sufﬁcient estimates of state variables within a conﬁgurable parameter space.
High Mach ﬂow cases often induce complicated phenomena, like discontinuous shockwaves, which are difﬁcult to represent numerically, even with standard techniques.
Using the 2D steady state Euler equations as an example, we were able to produce accurate estimates to the ﬂow ﬁeld for high Mach (10-30) scenarios.
These solutions were then used as initial conditions to a high-ﬁdelity CFD solver (CFD++) to improve the convergence, resulting in speed ups up toiterations.
As future work on the application side, we intend to extend these results to the full Navier-Stokes equations, while transitioning to a three-dimensional domain.
Along the way improvements will be made to the methodology to better understand network architecture performance as well as improving training techniques for the basis and unknown networks.
We also hope to develop a methodology using NBF (or other operator approaches) to handle problems where the spatial domain might vary across snapshots.
This would allow for the ability to do shape optimization or analysis of multiple geometric conﬁgurations.
On the learning side we hope to factor in an active learning feedback loop (Settles, 2009) that will be able to identify regions of the ψ parameter space with poor predictive performance, in order to execute more high-ﬁdelity scenarios in these regions to improve overall accuracy. Finally, some combination of DeepOnet and NBF seems like a reasonable path forward especially for speeding convergence of these techniques.
For example, one could create a set of trunk networks that are regressed from the orthogonal SVD basis, then freeze their weights and biases. Then adding a vanilla
DeepONet in combination with these bases could greatly speed the convergence and make it easier to implement an active learning type of approach for data generation.
A. Euler Equation Expansion
We can expand the Euler equations in terms of our state variables w :
[Equation]
B. Other Variable Field Visuals
The following set of plots provide a visual for the other variables of interest using the NBF approximation to the
Mach 25 scenario.
C. CFD++ Details
In our investigation, CFD++ was run with a second order spatial discretization scheme.
The inviscid ﬂuxes were calculated based on the minmod TVD limiter.
The ﬂuxes on the cell faces were reconstructed based on centroidal polynomials.
Spatial scheme blending was leveraged for the sake of numerical stability.
During the initial 250 iterations, ﬁrst order ﬂuxes were used exclusively.
Then a linear blending of the ﬁrst and second order discretization scheme was used until iteration 750, beyond which the second order ﬂuxes were used exclusively.
The time-residual from iteration to iteration depends on the integration scheme.
In this study, an implicit Gauss-Seidel relaxation time integration scheme was utilized with no additional convergence acceleration methods.
The time step sizes were based on the local (spatially varying) CFL (Courant Friedrichs-Lewy) number.
The CFL number is a dimensionless number that is equal to the time step size scaled by the time it takes for the fasts moving characteristics to travel across a cell
CFL = dt dx | u | + a , recalling | u | is the scaler speed ﬁeld.
a is the speed of sound ( a = √ T ∗ R ∗ γ )
In each simulations, the
CFL number was linearly increased from 0.01 to 10 over the ﬁrst 1000 iterations beyond which the CFL number was kept constant.