I
NCOMPLETE
TO COMPLETE MULTIPHYSICS FORECASTING - A HYBRID APPROACH FOR LEARNING UNKNOWN PHENOMENA
Anonymous authors
Paper under double-blind review
A BSTRACT
Modeling complex dynamical systems where only partial knowledge of their physical mechanisms is available is a crucial problem across all scientific and engineering disciplines.
Purely data-driven approaches, which only make use of an artificial neural network and data, often fail to accurately simulate the evolution of the system dynamics over a sufficiently long time and in a physically consistent manner.
Therefore, we propose a hybrid approach that uses a neural network model in combination with an incomplete PDE solver that provides known, but incomplete physical information.
In this study, we demonstrate that the results obtained from the incomplete PDEs can be efficiently corrected at every time step by the proposed hybrid neural network – PDE solver model, so that the effect of the unknown physics present in the system is correctly accounted for.
For validation purposes, the obtained simulations of the hybrid model are successfully compared against results coming from the complete set of PDEs describing the full physics of the considered system.
We demonstrate the validity of the proposed approach on a reactive flow, an archetypal multi-physics system that combines fluid mechanics and chemistry, the latter being the physics considered unknown.
Experiments are made on planar and Bunsen-type flames at various operating conditions.
The hybrid neural network - PDE approach correctly models the flame evolution of the cases under study for significantly long time windows, yields improved generalization, and allows for larger simulation time steps.
I NTRODUCTION
Modeling and forecasting of complex physical systems described by nonlinear partial differential equations (PDEs) are central to various domains with applications ranging from weather forecasting (Kalnay, 2003), design of airplane wings (Rhie & Chow, 1983), to material science (Wheeler et al., 1992).
Typically, a chosen set of PDEs are solved iteratively until convergence of the solution.
Modeling complex physical dynamics requires a good understanding of the underlying physical phenomena.
For cases where the complete physics information is missing, deep learning models can be employed to complete the physical description when additional data of the system is available.
Deep learning methods have shown promises to account for these unknown components of the system (Yin et al., 2021).
We consider a set of partial differential equations with partially unknown physics represented.
The corresponding PDE model for a general state ϕ is given by
[Equation]
where P i models the known but incomplete PDE description, S ϕ represents the unknown physics.
Within the scope of this model, the influence of S ϕ can lead to fundamentally different scenarios.
A commonly targeted case is when the governing equations of the complete PDE description are computationally too expensive to solve, turbulence modeling in computational fluid dynamics (CFD) being a good example.
In CFD, a spacial filtering is performed on the original governing PDEs.
This step introduces unclosed terms in the model equations that correspond to unrepresented physics in equation 1, due to the effects of the filtered scales.
This is a widely studied problem, where the use of deep learning models is currently being explored (Lapeyre et al., 2019; Kochkov et al., 2021; List et al., 2022; Stachenfeld et al., 2022).
In the following, we are targeting a more challenging problem, where increasing spacial resolution and/or reducing time-scales of the incomplete PDE solver does not lead to a converged full solution.
Rather, the incomplete and complete PDEs produce drastically different solutions due to the unknown physics.
The central learning objective is to correct this behavior and retrieve the evolution that would be obtained with the complete PDE description.
Our work expands on the combination of incomplete PDE solvers and neural networks (NNs) (Yin et al., 2021; Takeishi & Kalousis, 2021) to account for the effects of an incomplete physics model.
The NN aims to complete the PDE description, where the differences in complete and incomplete
PDE solutions are beyond the effects of spacial and temporal scales.
We showcase that combining the trained NN model with a differentiable solver for the incomplete PDE can accurately reproduce the physical solutions of the complete, multi-physics PDE solver with stable long-term rollouts.
We demonstrate the capabilities of this approach for an archetypal multi-physics system, namely a complex reactive flow which combines fluid mechanics and chemistry, where the latter is considered unknown.
Reactive flow modeling has applications in numerous domains such as combustion processes in gas turbines (Lieuwen, 2012), climate modeling (Jacobson, 1999; Rolnick et al., 2022) and astrophysics simulations (Gamezo et al., 2003).
Resolving the NavierStokes equations lies at the core of these problems, where additionally the transport of different species of relevance must also be accounted for, together with their production or consumption often following complex reaction mechanisms (Poinsot & Veynante, 2005).
For chemically reacting flows, generation or consumption of multiple species via some chemical reaction are modeled using a net source term.
It is a well-known fact that the incorporation of a detailed chemical kinetic mechanism in a reacting flow model can result in a stiff system of governing equations (Wanner & Hairer, 1996; Najm et al., 1998; Knio et al., 1999).
[Figure]
we consider can yield fundamentally different evolutions, as shown here for a sample temperature field over time.
We consider the reactive flow simulation to be the complete PDE description, while the non-reactive flow simulation represents the incomplete PDE basis, where the chemical kinetic mechanisms are collected in the unknown physics term of equation 1.
Figure 1 shows a visual example of the fundamental differences in system dynamics that can be caused by unknown reaction terms.
We showcase the effectiveness of our approach for different cases of planar 2D premixed methane-air flames, and the varying transient evolution of Bunsen-type flames.
We show that the proposed approach can handle large domains with highly resolved flames, which is closer to the practical flame domains used in many industrial applications.
Specifically we concentrate on training a NN model to correct the spatio-temporal effects of energy and species transport source terms.
We show that in addition to recovering the desired solutions, this approach overcomes inherent problems of temporal stiffness due to the complex reaction mechanism.
Lastly, we demonstrate the applicability of neural network model to control the flow inlet velocities to arrive at desired flame shapes, when combined with a differentiable flow solver.
R ELATED WORK
Deep learning methods have been widely used to model the solutions of partial differential equations (Lagaris et al., 1998;
Long et al., 2018; Han et al., 2018; Bar-Sinai et al., 2019) and in particular, the Navier-Stokes equations (Thuerey et al., 2020; Fukami et al., 2019).
These models can be very fast and do not suffer from the time-step stability issues associated with traditional numerical solvers.
Nevertheless, as these purely data-driven approaches lack the physical understanding of the system being modeled, they generally fail in generalizing to other operating conditions (Kim et al., 2019; Lapeyre et al., 2019).
To leverage the potential of deep learning in physical simulations, it is therefore necessary to incorporate some physical information within the deep learning framework.
Deep learning models can enforce physical constraints partially through the loss function (Bar-Sinai et al., 2019; Raissi et al., 2019) or changes in neural network architecture (Greydanus et al., 2019;
Lu et al., 2021).
However, these approaches struggle to enforce physical constraints such as boundary conditions or predict the strong unsteadiness and chaotic nature of flows.
Neural operators (Li et al., 2020b; Bhattacharya et al., 2021; Patel et al., 2021), specifically, the Fourier neural operator (FNO) of Li et al.
(2020a) introduce an interesting line of work by learning mesh-free, infinite-dimensional operators with neural networks, but do not necessarily offer advantages for longer term predictions.
Hybrid approaches that combine machine learning techniques with numerical PDE solvers (Wang et al., 2020), have attracted a significant amount of interest due to their capabilities for generalization (Chen et al., 2018).
In this context, neural networks are typically used to model or replace a part of the conventional PDE solver to improve aspects of the solving process.
For example, Tompson et al.
(2017), ¨Ozbay et al.
and Ajuria Illarramendi et al.
(2020) proposed a convolutional neural network based approach to solve the Poisson equation in CFD simulation.
In recent years, a number of deep learning based models have been introduced to accurately model turbulent flows (Pathak et al., 2020; Stachenfeld et al., 2022; Dresdner et al., 2022).
Um et al.
and BelbutePeres et al.
(2020) showed the advantages of training neural networks with differentiable physics to correct the numerical errors that arise in the discretization of PDEs.
Sirignano et al.
(2020) and
Kochkov et al.
(2021) additionally correct for closure error by integrating neural network with a differentiable CFD simulator.
These approaches demonstrate the capabilities of neural networks to correct errors in a fast, under-resolved simulation.
Similar to the goals of our work, Yin et al.
and Takeishi & Kalousis (2021) have introduced frameworks of augmenting incomplete physical dynamics with neural network models.
These approaches demonstrate the applicability on ODE/PDE systems, which are weakly nonlinear and the unknown dynamics are linear combinations of the underlying flow fields.
We expand on these works to explore the significantly more challenging scenario of reactive flows.
These are characterised by a multi-physics system with nonlinear advective terms and strongly nonlinear dynamics, described by exponential source terms that exhibit nonlinear combinations of the flow fields.
M ETHODOLOGY
We consider two different sets of PDEs with their associated numerical solver, which we denote as the incomplete PDE P i and the complete PDE P c .
By evaluating P c on an input state ϕ t at time t , we can compute the points of the phase space sequences; ϕ t +∆ t = P c ( ϕ t ) .
Without loss of generality, we assume a fixed time-step ∆ t and denote a state ϕ t +∆ t at next time instance as ϕ t +1 .
The main goal of this study is to learn a correction function that models the effects of the unknown physical system on the incomplete PDE solver to obtain the complete PDE solutions.
We train a correction model C ( ϕ ; θ ) :
X → X that maps P i ( ϕ t ) to ϕ t +1 , defined over a finite time interval [0 , T ] and a spacial domain Ω ⊂ R 2 .
The learning objective is to find the best possible correction function
C ( ϕ ; θ ) given the weights θ
and an input flow state, ϕ .
The model parameters θ are estimated from the complete PDE solution trajectories ( ϕ 0 , ϕ 1 , .., ϕ T ) .
The learned predictions obtained after repeatedly applying the corrector C and invoking P i are denoted by ( ˜ ϕ 0 , ˜ ϕ 1 , .., ˜ ϕ T ) .
3. P ARTIAL DIFFERENTIAL EQUATIONS FOR REACTIVE FLOWS
The physical system we investigate is a laminar premixed methane-air flame with a single step chemistry.
It is governed by the following Navier-Stokes equations (Poinsot & Veynante, 2005)
[Equation]
where ρ , u i , p , T , Y k denote the density, velocity, pressure, temperature, and species mass fractions of species k , respectively.
τ , D k , λ are the strain rate tensor, the diffusion coefficient of species k and mixture thermal conductivity.
In addition, C p denotes the mixture specific heat capacity. ˙
ω k and ˙ ω ′ T are the species reaction rate and heat release rate, respectively.
We consider a one-step chemical system of the type υ ′ F F + υ
O O → products , where υ ′ F and υ
O are the stoichiometric coefficients corresponding to fuel and oxidizer.
The simplifications proposed by Williams (1985) and Mitani (1980) are used to model the reaction rates ˙
ω k for each species.
The fuel consumption rate ˙ ω F is assumed to have the Arrhenius form.
[Equation]
The heat release source term ˙ ω ′ T and the fuel source term ˙ ω F are linked by ˙ ω ′ T = − Q ˙ ω F , where
Q is the heat of reaction per unit mass.
Following Poinsot & Veynante (2005), parameters corresponding to a real-world methane-air flame are chosen as: B 1 = 1 .
7 uSI ; β 1 = 0;
E a = 83600 J/mole ; n F = 1; n O = 0 .
Q = 50100 kJ/kg ;
C p = 1450 J/(kgK).
Taken together, the system of equations above is a challenging scenario even for classical solvers, and due to its practical relevance likewise a highly interesting environment for deep learning methods.
3. P ROBLEM FORMULATION
The incomplete PDE solver solves the set of equations 2 without the source terms and reaction rates ˙ ω k and ˙ ω ′ T , while the complete PDE solver solves the full set from equation 2.
The neural network model, denoted by C ( P i ( ϕ ) | θ ) , corrects the incomplete/non-reacting flow states P i ( ϕ ) to obtain the complete/reacting states
P c ( ϕ ) as shown in figure 2 (B2).
The neural network is trained to model the effects of the unknown chemistry using parameters θ given an input flow state, ϕ =
[ u i , p, T, Y f , Y o ] .
As seen from equation 2, and equation 3, the temperature and species mass fractions fields are strongly coupled which significantly increases the prediction problem complexity.
A slight error in one of the fields will quickly propagate into the other fields, making the predictions diverge.
In the following, a subscript C s ( ◦ ) will denote that the neural network C generates the field s , e.g., C T generating the temperature field.
˜
ϕ
t +1 =
[ u i,t +1 , p t +1 , C T ( P i (˜ ϕ t ) | θ ) , C Y f
( P i (˜ ϕ t ) | θ ) , C Y o ( P i (˜ ϕ t ) | θ )] where ˜ · indicates a corrected state.
u i,t +1 and p t +1 are the time-advanced velocity and pressure field, respectively, predicted using the incomplete PDE solver.
3. T RAINING M ETHODOLOGY
We employ a hybrid NN-PDE approach that augments a neural network model with a PDE solver (Um et al., 2020; Kochkov et al., 2021).
In contrast to previous work, we use the incomplete PDE solver as a basis, and hence the solver does not converge to the desired solutions under refinement, as explained in 3.1.
The neural network is integrated and trained in a loop with the incomplete PDE solver using stochastic gradient descent for m time steps, as shown in figure 2 (A,B2).
Here, the number of temporal look-ahead steps, m , is an important hyper-parameter of the training process.
Higher m provides the network with longer-term feedback at training time through the gradient rollouts.
This gives the model improved feedback on how the time dynamics of the incomplete PDE solver affect the input states, and hence which corrections need to be inferred by the model.
A central component of the hybrid NN-PDE model is the differentiable solver, which allows us to embed the solver for the incomplete PDE system in the training of a neural network.
The differentiable solver acts as additional non trainable layers in the network.
Instead, they provide derivatives of the outputs of the simulation with respect to its inputs and parameters.
Here, we use the differentiable PDE solver from the phiflow framework of Holl et al.
(2019) in combination with Tensorflow to obtain the non-reactive flow solver and reactive flow solver solutions.
The marker-and-cell method (Harlow & Welch, 1965) is adopted to represent temperature, pressure, density, species mass fraction fields in a centered grid, and velocities in a staggered grid.
A purely data-driven model (PDD) is used as a baseline.
It employs a neural network model to learn the complete flow states P c ( ϕ ) given an input flow state ϕ S where ϕ S =
[ T, Y f , Y o ] , as shown in figure 2 (B1).
The new state predicted by the trained neural network model is ˜ ϕ
St +1 = C
(˜ ϕ St | θ ) .
For the hybrid NN-PDE as well as PDD, the neural network part of the predictor block in figure 2 consists of a fully convolutional neural network model.
For both, we experimented with both the ResNet (He et al., 2016) and UNet (Ronneberger et al., 2015) architectures.
We found that the ResNet performed the best for the PDD setting, while for the hybrid NN-PDE approach the UNet performed consistently better.
Hence, the following results will use a ResNet for PDD models, and a UNet for the NN-PDE hybrids.
Additional details of the neural network architectures are provided in the appendix.
The network receives the input states as shown in figure 2.
The output of the neural network model is used to obtain the corrected state ˜ ϕ
t +1 as specified above.
We constrain the mass fraction fields Y k to contain physical values in the range Y f ∈ [0 , 0 . 05] and Y o ∈ [0 , 1] .
All models are trained for 100 epochs with a batch size of 3 and a learning rate of 0.0001.
We additionally compare against the Fourier neural operator (FNO) of Li et al.
(2020a)
as an example of a state-of-the-art neural operator method.
The new state predicted by the trained model is given by ˜ ϕ
St +1 = C
(˜ ϕ St | θ ) where ˜ ϕ S
=
[ T, Y f , Y o ] and C ( ◦ ) represents the Fourier neural operator.
All three approaches use an L 2 based loss that is evaluated for m steps as
L ( θ ) = (cid:80) t + m n = t +1 (cid:80) ϕ = { T,Y f ,Y o } || ˜ ϕ n , P c ( ϕ n ) || 2 .
N UMERICAL E XPERIMENTS
We consider a case of planar 2D premixed methane-air flame propagating in a quiescent mixture ( Planar-v0 ) and two cases of the transient evolution of initially planar laminar premixed flame into a Bunsen-type flame under different inlet velocity conditions ( uniform-Bunsen, nonUniformBunsen ).
We obtain the target data by considering the source terms as defined by equation 3 with the parameters mentioned in section 3.1.
Planar-v
For the most basic case, the planar 2D flame model setup, we consider the reacting
Navier-Stokes equations described in section 3 with zero inlet velocity
i.e. u x = 0 , u y = 0 .
We consider a square domain of size 0.05 m × 0.05 m with 32 × 32 resolution and closed boundary conditions.
The simulation is initialized using a steep transition between a premixed methane-air mixture and burnt gases.
Our training data consists of 6 simulations of 300 steps created by varying the equivalence ratio E .
It represents the stoichiometric mixture ( φ ) of fuel Y f and oxidizer Y o mass fractions, i.e., E = φ Y f Y o and thus fundamentally influences the dynamics of the chemical reaction.
For the training data we use E train = { 1 . 0 , 0 . 9 , 0 . 8 , 0 .
7 , 0 . 6 } , while the test dataset contains
E test = { 0 . 95 , 0 . 85 , 0 .
uniform-Bunsen
In contrast to the planar case, the premixed methane-air mixture is now fed with a constant inlet velocity.
The boundary conditions upstream, at y = 0 , are ( u x , u y ) x,y =
= (0 , κ ) where κ is a n -dimensional vector with constant amplitude.
The target simulation contains a heat release rate term, and in this case the initial temperature field evolves into different Λ -shaped flames at the end of the 300 th time step.
Training and testing datasets are created by varying the equivalence ratio and inlet velocity amplitude U : E train = { 1 . 0 , 0 . 9 , 0 . 8 } and U train = { 0 . 45 , 0 . 4 , 0 . 3 } .
The test dataset uses E test = { 0 . 95 , 0 .
85 } U test = { 0 . 43 , 0 .
The length and temperature of the flame significantly vary depending on the inlet velocity and equivalence ratio provided.
Further details on the boundary conditions and training data visualizations can be found in the appendix.
nonUniform-Bunsen
As a third case we consider the transient evolution of a premixed methaneair flame with non-uniform inlet velocity.
The boundary conditions upstream, at y = 0, are [Figure] [Table] ( u x , u y ) x,y =
= (0 , κ ) where κ is a n -dimensional vector whose elements are each sampled from a uniform distribution from [0.2, 0.65].
We experiment with two different domain sizes, 32 ×( nonUniform-Bunsen32 ) and 100 × 100 ( nonUniform-Bunsen100 ), as shown in figure 4.
The larger domain size used is closer to the practical reactive flow domain used in CFD applications (Jaensch et al., 2017) with a highly resolved flame.
These inlet velocity conditions generate complex flame shapes which increase the difficulty of the prediction problem.
We consider simulation sequences with 500 steps as it takes longer time for the flame to reach the steady-state solution.
R ESULTS
We demonstrate the capabilities of the proposed learning approach to represent the complete PDE description with the aforementioned cases of increasing difficulty.
We also study its ability to generalize to unseen operating conditions such as equivalence ratios, simultaneous variations in constant inlet velocity and equivalence ratio, and non-uniform inlet velocity profiles.
The source code of our work will be published upon acceptance.
As baselines, we compare against a purely data-driven approach; a neural network model with exactly the same look-ahead steps, and the Fourier neural operator of Li et al.
(2020a) that likewise includes m look-ahead steps as discussed in section 3.3.
5. Q UALITATIVE AND Q UANTITATIVE A NALYSIS
Table 1 compares the mean absolute percentage errors (MAPE) and mean squared errors (MSE) of temperature field for all the cases discussed in section 4.
For Planar-v0, the FNO and PDD approaches yield large errors with a MAPE of 8.27 % and 6.33 % , respectively.
On the other hand, the hybrid NN-PDE model trained with 32 look-ahead steps reduces the error to 1.4 % , and thus performs significantly better than the two baselines.
This behavior is visualized in figure 3 with a 1D transverse cut of the simulation domain over 300 time-steps.
The hybrid NN-PDE approach successfully captures the propagation of the flame.
In figure 3 (E), we also compare two important physical quantities, the flame temperature and relative displacement of the flame front, across different equivalence ratios.
It can be seen that the hybrid NN-PDE model (blue circles) accurately predicts the flame temperature for different test cases (solid [Figure] blue line).
The relative displacement of the flame front is computed as | ˜ x t − ˜ x 0 | / | x t − x 0 | , where x t is the position along the flame normal at time t on the 1200 K isotherm of the ground truth simulation, and ˜ x denotes the predicted position.
For all test cases, the hybrid approach accurately predicts the flame displacement over t = 300 steps, while the other approaches yield significant errors.
For uniform-Bunsen case, amongst the baseline models, PDD performs better with an error of 7.58 % compared to FNO model which yields an error of 15.57 % .
A neural network model combined with an incomplete PDE solver for 32 look-ahead steps yields a significantly lower error of 0.72 % as shown in table 1.
This means, the hybrid approach reduces the errors by a factor of 10 over the baselines considered.
Figure 4(A) shows the temporal evolution of the hybrid NN-PDE approach predicted over 300 time steps for the uniform-Bunsen case.
It shows the results for a test case with
U = 0 .
4375 and E = 0 .
It predicts a symmetric flame with accurate flame height and achieves very low instantaneous MAPE of 0.73 % at t =300, when compared with the ground truth data.
Next, we study a complex scenario of nonUniform-Bunsen flame with 32 × 32 resolution.
The hybrid NN-PDE approach outperforms the PDD approach and FNO with an improvement of ∼ 80% .
The MSE values show this trend even more clearly.
The large standard deviations of the MSE numbers indicate that the predictions made by the baseline approaches contain substantial deviations from target values.
Important to highlight is that, despite using the same training data and similar neural network architecture, with same look-ahead steps, the hybrid approach outperforms the PDD approach thanks to its learned collaboration with the incomplete PDE solver.
It accurately reproduces the complete PDE behavior.
Figure 4(B) highlights this with a visualization of the hybrid NN-PDE model predictions for nonUniform-Bunsen32 case.
The trained model accurately predicts the flame simulation over long roll-outs of 500 steps and achieves the complex flame shape with a low, instantaneous MAPE of 2.38 % .
Finally, we showcase the ability of hybrid approach to predict the temporal evolution of highly resolved flames with the uniform-Bunsen100 scenario.
Despite the increased complexity of the larger resolution, it achieves a very good overall MAPE of 3.23 % over 12 test cases of 500 simulation steps.
Figure 4(C) shows an example of physically accurate predictions made by the hybrid NN-PDE model.
We omit the evaluation of baselines for high resolution cases as they do not succeed to model the flame dynamics for low resolution cases.
Effect of training dataset size Figure 5 (A) shows that increasing the number of training sets has little or no effect on the prediction capabilities of the PDD models for nonUniform-Bunsen32 case. The hybrid model with 12 training sets achieves a MAPE of 12.49 ± 4.17%, an improvement of 38% over the PDD model with 32 training sets. This result strengthens the hypothesis that integrating the incomplete PDE solver into the neural network training yields a learning signal that fundamentally differs from that produced by training with precomputed data. The purely data-driven models cannot achieve the same level of accuracy even in the presence of large amounts of data.
Generalization to incorrect PDE parameters
The proposed hybrid NN-PDE model is capable of completing the PDE description even if the underlying incomplete PDE solver does not have access
[Figure]
[Figure]
to the correct parameters, referred to as ‘incomplete, incorrect PDE’. For this test we assume that the values of τ , D k , λ in equation 2 are unknown and set to incorrect values.
For the nonUniformBunsen32 case, the hybrid model with the incomplete, incorrect solver achieves an overall MAPE of 2.48 ± 1.20 %, a slight increase over the MAPE of 2.04 ± 1.39 % for the hybrid model with the incomplete, correct PDE. Additional details can be seen in appendix C [Figure]
5. R ELAXING TEMPORAL STIFFNESS IN THE PDE SOLVER
Traditionally, the source terms and reaction rate terms involved in modeling the fast chemistry of reacting flows require the use of very small time-steps in simulations due to the stiffness of the chemical mechanisms.
The incomplete PDE solver used in the hybrid NN-PDE approach, does not contain the source terms and reaction rate terms.
Therefore, the time scales associated with the chemical reactions play a less important role in maintaining numerical stability, and it becomes possible for the solver to employ larger time steps.
To illustrate this advantage, we train the neural network model with a time-step that is twice as large as the largest time-step ∆ t c required for the complete PDE solver to yield a stable result.
We mimic the setup described in section 4, but now the incomplete PDE solver uses a time-step of 2∆ t c , which is too large for the complete PDE solver by itself to converge to a solution.
Figure 6, shown as a first example, indicates that the results obtained from the models trained with a larger time-step at t = { 25 , 50 , 100 , 150 } are in good agreement with the target data for the difficult uniform-Bunsen case.
The flame dynamics are predicted accurately while using less simulation steps.
For the uniform-Bunsen case, the hybrid approach takes 8 .
23 ± 0 .
011 s to infer one simulation run, whereas the complete PDE solver requires 15 .
21 ± 0 .
003 s .
Similar performance gains are observed across the other cases studied.
Note that for the previous cases, the trained model incurs a negligible runtime overhead.
In these cases, even for costly runs with high resolutions the incomplete PDE solver would not converge to the solutions of the complete description.
The last column of table 1 ( Hybrid NN-PDE-dt ) summarizes the errors of the large time-step approach for all four reactive cases considered.
Despite an effectively doubled computational performance, this model achieves similar errors to the hybrid NN-PDE approach.
This highlights the capabilities of learned, hybrid PDE solvers, which can produce these solutions without the stability problems exhibited by the complete PDE solver, while at the same time being more accurate than pure data-driven predictions.
The PDD approach is not restricted by the time-step of the PDE solver. Therefore, we investigate the performance of the PDD approach on the nonUniform-Bunsen32 case with larger time-steps as
(a) t =
Initial Sha e T(b) t =(c) t =(d) t =
NN Prediction ̃T NÑ (e) t =
Target Sha e T ̃ ̃T NÑ (f) t =
T ̃ (g) t =80016002400 T [k] shown in the figure 5 (B). PDD achieves higher MAPE of 22.44 % for 8 ∆ t c setup as compared to the MAPE of 12.48 % for ∆ t c setup. The PDD approach does not predict the dynamics accurately for any of the larger time-steps considered. Additional visualizations can be found in appendix C.
5. F LAME SHAPE CONTROL
We lastly demonstrate the potential of our approach with a problem from multi-physics system control: we test the joint applicability of a neural network with reactive flow solver to obtain a desired flame shape.
We use a predictor-corrector approach (Holl et al., 2019) with a differentiable physics loss function and control sequence refinement for nonUniform-Bunsen case with 32 ×resolution.
The learning objective is to arrive at a target temperature field with the desired flame shape from an arbitrary, given initial configuration by controlling the velocity field.
We assume that the temperature and mass-fraction fields are observable and only the velocity field is controllable.
Figure 7 (a-e) shows the transition achieved and compares the target flame shape obtained using the neural network model with the target flame shape ( T ∗ ) for a test case.
Figure 7 (f,g) compares the flame shape obtained for additional test cases.
The neural network model is able to control the velocity of fuel-air mixture flow to obtain the desired flame shape.
To quantify these results, the learned model achieves MAPE of 1.87 ± 0.73% over 50 test cases considered.
It achieves an improvement of 52% compared to a baseline model without a predictor network for the long term prediction of temperature fields.
This baseline model has a MAPE of 3.91 ± 2.19%.
This case highlights the capabilities of differentiable multi-physics solvers, and that our full neural network model is able to reliably control the complex physics of the reactive flow over long time spans.
C ONCLUSIONS
These results demonstrate that neural networks can be employed for learning the completion of partial PDE solvers in the context of a multi-physics system.
The trained neural network model works alongside a partial solver to accurately account for and predict the effects of unknown physics.
The ability of the presented hybrid NN-PDE approach to predict the long-term temporal evolution of multi-physics systems was demonstrated on the specific case of a reacting flow in different configurations.
When compared with a purely data-driven approach and state-of-the-art FNO approach, the qualitative and quantitative performance of the hybrid NN-PDE model is found to be clearly superior.
The incomplete PDE description helps the neural network model recover the target simulation with a significantly improved accuracy.
This hybrid NN-PDE model is able to predict the correct evolution of the important features of the multi-physics system (in our case, the flame interface and flame shape) for longer simulation steps in all scenarios and is able to generalize to other (initial and boundary) conditions of the system.
This is demonstrated by variations in the equivalence ratio and inlet velocity forcing.
We also show that such hybrid approach may have the added benefits of allowing to relax numerical constraints linked to the potential stiffness of the unknown physics.
Our work represents a stepping stone for numerous avenues of future work, e.g., the proposed hybrid
NN-PDE solver could be further utilized to predict and control the dynamics of other tightly coupled systems (Levy, 1999; Dowell & Hall, 2001).
Here, the inclusion of transfer learning techniques would pose an interesting avenue for future work.
We note that the proposed approach does not seek to replace the classical physical simulation approaches.
However, it is an important step in the direction of harnessing the capabilities of neural network models using the knowledge of partial differential equations for complex multi-physical systems.
A PPENDIX A N EURAL N ETWORK A RCHITECTURE
All results provided in the main paper use a fully convolutional neural network.
Different input flow field quantities have different value ranges of the data e.g. T ∈
[400 , 2500] , Y f ∈ [0 , 0 . 05] .
Therefore, the input data provided to the neural network model is normalized using mean and standard deviation.
The output of the neural network model is then transformed back using the same mean and standard deviation.
When predicting correction of Y k fields, the output of the neural network is constrained to provide values in the range Y f ∈ [0 , 0 .
] , Y o ∈ [0 , 1] .
In the hybrid NN-PDE approach, the output of the neural network model is provided to the incomplete PDE solver as an input, as shown in figure 2.
Each network is trained for 100 epochs with learning rate of 0.0001.
An
L 2 based loss function is used as defined in section 3.3.
We additionally experimented with ResNet and UNet architectures for the neural network part of the purely data driven and hybrid NN-PDE approach discussed in the paper.
Table 2 provides details of the convolutional blocks, layers of the network, and activation function.
The UNet32 architecture with 2 layers is used for 32 × 32 cases and the UNet100 architecture with 3 layers is used for 100 × 100 case discussed in the paper.
Table 3 shows the comparison between MAPE achieved by ResNet vs. UNet architecture for the purely data-driven and the hybrid NN-PDE approach for different cases considered in the paper.
For the hybrid NN-PDE approach with 32 × 32 resolution cases, ResNet and
UNet achieve comparable performance with the UNet yielding the lowest error.
The main difference arises in the 100 × 100 resolution case.
The ResNet fails to converge during training, leading to high test error.
On the contrary, the UNet achieves low training and testing errors.
The main advantage of UNet comes from transforming high resolution input to low resolution representation and reconstructing it back to the high resolution output.
Therefore, we choose the UNet architecture for the hybrid NN-PDE approach.
For the purely data-driven approach ResNet was found to be better for 2 out of 3 cases.
Specifically for the nonUniform-Bunsen32 case, UNet performs very poorly for the PDD approach.
Therefore, we report ResNet errors for the PDD approach throughout the paper.
Figure 2 shows the details of the predictor block for PDD and hybrid NN-PDE approach.
[Table]
[Table] [Table] [Table]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
B D ATA G ENERATION
To generate input and target data for training, we simulate the temperature T , mass fractions Y f ,
Y o and velocity u x , u y fields of all flames under study.
Table 4 summarizes the boundary conditions applied for the planar and Bunsen-type flame cases discussed in the paper.
No-slip boundary conditions are used for the velocity field u y to obtain the Λ -shaped flames.
We consider variations in equivalence ratio E and inlet velocity amplitude u
y to obtain training and test datasets.
Figure 8 (A) shows the temperature field evolution of the incomplete PDE solver at different time instances for 3 different operating conditions for the uniform-Bunsen case.
Figure(B) shows the corresponding target training data for the uniform-Bunsen case.
All the simulations are run for 300 steps.
It shows the difference between non-reactive and reactive flow solver simulations for different operating conditions over 300 time-steps.
Figure 8 (C) shows the last snapshot ( t = 300 ) of 9 training simulations and 6 testing datasets with operating parameters interpolated between the training data parameters for uniform-Bunsen case.
These datasets are obtained by varying equivalence ratio and constant inlet velocity as discussed in section 4.
Flame temperature depends on the equivalence ratio used and flame height depends on the inlet velocity amplitude.
Figure 8 (D) showcases the snapshots of training and testing dataset at t = 500 for nonUniform-Bunsen32 case.
Non-uniform variations in inlet velocity profile leads to different complex flame shapes.
We usedatasets with 500 simulation steps to train the models and test it on 12 test cases shown in figure(D). [Figure] [Figure]
C A BLATION STUDY
Effect of training dataset size We study the effect of training dataset size on the accuracy of PDD predictions.
We train PDD models with different numbers of simulations in the training dataset.
Each simulation contains 500 time-steps. Figure 5 (A) shows the MAPE of PDD models trained with { 4 , 8 , 12 , 16 , 24 , 32 } training datasets, over a fixed testset. We compare the performance of these PDD models with an equivalent (trained with same look-ahead steps m = 2 ) hybrid NN-PDE model, trained with 12 simulation sets. Figure 5 (A) shows increasing the number of training sets has little or no effect on the prediction capabilities of the PDD models for nonUniform-Bunsen case.
The hybrid model with 12 training sets achieves a MAPE of 12.49 ± 4.17%, an improvement of 38% over the PDD model with 32 training sets. The purely data-driven models cannot achieve the same level of accuracy even in the presence of large amounts of data. Generalization to incorrect PDE parameters The proposed hybrid NN-PDE model is capable of completing the PDE description even if the underlying incomplete PDE solver has incorrect parameters.
We refer to the incomplete solver with incorrect parameters as ‘incomplete, incorrect PDE’.
We experiment with a modified hybrid NN-PDE approach wherein we combine an incomplete, incorrect
PDE solver with a neural network model.
We assume that the known values of the incomplete PDE parameters in equation 2, strain rate tensor ( τ ), the diffusion coefficient of species k
( D k ) and mixture thermal conductivity ( λ ), are incorrect. Figure 9 (A) shows the difference between temperature field evolution of the incomplete PDE solver with correct parameters ( [ τ, D k , λ ] = [0 . 1 , 0 . 1 , 0 . 1] ) and incorrect parameters ( [ τ, D k , λ ] = at different time-steps.
The hybrid NNPDE combines this incomplete, incorrect PDE solver with the neural network model to obtain the solutions of the complete PDE solver with correct parameters.
Figure 9 (B) compares the flame dynamics predicted by this hybrid NN-PDE model with that of the complete, correct PDE solver.
A good match is observed over various test cases.
The hybrid model with incomplete, incorrect solver achieves an overall MAPE of 2.48 ± 1.20 %, compared to the MAPE of 2.04 ± {+1.39 % for hybrid model with incomplete, correct PDE. M A P E (A) Planar-v FNO PDD Hybrid     
+}
M A P E (B) uniform-Bunsen
Testset number
M A P E
[Figure]
Effect of longer look-ahead steps We evaluate the effect of varying look-ahead steps on the performance of the hybrid model.
We show the comparison of models trained with m = { 2 , 4 , 8 , 16 , 32 } for different cases in figure 10.
Models with smaller m (2) do not learn to accurately correct the fields over long time and quickly diverge from the target simulation.
Using larger m improves the quality of prediction drastically as the model learns the correction via the gradients over longer simulation steps.
For Planar-v0 and uniform-Bunsen cases, iterating the NN and PDE solver for 32 timesteps improves the accuracy by 81.7% and 94% respectively compared to the m = 2 model.
For nonUniform-Bunsen32 and nonUniform-Bunsen100, the model performance improves by 84.2% and 70.9% respectively, by using m = 32 instead of m = 2 model.
Effect of temporal coarsening Figure 11 and table 1 compare the MAPE of temperature field predictions at time-step ∆ t c , for all three scenarios.
As seen from the figure 11, the hybrid NN-PDE approach consistently performs better than both the baselines for all three scenarios considered.
Higher standard deviations in Table 1 indicate the large differences in the prediction of flame temperatures for baseline approaches.
The multi-physics systems we study provide a substantially more difficult environment than regular fluids: the chemical reactions are numerically very stiff, and the resulting dynamic interactions are difficult to capture by numerical solvers.
To further illustrate the temporal behavior of the simulations, we perform additional experiments with the baseline approaches using twice the time-step size of the complete solver ( 2∆ t c ).
As shown in figure 12, for the
Planar-v0 and uniform-Bunsen case, PDD catches up with the hybrid approach whereas the PDD approach completely fails to predict the dynamics of the nonUniform-Bunsen32 case.
We further investigate the nonUniform-Bunsen32 case with larger time-steps (2 times, 4 times and 8 times) as shown in figure 5 (B).
It shows MAPE of the predictions over all the testsets and simulation steps by the PDD model.
The PDD approach does not predict the dynamics accurately for any of the larger time-steps considered.
Figure 13 visualize these results over different time-steps for one of the test cases.
The predictions made by FNO, PDD and hybrid NN-PDE model for nonUniform-Bunsen case at twice the time-step of 2∆ t c , are compared with the ground truth solutions coming from the complete PDE solver at time-step of ∆ t c .
FNO and PDD (upper two rows) fail to recover the correct flame shapes over 250 simulation steps.
The hybrid NN-PDE model (second row from below) predicts the flame shape accurately, thus relaxing the temporal stiffness of the complete PDE solver.
D A DDITIONAL R ESULTS
In this section, we present additional visualizations of all reactive flow scenarios considered in section 4 for different test cases.
Additionally, we discuss the results of the mass fraction fields we obtained.
Table 5 summarizes MSE values of fuel ( Y f ) and
oxidizer
( Y o ) mass fraction fields for [Figure] [Figure] different baselines and the hybrid NN-PDE approach.
We observe similar performance to the temperature field predictions.
The hybrid NN-PDE model consistently outperforms the FNO and PDD approaches for all the scenarios considered.
The hybrid NN-PDE model is able to simultaneously correct temperature and mass fraction fields, thus successfully modeling the effects of unknown chemical reactions on all the fields of the flow state [Figure]
D. P LANAR - V
Figure 14(A) shows the 2D visualization of the temperature field predictions for Planar-v0 case.
As seen from the ground truth images, methane-air mixture (black color) converts into the burned products (yellow color) due the chemical reaction at the flat flame surface (red color).
The dotted, horizontal red line helps to compare the transition of the flame interface, i.e. the displacement of the flame front.
Due to the chemical reaction, fuel-air mixture is consumed and turns into burned product as the simulation progresses.
The FNO approach completely fails to predict the propagation of the flame for the given test case.
Its output does not show any evolution from the initial temperature profile for the given operating condition.
The purely data-driven approach fails to capture the flame front displacement correctly, thus leading to an inaccurate prediction with large errors.
The
[Figure]
[Figure]
[Figure]
[Table]
hybrid NN-PDE model accurately captures this evolution of planar methane-air flame in a quiescent mixture.
Figure 15 shows the instantaneous MAPE w.r.t.
ground truth data for predictions shown in figure 14.
The absolute error shown in figure 15(A) exceeds 1100 K for the FNO and purely datadriven approaches as these do not predict the flame temperature correctly.
We use the upper limit of 1100 K for colorbar to highlight the errors in the hybrid NN-PDE approach more clearly.
Large errors in FNO and PDD results stem from their inability to reliably predict the flame temperature and flame front displacement.
D. UNIFORM -B UNSEN
Figures 14(B) and 15(B) show an example of uniform-Bunsen case; a test case with u y = 0 .
375 and
E = 0 .
The constant inlet velocity results in a symmetric, Λ -shaped flame.
Vertical, dotted, red line in figure 14(B) helps to assess the symmetric nature of the flame.
The purely data-driven model predicts a thicker flame ( t = 50 ) or a flame with a spurious tip ( t = 172 ) or an asymmetric flame ( t = 225 ).
Snapshots of the hybrid NN-PDE model predictions in figure 14(B) show that it adapts to this scenario very well and succeeds in obtaining the correct results for long term forecasts of the temperature field.
Furthermore, the flame shape and height are also better predicted, as shown in figure 14(B).
Very low error levels in figure 15(B), such as ϵ = 1 . 26 at t = 300 , indicate that the hybrid model recovers the temperature field satisfactorily.
Figure 16 compares the performance of the hybrid NN-PDE model for the Y f and Y o fields for two different test cases.
We again observe that the hybrid NN-PDE model predicts all the fields accurately with very low MSE.
D. NON U NIFORM -B UNSEN
Figure 4 compares the temporal predictions made by FNO, PDD and the hybrid NN-PDE model with ground truth data.
Compared to Planar-v0 and uniform-Bunsen case, FNO predicts qualitatively better results for this case.
Although highly still accurate, it predicts shapes that come closer to the target flame shapes for the later part of the simulation (for t > 300 ).
This improvement might be due to the larger training dataset used for this scenario compared to the previous two scenarios.
However, it fails to predict accurate temporal predictions over the entire simulation.
The hybrid
NN-PDE approach satisfactorily predicts the accurate flame evolution.
Additionally, we showcase the performance of the hybrid NN-PDE approach on two different test cases with complex flame shapes in figure 18.
The neural network model along with incomplete PDE solver reconstructs these complex flame shapes in an accurate manner.
Finally we showcase the performance of the hybrid NN-PDE approach on different test cases for highly resolved flame case of nonUniform-Bunsen100.
Figure 19 compares the predictions made by hybrid NN-PDE over { 0 , 100 , 200 , 250 , 300 , 325 , 350 , 400 , 450 , 500 } simulation steps with the ground truth data.
It also shows the absolute difference between them.
As the simulation progresses, higher errors are observed around the flame front.
However, the hybrid NN-PDE approach captures the flame shape very accurately for longer roll-outs of 500 simulation steps.
Currently this approach uses training dataset similar to nonUniform-Bunsen32.
Further improvements in accuracy can be
[Figure]
[Figure] [Figure] [Figure] achieved by increasing the training dataset size or training the hybrid model with longer look-ahead steps.
[Figure] [Figure]
t
[Figure] [Figure]
|
P c ( T t )
[Figure]
̃ T t |
[Figure]
r
[Figure]