From Deep to Physics-Informed Learning of Turbulence: Diagnostics
Abstract
We describe tests validating progress made toward acceleration and automation of hydrodynamic codes in the regime of developed turbulence by three Deep
Learning (DL) Neural Network (NN) schemes trained on Direct Numerical Simulations of turbulence.
Even the bare DL solutions, which do not take into account any physics of turbulence explicitly, are impressively good overall when it comes to qualitative description of important features of turbulence.
However, the early tests have also uncovered some caveats of the DL approaches.
We observe that the static DL scheme, implementing Convolutional GAN and trained on spatial snapshots of turbulence, fails to reproduce intermittency of turbulent ﬂuctuations at small scales and details of the turbulence geometry at large scales. We show that the dynamic
NN schemes, namely LAT-NET and Compressed Convolutional LSTM, trained on a temporal sequence of turbulence snapshots are capable to correct for the caveats of the static NN.
We suggest a path forward towards improving reproducibility of the large-scale geometry of turbulence with NN.
Introduction
In this manuscript, reporting ﬁrst results of the approach focusing on developing a Physics-Informed
Machine Learning (PIML) framework to improve turbulence model implementations in hydrodynamic codes, see also the Supplementary Material (SM) A, we aim to verify whether various statistical properties, constraints and relations not enforced explicitly within the DL training on the ground truth, Direct Numerical Simulation (DNS), data hold.
To do this, we compare results extracted from the training data and from the generated/synthetic data.
Three DL schemes, GAN of (1), LAT-NET of (2), generalized to arbitrary solvers and newly developed Compressed Convolutional Long Short-Term Memory (CC-LSTM) scheme are juxtaposed within the setting of the homogeneous, isotropic, stationary turbulence.
Speciﬁcally, we have veriﬁed (a) spectrum of energy ﬂuctuations over scales, (b) statistics of velocity gradient (small/viscous scale object), (c) anomalous exponents of higher order velocity increments, and (d) statistics of the coarse-grained velocity-strain alignment represented in the plane of the coarse-grained velocity gradient tensor.
Logic behind and details of the diagnostics, suggested in our prior work on the subject (1), are described in the SM B. Static
GAN DL scheme is trained on the spatial data (instantaneous snapshots) from the Johns Hopkins turbulence database (3).
The dynamic schemes, LAT-NET and CC-LSTM, are trained on the dynamic (sequence of snapshots) data from the SpectralDNS code (4).
Description of the DL schemes and results of applying the two step diagnostics to the the DL schemes are described in Section
We draw conclusions and suggest a path forward in Section 3.
[Figure]
Deep Learning for Turbulence
Generating Turbulence through Adversarial Training.
Recent advances in DL have proven remarkably successful on image-based problems such as generation, classiﬁcation, and denoising.
Much of the success of these DL techniques relies on the hierarchical identiﬁcation and abstraction of features present in images using deep networks.
A complex superposition of structures also occurs in turbulent ﬂows, naturally leading us to examine whether techniques developed for images can be used to learn turbulent ﬂow physics.
It was demonstrated in (1) that Generative Adversarial Networks (GANs) can be trained on high ﬁdelity DNS to generate high-quality synthetic images of turbulent ﬂows.
GAN was originally introduced by Goodfellow et al, (5), and relies on competition between two deep Neural Networks (NNs).
The ﬁrst network, the generator, attempts to create synthetic images that appear to be drawn from the correct high dimensional distribution.
The second network, the discriminator, compares the proposed images against a set of training data and estimates the likelihood of the image being genuine.
A min-max optimization problem is solved iteratively to train both networks’ parameters as they compete against each other.
Over time, the generator learns to draw more plausible images while the discriminator’s ability to distinguish real and synthetic images also improves.
Our GAN implementation makes use of the Convolution (C-GAN) for both the generator and discriminator using the architecture guidelines for stable deep C-GANs recommended by Radford et al (6) and Reed et al (7).
We train on 2D slices of 3D homogeneous isotropic turbulence from the
Johns Hopkins Turbulence Database (3).
The three velocity components map naturally to different red, green, and blue color channels in imaged-based deep learning networks.
We calculated a number of common turbulence quantities on the GANs output as part of our survey of DL techniques.
The results, summarized in Figure 1, suggest that the generator has successfully learned to sample from the full multi-point Probability Distribution Functions (PDFs) of valid turbulent ﬂow ﬁelds.
LAT-NET.
Dynamic approach to modeling turbulence is to make temporal predictions given an initial state of the ﬂow.
Toward this end, the test was a modiﬁed version of LAT-NET (2).
LAT-NET works by compressing a snapshot of the ﬂow onto a compact latent space and then mimicking the dynamics of the ﬂow on this latent space.
Doing this allows the network to generate the ﬂow simulation for signiﬁcantly less memory and computation then the ﬂow solver.
A convolutional autoencoder is used to create the mapping to and from the latent space while a separate convolutional network is applied iteratively on the latent space to reproduce the dynamics of the underlying ﬂow.
Fig. ??
presents the scheme of operations where the mapping on the latent space is referred to as the latent mapping.
Each application of the latent mapping is meant to ditto a ﬁxed amount of time in the simulation.
In our case, each compression mapping corresponds to 0 .
06 sec in the simulation thus applying the mapping 10 times propagates the predicted ﬂow forward in time 0 .
6 seconds.
The Lat-Net method was ﬁrst
Lat­Net
Iter
{+True Iter
+}
Lat­Net
{+Iter True Iter0 20 
 Lat­Net Iter0 20 
 True Iter proposed+} to predict lattice Boltzmann ﬂuid simulations however we have relaxed this constraint so that the network can be used on ﬂow data regardless of the underlying solver.
We have also removed the boundary conditions as our tests are on homogeneous ﬂow with no physical boundaries.
Training the network is also modiﬁed from the original work by splitting the training process into two phases.
First, the autoencoder is trained on snapshots of the data and then frozen.
Next, the latent mapping is trained on the compressed data in a sequential fashion in Fig.
Breaking up the training process both speeds up learning and reduces the memory usage allowing larger domains to be trained on.
Results of the four diagnostic tests, described above and SM B, applied to the dynamic NN LAT-NET are shown in Fig.
Compressed Convolutional LSTM.
Here we describe another dynamic scheme based on the Convolutional LSTM by Shi (27), exploiting the strength of the convolutional networks (which captures spatial features) and Long Short Term Memory i.e. LSTM networks (which capture temporal patterns) together in a combined architecture, and also extending it to the case of 3d images.
To reduce dimensionality of the data we also compress and decompress input and output images, respectively, by means of two 3d convolutional autoencoders.
The resulting Compressed Convolutional LSTM (CC-LSTM) architecture is illustrated in Fig.
The results from the CC-LSTM scheme can be seen in Fig.
Results for energy spectra and for statistics of the velocity gradient show good-to-reasonable consistency between the synthetic and training data. QR diagnostics show good reproduction by the synthetic data of the training data trends at the larger scales. Respective deviations are signiﬁcant at the smaller scales – this is a parasitic effect attributed to insufﬁcient spatial resolution of the spatio-temporal (thus more demanding then pure spatial) schemes.  Conclusions and Path Forward
In this manuscript we tested three DL schemes in their ability to reproduce turbulent ﬂows.
We introduce four tests of increasing complexity.
Our ﬁrst scheme, implementing C-GAN, is static, i.e. it takes as an input only instantaneous snapshots of the turbulence ﬁeld, completely ignoring any dynamical aspects of turbulence.
The scheme paths our ﬁrst test, testing distribution of energy over scales, with ﬂying colors.
The second test, checking statistics of the velocity gradient which is a small scale object, reveals that GAN underestimates tails of the PDF of the velocity gradient, making generated samples less intermittent then the DNS input.
Third test, of the intermittency (non-Gaussianity) at the intermediate scales (from inertial range of turbulence) via analysis of velocity increments, was passed by the C-GAN scheme well.
Finally, the most elaborate fourth test, consisting in checking statistics of the coarse-grained velocity tensor, reveals a difﬁculty of GAN to reproduce details of the vorticity-strain alignment at the largest (energy containing) scales of turbulence.
Based on the analysis of the static NN scheme we naturally came to the question – if dynamic NNs, trained on temporal sequence of snapshots, is capable of correcting for the static scheme deﬁciencies in reproducing (a) intermittency of the smallest (close to viscous) scales, and (b) statistics of the vorticity-strain alignment of the largest (close to the energy containing) scales?
To resolve the question we analyzed the dynamic LAT-NET and CC-LSTM NN schemes trained on temporal sequence of turbulence snapshots.
We observed that the dynamic schemes perform on the three-test diagnostics (excluding the anomalous scaling test) at least as well as the static [Figure] [Figure]
[Figure] NN scheme. Moreover, we discovered that the small scale intermittency (feature (a) above) is now reproduced well, therefore correcting for the caveat of the static NN.
We attribute this success of the dynamic schemes to the direct cascade nature of turbulence – turbulent dynamics makes small scale statistics universal, i.e. minimally sensitive to larger scale details of the vorticity-strain alignment.
We also note that CC-LSTM is reproducing geometry of the ﬂow better at the larger scales than at the smaller scales, while LAT-NET show the opposite tendency.
Sub-optimal performance of the two dynamic schemes in the anomalous scaling test is attributed to problems with insufﬁcient spatial resolution. This analysis suggests that dynamic networks bring in additional information which improves reproduction of intermittency at smaller scales and geometry of the ﬂow at large scales.
A Physics Informed Machine Learning: The Methodology
[Equation]
[Figure] closures
Physics Informed Tuning:
Computational Fluid Mechanics 1970 -
• class of physics-functions is narrow • few parameters, trained manually
Physics Informed Machine Learning: 2017- on going
• class of physics+NN-functions is broad • parameters are trained
automatically parametrized schemes, e.g. Neural Networks • equations free • many parameters • trained automatically
Physics Free 2015Machine Learning !"
[Equation]
$ ) Fully resolved Direct Simulation = data : " = " # + " ( sub- filtered low-pathfiltered : " # 0 → → " # ($) parametrizedmap model with parameters
I n t e r p r e t a b ilit y /
A cc u r ac y
[Figure]
Tuning
Direct
Simulations
Physics
Informed
Machine
Learning
Physics
Free
Machine
Learning train parameters = match data & mdel model
The essence of the Physics Informed Learning (PIML), illustrated in Fig.
(4), is in capturing physics with the right amount of domain speciﬁcity and interpretability of the Physics Informed Tuning (PIT) non-automatic approach, expressed e.g. in symmetries and constraints, while also providing prediction power and computational tractability which is on pier with state of the art Machine
Learning, such as Deep Learning (DL), which is typically Physics Free, thus PFML.
Development of the PIML approach, which we expect to be transferable to many areas of natural and engineering sciences, is anchored in this manuscript to ﬂuid mechanics.
Majority of ﬂuid ﬂows of interest are turbulent, i.e. containing rich spatio-temporal correlations.
Goverments, Industry and Science and
Engineering communities continue to make signiﬁcant investments in computational software and hardware to solve turbulence in multiple situations of interest.
We conjecture that, when developed, the PIML approach will allow to accelerate turbulent computations by orders of magnitude through separation of scales into macro-scales that are simulated and sub-scales that are efﬁciently emulated based on PIML learned dynamics.
Moreover, the acceleration will be automatic allowing program experts to focus on fewer ad-hoc adjustments of reduced models than is custom today.
Also, since a low-dimensional sub-scale state is known the development of closure models (coupling of scales) is greatly simpliﬁed.
In the simplest setting of interest we train parameters of a ML model on the data from Direct
Numerical Simulations (DNS) to predict important features of the ﬂow faster than the DNS.
Our main hypothesis is that tremendous acceleration is achievable with PIML.
The hypothesis is based on a number of recent reports (1; 9; 10; 2; 11; 12; 13; 14) that DL schemes developed originally for popular
IT industry applications, when applied to DNS data as is, do provide an acceleration.
Developing scientiﬁc diagnostics for testing and juxtaposing the bare DL approaches and then augmenting DLs based on the results of the diagnostics becomes critical.
Complementarily we expect, based on another set of recent publications (15; 16; 17; 18; 19) that a signiﬁcant acceleration is also achievable with properly relaxed and parameterized ﬂuid-mechanics phenomenological models, e.g. of Large
Eddy Simulations (LES) and Reynolds Averaged Navier Stocks (RANS) types, with much fewer parameters than in the DL models.
PIML connects the two approaches to extract robust winning schemes from a number of analyzed synergistic solutions.
Future PIML solutions should be customized to data from DNS of a sequence of turbulence models of increasing complexity, starting from steady homogeneous, isotropic turbulence and extending to non-stationary, non-isotropic turbulent ﬂows involving active (chemical and nuclear) mixing.
This strategy will allow us, in particular, to probe and validate transferability of the PIML models trained on simpler cases to more complex situations.
B Turbulence Metric
In this Section we review basic statistical concepts commonly used in the modern literature to analyze results of theoretical, computational and experimental studies of homogeneous isotropic incompressible turbulence in three dimensions.
Combination of these concepts are used in the main part of the manuscript as a metric to juxtapose results of the two (static and dynamic) DL methods.
We assume that a 3 d snapshot, or its 2 d slice, or a temporal sequence of snapshots of the velocity ﬁeld, v = ( v ( i ) ( r ) | i = 1 , · · · , 3) , is investigated.
We will focus here on analysis of static correlations within the snapshots.
We consider various objects of interest, e [Equation] g. correlation functions of second, third and fourth orders
[Equation]
Rich tensorial structure of the correlation functions carry a lot of information.
It also suggests a number of useful derived objects, each focusing on a particular feature of the turbulent ﬂows.
In particular, we may be to discuss structure functions, deﬁned as tensorial moments of the increments between two points separated by the radius-vector r :
[Equation]
Other objects of interest, derived from the correlation function by spatial differentiation and then merging the points are moments of the velocity gradient tensor, m ( i,j )
= ∇ ( i ) v
[Equation]
We may also be interested to study mixed objects, e.g. the so-called energy ﬂux
[Equation]
The remainder of this Section is organized as follows.
We describe important turbulence concepts mentioned in the main part of the paper one by one, starting from simpler ones and advancing towards more complex concepts.
4 / 5 Kolmogorov law and the Energy Spectra Main statement of the Kolmogorov theory of turbulence (in fact it is the only formally proved statement of the theory) is that asymptotically in the inertial range, i.e. at L (cid:29) r (cid:29) η , where L is the largest, so-called energy-containing scale of turbulence and η is the smallest scale of turbulence, so-called Kolmogorov (viscous) scale, F ( r ) does not depend on r .
Moreover, the so-called 4 / 5 -law states for the third-order moment of the longitudinal velocity increment
[Equation]
where ε = νD ( i,j ; i,j )/ 2 is the kinetic energy dissipation also equal to the energy ﬂux.
Self-similarity hypothesis extended from the third moment to the second moment results in the expectation that within the inertial range, L (cid:29) rη , the second moment of velocity increment scales as, S 2 ( r ) ∼ v L ( r/L ) 2 / 3 .
This feature is typically tested by plotting the energy spectra of turbulence (expressed via S 2 ( r ) ) in the wave vector domain, e.g. as shown in Figs.
(1a,3a) of the main text.
Normal and Anomalous scaling of velocity increments One expects moments of the velocity increment to show the following scaling behavior inside the inertial range of turbulence
[Equation]
(12) where L is the energy containing (largest) scale of turbulence, η ∼ ( ν/v L ) 3 / 4 L 1 / 4 is the Kolmogorov (viscous) scale, v L is the typical velocity ﬂuctuations at the energy containing scale, ν is the viscosity coefﬁcient and ∆ n is the so-called anomalous scaling.
∆ 3 = 0 , and ∆ n is a growing function of n .
In 1941 A. Kolmogorov (20; 21; 22) hypothesized self-similarity of turbulence, i.e. that the anomalous scaling is absent, ∀ n = 3 , · · · : ∆ n = 0 .
Criticized by L. Landau (see e.g. (23) for related history notes), Kolmogorov reconsidered self-similarity in 1962 (24), suggesting instead the so-called reﬁned (log-normal) similarity hypothesis, ∆ n = µn ( n − 3) / 16 , which was then conﬁrmed to be rather accurate in experiments and simulations, see e.g. (25).
The intermittency test was implemented for both static and dynamic NN schemes discussed in the main part of the manuscript.
Both static and dynamic NN schemes passed the test sucessfully.
We do not show results in the main part of the manuscript because of the space limitations.
Intermittency of Velocity Gradient Consistently with Eq.
(12), estimation of the moments of the velocity gradient results in
[Equation]
where dependence of the r/L - and v L - independent pre-factors in both Eq.
(12) and Eq.
(13) on n is ignored.
Intermittency (extreme non-Gaussianity) of turbulence is expressed the strongest in Eq.
Statistics of coarse-grained velocity gradients: Q − R plane.
Isolines of probability in the Q − R plane, expressing intimate feature of the turbulent ﬂow geometry, has a nontrivial shape documented in the literature.
See (8; 26) and references therein.
Different parts of the Q − R plane are associated with different structures of the ﬂow.
Thus lower right corner (negative Q and R ), which has higher probability than other quadrants, corresponds to a pancake type of structure (two expending directions, one contracting) with the direction of rotation (vorticity) aligned with the second eigenvector of the stress.
This tea-drop shape of the probability isoline becomes more prominent with decrease of the coarse-graining scale.