F REQUENCY - AWARE I NTERFACE D YNAMICS WITH G ENERATIVE A DVERSARIAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
We present a new method for reconstructing and reﬁning complex surfaces based on physical simulations.
Taking a roughly approximated simulation as input, our method infers corresponding spatial details while taking into account how they evolve over time.
We consider this problem in terms of spatial and temporal frequencies and leverage generative adversarial networks to learn the desired spatiotemporal signal for the surface dynamics.
Furthermore, we investigate the possibility to train our network in an unsupervised manner, i e. without predeﬁned training pairs.
We highlight the capabilities of our method with a set of synthetic wave function tests and complex 3D dynamics of elasto-plastic materials
I NTRODUCTION
Complex and chaotic physical phenomena such as liquids, gels and goo are still very challenging when it comes to representing them as detailed and realistically as possible.
A variety of numerical methods have been proposed to simulate such materials, from purely Eulerian methods (Harlow &
Welch, 1965; Stam, 1999), over particle based methods (Gingold & Monaghan, 1977; Ihmsen et al., 2014), to hybrids (Zhu & Bridson, 2005; Stomakhin et al., 2013).
Such simulations have also been targeted with deep learning methods (Tompson et al., 2017; Mrowca et al., 2018; Li et al., 2019), but despite signiﬁcant advances, they remain very time-consuming and highly challenging to solve.
One approach to speed up the necessary calculations and to allow for more control is to employ super-sampling .
This can be seen as a form of post-processing where one simulates only a lowresolution simulation and uses an up-sampling technique to approximate the behavior of a highresolution simulation.
Neural networks are of special interest here because of their capability to efﬁciently approximate the strongly nonlinear behavior of physical simulations.
Applying neural networks to space-time data sets of physical simulations has seen strongly growing interest in recent years (Ladicky et al., 2015; Kim et al., 2020), and is particularly interesting in this context to incorporate additional constraints, e.g., for temporal coherence (Xie et al., 2018), or for physical plausibility (Tompson et al., 2017; Kim et al., 2019).
An important aspect here is that methods based on simple distance losses, such as mean square errors, quickly reach their limits.
The generated data tends to be smooth without the necessary small-scale features.
Generative adversarial networks (GANs) have been proposed to overcome this issue (Goodfellow, 2016).
They are characterized by the fact that, apart from a generative network, they also make use of a discriminator that classiﬁes the results of the generator with respect to the ground-truth data.
Via a joint training, the distribution of solutions of the generator is guided to approximate the ground-truth data distribution.
As the quality of the results is primarily determined by the discriminator network, it remains an open problem to accurately evaluate the quality of the inferred results.
In our work we propose to evaluate the problem in the Fourier space.
In this way, we are able to evaluate the given methods reliably, and it allows us to design improved learning algorithms that more faithfully recover the small scale details of the reference data.
For the core of our method, we build on an existing GAN-based architecture that employs two discriminator networks, one for the spatial and one for the temporal behaviour (Xie et al., 2018).
In terms of ground truth data, we focus on multi-phase (solid-ﬂuid-air) interactions with a sharp ﬂuid-air interface.
Unlike single-phase ﬂow whose details are visible and relevant solely due to transparency throughout the volume, the details of our data are in most cases only visible on the surface.
Of course, the internal dynamics in the volume also play a role, but they are mostly hidden from the viewer, only the effects on the surface are visible.
Furthermore, we consider phenomena that build up and take place over the course of several frames.
Thus, as we will outline below, we employ a recurrent approach that is conditioned on a previous output in order to produce the solution for a subsequent timestep.
In order to represent and process ﬁne details, we treat such detail as high-frequency displacements of a low-frequency surface, and correspondingly formulate the problem in Fourier space.
The transformation into Fourier space yields an isolated view of the individual frequencies, and thus allows for a much improved analysis of the results achieved by different methods.
E.g., it robustly identiﬁes the strong smoothing behavior of L 2 metrics, and can detect mode collapse problems of adversarial training runs.
We also demonstrate how frequency information can be incorporated into the learning objective in order to improve results.
To summarize, the central contributions of our work are: (1) A method for frequency evaluation with a consideration of spatial properties, (2)
A novel frequency aware loss formulation, (3)
A simple, yet intuitive evaluation of different generative methods, (4) A time consistent spatio-temporal upsampling of complex physical surfaces.
Related Work Deep learning methods in conjunction with physical models were employed in variety of contexts, ranging from learning models for physical intuition (Battaglia et al., 2016;
Sanchez-Gonzalez et al., 2018), over robotic control (Schenck & Fox, 2018;
Hu et al., 2019) to engineering applications (Ling et al., 2016; Morton et al., 2018).
In the following, we focus on ﬂuid-like materials with continuous descriptions, which encompass a wide range of behavior and pose challenging tasks for learning methods (Mrowca et al., 2018; Li et al., 2019).
For ﬂuid ﬂows in particular, a variety of learning methods were proposed (Tompson et al., 2017; Prantl et al., 2017;
Um et al., 2018).
A common approach to reduce the high computational cost of a simulation is to employ super-resolution techniques (Dong et al., 2016; Chu & Thuerey, 2017; Bai et al., 2019).
In this context, our work targets the up-sampling for physics-based animations, for which we leverage the approach proposed by Xie et al.
However, in contrast to this work, we target phenomena with clear interfaces, which motivates the frequency-based viewpoint of our work.
For sharp interfaces, Lagrangian models are a very popular discretization of continuum mechanical systems.
E.g., smoothed particle hydrodynamics (SPH) (Gingold & Monaghan, 1977; Koschier et al., 2019) is a widely-used particle-based simulation method.
While points and particles are likewise frequently used representations for physical deep learning (Li et al., 2019; Ummenhofer et al., 2019; Sanchez-Gonzalez et al., 2020), Eulerian, i.e., grid-based representations offer advantages in terms of efﬁcient and robust kernel evaluations.
We employ generative adversarial networks (Goodfellow, 2016), as a powerful and established method for learning generative models.
Here, ”unconditional” GANs typically rely on a synthetic input vector from Gaussian noise to produce the desired output distribution, e.g., the DC-GAN approach (Radford et al., 2016).
Conditional GANs (Mirza & Osindero, 2014) were introduced to provide the network with an input that allows the neural network to steer the generation of the output.
Hence super-resolution tasks for natural images (Ledig et al., 2016), or image translation tasks (Isola et al., 2017) employ conditional GANs.
The time dimension was also taken into account in natural imaging works, e.g., by Saito et al.
in the form of a temporal generator (Saito et al., 2017), or via a stochastic sequence generator (Yu et al., 2017).
Other works have included direct L 2 loss terms as temporal regularizers (Bhattacharjee & Das, 2017; Chen et al., 2017), which, however, typically strongly restricts the changes over time.
Similar to ﬂow advection, video networks also often use warping information to align data over time (Liu et al., 2017; de Bezenac et al., 2017).
We will demonstrate that recurrent architectures similar to those used for video super-resolution (Sajjadi et al., 2018) are likewise very amenable for physical problems over time.
M ETHOD
The input for our method is a coarsely approximated source simulation, with the learning objective to infer the surface of a target simulation over space and time.
This target is typically computed via a potentially very costly, ﬁnely resolved simulation run for the same physical setup.
When it comes to the possibilities of simulation representations, there is a great variance.
In our case we have chosen an implicit representation of the data, by a signed-distance ﬁeld (SDF) denoted by g : R 3 !
R .
˜ Y t X t   t  [0, N t ) (a) Generator.
Y t  1 , Y t , Y t + ˜ Y t  1 ,˜
Y t ,˜
Y t +1 ˜
Y t  s t  [0, N t )
Y t (0,1)
t t  [1, N t  1) (0,1)   v t  v t
[Figure]
An SDF returns, for a given point, the signed distance to the surface, with negative being inside the medium.
Such a function is realized in practice by a grid
X 2 R M x ⇥ M y ⇥
M z , storing the precomputed signed distance values.
We have chosen this representation because most neural network layers are designed for array-like representations, and the loss functions on grid-based data are very efﬁcient to evaluate.
Additionally, an implicit representation via a grid can leverage tools from the ﬁeld of level-set processing (Adalsteinsson & Sethian, 1999), and facilitates the frequency viewpoint via a Fourier transformation.
Additional values, like the velocity, are also mapped on a grid V
R M x ⇥ M y ⇥ M z ⇥ 3 .
Our goal is to let a generative network G : R M x ⇥ M y ⇥ M z ⇥ 4 !
R N x ⇥ N y ⇥
N
z infer a grid ˜ Y which approximates a desired high-resolution simulation Y 2 R N x ⇥ N y ⇥ N z with
N ⇤ = kM ⇤ , N ⇤ 2 N { x,y,z } and up-sample factor k 2 N , i.e. G ( X ) =
˜ Y ⇡ Y .
As our method only requires position and velocity data from a simulation, it is largely agnostic to the type of solver or physical model for generating the source and target particle data.
2. N EURAL N ETWORK F ORMULATION
Our method is based on a generative, neural network with a 3D fully-convolutional ResNet architecture (He et al., 2016) that produces an output ﬁeld at a single instance in time.
The low-resolution input data is ﬁrst up-sampled with a tri-linear up-sampling and then processed with several convolutional layers, as shown in Figure 1a.
We use leaky ReLU as activation function after each layer, except for the last layer, where we use a tanh activation.
In our case, the input data consists of the implicitly represented geometry data X t , the velocity V t of the simulation as well as the results of a previous pass ˜ Y t .
The previously generated data is advected with the low-resolution velocity before further processing.
Through this feedback loop we train our network recurrently by iterating over a sequence of T = 10 frames.
This yields stability over longer periods of time and gives better insights about temporal behaviour.
Furthermore, the recurrent training is important to enable persistent behavior over time, such as the progression of ﬁne surface waves.
Unlike the process for generating the input data, the network training cannot resort to a physical simulation with full resolution, and hence cannot uniquely determine the evolution of future states.
Therefore, its main learning objective is to capture the dynamics of the target simulations beyond that basic motion computed with an advection step.
For initialization of the undeﬁned ﬁrst frame
˜ Y  1 we use a tri-linear up-sampled version of the input.
To train our network we have to deﬁne ﬁrst a loss function that allows us to evaluate the differences between generated and ground-truth data.
The most basic loss function is a simple mean squared error (MSE):
[Equation]
This has the big disadvantage that it is ill-suited to measure the similarity or differences of solutions.
For example, considering a function with multiple solutions for a given input, i.e., a multi-modal setting, a method that trains with an MSE loss will learn the expected value of the output distribution, i.e, the average of the different solutions.
However, the average is typically not a part of the solution set.
Thus, the MSE loss often does not correspond to the correct distance in solution space, based on signiﬁcant factors corresponding to the distribution of the solutions.
Our super-sampling setup is such a problem: Due to the low resolution input, the high resolution details cannot be determined uniquely, resulting in a variety of possible solutions when up-sampling.
Via physical properties of the material and its temporal sequence, some solutions can be eliminated, but nonetheless the space of solutions typically remains inﬁnitely large.
If an MSE loss is used, all such samples from the training data set are simply averaged to obtain a mean value, so that the result does no longer reﬂect the level of detail of the ground-truth data.
The MSE loss nevertheless gives a rough direction, and provides a stable learning target.
Hence, we still use it as a component in the ﬁnal loss formulation, in combination with an adversarial loss.
In contrast to a direct distance metric, the adversarial loss approximates the ground-truth distribution.
Hence, the network no longer learns one mean value, but chooses one valid solution out of the possible ones.
We deﬁne a discriminator D s that takes as input a high-resolution version of a simulation frame and classiﬁes it, distinguishing between ground-truth and generated frames.
It does this through a binary output, where 0 is ”fake” and 1 is ”real”.
Its task is to provide the generator with feedback on the correctness of the given data.
The special feature is that the discriminators are trained together with the generator, thus creating a competitive interaction where both parties improve each other.
As loss for the discriminator we use a binary cross-entropy:
[Equation]
where y is the ground-truth and ˜ y is the generated value from the discriminator.
For complex tasks,
GANs can be unstable and difﬁcult to control.
For this reason we additionally use the recent Spectral
Normalization (Miyato et al., 2018), which we found to provide more stable adversarial training.
While we have primarily focused on spatial content, i.e., the surface of the material so far, the temporal behavior likewise plays a crucial role, and poses similar difﬁculties in our multi-modal setting.
On the one hand, the generation of details can quickly lead to temporally incoherent results, which is characterized by unappealing ﬂickering.
On the other hand, our network also should be able to match and recreate spatial solutions over time that reﬂect the physical behavior.
Following previous work (Xie et al., 2018), we use an additional discriminator D t to classify the temporal behavior of data.
This is done by passing three corresponding frames, which are aligned with each other using advection A : R N x ⇥ N y ⇥ N z ⇥ 3 ⇥ R M x ⇥ M y ⇥ M z ⇥ 3 !
R N x ⇥ N y ⇥
N z ⇥ 3 .
Apart from this, the temporal discriminator closely follows the structure of the spatial discriminator.
Both discriminators (Figure 1b) use a typical funnel structure, where the dimension is increasingly reduced using strided convolutional layers, with a last fully connected layer computing the classiﬁcation result.
We likewise use leaky ReLU activations, with a sigmoid function for the last layer.
The classiﬁcation of the discriminators is included in the loss formulation of the generator:
[Equation]
which gives the ﬁnal loss function:
[Equation]
where ↵ and  indicate the weighting of the individual loss terms.
An additional beneﬁt of the adversarial loss is that it allows for learning from unpaired data.
A common problem for up-sampling methods is the generation of paired ground truth data for training.
Due to different numerical approximations, and hence potentially differing physical behavior, the easiest solution is to simulate at high resolution, and down-sample the data.
While at training time the down-sampled data is used, at test time, the model needs to be applied to data from a lowresolution simulation instead.
This typically leads to large distribution shifts, and correspondingly impaired inference quality.
Therefore, we take an unpaired training approach into account that decouples the low and high resolution data.
The feedback from the discriminators is still based on the ground-truth data, which makes the output conditionally dependent on the input, but also approximates the behavior of the reference data.
However, there is no direct supervision in the generator anymore: the output is no longer compared with a matching ground-truth in the loss, but [Figure] only related to the input.
This is done by down-sampling the output and comparing it with the input:
[Equation]
where p : R N x ⇥ N y [Equation] N z ⇥
[Equation]
R M x ⇥ M y ⇥ M z ⇥ 3 is a down-sampling function based on average pooling.
This effectively removes the need for paired low- and high-resolution samples at training time, and fully relies on the discriminator to match both distributions.
To indicate the focus on surface structures, we refer to the ﬁnal version of our generative network as surfGAN .
For a more detailed description of the training and the network architecture we refer to the appendix A.1 and A.2.
2. F REQUENCY E VALUATION
Given the formulation so far, an inherent difﬁculty that remains is a robust and reliable evaluation of the generated outputs.
In a GAN setting, the discriminator determines the content, but it is typically not possible to evaluate whether it has correctly learned the shape of the output distribution.
While obvious cases such as a mode collapse towards a constant signal are easy to detect, it remains challenging to reliably detect shifts in the data distributions, especially so for GANs (Arjovsky et al., 2017; Jolicoeur-Martineau, 2018).
In our setting, the outputs should on the one hand represent a high-resolution version of the input, i.e., the down-scaled output should correspond to the input.
This can be measured with a simple MSE.
In addition, we also expect the generation of details that match the ground-truth as closely as possible.
MSE is no longer usable for this, as even if the right details are generated, a slight translation would already lead to substantial errors.
In addition, smallscale features have little effect on the MSE, despite being crucial for a realistic simulation result.
Finally, the temporal behaviour should also correspond to that of the ground-truth.
Considering the problem in frequency space, the sought after detail consists of high-frequency features that cannot be represented by the low-resolution simulation.
In addition, we have to distinguish between spatial and temporal frequencies.
The frequency-based view has the advantage that it yields a simple but powerful performance evaluation that complements the discriminator of the GAN training.
For the frequency evaluation we consider the SDF of our implicit grid G t 2 R N x ⇥ N y ⇥ N z for t 2 [0 , T ) , which can be equated with the frequency behavior of the surface.
In order to retain spatial properties in addition to the local frequency behaviour of the surface, we divide our image into blocks with size b 3 whose Fourier coefﬁcients g k x ,k y , k z we evaluate, where k ⇤ 2 [0 , b ) .
The method is similar to the one used for JPEG compressions.
Therefore, we divide the domain into N 0 x ⇥ N 0 y ⇥ N 0 z regions for the spatial frequency evaluation, where N
⇤
=
N ⇤ /b .
Using a three-dimensional fast Fourier transformation (FFT), we then transform the extracted blocks into the Fourier components:
[Equation]
where !
=
e  i 2 ⇡ b .
An important point here is that we take the absolute value from the coefﬁcients in order to eliminate the phase, and focus only on the amplitude of the frequency component.
This makes the problem more robust for translated small-scale structures.
Equation 6 yields b ⇥ b ⇥ b a m p lit ud e frequency[Hz] a m p lit ud e [Figure] [Figure] [Figure]
Fourier components per box.
The individual components can now be grouped with the same components from other blocks so that we get one N x /b ⇥ N y /b ⇥ N z /b grid per Fourier component (Figure 2), with each grid corresponding to a certain frequency.
These grids can be further processed, e.g., via computational kernels, and in addition can be inspected by humans for veriﬁcation.
For the temporal frequencies we consider the changes per pixel over time via FFT.
For longer sequences, a block-wise evaluation would be conceivable for time as well.
The separation of spatial and temporal frequencies gives us the possibility to compensate for differences by adjusting the weighting.
Based on this setup we can now evaluate results.
On the one hand, we can directly compare the frequency range of generated data and identify missing details for single samples or mini-batches.
E.g., this is amenable to loss formulations for learning objectives.
On the other hand we can also create a histogram for the frequencies of the whole solution space and thus compare the frequency distribution of the generated and ground-truth data.
This is especially helpful for GANs and works also for unsupervised setups.
R ESULTS
For evaluation we consider two different data sets: a synthetic 2D case and a 3D particle-based simulation.
The 2D data is fully controllable such that the frequency spectrum of the surface can be evaluated reliably.
The basis is a wavy surface, based on a sine wave with varying frequency (Figure 3).
This for a wide range of analysis to isolate problems in the generative process and illustrate the aspects of the proposed method.
We then evaluate the established methodology in 3D for a more complex scenario with data generated from a highly viscous SPH simulation (Weiler et al., 2018).
For more information on both data sets, refer to the appendix A.3 and A.4.
3. F REQUENCY E VALUATION
We ﬁrst consider the controlled 2D data.
Training is performed with the full data set ofsamples with 30 frames, and we evaluate the resulting models on 10 simulations that are not part of the training set with 30 frames each.
Examples of the block-wise analysis from our frequency a m p lit ud e a m p lit ud e frequency[Hz] ( [Figure] frequency[Hz] ( [Figure]
) Frequency loss.
frequency[Hz]
[Figure]
) surfGAN.
frequency[Hz] (d) surfGAN unpair.
MSE
[Table]
evaluation (Figure 2) are shown in Figure 4.
From top to bottom, it compares ground-truth, a network with MSE loss, and the surfGAN result.
While the MSE-based variant is not able to reconstruct ﬁne details, the surfGAN can recover these, as highlighted by the frequency evaluation.
The highfrequency blocks clearly illustrate the differences into terms of reconstruction accuracy.
In Tableand Figure 5 we also compare the mean error values of the different blocks quantitatively and via histograms.
The mean errors likewise illustrate that the MSE version is not able to reconstruct the high frequencies of the target function.
The spectrum of the generated data hardly shows any deﬂection in the high frequency range (Figure 5(a)).
We additionally extended this fully supervised MSE loss with a term that takes into account differences in the frequency spectrum via an L2 norm between the Fourier components of the data.
Figure 5(b) shows that this improves the situation, but does not sufﬁce to reconstruct the full high frequency range.
This is caused by the inherent averaging of the MSE, which is still suboptimal in Fourier space.
In contrast, our surfGAN setup (Figure 5(c)) achieves the best results.
The spectrum of the prediction very closely matches that of the ground truth.
The GAN based method seems to be able to implicitly learn the underlying frequency distribution.
Finally, we repeat the evaluation with an unpaired surfGAN setup.
Despite the fundamentally more challenging learning setup, the network manages to recover the missing frequencies, as shown in Figure 5(d).
Interestingly, the discriminator feedback causes the generator to slightly overshoot in terms of high frequency content.
[Table]
For our 3D results we have collected the values in Table 2.
We compare our method with tempoGAN, whose performance suffers mainly from the missing recursion loop.
It is clearly visible that surfGAN has the smallest error in the spatial frequencies, which correlates to the spatial detail level.
The temporal frequency, on the other hand, is almost equal to the value of the NN with the
MSE loss.
This is probably due to the fact that the temporal behavior is relatively smooth.
[Figure]
Q UALITATIVE
[Figure]
As a qualitative evaluation we executed our network with test data and visualize the results.
Figureshows visual examples of the 2D frequency evaluation.
While a version based on MSE generates very smooth images in the random data set, the surfGAN versions are able to reconstruct the jagged edges.
In direct comparison with the ground-truth data there are differences, but this is due to the randomness of the data.
Therefore, the exact solution cannot be reconstructed, but the surfGAN is still able to generate a very plausible solution.
[Figure]
For the 3D case we focus on the paired surfGAN setup.
Figure 7(a) shows that our prediction is able to reconstruct most of the details, even if they are not present in the input.
We have deliberately chosen frames after a long run-time (190-220 time-steps) to show that details can persist and that they are the result of complex, physically-based behavior over multiple frames.
Again, we compared our method with a simpler MSE-based generator and tempoGAN.
With MSE, the results become comparatively smooth, as can be seen in Figure 7(b).
In Figure 7(c) however, the results of tempoGAN are shown, where details are generated quite randomly and display a rather chaotic behavior over time.
Due to the lack of a recurrent processing, the network cannot build an accurate internal state, which has the effect that the details are generated unnaturally.
C ONCLUSION
We have presented a learning-based method to infer spatio-temporal detail to complex physical simulations.
Our method puts special emphasis on high frequency content, and we present an approach for assessing GAN-based outputs in terms of the generated spatial and temporal frequencies.
Interestingly, our proposed surfGAN performs better than a direct supervision in terms in frequency space.
Our method provides a ﬁrst step towards evaluation and synthesis of physical space-time processes, and could be employed for other phenomena such as turbulence Ling et al.
(2016) or weather Zaytar & El Amrani (2016).
Furthermore, it will be interesting to employ it in conjunction with other frequency-based representations Sitzmann et al.