L AGRANGIAN F LUID S IMULATION WITH C ONTINUOUS C ONVOLUTIONS
A BSTRACT
We present an approach to Lagrangian ﬂuid simulation with a new type of convolutional network.
Our networks process sets of moving particles, which describe ﬂuids in space and time.
Unlike previous approaches, we do not build an explicit graph structure to connect the particles but use spatial convolutions as the main differentiable operation that relates particles to their neighbors.
To this end we present a simple, novel, and effective extension of N-D convolutions to the continuous domain.
We show that our network architecture can simulate different materials, generalizes to arbitrary collision geometries, and can be used for inverse problems.
In addition, we demonstrate that our continuous convolutions outperform prior formulations in terms of accuracy and speed.
I NTRODUCTION
Understanding physics can help reasoning about our environment and interacting with it.
Neural networks have emerged as a particularly promising approach to capture the complexity of natural phenomena from data (Ling et al., 2016;
Tompson et al., 2017; Morton et al., 2018).
An important aspect of learning physics with neural networks is the choice of representation.
Lagrangian representations based on particles are particularly popular and have supported recent results with rigid bodies, deformable solids and ﬂuids (Battaglia et al., 2016; Mrowca et al., 2018; Li et al., 2019).
Many of these approaches use graph structures to deﬁne interactions; the existence of an edge determines in a binary fashion whether two particles interact.
However a wide range of physical processes such as ﬂuid mechanics are described by continuous partial differential equations rather than discrete graph structures.
The continuous, volumetric, and tightly coupled nature of these processes causes inherent difﬁculties for graph-based approaches, such as a large number of edge connections that must be established, tracked and disengaged as the particles move.
In this work, instead of using graphs as the underlying representation, we adopt a continuous viewpoint.
We propose to use convolutional networks (ConvNets) with continuous convolutions on particles for learning ﬂuid mechanics.
We treat ﬂuids as spatially continuous functions sampled at a ﬁnite set of (continuously evolving) positions and process them with a novel continuous convolution layer.
This matches the continuous nature of the problem more closely and simpliﬁes the deﬁnition of neural networks by abstracting the underlying particle representation.
We extend the grid-based ﬁlter representation commonly used for discrete convolutions to the continuous domain by simple linear interpolation.
Linear interpolation of the ﬁlters allows efﬁcient lookup of spatially varying ﬁlter values at arbitrary positions while retaining the compactness and efﬁciency of the grid representation.
In addition, we use a window function to deﬁne the support of the ﬁlters and a ball-to-cube mapping to support spherical receptive ﬁelds.
We show that our convolutions, despite their simplicity, perform better than more sophisticated representations (Wang et al., 2018;
Schenck & Fox, 2018).
With the presented continuous convolution layer, we develop an efﬁcient ConvNet architecture for learning ﬂuid mechanics.
The network processes sets of particles.
We use dynamic particles to represent the ﬂuid and static particles to describe the boundary of the scene.
Modeling the scene boundary with particles makes it easy to apply our network to new scenes and allows the network to learn collision handling in a uniﬁed framework.
Our network generalizes to arbitrary obstacle conﬁgurations and can simulate a range of material behavior.
To demonstrate the usefulness of a learned – and hence differentiable – ﬂuid simulator, we show that material properties can be estimated from observed simulation data.
Experimental results indicate that the presented approach outperforms a state-of-the-art graph-based framework (Li et al., 2019).
R ELATED W ORK
Fluids encompass a range of materials that are important in everyday life and throughout science and engineering (Wilcox, 2006).
A prominent set of methods, known as Smoothed Particle Hydrodynamics (SPH), employs a Lagrangian viewpoint to simulate these phenomena.
SPH originated in particle-based models for astrophysics (Gingold & Monaghan, 1977) and has become extremely popular for simulating complex ﬂows in many ﬁelds of science (Monaghan, 1988).
Among others, it has been highly successful for modeling interface ﬂows (Colagrossi & Landrini, 2003), complex multi-phase phenomena (Hu & Adams, 2006), and even magnetohydrodynamics (Price, 2012).
SPH and its variants have been widely used to model complex real-world phenomena for visual effects.
Following the earlier development of physics-based ﬂuid simulation for special effects by
Foster & Metaxas (1996), the introduction of SPH (M¨uller et al., 2003) has led to a large class of powerful algorithms (Solenthaler & Pajarola, 2009; Bender & Koschier, 2015).
These applications often involve complex geometries at large scales, and extensions such as FleX (Macklin et al., 2014) and pressure-aware rigid-body coupling (Gissler et al., 2018) broaden the framework to encompass many physical phenomena.
Lagrangian ﬂows have also been considered in machine learning.
The pioneering work of Ladick ´ y et al.
(2015) demonstrated that ﬂow representations can be learned with regression forests.
CNNs were used by Tompson et al.
(2017) to accelerate the expensive pressure correction step of gridbased solvers, while other works have focused on super-resolution (Xie et al., 2018), learning time evolution via the Koopman operator (Morton et al., 2018), and learning reduced representations (Wiewel et al., 2019; Kim et al., 2019).
Differentiable SPH solvers were proposed to solve control tasks for robotic applications (Schenck & Fox, 2018).
Generic physics simulations for Lagrangian rigid and deformable bodies were considered in a series of works that developed graph-based representations (Battaglia et al., 2016; Sanchez-Gonzalez et al., 2018).
Such graph-based representations were recently applied directly to ﬂuid simulation (Mrowca et al., 2018; Li et al., 2019).
We share with these recent works the goal of modeling Lagrangian ﬂuids with differentiable neural networks, but take a different tack: rather than using graph-based representations, we work with point clouds and continuous convolutions over the spatial domain.
From a technical perspective, our approach is also related to existing works that apply convolutions on point clouds.
A number of methods transferred the convolution concept to point clouds in the context of semantic classiﬁcation and segmentation of 3D objects (Hua et al., 2018; Atzmon et al., 2018; Hermosilla et al., 2018; Li et al., 2018;
Su et al., 2018; Wu et al., 2019; Xu et al., 2018; Lei et al., 2019).
Particularly notable in our context is the work of Wang et al.
(2018), who used continuous convolutions to compute the scene ﬂow between two point clouds, and the aforementioned work of Schenck & Fox (2018), who used convolutions to implement a differentiable version of position-based ﬂuids (Macklin & M¨uller, 2013).
Both works deﬁne continuous convolution operators that can support regression tasks and we compare to them directly in Section 6.
Most closely related to our ﬁlter representation is the work of Fey et al.
They use B-splines to deﬁne continuous ﬁlters and propose to use spherical coordinates to implement spherical receptive ﬁelds.
Spherical coordinates are problematic due to singularities and require special treatment, which we avoid with a ball-to-cube mapping.
Furthermore, the output of the operator of Fey et al. (2018) can be discontinuous.
We show in our ablation study in Section 6 that applying a window function to guarantee a continuous output is advantageous for our task.
B ASICS
Fluids have been studied for centuries, and the Navier-Stokes equations for incompressible ﬂuids are well established (Batchelor, 1967):
[Equation]
A common approach to solve these partial differential equations is to approximate the ﬂuid with a set of smooth particles (Monaghan, 1988).
Each particle corresponds to a continuous blob of matter and carries the local properties of the ﬂuid, such as velocity and density, which move with the ﬂow.
This is motivated by the fact that a function A ( x ) can be represented by an integral interpolation
[Equation]
where δ ( x ) denotes the Dirac delta function.
This equation can be discretized as
[Equation]
where V i is the volume at the given point in space and
[Equation]
Here W ( x, h ) is a smooth kernel or convolution with radius h , usually in the form of a Gaussian distribution, but more complex functions can also be used.
In practice, the kernel is ﬁnite and yields localized neighborhoods of particles that interact via interactions weighted by the kernel of its derivatives.
In this way, the continuous description for ﬂuids from Equation 1 can be discretized in a Lagrangian fashion and solved numerically.
Typically, internal forces are calculated based on the local pressure, viscosity, and surface tension, which give an update for the position of each particle.
Below, we adopt the position-based ﬂuids (PBF) method (Macklin & M ¨ uller, 2013; Macklin et al., 2014), which likewise is based on SPH, but reformulates the updates as constraints on the positions.
C ONTINUOUS C ONVOLUTIONS
The discrete convolution operator as commonly used in ConvNets is deﬁned as
[Equation]
where f and g are the input and the ﬁlter function, x is the position, τ is the shift vector, and Ω is the set of shift vectors that deﬁnes the support of the ﬁlter function.
On regular data such as images, the positions x range over a regular grid and the shift vectors τ are integer-valued, i.e. x , τ ∈ Z d for some dimensionality d .
Analogously in the continuous domain, this convolution is deﬁned as
[Equation]
where x and τ are real-valued vectors, i.e. x , τ ∈ R d .
We now adapt this deﬁnition to unstructured point clouds.
In this setting we have a ﬁnite number of points that sample the function f
but do not lie on a grid.
For a point cloud with i = 1 , .., N points with values f i at positions x
i , we deﬁne the convolution at position x as
[Equation]
N ( x , R ) is the set of points within a radius R around x .
a is a scalar function that can be used for density normalization speciﬁc to the points x i and x as in Hermosilla et al.
In the simplest case, a can be constant: a = 1 .
In our case we want to ensure a smooth response of our convolution under varying particle neighborhoods, therefore we deﬁne a as a window function:
[Equation]
A similar function has been used by M¨uller et al.
(2003) in the SPH framework.
ψ is another scalar function for normalization, which can be set in our implementation as either
[Equation]
We use ψ ( x ) = 1 , since changes in the density of particles are an important feature for simulating
For the ﬁlter function g we simply use a regular grid to store the ﬁlter values but use linear interpolation to make g a continuous function.
In addition, we use a mapping Λ( r ) of a unit ball to a unit cube to implement spherical ﬁlters as shown in Figure 1.
We use the mapping described by Griepentrog et al.
and give the detailed function in the appendix.
The intermediate coordinate mapping Λ provides the ﬂexibility to implement different spatial shapes while keeping the advantages of a regular grid for the storage and lookup of ﬁlter values.
Note that Equation 7 uses a similar approximation as in the SPH framework (Equation 3).
Assuming that each point represents the same volume, V i is a constant factor in Equation 3, which we drop in our deﬁnition.
L EARNING F LUID M ECHANICS WITH C ONVOLUTIONAL N ETWORKS
Our goal is to learn ﬂuid mechanics from observing the motion of particles. The input to our
ConvNet is a set of particles with corresponding features.
Since position itself is not a feature but simply deﬁnes the particle’s position in space, we must assign a feature vector to each particle.
The feature vector we use is a constant scalar 1 accompanied by the velocity v and the viscosity ν .
A particle p ni at timestep n with its position and input feature vector is thus a tuple ( x ni , [1 , v ni , ν i ]) .
Deﬁning the velocity explicitly as an input feature allows us to compute intermediate velocities and positions as in Ladick
´ y et al.
and to apply external forces and pass this information to the network.
We compute the intermediate positions x n ∗ i and velocities v n ∗ i beginning with timestep n with Heun’s method as
[Equation]
The vector a ext is an acceleration through which we can apply external forces to control the ﬂuid or to simply apply gravity.
The intermediate positions and velocities lack any interactions between particles or the scene, which we are going to implement with a ConvNet.
To enable the network to handle collisions with the environment we deﬁne a second set of static particles s j .
We sample particles on the boundaries of the scene with normals n j as the feature vectors, i.e. s j = ( x j , [ n j ]) .
Our network implements the function
[Equation]
which uses convolutions to combine features from both particle sets
∆ x is a correction of the position which accounts for all particle interactions including the collision handling with the scene.
Finally, we apply the correction to update positions and velocities for n + 1 as
[Equation]
Note that the updated position x n +1 i depends on the output vector ∆ x i and allows us to directly deﬁne our learning objective on the particle positions.
5. N ETWORK A RCHITECTURE
We use a simple convolutional architecture with an effective depth of four.
An overview of the network is shown in Figure 2.
Since we want to compute the correction for all dynamic particles in our scene, we compute convolutions for the intermediate positions deﬁned in Equation 11.
Our network is a sequence of continuous convolutions (CConv), which are deﬁned by an input particle set, the positions at which we want to evaluate the convolution, its ﬁlters G , and the radius R .
For instance, to describe a convolution on the static particles s i at the intermediate positions x n ∗ i as used in the beginning of our network we can write
[Equation]
where f i are the computed output features for each position x n ∗ i .
G is a 5D array storing all ﬁlters in the layout [ width , height , depth , ch in , ch out ] .
In contrast to discrete convolutions on a regular grid, the spatial ﬁlter dimensions here do not deﬁne the receptive ﬁeld but the resolution of the ﬁlters.
The receptive ﬁeld depends only on the radius R , which speciﬁes the spatial extent.
Throughout our network we use ﬁlters with a spatial resolution of [4 , 4 , 4] and a radius of 4 .
5 times the particle radius.
For convolutions within the dynamic particles we exclude the particle at which we evaluate the convolution and instead process the particle’s own features in a stream of fully-connected layers.
After each depth level we then combine the result from the convolutions and the fully-connected layers by addition.
This can be interpreted as a convolution with a spatial resolution of 4 × 4 × 4 + 1 .
We found that this design improves accuracy and allows us to use smaller ﬁlters with even sizes (see
Table 2).
5. T RAINING P ROCEDURE
We train our ﬂuid simulation network in supervised fashion based on particle trajectories produced by classic (“ground-truth”) physics simulation.
Our loss is deﬁned as follows:
[Equation]
The ground-truth position at timestep n + 1 is denoted by ˆ x n
+1 i and the predicted position from the network is denoted by x n +1 i =
x n ∗ i + ∆ x i , where ∆ x i is provided by the network.
φ i is an individual weight for each point. We use φ i = exp( − 1 c |N ( x n ∗ i ) | ) , which emphasizes the loss for
[Figure]
particles with fewer neighbors.
We choose c = 40 , which corresponds to the average number of neighbors across our experiments.
Particles with few neighbors are close to the surface or interact with the scene boundary.
Both cases are important for ﬂuid simulation because particles near the surface deﬁne the liquid-air interface, which is particularly salient, and particles near the scene boundary require collision handling.
The parameter γ = 0 .
5 makes our loss function more sensitive to small particle motions, which is important for increasing the accuracy and visual ﬁdelity for small ﬂuid ﬂows.
During training we predict particle positions for two future timesteps, namely n + 1 and n + 2 .
The combined loss L is
[Equation]
[Equation]
We found that optimizing a loss deﬁned over two frames improves the overall quality of the simulation.
(Optimization for three frames did not result in further improvements.)
We optimize L over 50,000 iterations with Adam (Kingma & Ba, 2015) and a learning rate decay with multiple steps, starting with a learning rate of 0 .
001 and stopping with 1 .
· 10 − 5 .
D ATASETS
We have trained our network on multiple datasets.
For quantitative comparisons with prior work we trained our network on the dam break data from Li et al.
The scene simulates the behavior of a randomly placed ﬂuid block in a static box.
We generate 2000 scenes for training and 300 for testing.
The data was generated with FleX, which is a position-based simulator that targets real-time applications (Macklin et al., 2014).
We also trained our network on more challenging data generated with DFSPH (Bender & Koschier, 2015), which prioritizes simulation ﬁdelity over runtime.
DFSPH can generate accurate ﬂuid ﬂows with very low volume compression: a desired property.
We generate ground-truth data by randomly placing multiple bodies of ﬂuid in 10 different box-like scenes and simulating them for 16 seconds each with an adaptive timestep of up to 1 kHz .
The time resolution of the generated data is 50 Hz .
We show a qualitative comparison of our method to the ground truth in Figure 3.
We generate 200 scenes for training and 20 scenes for the test set.
To train networks that can deal with multiple materials, we additionally generate 200 scenes with ﬂuids of varying viscosity.
For estimating material properties, we generate 7 test scenes that only differ in the viscosity parameter.
E VALUATION
Baselines.
We compare our method to DPI-Nets (Li et al., 2019), which were previously shown to signiﬁcantly outperform prior formulations such as hierarchical relation networks (Mrowca et al., 2018).
Figure 4 provides a qualitative comparison.
Quantitative results are reported in Table 1.
To analyze the accuracy of the forward step for each method, we compute the average error of the [Figure] particle positions with respect to the ground truth.
We use every 5 th frame for initialization and compute the deviation from the ground truth for two subsequent frames, denoted by n +1 and n +2 .
In addition, we report the average distance from the ground-truth particles to the closest particle in the prediction for the whole sequence, to measure long-term similarity.
We compute the distance for frame n as
[Equation]
[Equation] where X n is the set of predicted particle positions for frame n , ˆ x ni is the ground-truth position for particle i , and N is the number of particles.
We also compare our continuous convolution formulation to continuous convolution representations used in SPNets (Schenck & Fox, 2018), PCNN (Wang et al., 2018), KPConv (Thomas et al., 2019), and SplineCNN (Fey et al., 2018).
To this end, we plug the respective convolution operators into our network architecture as shown in Figure 2.
This facilitates controlled comparisons in which the overall architecture and simulation setup are ﬁxed and only the convolution operators are varied.
As shown in Table 1, our method outperforms all baselines with respect to both accuracy and inference time.
We have trained and tested all methods on the dam break dataset from Li et al.
(2019) as well as our data (generated with a high-ﬁdelity simulator (Bender & Koschier, 2015)).
To facilitate the training on our data for all methods, we generated a simpliﬁed version with a constant number of particles (6,000) and a single box environment (as shown in Figure 4).
We train DPI-Nets for 5 epochs and all other networks for 50,000 iterations, which corresponds to 2.7 epochs for the dam break data andepochs for our data.
Training takes about a day for our method with our convolutions on an NVIDIA RTX 2080Ti.
Training with PCNN convolutions (Wang et al., 2018) takes about 2 days on 4 GPUs.
Note that our reimplementation of PCNN convolutions uses Tensorﬂow’s built-in functions, which consume a lot of memory and necessitate multi-GPU training.
The DPI-Nets model trains in about one day.
Training with SplineCNN Convs takes 3 to 4 days on a single GPU.
We got the best results for this method with spherical kernel coordinates and closed splines.
For KPConvs we used a Quadro RTX 6000 with 24 GB of RAM due to the higher memory requirements.
Training took about 1 day with 15 kernel points.
For SPNets convolutions, we estimated a training time of more than 29 days with 3 × 3 × 3 ﬁlters by extrapolating from timing of a smaller number of iterations.
We thus only report inference time for this method.
Note that the convolutions of SPNets were designed to implement the position-based ﬂuids algorithm, while we use them here in a more general network architecture with a much larger number of channels, which explains the very long runtime.
[Table]
Ablation study.
We perform an ablation study to evaluate our decisions in the design of the continuous convolution operator and the network.
We study the importance of the interpolation, the window function, the loss design [Table]
and the architecture choice. The results are reported in Table 2. This table reports error measures (averaged over the test sequences) that were used in Table
[Table]
and also reports the errors for two predicted frames initialized with the frames at the end of each sequence to measure the errors for small ﬂow velocities.
Large errors for small velocities can yield perceptually salient artifacts: rather than being still, ﬂuid particles jitter or churn.
[Table]
Generalization.
In Figure 5 we show that our network generalizes well to scenes with drastically different geometry than seen during training.
(The training set uses only box-like containers.
See the appendix for visualization.)
These scenarios demonstrate that we can emit particles during simulation, which can be costly for methods that build and maintain explicit graph structures.
We compare generalization performance quantitatively to DFPSH on a complex scene in Figure 6.
Figuredemonstrates generalization along a different dimension.
Here we show that we can set the viscosity of the ﬂuid at test time to a value not seen during training.
The ﬂuid shape used in this example was also not seen during training. [Figure] [Figure]
Material estimation.
In this experiment we apply our network to an inverse problem: material estimation from observation.
We use our network to estimate the viscosity parameter of a ﬂuid from the particle movement.
The training data for this experiment contains 200 sequences with random viscosity parameters between 0 . 01 and 0 . 3 .
For testing we generate 5 new sequences with 100 frames and random viscosity within the same range as used during training and use an initial ﬂuid shape not present in the training data.
To test generalization we generate two sequences with viscosity values outside the training range, namely 0 .
35 and 0 . 4 .
To estimate the viscosity we backpropagate through the trained network and optimize ν with gradient descent.
Table 3 reports the results, which indicate that our network can be used to estimate material properties from observed data. [Table]
C ONCLUSION
We have developed continuous convolutional networks for Lagrangian ﬂuid simulation.
We have introduced a simple formulation for continuous convolutions and demonstrated its accuracy and speed.
Our model captures a wide range of complex material behavior, offers long-term stability, and generalizes to new situations such as varying particle counts, domain geometries, and material properties.
There are numerous directions for future work, such as extending the framework to incorporate rigid and deformable solids.
We will release the code to facilitate such development.
Our continuous convolution implementation will be made available as part of Open3D (Zhou et al., 2018).
Acknowledgements.
We thank Jan Bender for his support with the SPlisHSPlasH framework
A A PPENDIX A. I MPLEMENTATION D ETAILS
To accelerate the computation of Equation 7 we use existing general matrix multiplication primitives.
Similar to standard convolutions in deep learning frameworks we build a matrix with patches that we then multiply with the ﬁlter matrix.
We can account for the linear interpolation by applying the interpolation to the patch matrix instead of the ﬁlters.
For 3D point clouds a single point contributes to up to 8 voxels in a patch.
Another crucial part of our implementation is the nearest neighbor search.
Since the positions of the ﬂuid particles update with each timestep, we have to rebuild the neighborhood information for every frame.
We implement the neighborhood search with spatial hashing (we use the hash function proposed in Teschner et al. (2003)).
We explicitly store all particle neighbors in a compact list, which allows us to reuse the information for multiple convolutions operating on the same point sets.
Table 4 compares the average frame runtimes of our method with the baselines.
A. D ATASET G ENERATION
To enable generalization to new environments we use 10 different containers (see Figure 8) in our data generation process.
During scene generation we randomly sample an environment and place up to 3 ﬂuid bodies with different shapes and sizes (see Figure 9) and random initial velocities in [Table]
compared to the average 40 neighbors on our datasets. All runtimes were measured on a system with an Intel Core i9-7960 and an NVIDIA RTX 2080Ti. the scene. We simulate each generated scene with DFSPH using the SPlisHSPlasH framework 1 for 16 seconds to ensure that each scene contains frames with small particle velocities.
To create the particle representation of the box-like containers, we do Poisson-Disc sampling on the mesh surface with the tools provided by DFPSH and add surface normals to each particle. [Table]
A. T RAINING D ETAILS
We use the Tensorﬂow framework for implementing the training procedure.
We use Adam as optimizer and train with a batch size of 16 and an initial learning rate of 0.001.
We half the learning rate at steps 20000, 25000, ..., 45000.
For the convolutions we use the random uniform initializer with range [-0.05, 0.05].
All other weights are initialized with the respective default initializers of
Tensorﬂow version 1.12.
The output of the network is scaled with 1128 to roughly adjust the output range to the ground truth position correction of the training data.
The unit of length of the training data is meter.
The particle radius used in DFSPH is h = 0 .
025 m .
For our convolutions, we use spherical ﬁlters with an empirically determined radius of R = 4 .
5 h .
A. W INDOW F UNCTION
We compare 3 different choices for the window function a from Equation 8 in Table 5.
The results show that enforcing a continuous output with a window function yields better results.
Further, using a window function similar to kernels used in SPH codes gives better results than a simple triangular window.
Learning the window function is therefore a possible direction to extend our framework to further improve the accuracy.
A. C OORDINATE M APPING F UNCTION
We use the ball to cube mapping described in (Griepentrog et al., 2008) to map a position within a spherical region to the ﬁlter values stored in a regular grid.
We give here the functions used for the 3-D case as used in our implementation.
For more details see (Griepentrog et al., 2008).
The function Λ is a composition of the functions Λ ball → cyl and Λ cyl → cube , which map a sphere to a cylinder and a cylinder to a cube respectively.
We deﬁne Λ ball → cyl for vectors r = ( x, y, z ) as
[Equation]
The cylinder to cube mapping is deﬁned as
[Equation]
We assume that vectors r are normalized with the search radius R such that (cid:107) r (cid:107) 2 ≤ 1 .
This yields the following Λ , which maps from a unit ball to the normalized coordinates of a cube:
[Equation]
[Equation]