I NFERRING F LUID D YNAMICS VIA INVERSE RENDERING
Anonymous authors
Paper under double-blind review
A BSTRACT
Humans have a strong intuitive understanding of physical processes such as ﬂuid falling by just a glimpse of such a scene picture, i.e. , quickly derived from our immersive visual experiences in memory.
This work achieves such a photo-to-ﬂuiddynamics reconstruction functionality learned from unannotated videos, without any supervision of ground-truth ﬂuid dynamics.
In a nutshell, a differentiable
Euler simulator modeled with a ConvNet-based pressure projection solver, is integrated with a volumetric renderer, supporting end-to-end/coherent differentiable dynamic simulation and rendering.
By endowing each sampled point with a ﬂuid volume value, we derive a NeRF-like differentiable renderer dedicated from ﬂuid data; and thanks to this volume-augmented representation, ﬂuid dynamics could be inversely inferred from the error signal between the rendered result and groundtruth video frame ( i.e. , inverse rendering).
Experiments on our generated Fluid
Fall datasets and DPI Dam Break dataset are conducted to demonstrate both effectiveness and generalization ability of our method.
I NTRODUCTION
Simulating and rendering complex physics ( e.g. , ﬂuid dynamics, cloth dynamics, hair dynamics) are major topics in computer graphics, with prevailing applications in games, movies and animation industry.
To pursue better adaptability, generalization ability and efﬁciency, many works (Wiewel et al., 2019;
Kim et al., 2019; Li et al., 2019; Ummenhofer et al., 2019; Sanchez-Gonzalez et al., 2020; Pfaff et al., 2021; Wandel et al., 2022; de Avila Belbute-Peres et al., 2018;
Hu et al., 2019;
Loper & Black, 2014; Liu et al., 2019; Mildenhall et al., 2020; Liu et al., 2020) pay attention to either learning-based (differentiable) simulation or rendering recently, especially to learnable simulation.
However, most of these works need large amounts of ground-truth data simulated using traditional physical engines to train a robust and general learning-based simulator.
Meanwhile, most existing works rely on the traditional pipeline; namely, it requires stand-alone modules including physical dynamics simulation, 3D modeling/meshization and rasterization/raytracing based rendering, thus the complication makes it tedious and difﬁcult for live streaming deployment.
Few works (Wu et al., 2017a;
Guan et al., 2022; Li et al., 2021b) attempt to associate simulation and rendering into an integrated module.
VDA (Wu et al., 2017a) proposes a combined framework, where a physics engine and a non-differentiable graphic rendering engine are involved to understand physical scenes without human annotation.
However, only rigid body dynamics are supported.
NeuroFluid (Guan et al., 2022) is a concurrent work that proposes a fully-differentiable two-stage network and grounds Lagrangian ﬂuid ﬂows using image supervision.
However, image sequences provide supervision that only reveals the changes in ﬂuid geometry and cannot provide particle correspondence between consecutive frames ( i.e. , temporal/motion ambiguities) to support
Lagrangian simulation.
Moreover, lots of spatial ambiguities between particle positions and spatial distribution of ﬂuid ( e.g. , different particles positions may result in the same ﬂuid distribution used for rendering) are introduced by the proposed Particle-Driven NeRF (Guan et al., 2022).
In reverse rendering, ambiguity is a terrible problem that makes the optimization process prohibitively ill-posed (Zhao et al., 2021).
Thereby, NeuroFluid can only overﬁt one sequence every time and cannot learn ﬂuid dynamics.
(Li et al., 2021b) develops a similar framework, while it learns dynamics implicitly on latent space and represents the whole 3D environment using only a single vector.
Such a rough and implicit representation limits its ability for generalization.
We make detailed discussions and comparisons with (Guan et al., 2022; Li et al., 2021b) in Section 4.4.
To explicitly address these fundamental limitations, in this work, we propose an end-to-end/coherent differentiable framework integrating simulation, reconstruction (3D modeling) and rendering, with image/video data as supervision, aiming at learning ﬂuid dynamic models from videos.
In a nutshell, a ﬂuid volume augmented representation is adopted to facilitate reconstruction-dynamics linkage between simulation and rendering.
More concretely, we design a neural simulator in the Eulerian view, where 3D girds are constructed to save the velocity and volume of ﬂuid.
In addition to advection and external forces application, we develop a ConvNet-based pressure projection solver for differentiable velocity ﬁeld updating.
Naturally, the ﬂuid volume ﬁeld is updated using advection based on the updated velocity ﬁeld.
Note that the advection is an efﬁcient and differentiable operator that involves back-tracing and interpolation.
In the meantime, as the Euler 3D grid saves the volume of ﬂuid, a NeRF-like neural renderer is thus proposed to capture the geometry/shape information of ﬂuid which retrieves supervision signals from images taken from multiple views.
Speciﬁcally, we assign a ﬂuid volume value to each point sampled from emitted rays by performing trilinear interpolation on the ﬂuid volume ﬁeld.
Then the sampled points equipped with ﬂuid volume properties are sent into the NeRF model to render an image.
The ﬂuid volume value of each point provides information (about how much ﬂuid material there is) for the renderer to capture the effects of ﬂuid on its density and radiance.
The simple and efﬁcient ﬂuid volume representation not only suits a neural Euler ﬂuid engine for ﬂuid motion ﬁeld estimation but also supports a differentiable NeRFlike renderer relating image with ﬂuid volume, achieving end-to-end error signal propagation to all rendering, ﬂuid volume reconstruction and simulation modules.
Note that the ﬂuid ﬁeld (Euler) representation naturally provides spatial distribution of ﬂuid and better correspondence between consecutive frames, which greatly reduces ambiguities for inverse optimization.
The whole forward simulation-modeling-rendering process and the inverse procedure are shown in Figure 1.
Unlike methods (Niemeyer et al., 2019;
Pumarola et al., 2021; Tretschk et al., 2021;
Ost et al., 2021; Li et al., 2021a; Guan et al., 2022) that typically ﬁt one sequence, our model can be trained on abundant sequences simultaneously and generalizes to unseen sequences with different initial conditions ( i.e. , initial shape, position, velocity, etc. ).
Besides, we model and simulate ﬂuid in an explicit and interpretable way, which is different from method (Li et al., 2021b) that learns 3D physical scenes in latent space.
As shown in Figure 10, such an explicit representation way endows our method with a strong ability to render images in extreme views that are far away from our training distribution and perform scene editing efﬁciently.
We conduct experiments on a part of DPI DamBreak dataset (Li et al., 2019) and two datasets that we generate using Mantaﬂow (Pfaff & Thuerey) and Blender.
Various experiments that involve baseline comparison, representation comparison, future prediction, novel view synthesis and scene editing are performed to prove both effectiveness and generalization ability of our method.
Detailed ablation studies are conducted to analyze important components and parameters.
Upon acceptance, all code and data will be publicly available.
We also discuss the limitations of our work in Appendix A.10.
R ELATED W ORK
Fluid Simulation .
Fluid simulation is a long-standing research area of great interest in science and engineering disciplines.
Various classical algorithms (Chorin, 1968; Stam, 1999; Macklin et al., 2014; Fedkiw et al., 2001; Monaghan, 1994; Solenthaler & Pajarola, 2009; Macklin & M¨uller, 2013;
Bender & Koschier, 2015; Bardenhagen et al., 2000; Zehnder et al., 2018; Ando et al., 2015; Zhang & Bridson, 2014; Brackbill et al., 1988; Jiang et al., 2015; Hu et al., 2018) are proposed to facilitate accurate and fast simulation.
To pursue better adaptability, generalization ability and efﬁciency, learning-based ﬂuid simulation (Li et al., 2019; Ummenhofer et al., 2019; Sanchez-Gonzalez et al., 2020; Pfaff et al., 2021; Tompson et al., 2017; Wiewel et al., 2019; Zhu et al., 2019; Kim et al., 2019; Thuerey et al., 2020; Wandel et al., 2020) has attracted increasing attention in recent years.
Most of these works usually bypass solving large-scale partial differential equations ( i.e. , PDE) via efﬁcient convolution operators.
Lagrangian ﬂows (Li et al., 2019; Ummenhofer et al., 2019;
Sanchez-Gonzalez et al., 2020) usually model the ﬂuid and rigid body as a set of particles with different material types.
Graph neural networks (Sanchez-Gonzalez et al., 2020; Pfaff et al., 2021) are also suitable to solve such problems.
Euler ﬂows (Tompson et al., 2017; Wiewel et al., 2019;
Kim et al., 2019) divide the space into regular grids and save physical quantities ( e.g. , density, mass, volume, velocity) of material in the divided grids.
(Tompson et al., 2017) accelerates Euler ﬂuid simulation using a convolution network to solve pressure projection.
(Wiewel et al., 2019) encodes the pressure ﬁeld into a latent space and designs a LSTM-based network to predict the latent code [Figure] E
[Figure]
n a l
F o
[Figure]
P
[Figure]
u
[Figure] e [Figure] ( C onv
[Figure]
Neural Euler Simulator
Volume-Augmented NeRF
[Figure]
for future time steps.
Ground-truth pressure ﬁelds are usually needed to train such a network.
We also propose a differentiable Euler simulator that solves pressure projection using a neural network in this work.
However, we associate our simulator with a differentiable renderer and drive the Euler simulator with 2D videos by solving an inverse rendering problem.
Differentiable Rendering .
Recently, many works (Kato et al., 2018;
Insafutdinov & Dosovitskiy, 2018; Loper & Black, 2014; Liu et al., 2019; Li et al., 2018) attempt to turn rendering into a differentiable process.
(Kato et al., 2018; Liu et al., 2019) design soft rasterizers to achieve differentiable render.
(Li et al., 2018) develops a differentiable monte carlo ray tracer.
Neural radiance ﬁeld ( i.e. ,
NeRF) (Mildenhall et al., 2020) is one of the most popular differentiable renderers, where a continuous radiance ﬁeld is saved in a MLP-based network.
To render one image, we just need to sample point on rays and query its volume density and color from the trained NeRF model.
Finally, a differentiable volumetric rendering is performed.
Novel views can be generated by emitting rays from new positions with new poses.
An important advantage of differentiable rendering is that it supports inverse rendering well, i.e. , inferring 3D geometry, lighting or material based on the rendered images.
In this work, we design a ﬂuid volume augmented NeRF model and take advantage of its ability of inverse rendering to learn ﬂuid dynamics and drive a generalizable simulator.
Inverse Graphics .
There are many works (Yuille & Kersten, 2006; Zhu & Mumford, 2007; Bai et al., 2012;
Kulkarni et al., 2015; Jimenez Rezende et al., 2016; Eslami et al., 2016; Wu et al., 2017b;
Yao et al., 2018; Azinovic et al., 2019; Munkberg et al., 2022) that focus on inverse graphics.
Recently, some researches (Gupta et al., 2010; Battaglia et al., 2013;
Jia et al., 2014;
Shao et al., 2014; Zheng et al., 2015; Lerer et al., 2016; Mottaghi et al., 2016; Fragkiadaki et al., 2016; Battaglia et al., 2016;
Agrawal et al., 2016; Pinto et al., 2016;
Finn et al., 2016; Ehrhardt et al., 2017; Chang et al., 2017; Zhu et al., 2018; Raissi et al., 2019;
Li et al., 2020; Chu et al., 2022) on this ﬁeld pay more attention to reasoning about physical properties of the scene from image/video.
In contrast to these methods that infer static properties, we infer a generalizable physical dynamic model via inverse rendering.
(Wu et al., 2017a;
Guan et al., 2022; Li et al., 2021b) are most relevant works to us.
VDA (Wu et al., 2017a) develops a paradigm for understanding physical scenes without human annotations.
A physical engine (differentiable or trained with REINFORCE (Williams, 1992)) for scene reasoning and a graphic engine for rendering are constructed to reconstruct the visual data.
However, only rigid body dynamics are considered.
NeuroFluid (Guan et al., 2022) develops a framework that supports grounding ﬂuid particles from images.
Lots of ambiguities are introduced by modeling the ﬂuid in Lagrangian view and make the optimization process very ill-posed.
Thus, their model only overﬁts one sequence every time and cannot learn ﬂuid dynamics.
Li et al.
(2021b) develops a similar framework that learns dynamic model on latent space.
They represent the dynamic 3D environment using only a single vector captured from images, which limits the model to generalize to cases never seen before.
In this work, we propose to reason about generalizable ﬂuid dynamics in an explicit and interpretable way with videos as supervision.
Strong generalization ability to unseen initial conditions are demonstrated.
M ETHODOLOGY
We develop an end-to-end differentiable framework integrating ﬂuid dynamics simulation, 3D modeling and video frame rendering, with image/video data as supervision.
To re-formulate conventionally independent simulation and rendering modules into an inter-related fully differentiable pipeline, a ﬂuid volume ﬁeld augmented representation is adopted, which not only provides information about the interface between air and ﬂuid for better free-surface simulation, but also supports differentiable
NeRF-like rendering based on its direct linkage with the density color in neural radiance.
Based on this construction, a neural ﬂuid simulator in Eulerian view together with a neural renderer are jointly established, which propagates the error signal between ground-truth video frame and the rendered image all the way from renderer to simulator, thus guiding the ﬂuid to move forward step-by-step.
Details about the proposed simulator, ﬂuid representation and renderer, as well as the overall learning framework are introduced as follows and the whole pipeline is shown in the Figure 1.
3. N EURAL E ULER F LUID S IMULATION
Traditional ﬂuid simulation often involves solving incompressible Navier-Stokes equations, one variant of which is formulated as:
[Equation]
where D u D t is the material derivative of velocity u , ρ is ﬂuid density, p is the pressure (a scale ﬁeld), ν is kinematic viscosity and g is the external force applied to ﬂuid such as gravity.
The material derivative of u represents the rate of change of velocity ( i.e. , the acceleration), which comes from three components: 1) the derivative of pressure p , 2) viscous force (is usually dropped except for highly viscous ﬂuid), 3) external force ( e.g. , gravity).
In the Eulerian view, the space (ﬂuid container) is divided into grid.
The physical quantities ( e.g. , volume V , velocity u ) of the ﬂuid are saved in the ﬁxed grid.
It is free for computing spatial derivatives (ﬁnite difference).
To solve such a complex dynamic equation, we often split it into three operators, i.e. , advection, applying external force and pressure projection (keep the velocity ﬁeld divergence-free).
The challenge lies in solving the pressure projection, which is a large-scale linear system.
In this work, we develop a neural pressure projection solver.
Speciﬁcally, we ﬁrst perform advection to the ﬂuid volume ﬁeld V t − 1 and velocity ﬁeld u t − 1 given u t − 1 at time step t − 1 .
The advection can be seen as an operator that moves ﬂuid with a constant velocity.
In the Lagrangian view, we achieve advection by just moving particles according to its velocity.
The physical quantities such as mass and velocity are bound to particles and moved with these particles.
However, the grids are ﬁxed in the Eulerian view.
To achieve advection and update the physical quantities (volume and velocity) of each ﬁxed grid, we take a commonly used semi-Lagrangian method to do advection, which involves two sub-steps ( i.e. , backtracing and interpolation).
Detailed explanation of advection module is shown in Appendix A.3. We also use Maccormack (MacCormack, 2003) method to perform advection for reducing numerical viscosity.
This process is denoted as:
[Equation]
where ∆ t is the time step for simulation.
Then we apply gravity to the ﬂuid, i.e. , accelerating the velocity ﬁeld using gravitational acceleration g :
[Equation]
Then the updated velocity ﬁeld u ∗∗ are sent into a ConvNet to regress the velocity difference introduced by the pressure that comes from the interaction between ﬂuid elements, ﬂuid and rigid bodies, ﬂuid and other medias, etc.
To better handle scenes of free surface ﬂuid simulation ( i.e. , the Euler space contains both ﬂuid and air), for each grid in the Eulerian view, we augment a volume value to indicate its material property, i.e. , a real value in the range of 0 to 1.0 with 0 representing it contains no ﬂuid and 1.
representing full of ﬂuid, and the whole space (container) could be regarded as a volume ﬁeld.
The advantage of using this augmented representation is obvious.
First, this volume ﬁeld illustrates the ﬂuid volume distribution in the space explicitly, hence it tells the neural rendering network where the interface between ﬂuid and air is to facilitate high ﬁdelity material interface rendering.
Second, in the
Euler framework, the information about the material interface is also helpful in solving the pressure projection step.
We, therefore, take the volume of the ﬂuid V t as the additional input of the ConvNet for pressure projection.
In addition, we assign each grid an input feature vector that represents the distance from it to the container boundary.
Given all these inputs, the ConvNet regresses the ∆ u induced by the pressure: F Θ : ( u ∗∗ , V t ) → (∆ u ) .
The updated velocity at next step is written as:
[Equation]
Moreover, our ﬂuid should be incompressible and the pressure projection has to keep the velocity divergence-free.
To this end, we apply regularization to the updated velocity ﬁeld u t , i.e. , to add a penalty for its divergence when training the pressure projection network.
This design helps the simulator to do better in alleviating volume loss.
The framework of our Euler ﬂuid simulation is presented in Figure 1.
The detailed structure of the ConvNet is presented in Appendix.
3. V OLUME -A UGMENTED R ADIANCE F IELD
Note that the ﬂuid volume representation also serves as a medium to bridge the simulator and renderer in a joint differentiable way.
Based on the volume ﬁeld, a ﬂuid volume augmented Neural
Radiance Field (NeRF) is developed in this work.
The original NeRF (Mildenhall et al., 2020) model works only on static scene.
Rays represented as r ( k ) =
o + k d are emitted from the cameras to the scene.
o and d denote the position of camera and view direction respectively.
3D points x i ( i = 1 , 2 , ..., N ) are sampled along each ray with near and far bounds.
These sampled points are combined with viewing direction d and sent into the NeRF model ( i.e. , MLP-based network) to regress their density σ and view-depended colors c .
The RGB values of each pixel C ( r ) are integrated by all sampled points along the corresponding ray using volumetric rendering algorithm.
This process is formulated as:
[Equation]
where T i denotes the accumulated transmittance along the ray from near bound to point x i and δ i represents the distance between adjacent samples.
However, the density and the color of each sample point change with the ﬂow of the ﬂuid in our dynamic scenes.
This change occurs as the spatial volume distribution of the ﬂuid changes temporally.
Hence we propose to bind value of ﬂuid volume for each input sample point for neural radiance rendering, and the volume value provides the rendering network additional information on how much ﬂuid material is contained here.
This information is highly correlated with density and radiance of sampled point.
Speciﬁcally, we assign the volume ﬁeld of ﬂuid V inferred by the neural
Euler simulator into the scene space and perform trilinear interpolation for each sampled point x i on rays to capture its corresponding ﬂuid volume value v i .
We combine this captured volume value with position of the sample point to learn its volume density and color for rendering image
ˆ I , which is written as F Ψ : ( x i , d , v i )
→ ( c i , σ i ) .
The Euler simulator rolls out the volume of ﬂuid at each time step and guides the NeRF to synthesize a sequence of images and ﬁnally renders a ﬂuid video.
3. R ADIANCE -D RIVEN F LUID S IMULATION
Most learning-based ﬂuid simulators are 3D data driven, where large-scale data is generated using traditional computational ﬂuid dynamic (CFD) tools such as SPH, FLIP, etc.
The simulated particle velocity/position or velocity/pressure ﬁeld are provided for model training.
Based on our ﬂuid volume representation, we naturally associate the differentiable simulator and renderer, further driving the ﬂuid simulation by scene radiance ﬁeld learned from images.
We have stated that our volumeaugmented radiance ﬁeld network is able to generate corresponding ﬂuid images based on the input ﬂuid volume, i.e. , forward computation.
When providing a sequence of ﬂuid ﬂow video frames, the volume-augmented NeRF model can enforce the input volume of ﬂuid to be close to real distribution for achieving better rendering results, i.e. , to minimize the appearance error between the rendered scene and the ground-truth video frame.
Almost no ambiguity is introduced by bridging the simulation and the render using this volume representation.
Hence, the error signal between rendered image and ground-truth image provides valid gradients for correcting both the renderer and simulator, through jointly training the neural Euler simulator and volume-augmented radiance ﬁeld (as the whole pipeline is fully-differentiable ).
Speciﬁcally, we advect the volume ﬁeld V t using the velocity ﬁeld u t inferred by the neural Euler simulator and get the updated volume ﬁeld
V t +1 = advect ( u t , ∆ t, u t ) .
Then the volume ﬁeld of ﬂuid V t +1 is assigned into the scene space to render image ˆ
I t +1 that is supervised by ground-truth image
I t +1 .
Namely, the simulator updates its parameters to better solve the pressure projection problem that is formulated as a regression task in our work, hence providing right velocity to advect the ﬂuid volume for better rendering.
As a result, a ﬂuid simulator can be learned from videos without other supervision.
3. O PTIMIZATION D ETAILS
The volume-augmented renderer does not have any prior knowledge of the ﬂuid geometry, thus it is hard to directly joint train the simulator and renderer.
To endow the volume-augmented NeRF with the ability to capture geometry information from ﬂuid and do inverse rendering preliminarily, we ﬁrst train the radiance ﬁeld network with initial volume ﬁeld of all training sequences (volume of ﬁrst frames serve as initial conditions for starting ﬂuid simulation).
After pre-training, the render model can reason about the geometry information of ﬂuid from ground-truth image and perform rendering preliminarily.
Then we jointly train the render model and the simulator on sequence data to learn ﬂuid dynamics, i.e. , train the simulator and renderer step-by-step.
We summarize the optimization function L for joint training as follows:
[Equation]
where β is set to 1e-6 to control the strength of the regularization term.
E VALUATION
We conduct experiments on three datasets that involve two Fluid Fall datasets that we generated and
DPI Dam Break dataset (Li et al., 2019).
Results of simulation and rendering on three datasets are then reported to demonstrate the effectiveness, generalization ability and robustness of our method.
Ablation studies are performed to analyze components and hyper-parameters.
Experimental settings and more analyses are shown in Appendix A.
4. D ATASETS
Fluid Fall .
We use synthetic data simulated using a CFD tool Mantaﬂow (Pfaff & Thuerey) and rendered using Blender.
Two views for each video are generated.
The simulation environment is where ﬂuid objects fall in a cubic container.
We vary the initial shape, position and velocity.
We generate two versions of datasets in that each sample contains one/two ﬂuid objects randomly sampled from 5 different shapes.
Each one of the two datasets contains 100 sequences with 50 steps. 90 sequences are used for training and the rest for test.
Please refer to Appendix for more details.
Dam Break .
We randomly sample 180 sequences with 50 time steps for training and 20 sequences for test from training set and test set of DPI Dam Break dataset respectively.
The mesh of each time step is constructed from particles using Marching Cube algorithm and rendered using Blender.
Two views for each video are generated.
Please refer to Appendix for more details.
4. T HE R ESULTS OF S IMULATION
In this section, we evaluate our method on simulation.
We report results of our method on both training and test set.
The mean squared error (MSE) between the predicted volume ﬁeld and ground- [Table] [Figure] truth is reported in Table 1, where the average error of all time steps is presented.
The results on training set not only prove that our method infers accurate ﬂuid geometry properties from images and performs inverse rendering well, but also demonstrate that our method learns ﬂuid dynamics from videos.
Meanwhile, our method shows strong ability to generalize to unseen data (test set) because our method learns real ﬂuid dynamics explicitly from training videos, while most of related methods can only ﬁt on one sequence every time.
Then we compare our radiance-driven simulator with the simulator trained using ground-truth volume ﬁeld.
We can conclude from the results on test set that the simulator trained with videos as supervision performs comparably to the model trained with ground-truth volume ﬁeld.
We present some visualization results in Figure 2, where predicted ﬂuid volume ﬁelds and ground-truth are shown.
The volume ﬁeld is visualized as voxel by setting a threshold.
Accurate ﬂuid volume ﬁelds are advected using our simulated velocity ﬁeld.
Following the common setting (Guan et al., 2022; Li et al., 2021b), we also compare our simulator with the model trained with GT in future prediction (51-60 steps).
We roll out the simulator for 10 more steps that are not covered by the training set.
Results are also reported in Table 1.
Please refer to
Appendix A for more results.
4. T HE R ESULTS OF R ENDERING
We evaluate our renderer in this section.
PSNR, SSIM (Wang et al., 2004) and LPIPS (Zhang et al., 2018) are used to evaluate the quality of the rendered images.
We ﬁrst report quantitative results of our method on test set in Table 2 and present rendered results in Figure 3.
Our method achieves better results among most metrics and generalizes to test set without obvious deterioration even though the model has never seen these test images.
Photorealistic images are synthesized using our method.
To compare with other methods for rendering dynamic scenes and evaluate performance on novel view synthesis (NVS), we train two variants of NeRF ( i.e. , T-NeRF and D-NeRF) on our [Table] , while our method is only trained on training set. [Figure]
[Figure] [Figure] benchmarks. T-NeRF processes dynamic scenes by just combining the input position of sampled point x with an additional input time t .
D-NeRF (Pumarola et al., 2021) takes an extra network to learn a transformation to canonical space for each sampled point at time step t , i.e. , ( x , t ) −→ ∆ x .
Due to that D-NeRF and T-NeRF cannot generalize to unseen data, we train D-NeRF and T-NeRF on each sequence of test sets respectively .
Our method is only trained on training set and evaluated on test set directly.
As shown in Table 2 and Figure 3, our method achieves better performance and recovers ﬁne details in both geometry and appearance.
The generalization ability of our method is also fully proved.
This indicates that our method reasons about real physical properties and geometric characteristics of the ﬂuid.
Future prediction experiments are also performed on test set using T-NeRF, D-NeRF and our method.
The quantitative results are shown in Table 2.
4. C OMPARISON WITH METHODS USING OTHER REPRESENTATIONS
We represent and simulate the ﬂuid in an explicit and Eulerian way. NeuroFluid
(Guan et al., 2022) is a concurrent work that represents ﬂuid using particles.
Such a way introduces/enhances two kinds t O u r s Generalize the trained model to unseen sequence t Results of Rendering
G T
N e u r o F l u i d
O u r s
G T
N e u r o F l u i d [Figure] of ambiguity ( i.e. , spatial ambiguity and temporal ambiguity) and thus formulates a prohibitively illposed inverse optimization problem.
Detailed analyses of ambiguity are presented in Appendix A.5.
In contrast, we represent ﬂuid using the volume ﬁeld and perform simulation in the Eulerian view. Thanks to this representation, accurate gradients with respect to ﬂuid volume can be computed to update it directly and further drive the simulator with almost no ambiguity being introduced. Therefore, so many ambiguities make NeuroFluid can only overﬁt one sequence every time and cannot learn real ﬂuid dynamics, while our model can be trained with many sequences together and learn generalizable ﬂuid dynamics.
To demonstrate our statements, we train their model using their published code on the training set of our DPI dataset and report rendering results on the test set in Table 3.
Our model generalizes to test set well and achieves signiﬁcantly better performance.
We also report the simulation results of NeuroFluid trained with videos and ground-truth particles respectively in Table 4 to prove that method NeuroFluid cannot infer dynamics from videos and transfer to unseen data.
The average error of the particle positions with respect to ground-truth is used for evaluation.
The visualization results of simulation and rendering are shown in Figure 4, where our results and NeuroFluid are both shown. We see that our model can be trained on multiple sequences simultaneously and transfers to unseen sequences, while NeuroFluid collapses when trained with multiple sequences and transferred to other sequences.
Moreover, we compare our method with (Li et al., 2021b) that represents ﬂuid in the implicit space.
This rough and implicit representation also limits its generalization ability as the authors of (Li et al., 2021b) also stated in their paper. We report rendering results on test set of our DPI dataset in Table 3.
Our method achieves signiﬁcantly better results.
[Table] [Table]
The average position error of 50 steps is shown.
C ONCLUSION
We propose an end-to-end and differentiable framework integrating ﬂuid simulation, 3D modeling and rendering.
The neural Euler simulator and NeRF-like renderer are associated using an explicit ﬂuid volume representation.
We achieve learning ﬂuid dynamic model with videos as supervision by propagating error signals between rendered and ground-truth images to all modules.
Various experiments involving simulation, NVS, future prediction and scene editing demonstrate both effectiveness and generalization ability of our method.
A A PPENDIX A. M ODEL D ETAILS
We ﬁrst present detailed model architecture of our proposed neural Euler simulator (pressure projection solver).
As shown in Figure 5, the inputs of the network contain three components: 1) the velocity ﬁeld updated after advection and external force application; 2) the volume ﬁeld of ﬂuid used to provide space distribution of different materials (ﬂuid and air); 3) distance ﬁeld that each grid saves the normalized distance between its center and container boundary.
These three parts are processed by three mini-networks respectively that contain two layers of 3D convolution (followed by BatchNormalization (Ioffe & Szegedy, 2015) and LeakyRelu (Maas et al., 2013)) and the output embeddings are then concatenated as a single feature.
Finally, we send the concatenated feature into subsequent four convolutional layers and output ∆ u used to update velocity ﬁeld
[Table]
h N o r m
C, K 3 S
L ea ky
R e l u
C 3 ,
K 3 S
Then we present detailed model architecture of our proposed volume-augmented neural radiance ﬁeld (NeRF).
As shown in Figure 6, the network is fully-connected and similar as NeRF (Mildenhall et al., 2020).
In addition to point position x ∈ R 3 and corresponding camera view d ∈ R 3 , we also send the ﬂuid volume value v of point x into the volume-augmented NeRF to regress density σ and
RGB color c .
Given the point x , the ﬂuid volume v is generated by performing trilinear interpolation on ﬂuid volume ﬁled V that is advected using simulated velocity ﬁeld.
The position vector x and view vector d are encoded using positional encoding before being sent into the NeRF.
A. D ATASET D ETAILS
Fluid Fall .
We generate two datasets by constructing a Fluid Fall environment.
The simulation environment is where ﬂuid objects fall in a cubic container.
The size of the container is 2m x 2m x 2m. Each sample of the two version datasets contains one or two ﬂuid objects randomly sampled from 5 different shapes (as shown in Figure 7).
We simulate ﬂuid fall data using a CFD tool Mantaﬂow (Pfaff & Thuerey).
The viscosity is set to 0.004 for Fluid Fall (1 object) and 0.
for Fluid Fall (2 objects).
60 time steps (50 steps for training) are simulated for each trajectory.
The simulated ﬂuid mesh is rendered using Blender.
Two views for each sequence are generated.
The ground-truth videos of these two datasets are rendered with different BRDF.
The Blender project used for generating videos will be publicly available upon acceptance.
To generate a dataset, we vary the initial shape, position and velocity.
Each one of the two datasets contains 100 sequences.
90 sequences are used for training and the rest for test.
[Figure]
An additional layer outputs the density σ (which is rectiﬁed using a ReLU to ensure that the output density is nonnegative) and a 256-dimensional feature vector. This feature vector is concatenated with the positional encoding of the input viewing direction γ ( d ) , and is processed by an additional fully-connected ReLU layer with 128 channels. A ﬁnal layer (with sigmoid ) outputs the emitted RGB radiance at position x , as viewed by a ray with direction d , given ﬂuid volume value v interpolated from volume ﬁeld V .
Dam Break .
We randomly sample 180 sequences of 50 time steps for training and 20 sequences for test from the training set and test set of DPI Dam Break dataset (Li et al., 2019) respectively.
This dataset is based on a Dam Break environment.
The size of the container is 1.62m particles
Marching Cube algorithm and rendered using generating publicly
[Figure] [Figure]
[Figure] [Figure] View Advection in Lagrangian View
[Figure] [Figure] [Figure]
[Figure] [Figure] We perform advection to the volume ﬁeld according the velocity ﬁeld u t − 1 . The advection process contains two steps, i.e. , back-tracing and interpolation.
We solve the complex dynamic equation by operator splitting. The time integration is split into three steps, i.e. , advection, applying external force and pressure projection. The advection can be seen as an operator that moves ﬂuid with a constant velocity.
The velocity ﬁeld after applying advection and external force is not divergence-free, which violates the incompressibility of ﬂuid. So the pressure projection is the key step to rectify the velocity and make the ﬂuid looks real. In the Lagrangian view, we achieve advection by just moving particles according to their velocity. The physical quantities such as mass and velocity are bound to particles and moved with these particles.
As shown in Figure 8(a), there is a ﬂuid object that contains three particles ( e.g. , particle q ) at time step ( t − 1) . We denote the mass and velocity of each particle as m and velocity v t −respectively. The velocity v t − 1 represents the average velocity over ( t − 1) → t .
The process of advection is just moving all the particles according to their velocities. The advected velocities are then applied with external force and pressure projection to get ﬁnal velocities at the next time step t . In the Eulerian view, the grids are ﬁxed. We cannot apply advection as we do in the Lagrangian view directly.
To achieve advection and update the physical quantities (volume and velocity) of each ﬁxed grid, we take a commonly used semi-Lagrangian method to do advection. It contains two sub-steps, i.e. , back-tracing and interpolation. As shown in Figure 8(b), there is a volume ﬁeld V t − 1 . We now want to apply advection to the volume ﬁeld V t − 1 and get the updated volume ﬁeld V t .
The key is to compute the volume value of each grid center point in the ﬁeld V t . For each grid center point of ﬁeld V t , we ﬁrst ﬁnd where it is at time step ( t − 1) and then borrow the volume value from V t − 1 to the grid center at time step t . The velocity ﬁeld u t − 1 represents the average velocity over ( t − 1)
→ t .
So we can back-trace from the grid center point at time-step t using the velocity ﬁeld u t − 1 to ﬁnd where it lies at time step ( t − 1) and interpolate its volume value. Overall, the whole advection is achieved by back-tracing and interpolation, which are both differentiable. The velocity ﬁeld is also advected in this way. But the advected velocities need to be applied with external force and pressure projection to get the ﬁnal velocity ﬁeld at the next time step.
A. E XPERIMENTAL S ETTINGS
The size of Euler grid is set to 40 x 40 x 40 for our generated datasets and 54 x 60 x 32 for DPI
Dam Break.
The image resolution is set to 256 x 256 and two views are used for training.
The initial velocity ﬁeld is generated by setting velocity of the grids to the initial velocity, and the initial volume ﬁeld is generated by setting the volume value to 1.0 if the center point of grid is inside the initial ﬂuid mesh otherwise to 0.
We ﬁrst pre-train the volume-augmented NeRF model using the ﬁrst frames of all sequences for 20k iterations with a learning rate 5e-4 (decay 0.5 every 5k iterations).
Then we jointly train the simulator and renderer with sequences data for 60k iterations.
The initial learning t Train and test on one sequence t Results of Rendering Results of Simulation
O u r s
N e u r o F l u i d O u r s N e u r o F l u i d
A. A MBIGUITY ANALYSES OF DIFFERENT REPRESENTATIONS
As we have stated in the main paper, NeuroFluid (Guan et al., 2022) is a concurrent work that represents ﬂuid using particles.
Such a way introduces/enhances two kinds of ambiguity ( i.e. , spatial ambiguity and temporal ambiguity) and thus formulates a prohibitively ill-posed inverse optimization problem.
Spatial ambiguity exists between ﬂuid particles and the image rendered using NeRF.
NeRF constructs an implicit continuous ﬁeld and renders image by sampling in this continuous space.
However, particle representation is highly discrete and cannot be combined with NeRF directly.
To make it continuous, NeuroFluid computes some complex and continuous distribution characteristics based on the discrete particles (through their proposed Particle-Driven NeRF (Guan et al., 2022)) to represent ﬂuid.
Then they can render image using NeRF to sample in this transferred and continuous ﬁeld ( i.e. , discrete particles → continuous ﬁeld → image).
Unfortunately, lots of spatial ambiguities between particles and the image are introduced by this transferring when doing inverse rendering, e.g. , many different combinations of particles may result in the same transferred ﬁeld and further the same pixel color.
Namely, even if we can infer the transferred ﬂuid ﬁeld from images, we cannot infer correct particle positions from the inferred ﬁeld.
Moreover, temporal ambiguity exists even if the inferred particles are accurate, e.g. , applying different motion vectors to the input particles may result in the same spatial distribution of particles at the next time step.
Namely, no temporal correspondence is provided by the gradients from the image to help infer particle motion for Lagrangian simulation, which has to trace each particle in the whole life of the sequence.
In contrast, we represent ﬂuid using the volume ﬁeld directly and perform simulation in the Eulerian view, which can be seen as a continuous ﬁeld approximately through linear interpolation.
Thanks to this representation, accurate gradients with respect to ﬂuid volume can be computed to update it directly and further drive the simulator with almost no ambiguity being introduced.
Therefore, so many ambiguities make NeuroFluid can only overﬁt on one sequence every time and cannot learn real ﬂuid dynamics, while our model can be trained with many sequences together and learn generalizable ﬂuid dynamics.
Note that all comparisons presented in the main paper are conducted under a transfer setting ( i.e. , training on multiple sequences and testing on unseen sequences).
For further comparison, we also train our model and NeuroFluid using only one sequence (under their setting).
As shown in Figure 9, not a bad rendering result is achieved by NeuroFluid due to that their designed complex input features make its renderer to be overﬁtted more easily.
However, such a modeling method cannot compute correct gradients for inverse optimization and learn ﬂuid dynamics (simulation).
A similar phenomenon is observed in Figure 3 and Figure 4 of their own paper (Guan et al., 2022).
Note that it is harder for NeuroFluid to overﬁt on a more difﬁcult sequence than what they use in their paper, even though we train it on only one sequence. A. A BLATION S TUDY
Component analysis and parameter analysis are performed in this section.
The effectiveness of our proposed radiance-driven ﬂuid simulation is proved in Section 4.2.
Our simulator shows comparable performance to the model trained with ground-truth Euler ﬂuid grid.
Then we drop the penalty used to regularize divergence of the updated velocity ﬁeld during training and report the results on test set of Fluid Fall (1 obj) in Table 5 and Table 6.
Such a regularization term stabilizes training and achieves signiﬁcantly better simulation.
We have mentioned in Section 3.4 that we ﬁrst train the renderer and then jointly train the renderer and the simulator.
We perform an ablation study to prove that the jointly training strategy does perform better than solely training the simulator and ﬁxing the renderer. The results are also shown in Table 5 and Table 6. We ﬁnally analyze the improvement achieved by using more views for training. We train our model using 2 views and we compare it with the model trained with only one view.
The results of the simulator and renderer on test set of Fluid Fall (1 obj) are shown in Table 5 and Table 6.
Our method also works well when given videos with only one view, while more views bring obvious performance improvements in both simulation and rendering.
[Table]
[Table]
We also analyze the robustness of our method to β that is used to weight the contribution of divergence regularization term in equation 9.
Experiments with different β are conducted on Fluid Fall ( obj) and the results of simulation are shown in Table 7.
Our model is not very sensitive to different values of β .
We further report more detailed results on Fluid Fall (1 obj) of training our model using different number of views in Table 8.
In summary, more views result in better performance because more views provide more detailed 3D cues and more accurate and ﬁner gradients for the simulator.
However, our model also works well even with only one view.
[Table] [Table]
A. S PACE AND TIME ANALYSIS
We report model size and timings in Table 9.
The timing of inference is evaluated on one 3090 GPU.
It takes about 48 hours to train the whole model on 180 sequences with four 3090 GPUs.
Although improving the inference speed is not the main concern of this work, there are some tricks that could be taken to improve the rendering speed, e.g. , guiding the sampling according to the simulated ﬂuid volume ﬁeld.
[Table]
A. E XTREME V IEWS G ENERATION AND S CENE E DITING
We explicitly simulate the ﬂuid and the geometry of ﬂuid is explicitly represented.
Such a representation method endows the renderer with the ability to generate ﬂuid videos from views that are far away from training distribution.
To prove this, we train our model with only one view and synthetic images from extreme views ( e.g. , overlook).
Visualization results are shown in Figure 10(a).
O r i g i n a l
V i e w t
D - N e
R F
V i e w (a) Extreme views synthesis
O r i g i n a l
E d iti ont
E d iti on
E d iti on(b) Scene editing
Furthermore, one extra advantage of our ﬂuid volume representation is that it is ideal and efﬁcient for scene editing , i.e. , moving the ﬂuid in the 3D space by simply applying a transformation to the volume ﬁeld.
As shown in Figure 10(b), we move the ﬂuid in the scene and render corresponding images.
We see that our model even handles occlusion well.
Moreover, our simulated ﬂuid can be easily placed into complex scenes by combining with other NeRF models.
A. M ORE V ISUALIZATION RESULTS
We present more results of our simulated ﬂuid volume ﬁeld and rendered images on test sets of the three datasets in Figure 11.
We develop an end-to-end framework integrating simulation, 3D modeling and rendering and learn ﬂuid dynamics with videos as supervision.
Although no ground-truth simulation data is provided, our method performs accurate simulation and renders photorealistic images.
Rendered videos are shown in supplementary materials.
Each video contains 60 frames,frames of which are results of future prediction.
We also provide some results of novel view synthesis (NVS) on the test sets in Figure 12.
Although we train our model on training set with only two views, our model generalizes to test set without obvious deterioration and synthesizes images from 50 different views that are out of training distribution.
The camera rotates along z-axis and we render ﬂuid of every step with different views.
A part of views is shown in Figure 12.
Rendered videos are shown in supplementary materials.
We have reported ﬂuid editing ( i.e. , moving the ﬂuid in the scene) results in Figure 4(b) of our paper.
We further rotate the ﬂuid in the scene by directly rotating the simulated ﬂuid volume ﬁeld during inference.
Visualization results are shown in Figure 13, where the ﬂuid falls, translates and rotates in the scene.
Videos of scene editing are shown in supplementary materials.
Note that the model used to perform scene editing is trained on one sequence with two views.
A. L IMITATIONS AND FUTURE WORK
Same as most of other methods for inverse rendering (Guan et al., 2022; Munkberg et al., 2022;
Hasselgren et al., 2022; Yu et al., 2022), we mainly focus on learning from synthetic data.
It remains a lot of great challenges from many research areas ( e.g. , differentiable photorealistic rendering, optics, etc. ) and requires extensive research efforts as well as technical development to achieve learning from real-world data, especially for ﬂuid that contains complex light-transport effects ( e.g. , t O u r s G T Rendered Images (Test Set) t
O u r s G T O u r s
G T O u r s
G T O u r s G T O u r s
G T Predicted Volume Field (Test Set) t V i e w R o t a ti on t
V i e w R o t a ti on refraction, reﬂection, etc. ). Although some differentiable renders (Wang et al., 2022; Bemana et al., 2022) are proposed to consider refraction in ﬂuid and perform forward rendering well, lots of ambiguities are introduced by refracting ( e.g. , different local shapes may result in the same local color) and make the optimization of inverse rendering task exceedingly difﬁcult.
Thereby, it still remains a great challenge to do inverse rendering for ﬂuid data from real world and beyond the current state-of-the-arts.
A potential solution is to extract and utilize geometry cues contained in [Figure] images to reduce ambiguities and infer ﬂuid dynamics.
Besides, embedding ﬂuid-solid coupled simulation in our framework and using our model to infer external forces from videos are also interesting research topics.