NeuPhysics: Editable Neural Geometry and Physics from Monocular Videos
Abstract
We present a method for learning 3D geometry and physics parameters of a dynamic scene from only a monocular RGB video input.
To decouple the learning ofunderlying scene geometry from dynamic motion, we learn a time-invariant signeddistance function (SDF) which serves as a reference frame, as well as an associateddeformation field that is conditioned on time.
We further design a two-way conversion between the neural field representation and its corresponding hexahedralmesh.
This hybrid representation allows us to bridge the neural geometry representation with a differentiable physics simulator, and therefore estimate physicsparameters from the source video by minimizing a cycle consistency loss.
It alsoallows a user to interactively edit 3D objects from the source video by editing therecovered hexahedral mesh, and propagating the operation back to the neural fieldrepresentation.
Experiments show that our method achieves higher-quality meshand video reconstruction of dynamic scenes compared to other competitive Neural
Field methods, and we provide extensive examples which demonstrate its abilityto extract useful 3D representations of dynamic scenes from videos captured withconsumer-grade cameras.
Introduction
Monocular RGB videos are easily accessible nowadays as smartphones are ubiquitous.
Can youimagine one day being able to build an interactive digital twin of your surrounding world using justa smartphone?
You could reconstruct a 3D portrait for yourself (Sec. 4.1); playback an existingrecording from a new angle (Sec. 4.2); edit your favorite cartoon video (Sec. 4.4); or estimate thematerial parameters and interact with it virtually (Sec. 4.3).
In this paper, we design a systemthat could possibly make all of these things come true.
Taking a single RGB video as input, ourmethod can estimate the geometry and dynamics parameters of the scene which is parameterized byinfinite-resolution neural fields [36] and enables editing and physically-based simulation.
In this paper we design an end-to-end differentiable rendering and simulation pipeline that candirectly learn the geometry and dynamics from a monocular video.
By assuming that there exists aone-to-one dense mapping in the 3D space between two temporal frames, our method decouples thistask into two orthogonal parts: a static reference frame, and a mapping function that encodes motion.
Within this pipeline, the static reference frame consists of three neural fields – appearance, rigidity,and an SDF for geometry.
Given a 3D spatial coordinate as input, these fields output its RGB color,rigidity mask, and SDF value respectively, in the reference frame.
Time-variant motion is encoded ina separate field, which takes as input a 3D spatial coordinate sampled along a ray, plus a learned latentcode associated with a particular time step, and outputs the displacement from the spatial coordinateback to its corresponding coordinate in the reference frame.
We utilize the density function from
NeuS [58] for rendering, and the Rigidity mask from NonrigidNeRF
[54] to regularize the motionand obtain foreground/background separation.
An advantage of factorizing a dynamic scene intostatic geometry + dynamic motion is that by querying the motion network, we can easily get pointwise correspondence across frames, from which we can infer the dynamics properties of the scene.
Specifically, physics parameters will be optimized by minimizing the difference between the resultsfrom the motion network and the differentiable simulator.
After learning reasonable parameters, thedifferentiable simulator can in turn be used to animate and edit the scene.
Experiments show that ourmethod outperforms comparison methods in surface reconstruction, video reconstruction, and novel view synthesis tasks, both qualitatively and quantitatively.
Multiple examples also demonstrate ourmethod’s scene understanding and user interaction capabilities.
In summary, the key contributions of this work are as follows:
• We present a framework for estimating and editing the geometry, appearance, and physics
parameters directly from a single monocular video (Figure 2).
• By decoupling the learning of static SDF geometry and dynamic motion, our method
better regularizes this under-constrained problem and enables reconstruction of high-qualitydynamic surface mesh sequences from monocular videos (Figure 3).
• By connecting the neural-fields-based geometry reconstruction to a differentiable physicsengine, this approach avoids manual construction of 3D models and poor initialization for
the differentiable simulator, empowering the users to estimate the modeling and physics
parameters from scratch (Figure 5).
• We design a conversion (Figure 1& 2) between the explicit hexahedral mesh and implicitneural fields in our end-to-end modeling, simulation and rendering pipeline, within which
users can (1) obtain a high-quality explicit mesh from the implicit fields, and (2) interact(e g. add, delete, move, deform, and simulate) with the implicit 3D field by animating ormanipulating the explicit mesh.
Related Work
Differentiable volume rendering with implicit neural representation have recently been used tosynthesize novel views, estimate geometry and appearance.
Neural Radiance Field (NeRF) [36], andits follow-on works [8, 50, 2, 64] have achieved photorealistic rendering results even on complexscenes.
The expressive coordinate-based multi-layer perceptron (MLP) representation equips thescene with high resolution [70] while remains regularized [69] and memory-efficient.
Comparedwith differentiable surface rendering [39, 40], volume rendering could backpropagate gradientsto the entire space so to better handle complex geometry optimization with topological changesand mitigate local minima.
The original density field from NeRF cannot directly reconstruct highquality surface mesh, so SDF-based neural fields [58, 66, 41] are proposed for multi-view geometryreconstruction.
Compared to other discrete representation like mesh [57, 24, 33], point cloud [13, 32]and voxels [5, 62], SDF-field has higher resolution.
And the the NeRF-based method can take raw
RGB images without pixel-lebel segmentation [38, 65, 25].
However, current SDF-based neuralfields cannot handle dynamic scenes.
We will show in Figure 3 that existing methods do not convergewell on dynamic input while our method can handle these cases.
Recent work has extended NeRFto videos of dynamic scenes [43, 1, 68], but they mainly focus on the 2D image synthesis, andfew of them can reconstruct the 3D geometry well.
They either learn a canonical field with timedependent motion [42, 44, 54, 4] or condition all the fields on the time axis [12, 15, 60].
We choosethe former strategy because decoupling motion from the geometry makes the formulation moreconstrained and it also provides us with point-wise correspondence.
All the mentioned dynamicextension of NeRF focus on the 2D video reconstruction and lack regularization on the 3D geometry.
Our method can reconstruct the geometry much better than the existing works.
Moreover, sincesingle-video 3D reconstruction is highly ill-posed, our geometry prior can also help improving theimage reconstruction quality, as shown in Section 4.1.
Differentiable simulators enable physics prior to better fit in data-driven approaches and thereforehave been applied to more and more robotics and graphics tasks [51, 10, 20, 29, 52, 63].
Thosephysically-based simulation methods have been developed for dynamical systems of different properties, including rigid bodies [6, 45, 7, 14, 35], soft bodies [23, 22, 46, 16, 11, 28, 27, 19], articulatedbodies [18, 59, 47], cloth [30, 31], and fluids [55, 56, 21, 53].
Methods
Given a monocular video consisting of N RGB image frames { I i } Ni =1 , our approach will estimatethe 3D geometry within the camera viewing frustum, foreground/background segmentation, andpointwise correspondence (across time) of the dynamic
3.1 Rendering Pipeline
SDF and color networks.
Similar to recent Neural Fields methods [36], we parameterize thegeometry and appearance of a scene using a coordinated-based Multi Layer Perceptron (MLP), whichcan further be used to render 2D images using the volume rendering technique proposed in NeuS [58].
The geometry of the entire scene is represented by an SDF network s θ SDF ( p ) : R 3 → R that maps aspatial coordinate p ∈ R 3 to its signed distance to the nearest surface.
For the appearance, we havea color network c θ color ( p ) : R 3 → R 3 that describes the RGB value of each spatial point.
Whenrendering this scene, the points in the ray emitted from a pixel can be written as { p ( t ) =
o
+ t v | t ≥ 0 } ,where o is the camera center, v is the unit direction of the ray, and t is the distance from the cameracenter.
The color of this pixel is then calculated by integration,
[Equation]
where w ( · ) [58] is a weight function describing the contribution of each point’s color based on its
SDF value.
More details can be found in the Appendix.
Motion network.
However, the above framework can only handle multi-view images of a static scene,which motivates another component to accommodate the temporal information.
Instead of addingan extra dimension to directly condition the existing networks upon time, we choose to decouplethe dynamic and static information.
Therefore, our method defines the SDF and color network on acanonical frame while estimating the offset field [54] of each frame.
For the i -th frame I i encodedby a latent feature l i ∈ R d l , the motion network b ′ θ motion ( l i , p ) :
R d l × R 3 → R 3 bends each queryray { p ( t ) } to { p ′ ( t )
= p ( t ) + b ′ ( l i , p ( t )) } .
We then compute SDF and color values of p ′ ( t ) on thereference frame.
Rigidity network.
Furthermore, since we want to separate and simulate dynamic objects withinthe scene in a later state, we apply a Rigidity network [54] r θ rigidity ( p ) : R 3 → R to distinguishdynamic foreground and static background.
The mask r ( p ) ∈ [0 , 1] defined on the canonicalframe indicates the likelihood that a point belongs to the swept volume of moving elements in thescene, with a value of 0 representing "completely static".
The offset value is then revised to beb θ rigidity θ motion ( l i , p ( t ))
=
r θ rigidity ( p ) · b ′ θ motion ( l i , p ( t )) .
During training, this formulation encouragesdynamic regions to have larger r ( · ) values while driving static areas toward zero.
3.2 Simulation Pipeline
Geometry reconstruction.
Given the signed distance field s ( · ) along with the rigidity mask r ( · ) , wecan reconstruct the geometry of the dynamic foreground and static background.
For example, theforeground in the i -th frame is an occupied space within the swept volume of moving objects.
Thisoccupied volume can be described by
[Equation]
where k sdf and k rigidity are constant thresholds for the SDF and rigidity values, respectively.
Withthis formulation, one can convert it into some type of explicit mesh representation.
For renderingpurposes, a surface mesh can be obtained using the marching cubes algorithm
Alternatively, avolume mesh of the dynamic elements is also reconstructed for the purpose of physical simulation.
We sample over a 3D grid to find vertices within volume A i .
If a grid point p satisfies Equation 2, weadd this voxel to the hexahedral volume mesh m i .
We choose a hexahedral rather than a tetrahedralvolume mesh due to its simplicity and regular shape, which can further be simulated by DiffPD [11],a differentiable FEM simulator for hexahedral mesh.
Triangle surface mesh and tetrahedral volumemeshes are also possible for future work.
Figure 1 visualize the conversion between the implicit andexplicit representation.
Physics Engine.
From the input video of a dynamic scene, we can extract a mesh sequence { m i } Ni =1with consecutive motion.
A mesh model m i is described by its vertices and hexahedral elements{ Q i ∈ R n v × 3 , E i ∈ N n e × 8 } , where the rows of Q i are the 3D coordinates of n v vertices, and eachrow in E i records the indices of eight vertices that form a hexahedron.
The objects in this paper are assumed to be soft bodies because (1) such bodies are important in applications such as metaverse and digital twins – human bodies, hands, and faces are deformable,and (2) compared to rigid bodies, soft bodies have more expressiveness to model objects and aretherefore more interesting and challenging to study.
We use Projective Dynamics [3] implemented by
DiffPD [11] as our physics engine.
The governing equation can be written as
[Equation]
where M is the mass matrix, Q i is the vertex locations at frame i , h is the time step, ˙
Q i is the velocity,
E is the potential energy due to deformation, θ physics are the physics parameters, and f ext is theexternal forces.
Note that the gradient of the potential energy ∇ E can be interpreted as the internalforces within the soft materials.
∇ E is defined on the frame i + 1 , so this system uses implicit timeintegration for better numerical stability.
Objects are modeled by homogeneous co-rotated materials,where the material parameters θ physics include Young’s modulus and Poisson’s ratio.
A detailedformulation of the elastic energy E can be found in the supplementary materials of [11].
Usually, it is difficult to estimate the states Q , ˙ Q , and the parameters θ physics and f ext in Equation 3from only a monocular video, due to the high-dimensionality and nonlinearity of the system.
However,such system identification tasks are useful for constructing digital twins, animation, and control.
Recent progress in differentiable physics tackle this challenge by making the entire simulationdifferentiable.
In DiffPD, gradients ∂ L ∂ Q i , ∂ L ∂ ˙ Q i , ∂ L ∂θ physics , and ∂ L ∂ f ext are all available, where L is a userdefined scalar loss function.
We can then efficiently estimate the parameters using a gradient-basedoptimizer like Adam [26].
Parameter Estimation.
By simulating the meshes in a differentiable physics engine sim ( θ physics , · )and comparing with the reconstruction results as a supervision signal, we can optimize the physicsparameters θ physics using gradient-based methods.
To illustrate, we run 1 simulation step fromthe i -th frame and get a new model at the i + 1 frame depending on θ physics : m ′ i
+1 ( θ physics )
=
sim ( θ physics , m i , i → i + 1) .
One idea is to design an error function (like Chamfer Distance) distance ( m i +1 , m ′
i +1 ( θ physics ))
between the extracted mesh m i +1 and the simulated mesh m ′
i +1 ( θ physics ) and search for the θ physicsthat minimizes the error.
However, this strategy requires us to perform sampling and mesh reconstruction for every time step, which is computationally expensive.
An ablation study and profiling numberscan be found in Figure 8.
Additionally, the reconstructed mesh might vary significantly across timesteps, thus cannot guarantee point-wise correspondence, making the dynamics discontinuous andnon-differentiable w.r.t.
the networks.
To save sampling time and make the gradient well-defined, we design a cycle-consistency loss to 173 compute the distance between the simulation results and neural fields.
Starting from an arbitrarymesh m i sampled at the i -th frame, we run our differentiable simulator on it for j steps and get 175 m ′
i + j ( θ physics ) =
sim ( θ physics , m i , i → i + j ) .
When they are deformed back to the referenceframe, m ′ i + j ( θ physics )
should correspond to the same shape as m i .
Note that as m ′ i + j ( θ physics ) issimulated from m i , they have an exact pointwise correspondence (same topology), so we can directly 178 subtract their point clouds to obtain their distance: [Equation]
[Equation]
) 180 where b ( l i , m i ) is the motion field at the frame i that can map the mesh
m i to the canonical frame.
From this expression, we note that L physics is differentiable w.r.t.
the rigidity network, motionnetwork, and physics parameters.
Even over a long sequence, we can sample m i only once and use itas a template to compare with all other frames.
Scene Editing.
The simulator in our pipeline can serve as a strong physics prior to forecast themotion and edit the scene.
The right part of Figure 1 shows how we edit the SDF and color field bymanipulating the hexahedral mesh.
Based on a hexahedral mesh m i extracted from the i -th frame,we can simulate it to get a modified one m ′
i +1 .
To render the modified scene, we will add one more‘bending’ layer b edit ( m ′ i , · ) on top of the primary motion field b ( l i , · ) .
When rendering a point p ′ for the edited image, we must first check if this point is being moved by ourediting operation.
An equivalent question is whether this point is located within the hexahedral meshm ′
i .
The inside-outside-test would be trivial if all hexahedra in m ′
i remained in a regular grid, however,the hexahedra could be deformed after simulation.
Therefore, we split each hexahedron into fivetetrahedra.
The query point p ′ is inside the hexahedron if and only if it is inside one of the tetrahedra.
𝜃 𝑚𝑜𝑡𝑖𝑜𝑛
MotionField (𝑡 1 ) 𝜃 𝑟𝑖𝑔𝑖𝑑𝑖𝑡𝑦
Rigidity Field 𝜃 𝑃ℎ𝑦𝑠𝑖𝑐𝑠
Diff.
Sim.
Object (𝑡 1 )
Canonical Object
Object (𝑡 2 ) 𝜃 𝑃ℎ𝑦𝑠𝑖𝑐𝑠
Diff.
Sim.
Object (𝑡 3 ) 𝜃 𝑚𝑜𝑡𝑖𝑜𝑛
MotionField (𝑡 2 ) 𝜃 𝑚𝑜𝑡𝑖𝑜𝑛
MotionField (𝑡 3 )
Canonical Object
CanonicalObject 𝐿 𝑝ℎ𝑦𝑠𝑖𝑐𝑠
Hex Mesh Reconstruction
WarpBack WarpBack
WarpBack
Cycle Consistency
Cycle Consistency 𝜃 𝑐𝑜𝑙𝑜𝑟
ColorField 𝜃 𝑆𝐷𝐹
SDFField 𝐿 𝑅𝑒𝑛𝑑𝑒𝑟𝑖𝑛𝑔
VolumeRendering
Non-Differentiable Operators
Trainable Parameters
Differentiable Operators
Loss Functions
With this principle in mind, our next step is running a KNN algorithm Q ′ p ′ =
KNN ( m ′ i , p ′ ) tolocate the nearest vertices of the mesh to the query point p and select the tetrahedra that Q ′ p ′ belongto as candidate tetrahedra.
Similar to [67], we decide whether p ′ is inside a tetrahedron according toits barycentric coordinates.
For a tetrahedron consisting of four vertices { p ′ i =
( x ′ i , y ′
i , z ′ i ) ⊤ }
i
=1 , thebarycentric coordinates { c i } 4 i =1 of p ′
=
( x ′ , y ′ , z ′ )
⊤ can be computed by c i =
Det i /Det 0 , where
Det 0 = (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) x ′y ′z ′ x ′y ′z ′ x ′y ′z ′ x ′y ′z ′ (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) , Det 1 = (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) x ′ y ′ z ′ x ′y ′z ′ x ′y ′z ′ x ′y ′z ′ (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) , Det 2 = (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) x ′y ′z ′ x ′ y ′
z ′ x ′y ′z ′ x ′y ′z ′ (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) , ..., (5)
[Figure]
and Det i is obtained by substituting the i -th row of Det 0with ( x ′ , y ′ , z ′ , 1) .
p ′ is inside or on the surface of thistetrahedron if and only if 0 ≤ c i ≤ 1 holds for i
=1 , 2 , 3 , 4 .
If p ′ does belong to a tetrahedron, then it is edited alongwith the mesh.
Assuming the modification can be linearlyinterpolated inside a tetrahedron, the original coordinate pin the i -th frame is then the weighted sum p = (cid:80) 4 i =1 c
i p i ,where p i is the original coordinate of the vertex p ′
i beforeediting.
Additionally, we need to consider the area that wasonce occupied by m i but is now empty after editing: wecan simply assign those points a large positive SDF valuesuch that they are effectively transparent during volumerendering and will not influence 3D reconstruction.
3.3 Training and Losses
To train our network, the main supervision comes from the video, that we would like to minimizethe distance between rendered and the ground truth images L color =
(cid:13)(cid:13)(cid:13)
I i − ˆ
I i (cid:13)(cid:13)(cid:13) 1 .
Besides that, wewill include extra terms to regularize the training.
First, one important property of SDF is its unitnorm.
Eikonal term [17] is therefore added as L Eikonal =
(cid:80) i,k ( ∥ p i,k ∥ 2 − 1) 2 .
Moreover, we wantthe motion in our scene to be more constrained (i.e. locally rigid, incompressible, and more staticregion), so we add offset loss L offset = (cid:80) i,k ∥ b ′ ( l i , p i,k ) ∥ 2 + r ( p i,k ) and divergence loss [54]
L divergence =
(cid:80)
i,k ∥ div ( b ( l i , p i,k ))
∥ 2 .
Finally, we will use the physics loss L physics when we trainfor physics parameters.
Therefore, the total loss function is
[Equation]
By default, we will set w Eik = 0 .
1 , w off = 50 . 0 , w div = 2 .
and it might vary depending onthe scenes.
Figure 2 demonstrates the data and gradients flow during training.
In our proposedcycle-consistency physics loss, the rendering and simulation modules are connected by the rigidityand motion networks.
Instead of jointly training all the parameters together, we adopt a sequential
Method
Sitting
Standing
Basketball
Hand
LPIPS ↓
SSIM ↑
PSNR ↑
LPIPS ↓
SSIM ↑
PSNR ↑
LPIPS ↓
SSIM ↑
PSNR ↑
LPIPS ↓
SSIM ↑
PSNR ↑
D-NeRF 0.
34.146 NeuS 0.
34.485 NRNeRF 0.
33.999 Ours 0.
[Figure]
training strategy where the rendering networks are first trained with rendering losses for 300,000epochs, and the physics parameters are then trained with L physics for 100 epochs..
The physicsmodule is trained independently after the networks converge because the differentiable simulationis much more expensive than querying of MLPs.
An ablation study for the training strategy isconducted in Section 4.5.
It would be an interesting future work to try a faster engine and leverage thesimulation engine as a physics prior to improve the quality of the rendering networks.
A full trainingcycle typically takes around 18 hours on an NVIDIA A5000 GPU.
For the training data (used withconsent), we note that the background must contain sufficient detail, or the Structure from Motionimplementation by COLMAP [48, 49] could fail, particularly since the dynamic foreground makescamera registration even more difficult.
Experiments
4.1 Dynamic Video Reconstruction
We visualize some mesh reconstruction quality in Figure 3.
Table 1 also shows the quantitativeresults.
Our method outperforms comparison approaches in many entries.
More video reconstructioncomparisions can be found in Appendix F.
4.2 Novel View Synthesis
We simulate four cases of deformable objects moving in a room.
For each scene, we render twovideos using different camera paths so that we can train on one path and use the other one to testnovel view synthesis.
Quantitative results in Table 2 show that our method achieves the best synthesisquality in all the metrics.
We show the datasets and results in Figure 4.
(a, b) are two frames from thetraining videos.
(c) is taken at the same time step as (b), but it is a novel view from the test path.
Wecan see that our method successfully synthesizes the dynamics scene from a novel view, while othermethods cannot reconstruct the geometry and texture properly.
Method
Tennis Ball
Basketball
Blue Ball
Blue Cube
[Table]
↑
PSNR ↑
LPIPS
[Table]
LPIPS
[Table]
PSNR
[Table]
NRNeRF
[Table]
[Figure] [Figure]
4.3 Parameter Estimation
[Figure]
In this section, we first use our network to reconstruct the dynamic scene and use an embeddeddifferentiable simulator to estimate the parameters.
After that, we further edit some of the physicalparameters to generate novel video frames.
More visual results can be found in the supplementaryvideo.
Figure 5 shows the initial frame of the Basketball sequence, where a synthetic basketball ballis released from the air and collides with the ground.
We utilize DiffPD [11] as our differentiableengine.
The ball is modeled by co-rotated material.
We estimate Young’s modulus and verticalacceleration respectively in two separate optimization tasks.
The physics loss is minimized for 100epochs (as described in Section 3.2) using Adam [26] optimizer with a learning rate of 0.01.
Theoptimization starts with Young’s modulus of 2 × 10 5 Pa and acceleration of 0 m/s 2 .
The estimated
Young’s modulus is 2 .
96 × 10 6 Pa and acceleration is 3 .
78 m/s 2 .
Based on this estimation, we canmodify some of the parameters in this virtual world and generate new videos.
For example, theoriginal basketball does not have horizontal speed, so we add 0 .
3 m/s left and right initial velocitiesin Figure 5 (b, c), respectively.
We also decrease the Young’s modulus in Figure 5 (d, e, f) from2 × 10 7 Pa to 2 × 10 5 Pa , where the deformation is getting more significant.
4.4 Editing
Besides the simulation, our hybrid representation provides a flexible interface for editing and rendering.
Users can directly modify the geometry on the 3D hexahedra mesh and render along a new
[Figure]
camera path in the neural radiance field.
In this section, we will then demonstrate how to interactwith a learned scene.
Region of Interest.
Similar to most graphic design software, the first thing before simulation andediting are to select the region of interest.
Our method naturally provides the rigidity mask and
SDF values, so a foreground area
A i at the i -th frame can be computed as Equation 2.
But, if usersrequire a more precise customized area, our method can take a bounding box or even more complexhexahedra bounding volume A ′ as input.
The final region of interest is then A ′ ∩
A i , from which ahexahedra mesh m can be reconstructed.
Delete.
Figure 6 (a, b) are two different frames from input videos.
The first video films a deforminghand in front of a wall and the second video is a bouncing ball in a room.
Our first goal is to deletethe foreground from view 1.
To delete an area, we can simply set the SDF value to a positive number(e.g.
1.0) for the query points inside the hexahedra mesh m .
Due to we use a bell-shaped densitydistribution centered at 0 [58], a high SDF area would be less likely to be sampled and therefore betransparent.
After removing the foreground, although some parts of the background are occluded bythe hand and the ball, view 2 can provide extra texture/structure information and help to complete thebackground in view 1.
Since all the editing is done in the 3D space, the result in 2D images can bevery sharp and clear as shown in (c).
Appearance.
We can also modify the output of the color network to change the appearance of a certainarea.
For example, in some applications, we would like to highlight the moving foreground of a videoand grey out the background.
For this purpose, we can change the original color [ r, g, b ]
=
c θ c ( p ) tobe
[ r + g + b, r + g + b, r + g + b] for all the query points outside m .
Figure 6 (d) shows the results wherethe background is turned into grayscale.
Duplicate.
Translating and scaling are basic operations in graphic design software.
In our pipeline,users can easily copy, move, and scale the hexahedra mesh m and get a new one m ′ .
During thevolume rendering, the SDF and color values of the points p ′ inside m ′ are set to be the same as itsinverse image p in m .
Figure 6 (d) is the result of duplicating an existing object, with its size andposition changed at the same time.
[Figure] [Figure]
4.5 Ablation Study
[Figure]
In this section, we conduct several ablation studies using the bouncing ball example to justify our choices of network designs and training strategies.
Sequential Vs.
Joint Optimization.
Our method learns the scenesequentially by first optimizing the rendering networks, followed bythe physics parameters.
A seemingly more concise strategy mightbe to jointly train all parameters from scratch.
Unfortunately, thejoint training strategy is impractical for two reasons: (1) the tablein Figure 7 indicates that the physics module converges fast thanksto the effectiveness of the differentiable simulator, but running theforward and backward simulation is costly ( ∼ 70 s) compared toquerying the rendering networks ( ∼ 0 .
22 s) – training the renderingmodule and physics module together would require many more iterations (300,000 epochs), with eachtaking much longer ( ∼ 70 sec per epoch), making the training infeasible, and (2) before the renderingmodule converges, the reconstructed geometry makes little sense for simulation, so the gradientsfrom the L physics are not helpful.
The smoothed plot in Figure 7 indicates that joint training slowsdown the rendering optimization early on.
Sequential optimization is a faster and more reasonablestrategy in our case.
Physics Module.
After choosing the sequential optimization strategy, there are still multiple optionsfor physical parameter estimation.
In our pipeline, we reconstruct the volume mesh m 1 of thefirst frame, then predict the following mesh sequence m ′
i = sim ( θ physics , m 1 , 1 → i ) by thedifferentiable simulator.
The physics loss for each frame can be computed in a cycle-consistencymanner as in Equation 4.
Note that our proposed cycle-consistency loss naturally guarantees thecorrespondence of each vertex and does not require mesh reconstruction for every time step.
Foreach epoch, we only reconstruct the volume mesh for the first frame, which reduces computation and
[Figure]
makes the loss differentiable w.r.t.
the rendering networks.
A more direct physics loss would be reconstructing the geometry of each frame and computing the
Chamfer distance between the simulated and reconstructed meshes.
We compare the two strategiesin Figure 8.
‘Ours’ is optimized with our proposed cycle-consistency loss L physics and ‘Chamfer’is optimized with the Chamfer distance.
(c) shows the value of acceleration , the optimizationvariable.
(a, b) indicate that these two losses have comparable performance in both metrics.
However,‘Chamfer’ takes much more time due to reconstructing every frame.
Therefore, using our proposedcycle-consistency loss runs faster and it maintains differentiability between the rendering and physicssimulation module.
Conceptually, decoupling the NeRF and differentiable physics has anotherdrawback: a mesh reconstructed purely from SDF cannot capture the interior stress and deformation .
Our proposed cycle-consistency loss addresses this concern, as it takes advantage of the learnedmotion field, so it is capable of finding the point-wise correspondence and describing the deformation.
Rigidity Map.
Finding the correct dynamic foreground object is a prerequisite for physical simulation.
The raw output of the SDF field can only reconstruct a global scene, so the rigidity map is used tosegment the foreground.
The rigidity map collects the motion information from all frames and assignsa high rigidity mask value to dynamic regions.
Figure 9 (c, d) visualizes the rough segmentationresults resulting from differend rigidity thresholds.
We therefore set the default threshold to 0.2to separate the dynamic foreground from the background.
If the rigidity map does not provide aclean separation, users may manually specify a bounding box covering the moving object sweptvolume, and get the final foreground as shown in (e, f).
An alternative segmentation approach isto use the magnitude of the motion field as a criterion to separate dynamic foreground, i.e. largemotion area corresponds to dynamic objects (and the converse).
However, in some frames, dynamicparts might only contain small offsets from the canonical frame.
For example, a cyclic bouncing ballcould overlap with the canonical position and thus have offset values of zero, even though it shouldbe classified as a dynamic area.
Figure 9 (a, b) shows results of filtering the scene using motionmagnitude.
It is not a good strategy, since a large portion of the background remains even as the ballis partially clipped off in (b).
Conclusion
[Figure]
In this paper, we design an end-to-end pipelinethat can simultaneously estimate and edit geometry, appearance, and physics using only onesingle video.
However, there are some limitations.
For example, similar to most of the NeRFlike methods, it does not run in real time yet.
Moreover, the reconstruction quality can stillbe improved, especially when the camera posesare somewhat constrained.
We could probablyincorporate more semantic information to improve the results.
Possible future directions include: (1) to further regularize the training and betterperceive the 3D structure, we can utilize some pretrained networks [9] as supervision.
For example,optical flow / scene flow networks can be used for motion detection, while depth estimation can beachieved with the SDF network.
(2) It might be possible to estimate the texture [61] and UV map,helping artists construct 3D models more easily with only a smartphone.
The current trainingand inference is still slow, we can possibly migrate to a much faster framework like [37].
(4) We cancouple this framework with more diverse sets of differentiable simulators to support a wider range ofobjects and scenes, e.g. cloth, tetrahedra mesh [46], or even robots [19].
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to 547 answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You arestrongly encouraged to include a justification to your answer , either by referencing the appropriate section of 549 your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the Checklist 554 section does not count towards the page limit.
In your paper, please delete this instructions block and only keepthe Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?
[Yes] See Section 4.
(b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Please refer to the supplementarycode and Section 3.3.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments [Table] multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,
internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] MIT License.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/cu-
rating?
[Yes] Used with consent.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
informa-
tion or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)
approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on
participant compensation?
[N/A]
[Figure]
The rigidity, color, and SDF are defined on a canonical frame, while the motion of each spatial point is time-dependent.
After fusing the two streams, volume rendering is used to reconstruct images and compute the color loss.
A differentiable simulator is embedded after the neural fields to learn the dynamics parameters and edit the scene.
Appendix A Overview
Figure 10 shows the workflow of our method.
Appendix B Weight Function
We follow the NeuS [58] paper to set the weight function w ( t ) in Equation 1.
First, a density function ρ ( t ) is 594 defined as:
[Equation]
where Φ h ( x ) =
(1 + e − hx ) − 1 is the Sigmoid function, and w ( t ) can be constructed as:
[Equation]
[Equation] ( t ) , where T ( t ) = exp (cid:18) − (cid:90) t
[Equation]
Appendix C Additional Implementation Details
The Color Network (MLP) consists of 4 dense layers, each with 256 hidden dimensions, while the SDF Network 598 has 8 dense layers of 256 hidden dimensions, the Motion Network has 5 dense layers of 64 hidden dimensions,and the Rigidity Network has 3 dense layers of 32 hidden dimensions.
Similar to NeRF++ [69], we use a separate NeRF network to render far-away points outside of a unit-sphere 601 region of interest.
The Motion and Rigidity networks are not applied to those regions since we assume thebackground to be static.
If we do not disable these networks on the background, we observe significant jittering 603 on those regions.
We have also tried enabling and disabling viewing direction as an input to the Color Network.
It turns out that 605 disabling viewing direction can result in smoother geometry, as it prevents over-fitting to the viewing direction asdiscussed in NeRF++.
However, we also observed that disabling the viewing direction led to incorrect geometries 607 (like holes) in some cases.
Appendix D Potential Applications
NeuPhysics enables high-resolution mesh reconstruction from a casual video captured by a smartphone.
It would 610 be easy for users to collect their own data to customize virtual body avatars for games and social applications.
Accurate facial and body models can also be used in virtual try-on, or even customized suits, furniture, and 612 sports equipment.
Moreover, the physically-based, dynamic, photorealistic editing technique could be used indesigning, content creation, and broader VR/AR applications.
[Figure]
Appendix E Other Supplementary Materials
More dynamic visualization results can be found in our supplementary video; we invite the readers to view these 616 videos for visual comparison.
A preliminary version of our code is included for reviewing only.
We will alsoopen-source a release version in the future for research reproduction with the paper publication.
Appendix F Video Reconstruction
For the video reconstruction task, we evaluate the methods on a dataset consisting of both real-world captured 620 and synthetic videos.
The synthetic data are animated by Blender, and the real-world videos are taken by asmartphone in both indoor and outdoor scenes.
Figure 11 visualizes the datasets as well as the reconstruction 622 videos by different methods.
(a, b) are two frames from the input videos where both the scene and the camerapositions have changed.
(c) is by our method, while (d) NRNeRF [54] (e) D-NeRF [44], and (f) NeuS [58] are 624 SOTA methods for comparison.
Note that (d) and (e) are designed for dynamic videos as input, but not designedfor geometry reconstruction.
(f) learns SDF field during the video reconstruction as ours does, however, it is 626 originally designed for static scenes so inconsistency among views would likely undermine its performance.