NeuPhysics: Editable Neural Geometry and Physics from Monocular Videos
Anonymous Author(s)
AffiliationAddress email
Abstract
We present a method for learning geometry and physics parameters of a dynamicscene requiring only a monocular RGB video.
Our approach uses a hybrid representation of neural fields and hexahedra mesh, enabling objects in the scene to beinteractively edited, and synthesized from novel views.
To decouple the learning ofunderlying scene geometry from dynamic motion, we learn a time-invariant signeddistance function which serves as a reference frame, as well as an associated deformation field that is conditioned on each time step.
We design a two-way conversionbetween the neural field and corresponding mesh representation, which allows usto bridge the neural representation with a differentiable physics simulator, andtherefore estimate physics parameters from the source video, by minimizing a cycleconsistency loss.
This flexible, hybrid representation also allows a user to easilyedit 3D objects from the source video by directly editing the recovered hexahedramesh, and propagating this operation back to the neural field.
In Experiments,our method achieves higher-quality mesh and video reconstruction of dynamicscenes compared to other competitive Neural Field methods.
Finally, we provideextensive examples which demonstrate our method’s ability to extract useful 3Drepresentations of dynamic scenes from videos captured with consumer-gradecameras.
Introduction
Monocular RGB videos are easily accessible nowadays as almost everyone has a smartphone.
Can youimagine one day you might be able to build a digital twin of your surrounding world and interact withit with merely a smartphone?
You could reconstruct a 3D portrait for yourself (Sec. 4.1); playback anexisting recording from a new angle (Sec. 4.2); edit your favorite cartoon video (Sec. 5.2); or estimatethe material parameters and interact with it virtually (Sec. 5.1).
In this paper, we are designing asystem that could possibly make all those things come true.
Taking a single RGB video as input, ourmethod can estimate the geometry and dynamics parameters of the scene which is parameterized byinfinite-resolution neural fields [38] and is capable of editing and physically-based simulation.
Although a human brain can easily infer the 3D shapes and physics properties of the environment from even a single eye, parsing a scene from monocular video is still very challenging today for computer algorithms.
The first challenge is the complex geometry of a scene.
Explicit representations like surface mesh are widely used for modeling scenes [3, 39], however, they suffer from limitedresolution and varying topology in a dynamic environment.
Implicit representations like Signed
Distance Function (SDF) describe the complex structure in a more elegant way but are hard to beused for reasoning about dynamics properties. Geometry reconstruction from a single view is an intrinsically underconstrained problem. There is usually more than one 3D structure that corresponds to a 2D layout. To this end, dynamics parameter estimation is even harder because it depends upon correct geometry reconstruction and requires knowledge of temporal information.
Differentiable surface-based rendering [35, 29] enables detailed geometry and appearance reconstruction by backpropagating the gradients from the image space to the 3D model.
However, thediscontinuity of the mesh representation leads to severe local minima and makes such an algorithmsensitive to the initial shape.
Recent coordinate-based Multi-Layer Perceptron (MLP) representationaccompanied by differentiable volume rendering (like NeRF [38]) has also achieved photorealisticimage reconstruction and novel view synthesis while getting rid of the heavy reliance on the initialization.
The volume rendering provides gradients for the entire space, and the MLP field acts as a 3Danalogue to vector graphics, in that it is a compact representation with infinite resolution.
NeuS [61]and some other concurrent work [44, 69] estimate SDF fields during image reconstruction, fromwhich the surface mesh can then be extracted.
However, these works assume a static scene and failto generalize to dynamic scenes (shown in 4.1).
NeRF has also been extended to videos [46, 32] byincorporating time-dependent terms, but prior works mainly focus on video reconstruction and not3D geometry estimation.
To the best of our knowledge, none of these works can estimate the physicsproperties in the video.
Instead, differentiable physics [37, 27] has been coupled with gradient-basedoptimization to perform physics parameter estimation, but these works also require an informedinitialization and a pre-built 3D model.
None of these approaches can learn geometry and dynamicsfrom scratch.
In this paper, we design an end-to-end differentiable rendering and simulation pipeline that candirectly learn the geometry and dynamics from a monocular video.
By assuming that there exists aone-to-one dense mapping in the 3D space between two temporal frames, our method decouples thistask into two orthogonal parts: a static reference frame, and a mapping function that encodes motion.
Within this pipeline, the static reference frame consists of three neural fields – appearance, rigidity,and an SDF for geometry.
Given a 3D spatial coordinate as input, these fields output its RGB color,rigidity mask, and SDF value respectively, in the reference frame.
Time-variant motion is encodedin a separate field, which takes as input a 3D spatial coordinate sampled along a ray, plus a learnedlatent code associated with a particular time step, and outputs the displacement from the spatialcoordinate back to its corresponding coordinate in the reference frame.
We utilize the density functionfrom NeuS [61] for rendering, and the Rigidity mask from NonrigidNeRF [57] to regularize themotion and get foreground/background separation.
An advantage of factorizing a dynamic scene intostatic geometry + dynamic motion is that by querying the motion network, we can easily get pointwise correspondence across frames, from which we can infer the dynamics properties of the scene.
Specifically, physics parameters will be optimized by minimizing the difference between the resultsfrom the motion network and the differentiable simulator.
After learning reasonable parameters, thedifferentiable simulator can in turn be used to animate and edit the scene.
Experiments show that ourmethod outperforms comparison methods in surface reconstruction, video reconstruction, and novelview synthesis tasks, both qualitatively and quantitatively.
Multiple examples also demonstrate ourmethod’s scene understanding and user interaction capabilities.
In summary, the key contributions of this work are as follows:
• We present the first end-to-end differentiable framework capable of simultaneously estimating and editing the geometry, appearance, and physics parameters from a monocular video.
• By decoupling the learning of static geometry and dynamic motion, our method better
regularizes this under-constrained problem and allows us to infer the physics from point- wise correspondence across frames.
• By connecting the NeRF-like geometry reconstruction pipeline to the differentiable physicsengine, our approach avoids manual construction of 3D models and poor initialization for
the differentiable simulator, enabling the network to learn the geometry and dynamics from
scratch.• We design a conversion between the explicit hexahedra mesh and implicit neural fields in our end-to-end simulation and rendering pipeline, within which users can interact (add,
delete, move, deform, simulate, etc.) with the 3D scene and obtain high-quality rendering from novel camera views.
Related Work
2.1 Neural Implicit Field and Geometry Reconstruction
Differentiable volume rendering with implicit neural representation have recently been used tosynthesize novel views, estimate geometry and appearance.
Neural Radiance Field (NeRF) [38], andits follow-on works [8, 53, 2, 67] have achieved photorealistic rendering results even on complexscenes.
The expressive coordinate-based multi-layer perceptron (MLP) representation equips thescene with high resolution [73] while remains regularized [72] and memory-efficient.
Comparedwith differentiable surface rendering [42, 43], volume rendering could backpropagate gradients tothe entire space so to better handle complex geometry optimization with topological changes andmitigate local minima.
The original density field from NeRF cannot directly reconstruct high-quality surface mesh, so SDFbased neural fields [61, 69, 44] are proposed for multi-view geometry reconstruction.
Compared toother discrete representation like mesh [60, 24, 35], point cloud [13, 34] and voxels [5, 65], SDF-fieldhas higher resolution.
And the the NeRF-based method can take raw RGB images without pixel-lebelsegmentation [41, 68, 25].
However, current SDF-based neural fields cannot handle dynamic scenes.
We will show in Figure 4 that existing methods do not converge well on dynamic input while ourmethod can handle these cases.
Recent work has extended NeRF to videos of dynamic scenes [46, 1, 71], but they mainly focus onthe 2D image synthesis, and few of them can reconstruct the 3D geometry well.
They either learna canonical field with time-dependent motion [45, 47, 57, 4] or condition all the fields on the timeaxis [12, 15, 63].
We choose the former strategy because decoupling motion from the geometry makesthe formulation more constrained and it also provides us with point-wise correspondence.
All thementioned dynamic extension of NeRF focus on the 2D video reconstruction and lack regularizationon the 3D geometry.
Our method can reconstruct the geometry much better than the existing works.
Moreover, since single-video 3D reconstruction is highly ill-posed, our geometry prior can also helpimproving the image reconstruction quality, as shown in Section 4.1.
2.2 Differentiable Physics for Dynamics Estimation
Differentiable simulators enable physics prior to better fit in data-driven approaches and therefore have been applied to more and more robotics and graphics tasks [54, 10, 20, 30, 55, 66].
Those physically-based simulation methods have been developed for dynamical systems of differentproperties, including rigid bodies [6, 48, 7, 14], soft bodies [23, 22, 49, 16, 11, 28], articulatedbodies [18, 62, 50], cloth [31, 33], and fluids [58, 59, 21, 56].
Physics parameter estimation is an important application for differentiable simulators.
Gradsim [27]connects a differentiable surface renderer to a differentiable simulator and directly learn parametersfrom videos.
Song and Boularias [54] learns frictional coefficients to help robot manipulation.
Material parameters are estimated in simulation by [19] and further be used to help robot cutting. RISP [37] designs a rendering-invariant loss using the gradients of differentiable renderer for crossdomain parameter estimation. The aforementioned differentiable physics methods rely on pre-builtobject geometry models, while our method is fully automated and can learn to construct the geometrymodel from scratch without any prior.  Methods
Given a monocular video consisting of N RGB image frames { I i } N i =1 , our approach will estimatethe 3D geometry within the camera viewing frustum, foreground/background segmentation, andpointwise correspondence (across time) of the dynamic 3D scene.
Moreover, combined with thedifferentiable physics module, NeuPhysics can learn physics parameters from the dynamic motionsequence.
The learned physics model can be further used to do video editing and forecasting.
Figure 1shows the workflow of our method.
[Figure]
3.1 Rendering Pipeline
SDF and color networks.
Similar to recent Neural Fields methods [38], we parameterize thegeometry and appearance of a scene using coordinated-based Multi Layer Perceptron (MLP), whichcan further be used to render 2D images using the volume rendering technique proposed in NeuS [61].
The geometry of the entire scene is represented by an SDF network s θ s ( p ) : R 3 → R that maps aspatial coordinate p ∈ R 3 to its signed distance to the nearest surface.
For the appearance, we have acolor network c θ c ( p ) : R 3 → R 3 that describes the RGB value of each spatial point.
When renderingthis scene, the points in the ray emitted from a pixel can be written as { p ( t ) =
o + t v | t ≥ 0 } , whereo is the camera center, and v is the unit direction of the ray.
The color of this pixel is then calculatedby integration,
[Equation]
where w ( · ) [61] is a weight function describing the contribution of each point’s color based on its
SDF value.
More details can be found in the Appendix.
Motion network.
However, the above framework can only handle multi-view images of a staticscene. We need to have another design to accommodate the temporal information.
Instead of addingan extra dimension to directly condition the existing networks upon time, we choose to decouple thedynamic and static information.
Therefore, our method has the SDF and color network defined on acanonical frame while estimating the offset field [57] of each frame.
For the i -th frame I i encoded bya latent feature l i ∈ R d l , the motion network b θ b ( l i , p ) :
R d l × R 3 → R 3 is actually bending eachquery ray { p ( t ) } to { p ′ ( t )
= p ( t ) + b ( l i , p ( t )) } .
And we can compute SDF and color values ofp ′ ( t ) on the reference frame.
Rigidity network.
Furthermore, since we want to separate and simulate the dynamic objects withinthe scene in a later state, we thus apply a Rigidity network [57] r ( p ) θ r : R 3 → R to distinguishdynamic foreground and static background.
The mask r ( p ) ∈ [0 , 1] defined on the canonicalframe indicates the likelihood that a point belongs to the swept volume of moving elements in thescene, with a value of 0 representing "completely static".
The raw offset value is then revised to beb ( l i , p ( t ))
=
r ( p ) · b ′ ( l i , p ( t )) .
During training, this formulation will encourage dynamic parts tohave larger r ( · ) values and drive the static areas toward zero.
3.2 Simulation Pipeline
Geometry reconstruction.
Given the signed distance field s ( · ) along with the rigidity mask r ( · ) , wecan reconstruct the geometry of the dynamic foreground and the static background.
For example, theforeground in the i -th frame is an occupied space within the swept volume of moving objects.
Thisarea can be described by
[Equation]
where k sdf and k rigidity are constant thresholds for the SDF and rigidity values, respectively.
Withthis formulation, one can convert it into some type of explicit representation for the objects.
Thesurface mesh can be obtained using a marching cubes algorithm
Alternatively, we can simplysample over the grid to find the vertices within area A i to reconstruct the tetrahedral or hexahedralmesh m i .
We choose a hexahedral volume mesh for its simplicity and regular shape, which can alsobe further simulated by DiffPD [11], a differentiable FEM simulator for hexahedral mesh.
Trianglesurface mesh and tetrahedral volume meshes are also possible for future work.
Figure 2 visualize theconversion between the implicit and explicit representation.
Parameter Estimation From the input video of a dynamic scene, we can extract a mesh sequence{ m i } Ni =1 with consecutive motion.
A mesh model is described by its n v vertices and n e hexahedraelements { V i ∈ R n v × 3 , E ∈ N n e × 8 } .
By simulating the meshes in a differentiable physics enginesim ( θ physics , · ) and comparing with the reconstruction results as a supervision signal, we canoptimize the physics parameters θ physics using gradient-based methods.
To illustrate, we run 1simulation step from the i -th frame and get the new model m
′ i
{++
+}
= sim ( θ physics , m i ) .
A direct idea is to design an error function (like Chamfer Distance) distance θ physics ( m i +1 , m ′ i +1 )between the extracted mesh
m i +1 and the simulated mesh m ′
i +1 and search for θ physics to minimizethe error.
However, this strategy requires us to do sampling and mesh reconstruction every timestep, which is expensive.
Additionally, the reconstructed mesh might vary significantly across timesteps, thus cannot guarantee point-wise correspondence, making the dynamics discontinuous andnon-differentiable.
To save the sampling time and make the gradient well-defined, we design a cycle-consistency loss tocompute the distance between the simulation results and neural fields.
Starting from mesh m i sampledat the i -th frame, we run our differentiable simulator on it for j steps and get m ′
i + j . m
′ i
+ j shouldcorrespond to the same shape as m i in the reference frame.
Note that as m ′ i + j is simulated from m i ,they have an exat pointwise correspondence (same topology), so we can directly subtract their pointclouds to obtain their distance: L physics =
(cid:13)(cid:13)
( m i + b ( l i , m i )) − ( m ′ i + j + b ( l i + j , m ′ i + j ))
(cid:13)(cid:13) 2 .
Evenin a long sequence, we can sample m i only once and use it as a template to compare with all otherframes.
[Figure]
Scene Editing The simulator in ourpipeline can serve as a strong physicsprior to forecast the motion and editthe scene.
The right part of Figure 2shows how we edit the SDF and colorfield by manipulating the hexahedralmesh.
Based on a hexahedral meshm i extracted from the i -th frame, wecan simulate it to get a modified onem ′
i +1 .
To render the modified scene,we will add one more ‘bending’ layerb edit ( m ′ i , · ) over the top of the offsetfield b ( l i , · ) .
Given a query point p ′ in the editedscene, the first thing is to checkwhether it is inside the hexahedralmesh m ′
i .
This step would be easy ifthe all the hexahedra in m ′
i are kept asregular grids, however, the hexahedraafter simulation could be deformed.
Therefore, we split each hexahedroninto five tetrahedra.
The query point p ′ is inside hexahedron if and only if it is inside one of the tetrahedra.
With this principle in mind, our next step is running a KNN algorithm V ′ p ′ =
KNN ( m ′ i , p ′ )to locate the nearest vertices of the mesh to the query point p and select the tetrahedra that V ′ p
belongto as candidate tetrahedra.
Similar to [70], we decide whether p ′ is inside a tetrahedron according toits barycentric coordinates.
For a tetrahedron consisting of four vertices { p ′ i =
( x ′ i , y ′
i , z ′ i ) ⊤ }
i
=1 , the [Figure] barycentric coordinates { c i } 4 i =1 of p ′
=
( x ′ , y ′ , z ′ )
⊤ can be computed by c i =
Det i /Det 0 , where
Det 0 = (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) x ′y ′z ′ x ′y ′z ′ x ′y ′z ′ x ′y ′z ′ (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) , Det 1 = (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) x ′ y ′
z ′ x ′y ′z ′ x ′y ′z ′ x ′y ′z ′ (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) , Det 2 = (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) x ′y ′z ′ x ′ y ′
z ′ x ′y ′z ′ x ′y ′z ′ (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
and Det i is obtained by substituting the i -th row of Det 0 with ( x ′ , y ′ , z ′ , 1) .
p ′ is inside or on thesurface of this tetrahedron if and only if 0 ≤ c i ≤ 1 holds for i
= 1 , 2 , 3 , 4 .
Once p ′ belongs to a tetrahedron, it means that this point has been edited along with the mesh.
Assume the modification can be linearly interpolated inside a tetrahedron, the original coordinates pin the i -th frame is then the weighted sum p = (cid:80) 4 i =
1 c i p i , where p i is the original coordinate of thevertex p ′ i before editing.
In addition, we need to consider the area that is once occupied by the m ibut now empty after editing. We can simply assign those points a big SDF value such that they areeffectively transparent during volume rendering and will not influence the 3D reconstruction either.
3.3 Training and Losses
To train our network, the main supervision comes from the video, that we would like to minimizethe distance between rendered and the ground truth images L color =
(cid:13)(cid:13)(cid:13)
I i − ˆ
I i (cid:13)(cid:13)(cid:13) 1 .
Besides that, wewill include extra terms to regularize the training.
First, one important property of SDF is its unitnorm.
Eikonal term [17] is therefore added as L Eikonal =
(cid:80) i,k ( ∥ p i,k ∥ 2 − 1) 2 .
Moreover, we wantthe motion in our scene to be more constrained (i.e. locally rigid, incompressible, and more staticregion), so we add offset loss L offset = (cid:80) i,k ∥ b ′ ( l i , p i,k ) ∥ 2 + r ( p i,k ) and divergence loss [57]
L divergence =
(cid:80)
i,k ∥ div ( b ( l i , p i,k ))
∥ 2 .
Finally, we will use the physics loss L physics when we trainfor physics parameters.
Therefore, the total loss function is
[Equation]
By default, we will set w Eik = 0 .
1 , w off = 50 . 0 , w [Equation] = 2 .
and it might vary depending onthe scenes.
L physics is usually trained independently after the networks converge because ourdifferentiable simulation is much more expensive than the query of MLP.
It would be an interestingfuture work to try a faster engine and have the simulation engine as a physics prior. A full trainingcycle typically takes around 18 hours on an NVIDIA A5000 GPU. For the training data (used with
Method
Sitting
Standing
Basketball
Hand
LPIPS ↓
SSIM ↑
PSNR ↑
LPIPS ↓
SSIM ↑
PSNR ↑
LPIPS ↓
SSIM ↑
PSNR ↑
LPIPS ↓
SSIM ↑
PSNR ↑
D-NeRF 0.
34.146 NeuS 0.
34.485 NRNeRF 0.
33.999 Ours 0.
[Figure]
consent), we note that the background needs to have enough details.
The reason is that the dynamicforeground would make the camera registration harder, and the Structure from Motion implementationby COLMAP [51, 52] could possibly fail if the background is too empty.
Results: Scene Reconstruction
4.1 Dynamic Video Reconstruction
For the video reconstruction task, we evaluate the methods on a dataset consisting of both real-worldcaptured and synthetic videos.
The synthetic data are animated by Blender, and the real-world videosare taken by a smartphone in both indoor and outdoor scenes.
Figure 3 visualizes the datasets as wellas the reconstruction videos by different methods. (a, b) are two frames from the input videos whereboth the scene and the camera positions have changed. (c) is by our method, while (d) NRNeRF [57](e) D-NeRF [47], and (f) NeuS [61] are SOTA methods for comparison. Note that (d) and (e) aredesigned for dynamic videos as input, but not designed for geometry reconstruction.
(f) learns SDFfield during the video reconstruction as ours does, however, it is originally designed for static scenesso inconsistency among views would likely undermine its performance.
Table 1 shows the quantitative results.
Three standard error metrics are reported here: structuralsimilarity index measure (SSIM), peak signal-to-noise ratio (PSNR), and perceptual similarity through LPIPS [74]. Our method gets the best scores in 9/12 entries. One thing we notice that is our methodperforms a lot better than other methods in the scene where the motion is more significant (e.g. thebasketball).
Moreover, in the Standing sequence, all the three comparison methods fail to reconstructthe waving arms, which are smaller and move faster compared to the torso. We also visualize somemesh reconstruction quality in Figure 4. Basically, NeuS is designed for static scenes so ambiguityon the moving parts would lead to incorrect geometry. And similar to other NeRF-like methods, thereconstruction of D-NeRF can be noisy because of the uncertainty in random sampling.
4.2 Novel View Synthesis
In this section, we simulate four cases of deformable objects moving in a room.
For each scene, werender two videos using different camera paths so that we can train on one path and use the otherone to test novel view synthesis.
Quantitative results in Table 2 show that our method achieves thebest synthesis quality in all the metrics.
We show the datasets and results in Figure 5.
(a, b) are twoframes from the training videos.
(c) is taken at the same time step as (b), but it is a novel view fromthe test path.
We can see that our method successfully synthesizes the dynamics scene from a novelview, while other methods cannot reconstruct the geometry and texture properly.
Method
Tennis Ball
Basketball
Blue Ball
Blue Cube
LPIPS ↓ SSIM ↑ PSNR ↑ LPIPS ↓
SSIM ↑
PSNR ↑
LPIPS ↓
SSIM ↑ PSNR ↑
LPIPS ↓
SSIM ↑ PSNR ↑
NRNeRF 0.
29.044 D-NeRF 0. 27.855 Ours 0.
[Figure]
Results: Physically-based Understanding and Editing
5.1 Parameter Estimation
[Figure]
differentiable
NeuPhysics can estimate the acceleration and Young’s Modulus, generate new videos with different initial velocities and material parameters.
In this section, we first use our network to reconstruct the dynamic scene and use an embedded differentiable simulator to estimate the parameters.
After that, we further edit some ofthe physical parameters to generate novel videoframes.
More visual results can be found inthe supplementary video.
Figure 6 shows theinitial frame of the Basketball sequence, wherea synthetic basketball ball is released from theair and collides with the ground.
We utilize DiffPD [11] as our differentiable engine.
Wechoose to use coratated material to model thebouncing ball and set the gravity and materialparameters as trainable parameters.
We minimize the physics loss (as described in Section 3.2) using Adam [26] and get that the acceleration is 3 .
78 m/s 2 and the Young’s modulus is2 . 96 × 10 6 Pa .
Based on this estimation, we canmodify some of the parameters in this virtualworld and generate new videos.
For example, the original basketball does not have horizontal speed,so we add 0 .
3 m/s left and right initial velocities in Figure 6 (b, c), respectively.
We also change Young’s modulus to be 2 × 10 5 and use a virtual plane to collide at its right. Figure 6 (d) shows thatthe ball is much softer than before.
5.2 Editing
Besides the simulation, our hybrid representation provides a flexible interface for editing and rendering.
Users can directly modify the geometry on the 3D hexahedra mesh and render along a newcamera path in the neural radiance field.
In this section, we will then demonstrate how to interactwith a learned scene.
[Figure]
Region of Interest.
Similar to most graphic design software, the first thing before editing is toselect the region of interest.
Our method naturally provides the rigidity mask and SDF values, so aforeground area AA i at the i -th frame can be computed as Equation 2.
But, if users require a moreprecise customized area, our method can take a bounding box or even more complex hexahedrabounding volume A ′ as input.
The final region of interest is then A ′ ∩
A i , from which a hexahedramesh m can be reconstructed.
Delete.
Figure 7 (a, b) are two different frames from input videos.
The first video films a deforminghand in front of a wall and the second video is a bouncing ball in a room.
Our first goal is to deletethe foreground from view 1.
To delete an area, we can simply set the SDF value to a positive number(e.g.
1.0) for the query points inside the hexahedra mesh m .
Due to we use a bell-shaped densitydistribution centered at 0 [61], a high SDF area would be less likely to be sampled and therefore betransparent.
After removing the foreground, although some parts of the background are occluded bythe hand and the ball, view 2 can provide extra texture/structure information and help to complete thebackground in view 1.
Since all the editing is done in the 3D space, the result in 2D images can bevery sharp and clear as shown in (c).
Appearance.
We can also modify the output of the color network to change the appearance of a certainarea.
For example, in some applications, we would like to highlight the moving foreground of a videoand grey out the background.
For this purpose, we can change the original color [ r, g, b ]
=
c θ c ( p ) tobe
[ r + g + b, r + g + b, r + g + b] for all the query points outside m .
Figure 7 (d) shows the results wherethe background is turned into grayscale.
Duplicate.
Translating and scaling are basic operations in graphic design software.
In our pipeline,users can easily copy, move, and scale the hexahedra mesh m and get a new one m ′ .
During thevolume rendering, the SDF and color values of the points p ′ inside m ′ are set to be the same as itsinverse image p in m .
Figure 7 (d) is the result of duplicating an existing object, with its size andposition changed at the same time.
Conclusion
In this paper, we design an end-to-end pipeline that can simultaneously estimate and edit geometry,appearance, and physics using only one single video.
However, there are some limitations.
Forexample, similar to most of the NeRF-like methods, it does not run in real time yet.
Moreover, thereconstruction quality can still be improved, especially when the camera angles are limited.
We couldprobably incorporate more semantic information.
Some possible future directions include: (1) tofurther regularize the training and better perceive the 3D structure, we can utilize some pretrainednetworks [9] as supervision.
For example, optical flow / scene flow networks can be used for motiondetection and depth estimation can be used with the SDF network.
(2) It might be possible to estimatethe texture [64] and UV map, helping artists build 3D model easily with only a smartphone.
The current training and inference is still slow, we can possibly migrate to a much faster frameworklike [40].
(4) We can couple this framework with more differentiable simulators to support a widerrange of objects and scenes, e.g. cloth, tetrahedra mesh [49], or even robots [19].
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to 535 answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You arestrongly encouraged to include a justification to your answer , either by referencing the appropriate section of 537 your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the Checklist 542 section does not count towards the page limit.
In your paper, please delete this instructions block and only keepthe Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu-
tions and scope?
[Yes] See Section 4.
(b) Did you describe the limitations of your work?
[Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimentalresults (either in the supplemental material or as a URL)?
[Yes] Please refer to the supplementarycode and Section 3.3.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experimentsmultiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,
internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] MIT License.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/cu- [Table]
rating?
[Yes] Used with consent.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
informa-
tion or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
[Table] b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)
approvals, if applicable?
[N/A]
[Table]
c) Did you include the estimated hourly wage paid to participants and the total amount spent on
[Table]
participant compensation?
[N/A]