Paper: Neural SDF Flow for 3D Reconstruction of Dynamic Scenes (ICLR-2024)

Reviewer comment (Writing):
The paper should more clearly state that tangential motion causes issues when converting a time-dependent SDF into scene flow.

Author response:
- **Tangential Motion**

Thank you for the suggestion. We have updated the exception analysis in Section A2 of the main script as follow.

Given the SDF flow of 6 points on a surface that moves smoothly and rigidly, one can obtain the scene flow $\begin{bmatrix}\\boldsymbol\\omega\\\\\\mathbf{v}\end{bmatrix}$ of the surface by solving a linear equation
$$
\\mathbf{d} = -\\mathbf{A}\begin{bmatrix}
                \\boldsymbol\\omega\\\\
                \\mathbf{v}
            \end{bmatrix},
$$
where $\\mathbf{d}\\in\\mathbb{R}^6$ is the SDF flow of those 6 points. $\\mathbf{A}\\in\\mathbb{R}^{6\\times6}$ whose rows are computed from the normals and locations of those 6 points. To the best of our knowledge, the only exception when we cannot obtain the correct scene flow is that when the scene motion does not lead to any SDF changes to any 6 points on the surface, thus, $\\mathbf{d}=\\mathbf{0}$. In this case, there are many solutions to the linear equation. For example, a plane moving along its tangent direction or a ball rotating around its centre. For the former, any velocity on the tangent plane could be the solution. For the latter, any angular velocity by the ball centre will work. Note that, the linear relation between scene flow and SDF flow defined in Equation 16 still holds on these exceptions. In fact, as long as the surface and scene motion is smooth, Equation 16 holds.

We would also like to clarify that tangential motion does not necessarily lead to degenerate case when using the above linear equation to solve for scene flow. For example, given a ball translating in the space, there are always points on the ball where the velocity direction is along their tangent planes. However, we can always find another 2 points, together with any of those points, to solve for the velocity of the ball (we use 3 points because there is only translation.).

Reviewer comment (Evaluation):
The results don't show any good instances of topology change. I do not understand this claim. The best example I could find is at 0:45. Are there any other good instances? I'm also lost as to which aspect of the method helps with topology change. Please mark that more clearly.

Author response:
- **Topology Change**

There are a few topology change instances in the supplementary video: at 00:10 when the woman's hands leaving the table, at 00:19 when the two girls shaking hands, at 00:24, when the two hands of the guitarist in the middle are put apart. 

We acknowledge that all works based on SDF, such as Shao et al. (2023), including ours can handle the topology change. For those works that map observation frames to canonical frame such as Cai et al. (2022); Park et al. (2021a), they require to bring the input space into a higher dimension space to handle the topology change as has done in HyperNeRF (Park et al., 2021b). For other works that estimate per-frame densities and use scene flow to enforce the temporal constraints of the scene such as Li et al. (2021); Song et al. (2022), they can also handle the topology change due to the per-frame density estimation.

At last we would like to emphasize that handling topology change is not the focus nor the contribution of our work. Our main contribution is the novel dynamic scene representation: SDF flow which can do whatever SDF-based representations can and we further derive the mathematical relation between SDF flow and scene flow which bridges the scene geometry with scene motion. We believe such relation will benefit the community of dynamic scene reconstruction. To avoid misunderstanding, we have revised paragraphs with topology changes

Reviewer comment (Evaluation):
The paper doesn't show any appearance/novel view results. That's okay since it focuses on the geometry, and appearance is only used as an auxiliary to deal with the RGB input. However, I would still be curious to see video results where the appearance is taken from t=0 and propagated via the SDF flow. That would allow to visualize longer-term correspondence drift and whether tangential motion causes issues in practice. Currently, the SDF flow (and thus the derived scene flow) only cares about short-term "geometrical" correspondences. This is relevant because the paper doesn't just claim that the SDF flow is a helpful parametrization but rather that it enables scene flow, i.e. correspondences. Currently, the qualitative and quantitative scene flow evaluation is only short-term, even though the method requires long-term correspondences (equation 7). The quantitative scene flow evaluation is also only a single sequence.

Author response:
- **Scene Flow Evaluation**

**Render results.** Thank you for the suggestion. We provide the meshes with texture and rendered videos in the updated supplementary material (named *supp\_vid\_rebuttal.mp4* and *textured_meshes/*). The color of a surface point is obtained either from the current time step or from the corresponding point in future time steps (specified by the title of the file. For example, for the mesh file named *geo_1_col_2.ply* means that the geometry is from time step 1 and the color is from time step 3). The correspondences are defined by the scene flow computed from SDF flow.

**Long-term scene flow evaluation.** We now provide the full optical flow evaluation results on the CMU Panoptic dataset and the long-term scene flow evaluation results on the synthetic sequence (up to future 5 frames).  

Since we cannot obtain the ground truth scene flow for real-world sequences of the CMU Panoptic dataset, we try to evaluate the scene flow with optical flow by projecting it onto the image plane. In the table below, we provide such evaluation results on all 5 sequences of the CMU Panoptic dataset. The optical flow computed from our scene flow is consistently better than that of NDR.

| EPE (pixel) | Ian3 | Haggling\_b2 | Band1 | Pizza1 | Cello1 | avg |
|---|:---:|:---:|:---:|:---:|:---:|:---:|
| NDR (Cai et al., 2022) | 6.86 | 4.43 | 1.66 | 3.11 | 2.41 | 3.69 |
| Ours| **3.49** | **2.97** | **1.18** | **1.18** | **1.34** | **2.04** |

In addition, we follow the advice of Reviewer jfGg to directly evaluate the scene flow on the synthetic sequence shown in Figure 7 of the main script. The results are reported in the table below. We compare the results of three models: NDR (Cai et al., 2022), Ours, Ours with scene flow regularization which includes several regularization computed from SDF flow and scene flow (Please refer to the response to Reviewer nuCZ W-3 or Section A6 of the main script for more details). We also evaluate the long-term correspondences up to 5 frames. In particular, for every surface point in a frame, we compute the scene flow to the future first, second, and 5-th frames ("+1", "+2", and "+5"). We report the average end-point-error (EPE) in millimeter.  Our model without any explicitly scene flow regularization consistently outperforms NDR (Cai et al., 2022). With further regularization, the scene flow can be drastically improved. 

| EPE (mm) | +1 | +2 | +5 |
|---|:---:|:---:|:---:|
| NDR (Cai et al., 2022)| 16.8 | 34.8 | 90.2 |
| Ours | 11.7 | 24.7 | 71.4 |
| Ours w/ reg | **5.8** | **13.0** | **51.9** |

Reviewer comment (Experiments):
How does the temporal integration scale with scene length? Presumably linearly? Is that why only 24 frames are used? How many function evaluations does the Runge-Kutta solver need for the last (24th) frame?

Author response:
- **Integral Evaluation**

Yes, the temporal integration scales linearly with scene length. The second-order Runge-Kutta solver requires $23*2=46$ function evaluations for the 24-th frame.

Reviewer comment (Novelty):
The related work section isn't that thorough. For example, the dynamic NeRF papers Fang et al. Fast dynamic radiance fields with time-aware neural voxels and Li et al. Neural 3D Video Synthesis are missing. Li et al., like NSFF, can also handle topology changes. Furthermore, shape-from-template methods aren't mentioned. Scene flow methods (e.g. Song et al. PREF: Predictability Regularized Neural Motion Fields) aren't discussed. Also, since NDR is compared to, mentioning some of the papers in the RGB-D line of work would be good, e.g. DynamicFusion, VolumeDeform, KillingFusion, OcclusionFusion. And TARS (Duggal et al. Topologically-aware deformation fields for single-view 3d reconstruction) seems relevant since it isn't restricted to blend skinning (a typo ("blender") in the submission).

Author response:
- **More Related Works**

Thank you for the suggestion. We have updated the related work section to discuss those papers.

Reviewer comment (Evaluation):
Please discuss limitations.

Author response:
- **Limitations**

Thank you for the suggestion. We add the following discussion about the limitation of our work in the main script.

Similar to all other multi-view reconstruction works, the reconstruction quality highly rely on the number of views. Another limitation of our SDF flow is the time complexity. To solve for the integral (Equation 7) for any query point, the number of function evaluation grows with respect to time. We aim to address those limitations in the future work. For the first limitation, since we now have an explicit relation between the scene geometry and scene motion, we may be able to achieve better geometry with limited number of views by seeking extra regularization from motion such as optical flow. For the second limitation, a potential solution could be that instead of integrating from the initial time step, we can integrate within a fixed time window.

Reviewer comment (Theory):
Since the mathematical derivation matters here, please state what assumptions go into the paragraph between equation 5 and 6, which talks about the continuity and differentiability of a time-dependent SDF. In theory, there is nothing preventing an object from appearing out of nowhere, an SDF does not inherently have any restrictions on its temporal evolution (while it does have restrictions in space, namely the Eikonal equation). In real-world cases, geometry noise (say, due to the sensor or an imperfect reconstruction) pops randomly into existence and vanishes randomly over time, which leads to discontinuities w.r.t. the time parameter. Unless point x is meant to be a Lagrangian particle rather than an Eulerian grid coordinate? Figure 2 looks Eulerian though. Please state the assumptions that go into that paragraph.

Author response:
- **Input Assumptions**

We acknowledge that the SDF itself does not have any restrictions on its temporal evolution. The continuity and differentiability of the SDF as a function of time for a given 3D point come from the assumption that the scene geometry evolves in a continuous way such as cars running on the road, and all human activities. Although such assumption does not hold when an object appears out of nowhere, we would like to argue that such case is scarcely possible in our daily life.

Reviewer comment (Theory):
The argument at the end of Sec. 3.1 is that most dynamic NeRF methods have issues with topology changes. That's not the case for most dynamic NeRF methods that handle general objects since most condition the canonical model in some manner on time (e.g. HyperNeRF). An extreme case of that is Neural Scene Flow Fields. The argument only holds for D-NeRF-style methods like Nerfies or NR-NeRF. For the others, the better argument would be to say that they use density to parametrize geometry rather than SDF and hence the geometry tends to be very noisy and lack a clearly defined surface.

Author response:
- **Wrong Argument**

Thank you for pointing it out. We have updated that paragraph as follow to avoid confusion.

To extend NeRF to dynamic scenes, the most commonly adopted strategy is to jointly optimize an additional function that models the deformation in 3D space such function either maps all observation spaces to a canonical one such as (Park et al., 2021a) or models the temporal motion of the scene such as (Li et al., 2021). In the next section, we propose a drastically different representation, i.e., SDF flow, which can also model dynamic scenes.

Reviewer comment (Presentation):
Figure 2 would benefit from labelling the two ellipses with their respective timesteps (t=0 and t=6?).

Author response:
- **Labelling**

Thank you for pointing it out. We have updated the Figure.

Reviewer comment (Evaluation):
The optimisation takes too long, which limits its practical usage.

Author response:
- **Time consuming**

Thank you for pointing it out. We acknowledge that the time complexity is one of our limitations and will try to improve it in our future research. We also add the following discussion about the limitation in the script.

Similar to all other multi-view reconstruction works, the reconstruction quality highly rely on the number of views. Another limitation of our SDF flow is the time complexity. To solve for the integral (Equation 7) for any query point, the number of function evaluation grows with respect to time. We aim to address those limitations in the future work. For the first limitation, since we now have an explicit relation between the scene geometry and scene motion, we may be able to achieve better geometry with extra regularization from motion such as optical flow. For the second limitation, a potential solution could be that instead of integrating from the initial time step, we can integrate within a fixed time window.

Reviewer comment (Experiments):
The flow evaluation in the paper is very weak. The method only compares to NDR on 2D project error using RAFT as pesudo ground truth. This should not be hard to achieve if using a synthetic dataset, if using the rigid moving toy example being shown in the paper.

Author response:
- **Scene flow evaluation**

Thank you for the suggestion. We now provide the full optical flow evaluation results on the CMU Panoptic dataset and the scene flow evaluation results on the synthetic sequence. 

Since we cannot obtain the ground truth scene flow for real-world sequences of the CMU Panoptic dataset, we try to evaluate the scene flow with optical flow by projecting it onto the image plane. In the table below, we provide such evaluation results on all 5 sequences of the CMU Panoptic dataset. The optical flow computed from our scene flow is consistently better than that of NDR.

| EPE (pixel) | Ian3 | Haggling\_b2 | Band1 | Pizza1 | Cello1 | avg |
|---|:---:|:---:|:---:|:---:|:---:|:---:|
| NDR (Cai et al., 2022) | 6.86 | 4.43 | 1.66 | 3.11 | 2.41 | 3.69 |
| Ours| **3.49** | **2.97** | **1.18** | **1.18** | **1.34** | **2.04** |

In addition, we directly evaluate the scene flow on the synthetic sequence shown in Figure 7 of the main script. The results are reported in the table below. We compare the results of three models: NDR (Cai et al., 2022), Ours, Ours with scene flow regularization includes several regularization computed from SDF flow and scene flow (Please refer to the response to Reviewer nuCZ W-3 or Section A6 of the main script for more details). We also evaluate the long-term correspondences up to 5 frames. In particular, for every surface point in a frame, we compute the scene flow to the future first, second, and 5-th frames ("+1", "+2", and "+5"). We report the average end-point-error (EPE) in millimeter.  Our model without any explicitly scene flow regularization consistently outperforms NDR (Cai et al., 2022). With further regularization, the scene flow can be drastically improved. 

| EPE (mm) | +1 | +2 | +5 |
|---|:---:|:---:|:---:|
| NDR (Cai et al., 2022)| 16.8 | 34.8 | 90.2 |
| Ours | 11.7 | 24.7 | 71.4 |
| Ours w/ reg | **5.8** | **13.0** | **51.9** |

Reviewer comment (Reproducibility):
The flow estimation also lacks details. Solving the least-square optimization requires sample multiple points, which the author say "we select more than 6 points" and sometimes it still be ill-conditioned depending on the property of the sampled points, but I don't see any discussion related to this and how the results are generated.

Author response:
- **Details about the scene flow optimization**

Thank you for the comments, we clarify the details as below and add them to the Section A2 of the main script.

Given a set of points $\\mathcal{P}$ sampled uniformly from a reconstructed mesh, for any point $\\mathbf{x}$ in this set, we first obtain its $K$ nearest neighbours $\\{\\mathbf{y}_1,\\mathbf{y}_2,\cdots,\\mathbf{y}_K\\}$, where $\\mathbf{y}_i\\in\\mathcal{P}$. For every neighbouring points $\\mathbf{y}_i$, we query its surface normal $\\mathbf{n}(\\mathbf{y}_i)$ and SDF flow $d(\\mathbf{y}_i)$ from the MLPs. According to Equation 17, we obtain a matrix $\\mathbf{A}\\in\\mathbb{R}^{K\\times 6}$ where the $i$-th row $\\mathbf{A}_i=[(\\mathbf{y}_i\\times\\mathbf{n}(\\mathbf{y}_i))^T,\\mathbf{n}(\\mathbf{y}_i)^T]$ and the SDF flow vector $\\mathbf{d}\\in\\mathbb{R}^K$ where $\\mathbf{d}_i=d(\\mathbf{y}_i)$. We can then solve such problem as

$$
\begin{aligned}
    \begin{bmatrix}\\boldsymbol\\omega^*\\\\ \\mathbf{v}^*\end{bmatrix} 
    &= \text{arg min}_{\\boldsymbol\\omega, \\mathbf{v}} || \\mathbf{d} + \\mathbf{A}\begin{bmatrix}
                \\boldsymbol\\omega\\\\
                \\mathbf{v}
            \end{bmatrix} ||_2^2 \\\\
            &= - (\\mathbf{A}^T\\mathbf{A})^{-1}\\mathbf{A}^T\\mathbf{d}
\end{aligned}
$$
where $\boldsymbol\omega^*, \mathbf{v}^*$ is the optimal angular velocity and velocity.

In practice, we uniformly sample around $2000$ points from a reconstructed mesh and choose around $200$ neighbouring points to compute the scene flow of one surface point. To further study the influence of the number of neighbouring point, we compare the scene flow error with respect to the number of neighbouring points in the table below. 

| K | 50 | 100 | 200 | 500 | 1000 |
|---|:---:|:---:|:---:|:---:|:---:|
| EPE (mm) | 13.7 | 12.6 | 11.7 | 11.3 | 12.0 |

Reviewer comment (Evaluation):
Though the method is outperforming previous methods by big margin in the qualitative comparisons, the difference in qualitative comparison is less clear to me from all figures in the paper. In particular compared to Tensor4D, I can see each method wins in different level of details being captured. The paper currently summarizes it as "all of its results share similar artifacts which we conjecture are due to the tensor decomposition. Our method performs comparably to the baselines, with fewer artifacts." I am not sure I can capture this from the current presentation. Will love to know a concrete summary of the areas where the model increase performance best.

Author response:
**1. Detailed qualitative comparison to Baselines** 

The artifacts refer to as the check board effects on the surface of the mesh such as the thigh of the first sample and the dress of the second sample. We have updated the main script to highlight those regions.

Reviewer comment (Theory):
I did not see any discussions about the limitations of existing methods or representations. As all current multi-view dynamic reconstruction work, I assume this work will face the same limitations in large motion (which will also break the linear motion relation between scene flow and sdf flow) and number of views. But I'd like to some more technical insights of potential downside using SDF flow in some scenarios compared to other representation.

Author response:
**2. Limitations**

 Thank you for the suggestion. We add the following discussion about the limitation of our work in the main script and clarify about the large motion below. 

Similar to all other multi-view reconstruction works, the reconstruction quality highly rely on the number of views. Another limitation of our SDF flow is the time complexity. To solve for the integral (Equation 7) for any query point, the number of function evaluation grows with respect to time. We aim to address those limitations in the future work. For the first limitation, since we now have an explicit relation between the scene geometry and scene motion, we may be able to achieve better geometry with limited number of views by seeking extra regularization from motion such as optical flow. For the second limitation, a potential solution could be that instead of integrating from the initial time step, we can integrate within a fixed time window.

We would also like to clarify that as long as the surface and the motion are smooth, even large motion will not break the relation between the scene flow and SDF flow. Because the derivation of such relation is the limit as time approaches zero.

Reviewer comment (Evaluation):
Despite the novel idea, the reconstruction quality is far below my expectation. Even though the results from NDR (Neurips 2022) are also not that good enough, there are not that much dramatic change after the authors applied the proposed SDF flow. Of course, I can see the sceneflow visualization in Fig8 of the manuscript, the reconstruction quality is not really good enough. Moreover, we cannot judge whether the quality of the sceneflow is correct or not.

Author response:
- **W-1 Low fidelity results**

The reason for low fidelity results is because of the challenging data. We try to reconstruct the geometry of a dynamic scene with only 10 sparse views. In Section A5, Figure 17, we show all the 10 views at particular time step. The view changes are very large making it extreme hard to reconstruct the geometry not only for NeRF-based methods but also for other multi-view stereo ones. In fact, even COLMAP (Sch ̈onberger & Frahm, 2016) fails on those images due to “no good initial image pair found”.

**Scene flow evaluation.** Since we cannot obtain the ground truth scene flow for real-world sequences of the CMU Panoptic dataset, we try to evaluate the scene flow with optical flow by projecting it onto the image plane. In the table below, we provide such evaluation results on all 5 sequences of the CMU Panoptic dataset. The optical flow computed from our scene flow is consistently better than that of NDR.

| EPE (pixel) | Ian3 | Haggling\_b2 | Band1 | Pizza1 | Cello1 | avg |
|---|:---:|:---:|:---:|:---:|:---:|
| NDR (Cai et al., 2022) | 6.86 | 4.43 | 1.66 | 3.11 | 2.41 | 3.69 |
| Ours| **3.49** | **2.97** | **1.18** | **1.18** | **1.34** | **2.04** |

In addition, we follow the advice of Reviewer jfGg to directly evaluate the scene flow on the synthetic sequence shown in Figure 7 of the main script. The results are reported in the table below. We compare the results of three models: NDR (Cai et al., 2022), Ours, Ours with scene flow regularization which we will introduce in the response to W-3. We also evaluate the long-term correspondences up to 5 frames. In particular, for every surface point in a frame, we compute the scene flow to the future first, second, and 5-th frames ("+1", "+2", and "+5"). We report the average end-point-error (EPE) in millimeter.  Our model without any explicitly scene flow regularization consistently outperforms NDR (Cai et al., 2022). With further regularization, the scene flow can be drastically improved. 

| EPE (mm) | +1 | +2 | +5 |
|---|:---:|:---:|:---:|
| NDR (Cai et al., 2022)| 16.8 | 34.8 | 90.2 |
| Ours | 11.7 | 24.7 | 71.4 |
| Ours w/ reg | **5.8** | **13.0** | **51.9** |

Reviewer comment (Theory):
Can the authors further elaborate the validity of the proposed 2nd assumption? What geometric insight reside within this hypothesis? "[Assumption 2] As the time period Δt approaches zero, the absolute SDF change |Δs| of a surface point x equals the distance from x to the tangent plane to the evolved surface at the corresponding point x′ and the sign of Δs is determined by the angle between that tangent plane’s normal and the scene flow (as shown in Figure 3)."

Author response:
- **W-2. Validity on the assumption 2.**

Let us denote a surface point as $\\mathbf{x}$, its corresponding point on the evolved surface as $\\mathbf{x}'$, the normal of the tangent plane at $\\mathbf{x}'$ as $\\mathbf{n}(\\mathbf{x}')$, and the closest point on the evolved surface to $\\mathbf{x}$ as $\\mathbf{x}''$. According to the definition of SDF, the absolute SDF change of $\\mathbf{x}$ i.e., $|\\Delta s|$ is equal to $||\\mathbf{x}-\\mathbf{x}''||_2$. With the assumption that the infinitely small surface region around $\\mathbf{x}$ as well as its deformation is smooth, as the time period $\\Delta t$ approaches zero, $\\mathbf{x}''$ becomes infinitely close to $\\mathbf{x}'$ which can be regarded as lying on the tangent plane to the evolved surface at $\\mathbf{x}'$. So that the absolute SDF change $|\\Delta s|$ of $\\mathbf{x}$ will equal to the distance from $\\mathbf{x}$ to such tangent plane i.e., $|(\\mathbf{x}'-\\mathbf{x})^T\\mathbf{n}(\\mathbf{x}')|$. The sign of $\\Delta s$ is determined by angle between the scene flow and the normal of the tangent plane.

Reviewer comment (Experiments):
While training the network, does the understanding of sec 3.3 is needed? While this paper proposes an algorithm for bridging the SDF flow and sceneflow, there are not much material or experiments that clearly demonstrate the accuracy of the sceneflow. Moreover, the proposed assumptions are not used when training the methods. Accordingly, it is quite confusing me to understand the precise pipeline of this paper.

Author response:
- **W-3. Necessity of Sec. 3.3.**

We acknowledge that currently the training of our model does not involve the computation of scene flow. However, bridging the gap between SDF flow and scene flow could potentially be applied to train a better model. We evidence this by the toy example shown in Figure 7 of the main script. In particular, we train a new model (named ``Ours w/ reg'') that outputs not only the SDF flow but also the scene flow, we then apply several priors:

1. SDF flow and scene flow consistency prior from Equation 16.

2. Piece-wise rigid prior: we encourage the scene flow of close points to be similar. 

3. Optical flow loss: we project the scene flow onto image plane to obtain the optical flow and compute optical flow loss with it.

More details about this experiment are added to Section A6 of the main script. The evaluation results are show in the table below. The scene flow regularization helps to improve the reconstruction quality. As also evidenced in the response to W-1, the estimated scene flow from SDF flow is also improved. We believe revealing the mathematical relation between the scene motion and scene geometry has more potential applications which we will explore in the future. 

| (mm) | acc | comp | overall |
|---|:---:|:---:|:---:|
| NDR (Cai et al., 2022) | 33.5 | 25.3 | 29.4 |
| Ours | 31.8 | 23.3 | 27.6 |
| Ours w/ reg | **27.2** | **19.1** | **23.2** |