Paper: MovingParts: Motion-based 3D Part Discovery in Dynamic Radiance Field (ICLR-2024)

Reviewer comment (Theory):
I am not convinced that "Eulerian" and "Lagrangian" are the right terms. Standard deformable NeRFs like D-NeRF or Nerfies use backward deformation modeling. The paper proposes forward modeling instead. These terms should at the very least be mentioned explicitly somewhere. Both forward and backward modeling focus on particles and are direct functional inverses of each other. They thus resemble the Lagrangian perspective. As for example mentioned in Tretschk et al. State of the Art in Dense Monocular Non-Rigid 3D Reconstruction, I think it's more accurate to think of methods like NSFF that do not have a proper canonical model and rather directly regress the radiance field at each coordinate point in time and space as Eulerian. Such methods do not have a notion of particle, while both forward and backward deformation methods do. Or, to be more precise, they only use temporal consistency losses with auxiliary scene flow between *neighboring* frames and thus only have a notion of *instantaneous* flow to *temporally neighboring* frames at coordinate points in world space, which is exactly what Eulerian flow is. All of this is of course not ultimately relevant for the point of the paper in any way, only the presentation/writing, and I'm willing to hear why the notions proposed in the paper are more appropriate.

Author response:
**(A) Concept of Eulerian and Lagrangian.**

It is a good suggestion to explicitly mention forward and backward deformation in the paper and we will add it in a revision. We also agree with the reviewer that both forward and backward methods have some notion of particle. In particular, in D-NeRF or Nerfies, with the backward deformation, their volume density and color (the radiance field properties) are defined in the canonical space and can be thought of being models on the object material points/particles. However, please note that we focus on motion modeling (when it comes to Eulerian and Lagrangian views) here, and the backward deformation function itself is defined in the world coordinate space at fixed locations and cannot be considered as Lagrangian flow. In contrast, our forward deformation is defined in the canonical object coordinate space (on material particles) and exactly corresponds to what the standard Lagrangian flow is. To be more specific, our forward deformation can directly be used to describe the trajectories or pathlines of any object particles through the temporal frames, which is a classical lagrangian description; however, this is what the backward deformation in D-NeRF or Nerfies cannot achieve at all.   
Therefore, though the backward deformation is not identical to standard Eulerian flow (as the one the reviewer described in NSFF), we still consider the backward deformation as motions in an Eulerian description since they are defined at fixed world-space locations. We are happy to further clarify this in the paper.

Reviewer comment (Theory):
The submission argues from the intractability of tracking a particle with backward deformation modeling. But for example Cai et al. Neural Surface Reconstruction of Dynamic Scenes with Monocular RGB-D Camera use an invertible deformation model that would allow to long-term track a particle in a straightforward manner. They lack expressivity and are computationally expensive but that's a more involved argument than the one in the submission.

Author response:
**(B) Intractability of tracking particles with backward deformation.**

The reviewer is indeed correct that an invertible backward deformation model can achieve a similar goal because the forward deformation can be derived analytically. However, we think this actually aligns well with our argument that a forward deformation module is needed and an invertible function is an alternative way to achieve so. As pointed out by the reviewer, the invertible function also has other costs (lacking expressivity and computationally expensive) and our approach avoids these by modeling the two deformations separately with a cycle consistency loss. We will add the discussion with the invertible deformation in the paper.

Reviewer comment (Novelty):
Liu et al. also propose cycle consistency of forward and backward modeling for dynamic NeRF in their NeurIPS 2022 paper DeVRF. It does this on the motion and not the feature level though. I think this should be discussed in the paper.

Author response:
**(C) Discussion on cyclic consistency loss of DeVRF.**

In Section 4.4, we have introduced a discussion on the cyclic consistency loss of DeVRF. Additionally, in Section A.1, we have included an ablation study to compare their implementation with ours.

Reviewer comment (Novelty):
It might be good to discuss how the proposed method relates to articulated NeRF methods like Wei et al. Self-supervised Neural Articulated Shape and Appearance Models.

Author response:
**(D, F) Discussions on Articulated NeRF and Human NeRF.**

We added references and discussions of articulated NeRF and human
NeRF in the related work section.

Reviewer comment (Novelty):
While not necessary, please consider citing Nerfies when introducing the idea of using an SE(3) parametrization, since they were the first to do that for dynamic NeRFs.

Author response:
**(E) Discussions on SE(3) parametrization of Nerfies.**

We added a discussion on Nerfies when using SE(3) as a motion parameterization, which adopts a similar representation to us (In Section 4.2).

Reviewer comment (Novelty):
Maybe mention that category-specific methods, e.g. those focusing on the human body or face, do quite easily allow for part understanding and editing/re-posing, etc. and that the proposed method focuses on general dynamic scenes.

Author response:
**(D, F) Discussions on Articulated NeRF and Human NeRF.**

We added references and discussions of articulated NeRF and human
NeRF in the related work section.

Reviewer comment (Reproducibility):
The lower half of page 5 talks about averaging all the features in a group. I assume all these features come from all the points in the batch? But what is in the batch? An entire image? I.e. how is good, reproducible coverage ensured such that averaging in this manner always gives pretty much the same final feature for a certain group across different views?

Author response:
**(G) Averaged group feature may be different across different views.**

As you pointed out, the motion features utilized for grouping are sampled from points along rays in a single image, with 4096 rays sampled per batch. While there might be slight variations in the corresponding motion features across different views, we haven't observed any convergence issues.

Reviewer comment (Reproducibility):
I don't understand what the group merging module is used for. My only guess is this: During training, the initial, too-large number of groups is fixed/always used. Then, after training is finished, the group merging module is used to determine the final number of groups? So the APE cost is never used for backpropagation?

Author response:
**(I) Group merging module.**

Indeed, as you think, the group merge module is not engaged in training; it serves as a post-processing step to consolidate redundant groups.
The APE cost is used as a quantitative measure of the disparities in motion patterns between groups, enabling the aggregation of similar groups.

Reviewer comment (Experiments):
I am concerned that video results are only shown for the synthetic D-NeRF dataset, which is effectively a multi-view dataset (as discussed in Gao et al. Monocular Dynamic View Synthesis: A Reality Check). I'd much prefer to see the video results of the real-world monocular scenes used in the paper. And the datasets from Figures 6 and 7 (although these also use the quasi-multi-view setting). Otherwise, I will assume that reconstruction only works for quasi-multi-view synthetic scenes and not for input that is truly monocular or real.

Author response:
**(K, L) Video results.**

We have visually represented all experimental results discussed in the paper, including the generated Kubric-based dataset and real-world data, etc, in the URL provided above. 
We hope these videos can help to address your concerns.

Reviewer comment (Experiments):
Please add video results for part discovery on multiple scenes of the HyperNeRF dataset. Currently, only a handful of still images are shown. Otherwise, I will assume that these are severely cherry-picked and part discovery does not work on real-world scenes.

Author response:
**(K, L, O, P) Video results, ablations and Watch-It-Move.**

We are preparing an additional webpage for presenting the video results, ablations, etc. All these experiments' results will be shown in the next feedback soon.

Reviewer comment (Presentation):
The same applies for tracking. Using a well-textured canonical space like here https://github.com/pablopalafox/npms for example would help in visualizing the tracking/correspondence quality.

Author response:
**(M) Tracking visualization.**

Thank you for your suggestion. We will consider adding this kind of visualization to our paper.

Reviewer comment (Evaluation):
Please add LPIPS scores everywhere. It's standard for NeRF papers (see e.g. the original NeRF paper) and PSNR and SSIM are quite similar, while LPIPS is much more perceptual.

Author response:
**(N) LPIPS score.**

In our revised version, we have incorporated LPIPS scores for all tabular data and discussions.

Reviewer comment (Experiments):
There appear to be no ablations. For example, applying the cycle consistency loss on features vs. positions. Or using SE(3) vs. offsets. And loss ablations, e.g.: Why is the per-point color loss useful? What happens without the total variation loss? Also, an ablation of whether the forward model helps *in reconstructing* would be good. Is this additional component useful? If it isn't, then the proposed part discovery portion of the method (i.e. the group merging module) might be added flexibly as a post-processing step to existing backward dynamic NeRFs to discover parts, which would be interesting to know.

Author response:
**(O) Ablation Study.**

In Appendix A.1, we have included ablations on the Lagrangian module, cycle consistency loss, and more. To demonstrate the visual difference of these ablated models, we have provided the corresponding video results of these ablation experiments on the provided webpage.
As you mentioned, the presence of the Lagrangian module causes a slight drop in the rendering quality, which we consider to be due to the constraints of the rigid body parts of the Lagrangian module. But the quality drop is very marginal (less than 0.2dB); in general, our Lagrangian module allows for part discovery without losing the high rendering quality.
It would be interesting to explore if a post-processing step with our module could also work for part discovery. But we think jointly training the two modules at the same time could likely better regularize the motion features, making them more meaningful for parts and leading to better part discovery.

Reviewer comment (Experiments):
Can Watch-It-Move, the most relevant related work, not be applied to the D-NeRF dataset? I'm curious about the parts it would discover.

Author response:
**(P) Watch-It-Move on D-NeRF dataset.**

On the provided webpage, we have included visualizations of the Watch-It-Move results on the D-NeRF dataset. Notably, the monocular setup of D-NeRF presents a substantial challenge for Watch-It-Move due to its reliance on multi-view information for scene construction.
Although it can approximate the images within the training views, it encounters a failure when applied to the test view.

Reviewer comment (Evaluation):
I assume that part segmentation is evaluated in 2D image space rather than in 3D due issues with the unobservable insides of objects?

Author response:
**(Q) Part segmentation evaluation.**

As you said, in section 5.3, we evaluated part segmentation on the 2D images. This is due to the fact that there is no guarantee that NeRF can model the geometry and motion of unobservable insides of objects.

Reviewer comment (Theory):
The type of deformation to be considered is a bit limited, as just piecewise rigid deformations are possible, even though the algorithm can also infer a segmentation by exploiting the local rigidity of the clusters.

Author response:
We are grateful to you for your time and the review. We will clarify some of your questions or concerns in this feedback.

**Non-rigid scenarios.**

Please note that the main objective of our method is to discover rigid parts in high-quality dynamic NeRF reconstruction.
The rigid motion patterns are used as the evidence of part discovery.
Therefore, we make explicit the assumption of our input here, which is the general object with piece-wise rigid motion.
We emphasize that the automatic discovery of 3D rigid parts (with rigid motion) in dynamic NeRF is already a highly challenging task and it remains relatively unexplored in prior research.
Watch-it-move is the (only) one previous work that has a similar goal to ours, but they can only handle articulated objects (even more constrained than our rigid assumption) and does not work with monocular input.
We not only achieve automatic part discovery from monocular input but also achieve high rendering quality on par with previous dynamic NeRF methods that primarily focus on rendering quality and do not address part discovery at all.
Please let us know if there are any additional experiments you would like us to undertake under the rigid motion assumption.

Reviewer comment (Experiments):
The type of motion in the tested datasets is piecewise rigid, which is not the same as non-rigid. This can be observed in the estimation of groups, where the number of groups is very reduced and the global behavior of the deformations is quite simple.

Author response:
We are grateful to you for your time and the review. We will clarify some of your questions or concerns in this feedback.

**Non-rigid scenarios.**

Please note that the main objective of our method is to discover rigid parts in high-quality dynamic NeRF reconstruction.
The rigid motion patterns are used as the evidence of part discovery.
Therefore, we make explicit the assumption of our input here, which is the general object with piece-wise rigid motion.
We emphasize that the automatic discovery of 3D rigid parts (with rigid motion) in dynamic NeRF is already a highly challenging task and it remains relatively unexplored in prior research.
Watch-it-move is the (only) one previous work that has a similar goal to ours, but they can only handle articulated objects (even more constrained than our rigid assumption) and does not work with monocular input.
We not only achieve automatic part discovery from monocular input but also achieve high rendering quality on par with previous dynamic NeRF methods that primarily focus on rendering quality and do not address part discovery at all.
Please let us know if there are any additional experiments you would like us to undertake under the rigid motion assumption.

Reviewer comment (Novelty):
Using long-term motion trajectories is itself a traditional idea in computer vision, especially for motion segmentation (e.g. [a]). While I think that using Eulerian vs. Lagrangian views is a nice introduction to temporal motion representations, it would be nicer if they also briefly discussed the relationship of their idea to these traditional attempts. In my opinion, this would not sacrifice their technical contribution, but rather strengthen the theoretical and academic value of their approach.

Author response:
**Traditional motion segmentation with long-term motion trajectories.**

Thanks for pointing out the connection to motion trajectory and motion segmentation. We will cite the papers and add discussion in a revision. In general, different from the motion segmentation methods that mainly focus on recovering long-term point trajectories (spatio-temporal curves) in the 2D image plane from videos, our approach recovers 3D motion trajectories while ensuring multi-view consistency.

Reviewer comment (Experiments):
Since their technical contribution may be to combine short- and long-term motion trajectories, readers may want to see the ablation studies comparing with and without these modules.

Author response:
**Ablations of the proposed modules.**

Thank you for the suggestion, we are preparing an additional webpage for presenting more video results and ablations. It will be shown in the next response soon.

1. We have included ablation studies on our key modules and loss functions, and for detailed information, kindly consult Appendix A.1 in the revised version. Additionally, we have developed an anonymous webpage featuring video visualizations of the ablations and other experimental results.
We believe it will provide a more intuitive understanding of the paper.
(https://anonymous418de.github.io/MovingParts/)

Reviewer comment (Theory):
The approach is inspired by fluid simulation, with the scene motion being observed from both the Eulerian view and Lagrangian view, but it is unclear whether the method ensures that the relationship between the Eulerian and Lagrangian descriptions of a flow field, as defined by the material derivative, is verified or met.

Author response:
**About the material derivative.**

Our hybrid motion modeling is not directly connected to the material derivate. The material derivative describes the time rate of change of some physical property of a field, generally linking the Eulerian and Lagrangian descriptions of the same property's derivatives. While we adopt the Eulerian and Lagrangian perspectives from simulation, our hybrid model is designed to model the field motion/flow, instead of the time derivative of any additional field properties. We also do not define traditional field velocity in the model and hence there is no direct relationship with the material derivative. The goal of our designs is to build cycle consistency between the fixed (Eulerian) Cartesian grids and moving (Lagrangian) material points. It will be interesting to explore future extensions of our model to incorporate velocity and other material derivatives in dynamic fields.

Reviewer comment (Experiments):
Can you use other motion models, other than rigid ones?

Author response:
**Motion models other than rigid ones.**

In this paper, our goal is to achieve 3D rigid part discovery in dynamic NeRF reconstruction in an unsupervised manner. Please note that this is already a highly challenging task and relatively unexplored by prior arts. Watch-it-move is the (only) previous work that has a similar goal to ours, but they can only handle articulated objects (even more constrained than our rigid assumption) and does not work with monocular input. We not only achieve automatic part discovery from monocular input but also achieve high rendering quality on par with previous dynamic models that focus on rendering only. Potentially, our model can also be extended to handle more complex non-rigid motions in the future. However, the model might require other motion priors (e.g. motions of a specific kind of object), with ideally another low-dimensional motion parameter space, since we solve this problem in a per-scene optimization pipeline without additional part supervision.

Reviewer comment (Theory):
What is the relationship between the cycle consistency between the Eulerian and Lagrangian modules and the material derivative?

Author response:
**About the material derivative.**

Our hybrid motion modeling is not directly connected to the material derivate. The material derivative describes the time rate of change of some physical property of a field, generally linking the Eulerian and Lagrangian descriptions of the same property's derivatives. While we adopt the Eulerian and Lagrangian perspectives from simulation, our hybrid model is designed to model the field motion/flow, instead of the time derivative of any additional field properties. We also do not define traditional field velocity in the model and hence there is no direct relationship with the material derivative. The goal of our designs is to build cycle consistency between the fixed (Eulerian) Cartesian grids and moving (Lagrangian) material points. It will be interesting to explore future extensions of our model to incorporate velocity and other material derivatives in dynamic fields.