Paper: DynaVol: Unsupervised Learning for Dynamic Scenes through Object-Centric Voxelization (ICLR-2024)

Reviewer comment (Evaluation):
It is really good to see that the authors conduct a lot of ablation study on the work. However, some of them only consist of a chart with statistics, while lacking some insight analysis on them. It would further improve the quality of this work if more insights are discussed there.

Author response:
We appreciate your suggestion and have incorporated more discussion regarding the ablation study results in Section 4.5 of the revised paper. Specifically, we delve into the impact of different stages during the training process:
- **Warmup Stages:** We conducted an additional ablation study performing 4D voxel optimization from scratch with randomly initialized $\mathcal{V}{t=1}$ and $f_\phi(\cdot)$. This study clearly illustrates that excluding the warmup stages (which includes both the "3D voxel warmup stage" and the "3D-to-4D voxel expansion stage") has a substantial impact on the final performance, particularly for the scene decomposition results.
- **4D Voxel Optimization Stage:** We emphasize that the performance of DynaVol significantly degrades when excluding the 4D voxel optimization stage. These results demonstrate the importance of refining the object-centric voxel representation with a slot-based renderer.

Reviewer comment (Novelty):
The method proposed in the manuscript is pretty interesting, although it would be great to highlight the novelty and difference between the proposed method and previous literature. For example, on the dynamic modeling side, what is the relationship with [1, 2]. On the representation side, the design of volume slot attention is also not very well justified. It would be great if more insights and ablations are provided for that part of the design.

Author response:
We have included more discussion of the differences and relationships between our model and the suggested existing work [1,2] in Section 5 in the revision. We here provide a more detailed comparison of these methods on the dynamic modeling side:
- **DynaVol:** In addition to the rendering MLP, our model employs additional networks that learn $(\mathbf{x},t) \rightarrow \Delta \mathbf{x}\in \mathbb{R}^{3}$ to explicitly transport the density values in each voxel grid to the corresponding voxel grid at an arbitrary future time step.
- **Method from [1]:** This approach extends the original rendering MLP in NeRF to incorporate the dynamics information. Basically, it learns a mapping function of (RayPoint, ViewDirection, Time) $\rightarrow$ (Color, Density, Offset). Here, "offset" determines the 3D correspondence of the RayPoint at nearby time steps.
- **Method from [2]:** This approach has achieved significant improvements on dynamic scene benchmarks by representing motion trajectories by finding 3D correspondences for sampling points in nearby views.

Furthermore, while all of these methods leverage temporal-consistency constraints, unlike [1,2], which establish the constraint in pixel color space, we incorporate it in the canonical space to encourage coherent transportations of the density values across time.

Reviewer comment (Writing):
In the introduction part, I feel the importance of the problem that the paper is working on is not well explained. Instead, the author directly starts to focus on what method is proposed and what experiments are conducted. It would better enhance the quality of the paper if more background and impact of solving this problem are provided and discussed.

Author response:
We greatly appreciate the reviewer's suggestion and have revised the introduction accordingly. The importance of our work lies in three aspects:
* **A New Problem definition:** DynaVol aims to solve unsupervised object decomposition in a 3D dynamic scene with images collected by a monocular camera. It is a new research problem in computer vision.
* **Comparision to Previous Art:** Most existing approaches for unsupervised dynamic scene decomposition (such as SAVi++) as well as the models with large-scale supervised pretraining (such as SAM) ignore the 3D nature of the scene. In comparison, we present the first model for unsupervised 3D dynamic scene decomposition that naturally ensures 3D-consistent scene decomposition.
* **Impact on Downstream Tasks:** In addition to the ability to synthesize novel views and decompose 3D dynamic scenes, the learned object-centric occupancy grids in DynaVol can be easily used in downstream tasks such as scene editing (which is not supported by the 2D counterparts).

Reviewer comment (Experiments):
Although not a very big concern, the number of objects N seems to serve as input to the pipeline. This semantic could serve as additional supervision with strong prior, especially to the task of scene decomposition. It would be interesting to see how to address the problem of automatically determining the number of objects in the scene.

Author response:
(1) Clarification of the meaning of $N$:

First, we would like to clarify that $N$ represents **the number of slots** rather than the number of objects (sorry for the misleading words in the paper). In all experiments, $N$ is manually pre-defined as 10 and is assumed to be larger than the number of objects in the scene. In other words, **we do not know the exact number of objects**. We have clarified the meaning of $N$ in the revised paper.

Notably, using redundant slots naturally mitigates the *under-segmentation* problem. As shown in Figure 4 in the main text (*Per-slot decomposition results*) and Figure 8 in Appendix.F (**New!** *Per-slot rendering results*), the redundant slots can adaptively focus on capturing the background noises in the visual scene without making a substantial contribution to the final rendering results.

Additionally, this is a common practice to assume a known maximum value for the number of objects in the scene and set the number of slots to be larger than the number of objects. This setting is consistent with prior work, including Slot Attention, SAVi, SAVi++, etc.

(2) Automatically determining $N$ in the scene:

As suggested by the reviewer, we have tried to adaptively determine the value of $N$ using the connected components algorithm. Here are a few steps in this procedure:
- *Initial Filtering:* Following the initial warmup stage, we filter out empty voxels with low-density values.
- *Connected Component Identification:* We apply Algorithm 1 in the appendix of the revised paper to obtain $M$ connected components. However, $M$ tends to be larger than the number of objects in the scene.
- *Voxel Contribution Calculation:* To filter out noisy connected components, we compute the contributions of each voxel (denoted by $m_i$) to the rendering results: $m_i =T_i \cdot \alpha_i$, where $\alpha_i=1-\text{exp}(-\sigma_i \delta_i)$ represents the ray termination probability with volume density $\sigma_i$ and the distance $\delta_i$ between adjacent sampling points along the ray. Additionally, $T_i=\prod_{j=1}^{i-1}(1-\alpha_j)$ represents the accumulated transmittance from the camera position to the sampling point $i$. This approach aligns with the work of "*Compressing volumetric radiance fields to 1 MB*" by Li et al. (2023).
- *Accumulation and Sorting:* We accumulate the $m_i$ over all voxels in each connected component and sort the accumulated contribution values of the connected components.
- *Obtaining Top-$N$ Components:* We filter out the connected components with lower accumulated contribution values to ensure that the sum of $m_i$ of the remaining voxels is larger than 95% of that of all non-empty voxels. This process ultimately determines the number of slots, $N$, as the number of remaining connected components.

**Quantitative Comparison:** We compare the results obtained using the adaptively determined value of $N$ with those using the manually pre-defined value of $N$. The results are presented in terms of PSNR and FG-ARI, respectively:

| How to determine the number of slots? |       3ObjsFall (#Objects=4)        |     6ObjsFall (#Objects=7)      |       8ObjsFall (#Objects=9)        |
| --------------------- |:-----------------------------------:|:-------------------------------:|:-----------------------------------:|
| Manually pre-defined  | **32.11** $\mid$ **96.95** ($N=10$) | **29.98** $\mid$ 94.73 ($N=10$) | **29.78** $\mid$ **95.10** ($N=10$) |
| Adaptively determined |     31.68 $\mid$ 95.82 ($N=5$)      | 29.85 $\mid$ **95.29** ($N=8$)  |     29.51 $\mid$ 93.16 ($N=9$)      |

**Conclusion:** Our results suggest that manually pre-defining the number of slots can achieve comparable performance to the adaptive approach mentioned above, with the added benefit of being significantly easier to implement.

Reviewer comment (Presentation):
The proposed model DynaVol is complex. The text and the architecture figures were not easy to follow. Especially the parts about the slot updating and training are a bit opaque to me still. There are various unexplained variables in Fig. 1 which means the figure doesn’t really add much to explaining the concepts and architecture to me.

Author response:
We apologize for any difficulty you experienced in reading our paper. We have tried our best to organize the writing but found it challenging due to the complexity of the proposed method, which involves multiple training stages and many details. To improve the overall reading experience, we have made the following changes:
- We refined Section 3.1 and Figure 2 for a better overview of DynaVol.
- We revised Section 3.4 to reorganize the entire training pipeline and separate it into three stages, including a "3D voxel warmup stage", a "3D-to-4D voxel expansion stage", and a "4D voxel optimization stage". We further clarified the key insight of each stage and the connections between these stages. 
- We introduced Algorithm 1 in Appendix.B to present the details of 3D-to-4D voxel expansion, outlining how to initialize the 4D voxel representation using the connected component algorithm.

Please refer to the updated paper for the details.

Reviewer comment (Evaluation):
DynaVol has to be overfit to each individual single scene. As such this is similar to NERF and okay. But when comparing such an overfit model to a fully generalizing model like SAM, caution has to be used to clarify very precisely.

Author response:
Thanks for the suggestion! We have clarified the differences in the experimental setups of DynaVol and SAM in the caption of Table 3: "*The compared models, including SAM, are fully generalizing models trained beyond the test scenes. In contrast, our model enables 3D scene decomposition by overfitting each individual single scene in an unsupervised manner.*" Despite the distinct training setups, the comparison in Table 3 aims to showcase DynaVol's ability for 3D scene decomposition by leveraging knowledge of the segmented scene. 

Note: Both the quantitative results in Table 3 and the qualitative results in Figure 4 correspond to images of novel views outside the scope of DynaVol's training set.

Reviewer comment (Experiments):
My primary concern is the robustness of DynaVol when applied to intricate real-world scenes. The paper predominantly employs what can be described as a 'toy' dataset to validate its model. The observations made from this kind of dataset often fail to generalize to real-world applications. The mention of the HyperNeRF dataset application does lend some credence to its real-world viability. However, a more thorough exploration, perhaps using more diverse and challenging datasets (one or two demos would be enough), would provide stronger evidence of the model's capability.

Author response:
> Q1. My primary concern is the robustness of DynaVol when applied to intricate real-world scenes.

We have added new real-world experiments on the ENeRF-Outdoor dataset. Please refer to Figure 7 in Appendix.C for the qualitative results. Below, we present the quantitative comparisons in PSNR and SSIM:

| Real-world scenes |       Actor1_4        |     Actor2_3      |
| --------------------- |:-----------------------------------:|:-------------------------------:|
| HyperNeRF  | 22.79 $\mid$ 0.759 | 22.85 $\mid$ 0.773 | 25.22 $\mid$ 0.806|
| Ours |    **26.44** $\mid$ **0.871**      |  **26.24** $\mid$ **0.864** |

Reviewer comment (Experiments):
I wonder how important the background loss is in the pipeline. Will this method work without this loss? Will the slot decompose the background?

Author response:
> Q2. (1) How important the background entropy loss is? (2) Will the slot decompose the background?

(1) On the background entropy loss:

The background entropy loss is borrowed from DVGO, which aims to encourage the renderer to concentrate on either the foreground or background information. We further analyzed the impact of the background entropy loss using different loss weights (metrics: PSNR/FG-ARI):

 | $\alpha_e$ |     3ObjsFall     |     6ObjsFall     |      8ObjsFall      |
 | ---------- |:---------------------------:|:--------------------------:|:---------------------------:|
 | 0          | 31.38 $\mid$ 94.98 |   **30.43** $\mid$ 94.25  | **30.51** $\mid$ 92.59 |
 | 0.01 (Ours)          | **32.11** $\mid$ **96.95** |  29.98 $\mid$ **94.73**     | 29.78 $\mid$ **95.10** |
  | 0.1          |18.20 $\mid$ 4.65  |  19.68 $\mid$ 31.63   |23.05 $\mid$ 59.01  | 

**Findings:** Incorporating the background entropy loss with an appropriate weight is beneficial to the scene decomposition results as it penalizes ambiguous geometric estimations. However, a higher $\alpha_e$ value can introduce instability to the training procedure.
   
(2) Will the slot decompose the background?

Yes, our proposed method is motion-aware and is capable of decomposing the background, as the background can be treated as a static component. Additionally, during the 3D-to-4D voxel expansion stage, voxel grids with significantly distinct velocities estimated by the dynamics modules tend to be initiated with different slot indices by the connected components algorithm (Please refer to Algorithm 1 in the appendix).

As illustrated in Figure 3(b), our model effectively decomposes backgrounds in real-world scenarios. In the synthetic scenes in Figure 4, our model successfully decomposes the background table with an individual slot. For additional visualization results of per-slot rendering, please refer to the newly added Figure 8 in Appendix.F.

Reviewer comment (Evaluation):
Based on the results presented in Table 1, it seems that models utilizing 5 views (V=5) don't consistently surpass the performance of those with just a single view. Could the authors shed light on the underlying reason for this unexpected behavior?

Author response:
> Q3. Models utilizing 5 views don't consistently surpass the performance of those with just a single view.

Our motivation for using multiple views at $t=1$ stems from empirical observations in particular challenging scenarios. For instance, in 3ObjMetal, the color of a moving object with a metal surface may change rapidly across different time steps due to the drastic changes in specular reflection, which presents difficulties in learning the geometries. Incorporating additional views at the initial time step can enrich the geometric information, thereby improving the quality of the learned 3D voxel grids $\mathcal{F}\_{t=1}$ and corresponding 4D object-occupancy grids $\mathcal{V}\_{t=1}$.

Reviewer comment (Evaluation):
It's intriguing to observe that the 'warmup-stage-only' variant appears to perform quite competently even though it's restricted to initial timestep observations. It's even better than other baselines reported in Table 1. Could the authors explain the factors or mechanisms behind this seemingly robust performance?

Author response:
> Q5. It's intriguing to observe that the 'warmup-stage-only' variant appears to perform quite competently even though it's restricted to initial timestep observations. 

First, it's important to clarify that, even in the warmup stage, **the model is trained with the entire image sequence** rather than being restricted to the initial time step. As shown in the updated Figure 2, it also involves an end-to-end optimization of the density grids, the bi-directional dynamics networks, and a DVGO-style renderer (more efficient than the original renderer used in D-NeRF). Therefore, considering the training data and specific model designs, it is reasonable that the "warmup-stage-only" baseline can achieve good performance.

Additionally, it's essential to note that the "warmup-stage-only" baseline involves the "3D-to-4D voxel expansion" operations outlined in Algorithm 1 in the paper. Accordingly, the FG-AGI results validate the effectiveness of the connected component algorithm in initializing the 4D object-centric representation from the warmup of the density values in the 3D voxel grids. 

Further discussions have been incorporated in Section 4.3. For more details, please refer to our revised paper.