Paper: Consistent4D: Consistent 360° Dynamic Object Generation from Monocular Video (ICLR-2024)

Reviewer comment (Experiments):
In the paper, the authors compare the proposed approach to D-NeRF and K-planes, which are regression-based methods. I think the authors should add at least one generative-based baseline to convince the effectiveness. Concretely, I think a valid baseline is to run a single-view reconstruction pipeline per frame [a] and see whether they can be consistent. [a] https://github.com/threestudio-project/threestudio#zero-1-to-3-

Author response:
* **Generative Baseline**: We add zero123-per-frame as suggested, please refer to response 1 in “Responses to All Reviewers” for more details.

Reviewer comment (Experiments):
Currently, the only ablation in the paper is about whether to use interpolation consistency loss or not. Such ablation is only conducted with a user study.

Author response:
* **Ablations**: **Previously, we presented extensive ablations for all of the novel modules introduced in our paper**, including Cascade DyNeRF, ICL, and the video enhancer, as detailed in **Sec. 5.3**. **Quantitative ablations** are newly added, as shown in response 2 in “Responses to All Reviewers”. For other commonly adopted losses in image-to-3D methods, such as the reconstruction loss and 3D normal smoothness loss, we maintained them as **default settings** and did not provide ablations. However, we are more than willing to conduct additional ablations to provide readers with a more comprehensive understanding of our work:

Reviewer comment (Experiments):
There are quite a few ablations missing such that it is unclear which module contributes to the performance. One missing ablation is how each of the following losses contributes to the performance: foreground mask loss, normal orientation loss, and 3D smoothness loss. This could be provided with quantitative numbers.

Author response:
1. **The ablation of reconstruction loss, foreground mask loss, normal orientation loss and 3D smoothness loss**: While these losses have been adopted from Zero123 implemented in Threestudio and do not constitute our original contributions, we understand the importance of providing a comprehensive analysis. Below is the ablation for losses:

      |  | LPIPS &darr; | CLIP &uarr; | FVD &darr; |
      | -------- | -------- | -------- | -------- |
      | ours w/o rgb loss    | 0.16   | 0.85   |  1260.39  |
      | ours w/o mask loss    | **0.15**   | **0.87**   | 1157.97   |
      | ours w/o normal orientation loss    | **0.15**   | 0.86   | 1146.39 |
      | ours w/o 3d normal smooth loss    | 0.16  | 0.86| 1191.75 |
      | ours w/o SDS loss    | 0.19  | 0.81 | 1699.83  |
      | ours | 0.16 | **0.87** | **1133.44** |
      
      It is evident from our metric analysis that the SDS loss holds a pivotal role, exerting a significantly greater impact compared to the RGB reconstruction loss. This observation underscores the generative nature of our method, emphasizing the importance of the SDS loss.

Reviewer comment (Experiments):
Another missing ablation is the effect of varying the probability of using interpolation consistency loss during training. Currently, it is used with 25% probability. It would be more informative if performance could be shown for higher or lower probabilities, possibly as a performance curve.

Author response:
3. **Probability of ICL**: we provide the ablation of probability of ICL as bellow:
      |  | LPIPS &darr; | CLIP &uarr; | FVD &darr; |
      | -------- | -------- | -------- | --------
      | probability=0.    |  **0.16** | **0.87**   | 1226.92 |
      | probability=0.1    | **0.16** |  **0.87** | 1190.01   |
      | probability=0.25 (ours) | **0.16**   | **0.87**   | **1133.44** |
      | probabilityt=0.5    | **0.16** | 0.86 |  1179.67 |

      FVD metric suggests that a probability value of 0.25 is an appropriate setting.

Reviewer comment (Experiments):
Another missing ablation is the effect of changing the weight for the SDS loss. The weights for different losses differ quite a lot: the reconstruction loss weight is 500 while the SDS weight is only 0.01. It is unclear how changing the SDS weight would affect performance.

Author response:
4. **SDS weight**: We provide experiments about changing sds weight as bellow:
   |  | LPIPS &darr; | CLIP &uarr; | FVD &darr; |
   | -------- | -------- | -------- | -------- |
   | sds_weight=0.    |  0.19 |  0.81  |  1699.83   |
   | sds_weight=0.01    | **0.16** | 0.85  | 1321.19  |
   | sds_weight=0.1 (ours) | **0.16**   | **0.87**   | **1133.44** |
   | sds_weight=1.    | **0.16** | 0.86 |  1270.86 |
   The results indicate sds_weight=0.1 is a suitable setting.

Reviewer comment (Experiments):
As mentioned in Point 6, the proposed weight for SDS loss is only 0.01 while the reconstruction loss weight is 500. This setup essentially makes the framework regression-based. It is unclear whether the same reconstruction loss is applied in baselines, including foreground mask loss, normal orientation loss, and 3D smoothness loss. If not, the comparison might be misleading.

Author response:
First and foremost, we sincerely apologize for the typographical error in the SDS weight. **It should be 0.1 instead of 0.01**. We regret any confusion this mistake may have caused regarding our method.

The hyperparameters of reconstruction weight and sds weight, i.e., 500 and 0.1,  are **exactly the same as the default config of Zero123 implemented in Threestudio**[1]. The two loss weights generate good results in Zero123, so we **did not finetune them in our experiment**. Other losses, including normal orientation loss and 3D smoothness loss, **were kept exactly unchanged in all experiments** in the paper to ensure a fair comparison.

Reviewer comment (Novelty):
The difference between D-NeRF and K-planes should be clarified. Is the only difference that D-NeRF uses MLP while K-planes use the plane representation?

Author response:
* **The difference between D-NeRF and K-planes**: Sorry that we did not introduce D-NeRF in the paper. Besides architectural differences, there is another noticeable difference between D-NeRF and K-planes. D-NeRF needs to first reconstruct a static object and then learn offsets of each point in future frames. In other words, it adopts a two-stage training. In contrast, K-planes doesn’t need to reconstruct the static scene first. It directly models the dynamic scene.

Reviewer comment (Writing):
In Eq. (6), it is unclear if there is a typo. Should it be $\hat{x}_j = \phi (x_0, x_J, \gamma_j)$, i.e., the input is $x_J$ instead of $x_j$?

Author response:
* **Typo**: thanks for pointing it out, we have modified it.

Reviewer comment (Experiments):
More comparisons are needed in this setting, are there other datasets which can be evaluated on this tasks. Lack of training time and inference time comparison with other methods.

Author response:
*  **Comparisons on other datasets**: Existing multi-view dynamic scene datasets are usually scene-level, however, currently our method can only tackle object-level generation. A few datasets are object-level, however, the dynamic objects in those datasets are so complicated that even single-image-to-3D methods fail to generate the static 3D object from a single frame in the video, let alone our methods for dynamic object generation.  We will improve our work and extend it to the scene level and complicated objects in the future. 

* **Training time and inference time comparison with other methods**:
Since Zero123 requires more than 40GB GPU memory for training, we compare the training and inference time of all methods on **an 80G A100 GPU**. We provide the averaged results on 4 objects as follows (inference speed is measured under image resolution of **256x256**):
|  | training time | inference speed |
|-----------------|-----------------|-----------------|
| D-NeRF   | 61 min   | 2.8 fps   |
| K-planes   | 76 min   | 19.2 fps  |
| Zero123(per-frame)   | 79 min   | 41.8 fps   |
| ours w/o video enhancer   | 91 min  | 63.8 fps  |
| ours w/ video enhancer   | 97 min  | 62.9 fps  |

The training time of our method is a little longer than per-frame Zero123, whilst we have fast inference speed.

Reviewer comment (Reproducibility):
The detailed usage of the diffusion model in the method part are needed for better understanding this paper.

Author response:
* **Detailed usage of the diffusion model**: We illustrated the use of the diffusion model in the Preliminaries (Sec. 3.1)  and we elaborate on it below: 
The diffusion model adopted in this work is Zero123-xl, an image-to-image translation model. In the original design, this diffusion model takes a reference view image and relative camera position between the target view and the reference view as the input and outputs the target view image.  In our pipeline, we use images from the monocular video as the reference images. Following image-to-3D methods, we random sample target views for rendering images from dynamic NeRF, and calculate the relative position between the target views and the reference view to serve as the camera position input of the diffusion model. Then, the diffusion model can calculate SDS loss for the rendering images of the target view via Eq. 2 in the paper. The gradient is backward to dynamic NeRF through the rendering images. 

   For easy understanding, we now provide another interpretation of SDS loss in Eq. 2. SDS loss is equal to the L2 loss between the noisy image (obtained by adding random noise to rendered target view images) and the 1-step denoised image output by the diffusion model. Thus, the distribution of rendered images from target views is always optimized toward the distribution of target images in the diffusion model.

Reviewer comment (Writing):
Writing of the paper can improve a lot. Paper's readability is significantly hampered due to informal writing style, usage of obscure terminology and grammatical errors. Crucial details are left out leaving reader to fill gaps by extrapolating prior art. Also, notations are misplaced in a few places. (I did not check all but see questions section for the details on these)

Author response:
* **Paper writing**: we have re-origanize the paper as you suggested. We list the changes we made below:

  1) Confusing sentences about Cascade DyNeRF are removed
  2) Video enhancement is claimed as an optional step
  3) Loss function part is moved to the main body of the paper
  4) **Comprehensive implementation details** regarding both DyNeRF training and video enhancer training are provided in the appendix (Sec. A.4)
  5) Zero123 is added in comparison with state-of-the-art, and quantitative ablations are added as well

Reviewer comment (Experiments):
Some of the experimental evaluation, especially quantitative evaluation is not appropriate and given synthetic datasets are available on which authors evaluate the proposed method already, the same can be used for more appropriate Comparisions and ablations.

Author response:
* **Comparison with Zero123**: Zero123 lags behind us in terms of temporal consistency. Please refer to response 1 in “Responses to All Reviewers” for more details.

* **Quantitative ablation**: The improvements in the metric FVD validate the effectiveness of each proposed module. Please refer to response 2 in “Responses to All Reviewers” for more details.

Reviewer comment (Presentation):
Suggestion on paper's structure: The paper attempt to focus on too many incremental changes within individual components the 4D reconstruction pipeline uses. Some of these changes make no significant difference in the final results e.g. video enhancer though. Trying to pack and justify these incremental changes is difficult in limited space to which author did not do justice. I would highly recommend leaving some of these details out to appendix and incorporate more critical explanation of the pipeline -- which I reiterate is sufficiently novel for publication once explained clearly. Things such as loss function used to train the entire system must be included in the main body of the paper and clearly written. I would also highly recommend avoiding a storytelling narrative currently used in the paper where a series of failed/suboptimal experiments are listed one after the other without letting the reader know what is eventually used. Instead, explaining what the proposed pipeline and methodology is, followed by reiterating novel changes will be more clear. Incremental changes in K-plane with feature concatenations, cascade DyNerf justification followed by statement that Cascade DyNeRF alone is not enough for spatiotemporal consistency, so we resort to extra regularization, please see in the next section" confuses reader regarding what is novel.

Author response:
* **Paper writing**: we have re-origanize the paper as you suggested. We list the changes we made below:

  1) Confusing sentences about Cascade DyNeRF are removed
  2) Video enhancement is claimed as an optional step
  3) Loss function part is moved to the main body of the paper
  4) **Comprehensive implementation details** regarding both DyNeRF training and video enhancer training are provided in the appendix (Sec. A.4)
  5) Zero123 is added in comparison with state-of-the-art, and quantitative ablations are added as well

Reviewer comment (Evaluation):
While the reconstructions look impressive, I am not convinced with the quantitative evaluations presented in the paper. First, it is not fair to compare the proposed approach with Dynamic NeRF literature given that the baselines are designed to work on videos with sufficient camera motion only and not use many foundation models. As the proposed method introduces the 4D consistency via video interpolation module in 3D generation frameworks, would it not be more suitable to quantitatively evaluate the results with a baseline like Zero123? Ablations of ICL gives a flavour of the same but it is only qualitative. I would strongly recommend the 4D consistency and accuracy of the method is quantitatively evaluated on the synthetic sequence.

Author response:
* **Comparison with Zero123**: Zero123 lags behind us in terms of temporal consistency. Please refer to response 1 in “Responses to All Reviewers” for more details.

* **Quantitative ablation**: The improvements in the metric FVD validate the effectiveness of each proposed module. Please refer to response 2 in “Responses to All Reviewers” for more details.

* **Evaluation on previous dynamic NeRF settings**: Although conventional reconstruction is not the focus of the proposed method, we are glad to test our model in that setting. We will do this experiment.

Reviewer comment (Evaluation):
Besides, given that the authors use the image reconstruction loss used in NeRF (I struggled to verify the same), I disagree that traditional novel view synthesis losses (PSNR, SSIM) should be avoided. In fact, I do wonder why some of the generated novel views for example T-rex roar have Seagull turning around have texture don't have texture consistent with the input views. Perhaps the authors can elaborate. In fact a thorough 4D reconstructions evaluation requires not only good novel view synthesis (consistent with input appearance) but a good quality key-point tracking as well as accurate structure. Given that the authors have used synthetic datasets some of this can be evaluated as well. I understand that the paper focuses on the challenging scenario of stationary camera for 4D reconstructions but in principle the methodology can be evaluated on any sequence Dynamic NeRFS are evaluated on and encourage authors to do so.

Author response:
* **PSNR and SSIM as losses**: Thanks for suggesting using the two losses. Actually, we had conducted experiments using L1/PSNR/SSIM as the rendering loss. We empirically find that the L2 loss generates the best result,  and that is the reason for choosing the L2 loss in our method.

* **Inconsistent texture**: The texture inconsistency problem is widely observed in generative image-to-3D studies [1,2,3,4], which might be related to the SDS optimization. It will be our future work to improve the texture consistency.

* **Key-point tracking**: Currently, our model employs an implicit 4D field to model dynamic objects, which precludes the possibility of key-point tracking. This could be potentially addressed by using a point-based 4D model (e.g., dynamic 3D Gaussians) and we leave it as a  future work.

* **Evaluation on previous dynamic NeRF settings**: Although conventional reconstruction is not the focus of the proposed method, we are glad to test our model in that setting. We will do this experiment.

Reviewer comment (Writing):
Informal writing and notational inconsistency requiring explanation: “DyNeRF methods mainly assume the supervision signals are temporally coherent, however, this assumption does not hold in our task,” The author never formally defines what is "temporal coherence." However, the task paper solves is same as that of Dynamic NeRF, i.e., reconstructing deforming scene. Perhaps authors are trying to allude to the challenges in incorporating diffusion models as priors which have no notion of time?

Author response:
* **Explanation**

   a）Temporal coherence as the main challenge:  You’re right. Temporal coherence is a unique challenge in our pipeline, cause we adopt an image diffusion model without the notion of time to provide the supervision signal.

Reviewer comment (Writing):
Informal writing and notational inconsistency requiring explanation: “In order to minimize the impact of temporal discontinuity in the supervision signals, we are prone to DyNeRF methods with low-rankness.” Are authors suggesting they prefer to use a framework like K-planes – which imposes rank constraints – as the suitable dynamic NeRF framework? It is unclear to me why low rank will specifically help the proposed approach beyond being a useful prior for 4D reconstruction in general case. Please specify.

Author response:
* **Explanation**

   b)  K-planes with low-rankness: Thanks for pointing out. We agree that the use of “low-rankness” is inaccurate and confusing, so we have removed it in the revised paper. What we want to express is, K-planes is naturally prone to temporal coherence given that it has temporal interpolation operation. Considering temporal coherence is the main challenge in our pipeline, we choose to build our dyNeRF on K-planes.

Reviewer comment (Writing):
Informal writing and notational inconsistency requiring explanation: Equation 6: Does the indices j and batch size J are being confused in the equation? I suppose that a sensible interpolation loss will take image x_1 and x_J as input with \gamma_j and return the image \hat{x_j}? Also, while the text describes the input images to be indexed 1 and J the equation itself uses x_0 and x_j? The author does not differentiate between spatial and temporal constancy. Is the spatial consistency enforcing 3D consistency of rendered views using virtual cameras at a single time or something else?

Author response:
* **Explanation**

   c) Typos in Eq.6 and questions about spatial consistency: Thanks for pointing out. Yes, you’re right about Eq.6 and we have fixed those typos. For spatial consistency, we render batch images using virtual cameras at a single timestamp when applying ICL loss.

Reviewer comment (Writing):
Informal writing and notational inconsistency requiring explanation: “The spatial and temporal resolution of Cascade DyNeRF are configured to 50 and 8 for coarse-level, and 100 and 16 for fine-level, respectively. We first train DyNeRF with batch size 4 and resolution 64 for 5000 iterations. Then we decrease the batch size to 1 and increase the resolution to 256 for the next 5000 iteration training.” I am not certain what the resolution means in the second statement. Technical details in this paragraph are very hard to understand in general and can be explained better.

Author response:
* **Explanation**

   d) The meaning of resolution in implementation details: The resolution in the second statement stands for the resolution of rendering images during training. Due to limited GPU memory, we can only render 64x64 images when using batch_size=4.

Reviewer comment (Writing):
Informal writing and notational inconsistency requiring explanation: “we also apply foreground mask loss, normal orientation loss, and 3D smoothness loss.” These losses should be explained in appendix (I prefer within the body of the paper) and cited in the paper if they are coming from literature directly.

Author response:
* **Explanation**

   e)  Loss functions: We have moved them to the main body of the paper and have cited corresponding literatures in the revised paper.