Paper: DyST: Towards Dynamic Neural Scene Representations on Real-World Videos (ICLR-2024)

Reviewer comment (Novelty):
The method is quite similar to RUST[1]. The encoder, decoder and camera estimator are almost the same as the ones proposed in RUST.

Author response:
**Similarity to RUST**

We fully agree with the reviewer that DyST has a similar network architecture to RUST. This is intentional, as it should be noted that our major contributions are not the network architecture (besides e.g. the Camera & Dynamics estimators), but rather, that the contribution of our work lies in the novel swap-training scheme, co-training between the (newly proposed and to be released) DySO dataset and real-world videos, and the investigation and validation thereof. We believe that focusing on these crucial factors enabling separation into scene content, camera, and scene dynamics is a strength of the approach. While investigations in modifying the network architecture are certainly possible (e.g., to improve render quality even further in future work), they could distract from the novel capabilities of our model that our presented work is focused on, and make it harder to credit the new model capabilities to the novel components that we propose.

We apologize for the delayed rebuttal – some of the requested experiments needed more time to finish before we could summarize our findings. We address all reviews in the comments above. The point on the similarity with RUST is also addressed above in the comment to reviewer `XJou`. We note that our ablation in Sec. 4.4 (no swap) could be seen as the most similar model to RUST. From that experiment, we can conclude that it achieves a significantly lower PSNR (18.6 vs 26.0), worse LPIPS (0.45 vs. 0.34) and no separation into camera pose and scene dynamics (despite this ablation having a split into camera & scene dynamics components which RUST lacks).

We appreciate the engagement of the reviewer and welcome further comments or questions during the rebuttal phase.

Reviewer comment (Reproducibility):
Inference procedure. From the method architecture, the target view is required to obtain the camera latents and dynamic latents. In this case, I wonder if the specific novel view image is needed as the input to generate the novel view?

Author response:
**Inference procedure & control of the latent code**

We thank the reviewer for the question. DyST inference can be realized in several ways that do not include a dedicated novel target view. Depending on the use case, one of the input views can serve as the "target" view that is used to get the (initial) control latents. The PCA-analysis demonstrated in Sec. 4.2 and Fig. 6 show how one can control camera and scene dynamics (for any arbitrary video) in intuitive ways. Another possibility is to transfer the camera and / or scene dynamics from one video (e.g., a DySO scene) to another one (e.g., a real video), to have full control over the camera and scene dynamics. Finally, we show an example of scene control over a large number of videos in [1].

[1] https://ml-anon-1.github.io/#cam-ood

Reviewer comment (Experiments):
Some quantitative results on real data should be provided.

Author response:
**Quantitative results on real data**

We agree with the reviewer that quantitative results on real-world videos would be great, however it is non-trivial, since this would require access to synchronized multi-view videos. We will investigate this possibility and calculate quantitative metrics on real-world videos.

Reviewer comment (Presentation):
Reproducibility Statement should be put in the appendix.

Author response:
**Position of the Reproducibility Statement**

We thank the reviewer for noting this. It is our understanding that the official guide suggests placing the reproducibility statement "at the end of the main text (before references)" [1]. We are happy to correct the placement if there was a misunderstanding on our side.

[1] https://iclr.cc/Conferences/2024/AuthorGuide#:~:text=Reproducibility,been%20made%20to%20ensure%20reproducibility.

Reviewer comment (Experiments):
[Generalization in different types of motions] Additional experiments are needed to see if the proposed DyST model can generalize to camera poses and scene dynamics that were not seen during training. So, it would be better to provide qualitative results on how the controlled view looks like when horizontal shifts are input after training without horizontal shifts. (DySO’s camera motions consist of 4 horizontal shifts, panning, zooming motions, and random camera points)

Author response:
**Generalization to novel types of motions**

In theory, DyST will only learn to model camera motion and scene dynamics that exist in the data, as this is strongly encouraged by the model architecture. However, in practice, two types of out of distribution (OOD) generalization are observed:
Between-scene OOD generalization. The model learns dynamics that may exist only in a subset of videos, and at inference time, these dynamics can then be applied to other videos too.
Full-dataset OOD generalization. Since the model learns a decomposed representation of camera and scene dynamics, it exhibits OOD generalization in terms of novel dynamics that did not exist in the dataset. For example, the rotation of the camera demonstrated in [1] is neither existent in DySO, nor did we find examples in SSv2 that show this kind of camera motion.
For the camera-ready version, we can try to include a further experiment on a DySO variant that e.g. does not exhibit any horizontal shifts to see if the model generalizes to capture this at test time. That said, we believe that the result in [2], showing the rotating motion which was non-existent in DySO nor SSv2, already shows a much significantly stronger generalization capability in the same vein.

[1] https://ml-anon-1.github.io/#cam-ood

Reviewer comment (Experiments):
[Cluttered background] Since the backgrounds of the DySO dataset in Fig. 2 and 3 are clean, the authors need to experiment to see if DyST can robustly control the view even when using videos with cluttered backgrounds. In addition, it would be better to have a distance analysis for unclean scenes to see how distinctly it separates camera pose and scene dynamics like the experiment in Figure 5.

Author response:
**DyST on cluttered backgrounds & distance analysis**

We thank the reviewer for bringing this to our attention. Indeed the background of some scenes from DySO in our figures may look simple, however, DySO includes a large number of nontrivial HD backgrounds that include clutter. We have added many more results in [1] to show a better representation of the dataset. Further, the SSv2 dataset contains a large number of videos with cluttered backgrounds, where the model still exhibits robust view control. To highlight this further, the video in [2] includes videos from SSv2, and more scenes that we recorded ourselves, with cluttered backgrounds, e.g. additional static objects in the background.

We note that the distance analysis on the left side of Fig. 5 was calculated over a large number DySO scenes, including ones with cluttered backgrounds. The video on the right that we recorded ourselves already includes quite a lot of clutter, please note the static objects on the table. We also have included more distance analysis results in [3] for convenience.

[1] https://ml-anon-1.github.io/#dyso

[2] https://ml-anon-1.github.io/#cam-ood

[3] https://ml-anon-1.github.io/#latent-distances

Reviewer comment (Evaluation):
[Quantitative comparison] As the authors mentioned in Sec. 5 Conclusion, unlike NeRF's output, the output of the proposed method has a quality gap, such as objects disappearing or blurring. Therefore, quantitative comparison results such as PSNR and LPIPS between NeRF and DyST are needed.

Author response:
**Quantitative comparison**

We thank the reviewer for suggesting a comparison to NeRF. We'd like to emphasize that common high-quality NeRF videos have been generated in quite different setups, with access to high-quality camera poses and usually for static scenes. Furthermore, NeRF methods require expensive optimization per scene (or in this case, per video), and do not generalize in real-time to novel videos. Finally, NeRF methods usually overfit a network to the specific scene and therefore neither provide a handy latent representation of the scene content (in our case, the SLSR), nor scene dynamics.

With the caveats above, we agree with the reviewer that a comparison would still be valuable. We identified the strongest NeRF model that works on dynamic videos without requiring camera poses, Robust DynRF, and use the official implementation from [1]. We train the Robust DynRF for 50k steps on 6 scenes from SSv2, and find that it achieves an average PSNR of 26.1 and LPIPS of 0.34 on those scenes. In comparison, DyST achieves a significantly better PSNR of 27.9 and LPIPS of 0.18. Notably, we find that Robust DynRF produces artifacts around moving objects (see [2]), and is not able to generalize out of distribution (OOD) when moving the camera out of the frame of the video, whereas DyST is able to smoothly fill-in a reasonable background. Moreover, training the model on a single scene takes around 3.5 hours on a 4080 GPU, whereas a trained DyST model provides inference in a few milliseconds. We conclude that in the complex setting we target in this work, DyST has significantly better render performance than SotA NeRFs, while our model provides superior OOD performance, vastly faster inference, and handy latent representations of the videos.

[1] https://github.com/facebookresearch/robust-dynrf/

[2] https://ml-anon-1.github.io/#nerf-comparison

Reviewer comment (Experiments):
[Multiple objects] Also, as mentioned by the authors in the same section, the authors did not provide results for multiple object scenes. It would be helpful to see the results of latent distance analysis in Figure 5, PSNR, and LPIPS in Figure 3 for multiple object scenes.

Author response:
**Multiple objects**

We thank the reviewer for bringing this up. Our statement in Sec. 5 (conclusion) was meant to address multiple independent moving objects in a single video, we have improved the formulation in the draft. Indeed, closer inspection of the results we have provided reveals that a number of our real-world videos already show multiple moving objects, even in Fig. 5 on the right, with the hand and the cup being two moving objects. As the figure shows, the model can, to some extent, therefore generalize to multiple objects. Running an experiment on synthetic DySO scenes with multiple objects involves creating a new dataset and will therefore take more time, however we will try to include this in the final version of the manuscript.

Reviewer comment (Experiments):
The latent control swap training scheme needs 3 input views. It would be helpful to explain why 3 input views are needed and how the performance changes with less or more than 3 input views.

Author response:
**Number of input views for control swap**

We thank the reviewer for proposing this experiment.
It should be noted that the proposed control swap scheme has no requirements on the number of input views. In early experiments, we ran experiments with fewer (down to 1) or more input views, and found that the model largely behaved the same, with the expected difference that reconstruction quality would degrade or improve subtly depending on how much information the model receives. We chose 3 input views at training time to strike a good balance between efficiency and quality. It should be noted that the number of input views to DyST can be varied at inference time (in particular, it can be different from training), however the comparison would not be apples-to-apples, since more information would be given to the model.
For completeness, we note that there is a requirement on the number of target views during training: a minimum of 2, so a swap can be performed. However, this only applies on the synthetic dataset for co-training. On the real-world dataset (SSv2 in our case), even 1 input view and 1 target view would suffice for training.

Reviewer comment (Evaluation):
It would be better if the authors discuss why it needs the contrastiveness metric and what the authors are trying to show with the swap in Table 1.

Author response:
**Contrastiveness metric & Swap in Tab. 1**

With the contrastiveness metric and the different swap training scheme ablations, we aim to investigate quantitatively how cleanly the model separates camera motion from scene dynamics, i.e. the first question in Sec. 4.2: "First, does our training with latent
control swapping induce a meaningful separation of camera pose and scene dynamics on synthetic data?"
The contrastiveness measures the similarity between pairs of camera and scene dynamics latents for scene views that exhibit various combinations thereof. Intuitively, R^cam will be exactly zero if the latent camera pose is perfectly consistent for views rendered for the same camera, fully independently of the scene dynamics (and vice versa). In practice, the camera and scene dynamics latents are never perfectly separated and independent of each other, therefore the R^cam and R^dyn values are positive.

Tab. 1 analyzes the results of different training schemes for the proposed DyST model. We show that our proposed swap method leads to the cleanest separation between camera and scene dynamics, and that the same model trained without our swap scheme ("No swap") learns practically no separation, thereby providing quantitative evidence that our training scheme is helpful.