Paper: Pseudo-Generalized Dynamic View Synthesis from a Video (ICLR-2024)

Reviewer comment (Evaluation):
The most obvious downside of the paper is the result quality, unfortunately. The supplemental videos show rather low-quality results. Given that this is the first method in its problem setting, this is not necessarily a reason for rejection, as long as the experimental evaluation is great. The remaining weaknesses all concern the evaluation.

Author response:
Thanks a lot for pointing this out. After double-checking, we found that we indeed uploaded the wrong videos. We corrected this in the updated supplementary material. Videos exhibit much fewer artifacts.

Reviewer comment (Evaluation):
Qualitative video comparisons: I don't understand the results on the Nvidia Dynamic Scenes dataset. Quantitatively, Neural Scene Flow Fields seems to be about on par with the submission. However, qualitatively, the results of NSFF (on their website) are much better. Where does that large discrepancy come from? Qualitative video comparisons to other works would help.

Author response:
Note that the videos in the supplementary material are for **spatio-temporal interpolation**, which does not perfectly reflect the quantitative results for **novel-view synthesis** in the main paper.

To be specific, the quantitative results on the multi-view NVIDIA Dynamic Scenes dataset reported in Tab. 1 don’t need temporal interpolation (see the paragraph below Eq. (3)). Essentially, quantitative evaluations are for spatial interpolations. However, when synthesizing the videos in the supplementary material, we conduct both spatial and temporal interpolations to demonstrate the ability of our study-found approach to synthesize frames between observed times. As expected, results are not as good as some methods which use scene-specific optimization. Nonetheless, we think it is valuable to show these results and understand present day capabilities.

To provide illustrations for the quantitative results provided in the main paper, we included videos composed of evaluation frames in the updated supplementary. See folder section “[NVIDIA Dynamic Scenes] Videos from Frames for Quantitative Evaluations” in the updated html. 

Note, the original qualitative results of NSFF and DVS cannot be directly compared here due to the different setup. Originally, NSFF and DVS only selected 24 frames from the full video for training and evaluation. In contrast, we follow DynIBaR and evaluate on the whole video. Arguably, NSFF’s inability to handle long videos is one of the motivations for the development of DynIBaR (see DynIBaR’s Sec. 1 and NSFF results on the DynIBaR website).

Reviewer comment (Experiments):
Fast scene-specific dynamic NeRFs: I would like to see a comparison with fast scene-specific dynamic NeRF methods. In terms of utility for novel view synthesis, generalizable NeRFs have two main advantages over scene-specific NeRFs: speed and learned prior knowledge. The latter is not exploited by the submission, as the results show novel view synthesis that sticks closely to the input camera path (instead of revealing hidden areas that scene-specific methods could not handle). Which leaves speed and I'd hence like to see a comparison with Fang et al. TiNeuVox '22 (code is available), which optimizes a dynamic NeRF in a few minutes, unless there is a reason why such a comparison is unnecessary.

Author response:
Thanks a lot. Great suggestion. We add quantitative and qualitative results for TiNeuVox on both datasets to the paper and the supplementary html respectively. For qualitative videos, please refer to sections “[NVIDIA Dynamic Scenes] Videos from Frames for Quantitative Evaluations” and “[DyCheck iPhone] Videos from Frames for Quantitative Evaluations” on the updated html.

Specifically, we run TiNeuVox on each video for 40k steps, which is double the steps than the default 20k iterations. This gives TiNeuVox an additional advantage. We evaluate on its checkpoints at both 20k and 40k steps of optimization, which takes 45min and 1.5 hours on a V100.

Note, TiNeuVox can only complete optimization “in a few minutes” on the D-NeRF dataset. According to DyCheck (Gao et al., 2022a), the D-NeRF dataset is arguably one of the easiest datasets for dynamic novel view synthesis as it is “effectively multiview” (see Fig. 3 of DyCheck’s paper). In contrast, the NVIDIA Dynamic Scenes and DyCheck iPhone data are much more challenging.

Meanwhile, we want to clarify that TiNeuVox also requires consistent depth for its regularization loss on background 3D points (https://github.com/hustvl/TiNeuVox/blob/d1f3adb6749420d10ecc074806f06459f189acbd/lib/load_hyper.py#L79-L83). Therefore, our comparison to TiNeuVox is fair.

We observe that TiNeuVox results are inferior to our approach on both datasets. 
We updated Tab. 1, 2 and Fig. 1 to include the updated results.
Note that there are no big differences between results after 20k and 40k steps, i.e., the optimization converged.
We hypothesize the inferior results are due to TiNeuVox not disentangling static and dynamic content. Though a joint representation works well on “effective multiview” datasets, e.g., D-NeRF data and Nerfies data, it is more challenging to attain good results with a joint representation on NVIDIA Dynamic Scenes or DyCheck iPhone data.

Reviewer comment (Experiments):
Static generalizable per-frame NVS: Also, given the rather limited quality and temporal instability of the results, I'd like to see a comparison with static single-image generalizable novel view synthesis methods. For example, Sajjadi et al. Scene Representation Transformer '22. MIT-licensed code is here https://github.com/stelzner/srt and the authors say on their project page that this code is reliable.

Author response:
Thanks a lot for the suggestion. We find that the suggested SRT repo (https://github.com/stelzner/srt) only contains pretrained checkpoints on the NMR dataset (a subset of ShapeNet hosted by https://github.com/autonomousvision/differentiable_volumetric_rendering) and the MultiShapeNet (MSN) dataset:
- For NMR, as mentioned in the repo (https://github.com/stelzner/srt#known-issues), SRT can only render from fixed 24 camera poses, which is not suitable in our case.
- For MSN, which is only trained on ShapeNet objects, a comparison is not suitable due to the large domain gap between ShapeNet and our real world dynamic scenes.
Therefore, a comparison to SRT is not meaningful.

Reviewer comment (Experiments):
All qualitative results of the proposed method: Why are the qualitative video results on the DyCheck iPhone dataset not included in the supplement?

Author response:
Thanks for the suggestion. We add them in the updated supplementary material. See section “[DyCheck iPhone] Videos from Frames for Quantitative Evaluations”.

Reviewer comment (Writing):
What is "hundreds of GPU hours per video" in the introduction referring to? It sounds as if existing per-scene methods take hundreds of hours per video (incl. appearance optimization). But that's not the case with most monocular dynamic NeRFs, especially the recent fast methods that take minutes rather than hours.

Author response:
We apologize for overlooking this particular point and sincerely appreciate your friendly reminder. We have revised the pdf to rephrase “usually take hundreds of GPU hours” to “could take up to hundreds of GPU hours” in Fig. 1’s caption and the introduction.

Reviewer comment (Novelty):
The related work is covered very thoroughly. Only recent diffusion-based approaches for novel view synthesis could additionally be cited, e.g. Watson et al. Novel View Synthesis with Diffusion Models (ICLR '23).

Author response:
Thanks a lot for sharing, we added the missing citation to Sec. 2’s "Generalized static novel-view synthesis” paragraph.

Reviewer comment (Writing):
Please add a sentence to the main text that splat-/point-/mesh-based rendering of the point cloud is used to get the dynamic image rendering. The section on dynamic rendering feels incomplete currently, with some context/framing missing.

Author response:
Thanks a lot for the suggestion. We clarify in Sec. 3.3.1 of the updated pdf that we use either SoftSplat or point/mesh-based rendering.

Reviewer comment (Evaluation):
I would not call the results "high-quality" (e.g. caption of Figure 8). The videos are not high-quality.

Author response:
Thanks a lot for the suggestion. We changed the word to “reasonable” in the updated pdf.

Reviewer comment (Presentation):
I am confused by what's happening in Table 3. The final method is 3? And 5-1 and 5-2 differ in what way? Does the final method not use Sec. 3.3.2, while 5-1 and 5-2 do?

Author response:
To clarify, as stated in the title of “Methodology for Studying the Question”, Sec. 3 describes the framework we are using for our study.

Our best-performing ablation (row 3 in Tab. 3) does not involve temporal tracking discussed in Sec. 3.3.2, as we find that tracking isn’t as robust as necessary (row 5-1 and 5-2 in Tab. 3).

The difference between row 5-1 and 5-2 is the underlying tracking approach. Row 5-1 uses TAPIR while Row 5-2 uses CoTracker. Both are concurrent work and we think it is beneficial to try both to understand the limitations.

Reviewer comment (Presentation):
Presentation and clarity can be enhanced.

Author response:
Thanks a lot for your suggestions. We are more than happy to further polish the paper given specific suggestions.

Reviewer comment (Experiments):
A broader range of related works should be included in the comparisons.

Author response:
Thanks a lot for the suggestions. We are happy to add those missing references if specific pointers are given.

Reviewer comment (Writing):
Ambiguity about the role of consistent depth estimates in the final result.

Author response:
To clarify, we want to understand whether generalized dynamic novel view synthesis is possible based on state-of-the-art data priors. Based on our study, we find that we are unable to achieve a completely generalized approach. However, with consistent depth, we are able to get rid of scene-specific appearance optimization.

Reviewer comment (Evaluation):
Can the authors clarify the specific role and impact of consistent depth estimates in their method?

Author response:
To clarify, we want to understand whether generalized dynamic novel view synthesis is possible based on state-of-the-art data priors. Based on our study, we find that we are unable to achieve a completely generalized approach. However, with consistent depth, we are able to get rid of scene-specific appearance optimization.

Reviewer comment (Writing):
We find a pseudo-generalized … is possible -> We found that … is possible.

Author response:
We emphasize that we do not claim our approach is “generalized”. In contrast, we unambiguously highlight our findings: we were unable to achieve a completely generalized approach based on current state-of-the-art data priors (see our abstraction, introduction, and conclusion).

Meanwhile, as correctly mentioned in the review, consistent depth needs scene-specific optimization. Hence we can only achieve a **pseudo-generalized** approach which doesn’t need scene-specific **appearance optimization**. 

Importantly, we never state that we do not need appearance cues (note the difference between **appearance cues** and **appearance optimization**). In contrast, we think appearance cues are important and our analysis of GNT adaptation in Sec 3.2.1 is based on exploiting appearance cues.

Reviewer comment (Theory):
Authors have given explanations justifying the keywords such as generalized, scene-specific optimization, scene consistent depth, etc., used in the paper. Yet, it is rather weak as the approach itself relies on consistent depth estimates of a dynamic scene, which, in fact, is a very open problem and acceptable solutions generally rely on appearance cues and scene flow.

Author response:
We emphasize that we do not claim our approach is “generalized”. In contrast, we unambiguously highlight our findings: we were unable to achieve a completely generalized approach based on current state-of-the-art data priors (see our abstraction, introduction, and conclusion).

Meanwhile, as correctly mentioned in the review, consistent depth needs scene-specific optimization. Hence we can only achieve a **pseudo-generalized** approach which doesn’t need scene-specific **appearance optimization**. 

Importantly, we never state that we do not need appearance cues (note the difference between **appearance cues** and **appearance optimization**). In contrast, we think appearance cues are important and our analysis of GNT adaptation in Sec 3.2.1 is based on exploiting appearance cues.

Reviewer comment (Experiments):
Authors should clarify why MonoNeRF does not qualify the definition of pseudo-generalized approach, given that the paper mentions “it is unclear whether MonoNeRF is entirely generalizable and can remove scene-specific appearance optimization”. It is better to test and present clarity in the rebuttal phase.

Author response:
Sorry for the confusion. The word “unclear” reflects our belief that MonoNeRF does not fall in the “generalized” category, as it requires scene-specific optimization on test scenes. To clarify, we copy our explanation from Sec. 2:
> Notably, MonoNeRF only reports results for a) temporal extrapolation on training scenes; or b) adaptation to new scenes after fine-tuning. Both setups require scene-specific appearance optimization on test scenes.

We point to Fig. 1 and Fig. 5 in the MonoNeRF paper, which verify the need for appearance optimization: without scene-specific appearance optimization, results are not competitive.

Reviewer comment (Novelty):
The statement “we think it is possible to avoid scene-specific appearance optimization even if the monocular video input contains dynamic objects” is provided despite the paper relying on Varma et al. 2023 pretrained GNT which greatly benefits from appearance. Clarification is needed in the rebuttal as it is inconclusive how far the proposed methods benefit from Varma et al. 2023 work, given that the current method is aware of the dynamic subject mask. Hence, the contribution appears very little.

Author response:
As an approach proposed for static scenes, GNT cannot handle dynamic novel view synthesis (see row 0 in Tab. S4).

Our adaptation clearly improves the performance of novel view synthesis (see Tab. S4 row 3 vs. row 0 and Fig. S2).

In hindsight, our proposed adaptation for GNT in Sec. 3.2.1 is simple. Yet, we think it is a finding that’s beneficial to the community. As far as we know, there does not exist work that states how to adapt a pretrained generalized view synthesis approach designed for static scenes to dynamic scenes.

Reviewer comment (Writing):
Results are considerably lower in performance, leading to the conclusion that appearance is indeed an important cue for neural rendering. While it could take more time, appearance helps gain realism. It is unclear whether the research presented in the paper is about time optimization or towards photorealistic rendering of dynamic scenes. Clarification is needed.

Author response:
We want to emphasize that our contributions are to study and understand whether a generalized approach for dynamic novel view synthesis is possible nowadays. As a result, the following thought of the reviewer does not fully reflect the goal of our study:

> I am not sure whether the research presented in the paper is about time optimization or towards photorealistic rendering of dynamic scenes.

Without any doubt, with scene-specific appearance optimization we can obtain better results. But this isn’t our goal. As mentioned by reviewer YL5S: “the paper tries to solve a worthwhile problem that would be very impactful to many groups and companies (rerendering a dynamic video without per-video optimization)”. Further note the opinion from reviewer bWXg: this work is a “carefully executed study with good experiments” and “is informative for researchers in the field to see where things currently stand”.

Reviewer comment (Experiments):
There are missing experiments on outdoor dynamic scene datasets such as Cityscapes. The authors should evaluate results on this dataset and compare them with "Enhancing photorealism enhancement," TPAMI 2022.

Author response:
Thanks a lot for the suggestion. As our work focuses on dynamic novel view synthesis, we use the commonly used dataset in the field. Specifically, we evaluate on two datasets: NVIDIA Dynamic Scenes and DyCheck iPhone. According to DyCheck (Gao et al., 2022a), these two datasets are much more challenging than others in this field.