Paper: Mask-Based Modeling for Neural Radiance Fields (ICLR-2024)

Reviewer comment (Experiments):
Concerning the different masking strategies, i.e., RGB mask and two feature masks, it is curious whether ray-level or view-level has the more considerable performance gain or whether each masking strategy prefers a different level of masking.

Author response:
**Answer 1**:  In our experiments, we find that the combination of using ray-level as well as view-level masking operation brings the most significant improvements, no matter for our final proposal or for the other three masking variants (i.e., RGB mask and two feature masks). We reckon that masking along the two dimensions together will better exploit the inner correlations among points and across views, in which case the 3D scene prior knowledge will be learnt more adequately. We already search for the optimal masking level (i.e., masking ratio) for our final proposal as well as the three masking variants in the ablation study, and report the best for each in *Table 4*, which shows the superiority of our final proposal.

Reviewer comment (Reproducibility):
In the experiments, NeuRay is adopted as the MLP-based network. But, to my knowledge, the MLP-based NeRF processes each sampled 3D point independently. It means that it is not possible to utilize the correlations among point-to-point and across view-to-view and learn a 3D scene prior knowledge. Are there some architecture modifications for NeuRay?

Author response:
**Answer 1**:  We do not make any architecture modifications for NeuRay. Naturally, there exist inherent correlations among the sampled points and across the reference views for generalizable NeRFs, regardless of different network architectures. Our mask-based pretraining target is to well-exploit such correlations for improving NeRF's generalizability. Even for the MLP-based NeRFs, such correlations, as the 3D scene prior knowledge, can still be learnt through our proposed MRVM and encapsulated in the implicit MLP networks as well.  That's because MLP networks also have some kind of inner correlations among different input signals. Our  experiments demonstrate the applicability of our proposed MRVM for both MLP-based and Transformer-based backbones.

Reviewer comment (Experiments):
To validate the influence of different masking strategies, the authors conduct experiments with three masking variants. There is missing one variant: taking the same masking strategy as described in Section 3.2 but minimize the $\mathcal L_2$ distance between $z_i^f$ and $z_i^c$. Note that normalizing the vector to unit-length before calculating the distance. This additional experiment can validate the effectiveness of the BYOL-like module.

Author response:
**Answer 2**:  Thanks for your valuable suggestion, we add the ablation as the reviewer suggested and refer as **NeRFormer+MRVM (w/o BYOL)** in the table below:

| Method    | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
| --------- | ---- | ---- | ----- |
| NeRFormer | 27.58 | 0.920 | 0.091 |
| NeRFormer+MRVM (original)| **29.25** | **0.942** |**0.060**|
| NeRFormer+MRVM (w/o BYOL)| 28.09 | 0.924 |0.086|

It validates the effectiveness of the BYOL-like module in our final proposal. Intuitively, as many previous contrastive-learning methods [1,2] pointed out,  the usage of projector and predictor avoid the model collapsing into a trivial solution, that is, the two branches learn similar but meaningless representations, which impairs the effectiveness of pretraining.

[1] Jean-Bastien Grill, Florian Strub, Florent Altche, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. 

[2] Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton   A Simple Framework for Contrastive Learning of Visual Representations.

Reviewer comment (Reproducibility):
As shown in Weaknesses, there are some missing details about the architecture of the used backbones. It is better to provide these details in the paper.

Author response:
**Answer 3**:  Thanks for your valuable suggestion, we'll provide more details about the architectures of the used backbones in our revision to make them more clear. We'll put them in the revised appendix.

Reviewer comment (Experiments):
There are no major weaknesses, in my opinion, design choices are quite well motivated, and quantitative results do indicate improvements. I think it might be interesting to investigate the few-shot results in more detail (i.e. show that the pretraining objective now enables tasks that were previously not looked at by generalizable nerfs).

Author response:
Thanks for your valuable suggestion. We add our proposed Masked Ray and View Modeling to a well-known generalizable NeRF baseline GNT [1], and compare under the same few-shot settings as the reviewer mentioned:

| Models | LLFF (3-shot)       | LLFF (6-shot)     | Synthetic (6-shot) | Synthetic (12-shot) |
| ------ | ------------------- | ----------------- | ------------------ | ------------------- |
| GNT    | 19.58  0.653  0.279 | 22.36 0.766 0.189 | 22.39 0.856 0.139  | 25.25 0.901 0.088   |
| GNT+MRVM   | **20.88**  **0.672**  **0.257** | **23.54** **0.777** **0.175** | **23.52** **0.869** **0.120** | **26.10** **0.911** **0.079** |

The numerical numbers are PSNR, SSIM and LPIPS from left to right respectively. It shows our proposed MRVＭ still benefits under the few-shot settings.

[1] Mukund Varma T, Peihao Wang, Xuxi Chen, Tianlong Chen, Subhashini Venugopalan, Zhangyang Wang. Is attention all that nerf needs?

Reviewer comment (Experiments):
For example, since the model is trained on masked inputs - it must additionally be able to perform quite well on cross-scene generalization in a few-shot setting (w/o any finetuning). Even in the case of finetuning, [1] presents some results on few-shot finetuning with as little as 3-6 images on LLFF, and 6-12 on NeRF Synthetic. It might be worth comparing against the same.

Author response:
Thanks for your valuable suggestion. We add our proposed Masked Ray and View Modeling to a well-known generalizable NeRF baseline GNT [1], and compare under the same few-shot settings as the reviewer mentioned:

| Models | LLFF (3-shot)       | LLFF (6-shot)     | Synthetic (6-shot) | Synthetic (12-shot) |
| ------ | ------------------- | ----------------- | ------------------ | ------------------- |
| GNT    | 19.58  0.653  0.279 | 22.36 0.766 0.189 | 22.39 0.856 0.139  | 25.25 0.901 0.088   |
| GNT+MRVM   | **20.88**  **0.672**  **0.257** | **23.54** **0.777** **0.175** | **23.52** **0.869** **0.120** | **26.10** **0.911** **0.079** |

The numerical numbers are PSNR, SSIM and LPIPS from left to right respectively. It shows our proposed MRVＭ still benefits under the few-shot settings.

[1] Mukund Varma T, Peihao Wang, Xuxi Chen, Tianlong Chen, Subhashini Venugopalan, Zhangyang Wang. Is attention all that nerf needs?

Reviewer comment (Experiments):
If I understand correctly when applying to the proposed method, an additional fine branch would be added. I wonder if this will cause double parameters and inference costs.

Author response:
Most of the Neural Radiance Field works adopt coarse-to-fine sampling strategy, using two independent network modules handling sampled points at coarse and fine stage respectively. We reuse the two network module branches as target and online branch, aiming to introduce the fewest additional parameters. Therefore, the only additional parameters introduced by our MRVM are the *light-weight projector and predictor*, which brings negligible additional computational overhead (only +0.2% parameters compared to the baseline model). Please refer to *Table 4* for more details. Moreover, since the projector and predictor are discarded at inference time, the inference speed is consistent with the original baseline methods.

Reviewer comment (Presentation):
The explanation of different masking strategies in the ablation study of the main paper is not easy to understand. I see the details are well illustrated in the supplementary material. I recommend claiming these illustrations in the main paper as well.

Author response:
Thanks for your valuable suggestion. Due to the page limits, we are only able to describe the three masking variants briefly in the main paper, we'll try to make them more precise in our revision.

Reviewer comment (Experiments):
The two main additions - of EMA and masking are tested together. There is a lack in understanding how dependent they are. Firstly, if one could be used without the other. And if so, secondly, what are the individual contributions.

Author response:
We'd like to firstly explain our approach of using EMA and masking together. As we mentioned in Section 3.2, apart from the vanilla rendering loss, we use an additional feature-alignment loss $L_{mrvm}$ in latent space following the successful formulation introduced by BYOL [1]. This requires the gradient to be manually stopped at the target (coarse) branch, as we illustrated in Figure 1. Therefore the projector in target branch can not be updated with gradient decent algorithm, that's why we use EMA to update its parameters using Equation 5. If only using masking without EMA, as we understand, is to make a copy of the parameters of the projector from online branch to target branch every iteration. We find this will make the mask-based pretraining stage instable based on our experiments $-$ the loss term $L_{mrvm}$ fluctuates and is not easy to converge.

[1] Jean-Bastien Grill, Florian Strub, Florent Altche, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning.

Reviewer comment (Reproducibility):
Sampling is not well specified. The extra 'fine' samples - whether they are they taken, as in the classical NeRF, according to an initial estimation of density from the coarse sample. And in this respect - shouldn't the masking prediction be focused at the more important (close-to-surface) high density locations, rather than at the coarse sampling?

Author response:
As the reviewer understands, the extra fine samples are taken, as in the classical NeRF, according to an initial estimation of density from the coarse samples. We agree that the close-to-surface points sampled at fine stage are more important, but we do not treat them as prediction target due to the fact that : 1) We can not predict the latent embeddings of these points directly from the coarse branch since they are newly sampled at fine stage and do not exist at the coarse stage. That makes it impractical to treat coarse branch as online and fine branch as target. 2)  We do perform an ablation study of making a copy of the overall fine-branch to serve as target and update via moving average, dubbed as **Feat mask^2^** in Section 4.3. However we find this variant, with more additional parameters, exhibits inferior performance compared to our final proposal. Please refer to *Table 4*  for quantitative comparisons and refer to the **Discussion** part in **Section 3.2** for more detailed analysis.

Reviewer comment (Experiments):
Impact on training time is not clear. Even though training time and procedures are very clearly presented in the appendix, I am missing what is the addition in comparison to the baseline, and which components are responsible for it?

Author response:
The generalizable NeRF adopts the training strategy of cross-scene pretraining followed by per-scene finetuning. It is firstly pretrained across various scenes. After pretraining, the model can either be tested directly on a novel scene or further finetuned on that specific scene. Our proposed Masked Ray and View Modeling serves as an auxiliary task when performing cross-scene pretraining, which requires an additional finetuning stage without masking to maintain the consistency between training and testing. Overall it takes about 30% additional training time compared to the baselines.

Reviewer comment (Presentation):
Clarity [minor]: (i) I would suggest improving Figure 1 to include some of the notations (g, h, etc') ; (ii) What are the blue dashed arrows in Figure 2 supposed to represent?

Author response:
The blue dashed arrows represent the message-passing along sampled points on the ray. Thanks for your valuable suggestion, we'll include such notations in our revision.

Reviewer comment (Experiments):
Have you tried comparing to per-scene NeRFs?

Author response:
The **per-scene finetuning** results as shown in Table 3 have already be comparable to several per-scene NeRF methods, and it shrinks the gaps compared to SOTA performance on per-scene NeRFs.

Reviewer comment (Experiments):
Could you provide an ablation showing the contribution of each of the components? In particular - applying only one of EMA / Masking?

Author response:
The ablation for different mask-based pretraining strategies are presented in *Table 4*. We find only applying masking will make the mask-based pretraining instable. As mentioned in **Answer 1**, if copying the parameters from online projector to target ones every iteration, we observe fluctuation for the loss term $L_{mrvm}$, making it not easy to converge in our experiments. 

We do not try to only apply EMA without masking since we propose a mask-then-predict pretraining task, simply abandoning masking operation is irrelevant to our proposal. Besides, the rest of the network parameters except the target projector are updated using normal gradient-decent algorithm, with no need for the EMA parameter updating strategy.

Reviewer comment (Experiments):
How sensitive is the choice of \lambda, the weight of the mrvm-loss?

Author response:
Experimentally we find $\lambda$ suitable between 0.1~0.5.

Reviewer comment (Theory):
Why do the authors use masked-based modeling to learn high-level global information? What are the advantages and necessity of using masked-based modeling?

Author response:
In our **Abstract**, **Introduction** and **Method 3.1** Section, we have emphasized the motivation and benefits of using mask-based modeling for improving the generalizability of Neural Radiance Field. A mask-then-predict pretraining objective has been widely applied in MLM [1] and MIM [2] field for learning a high-level global representation. As in NeRFs, we reckon that the learnt 3D scene global information, which contains the correlations among sampled points and across reference views, endowing the model with better capacity to effectively generalize to novel scenes with limited observations.

[1]  Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding.  

[2]  Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollar, and Ross Girshick. Masked autoencoders are scalable vision learners.

Reviewer comment (Experiments):
The authors should add the masked modeling design to more generalizable NeRF models to demonstrate its wide applicability. In addition, the authors should provide qualitative and quantitative comparisons with SOTA generalizable NeRF models, such as FreeNeRF[1] and SparseNeRF[2].

Author response:
We add our mask-based modeling strategy to another well-known generalizable NeRF baseline GNT [3], under the cross-scene generalization setting, using LLFF and NeRF Synthetic as evaluation sets. The quantitative comparisons are shown below:

| Models | LLFF (PSNR$\uparrow$) | LLFF (SSIM$\uparrow$) | LLFF (LPIPS$\downarrow$) | Synthetic (PSNR$\uparrow$) | Synthetic (SSIM$\uparrow$) | Synthetic (LPIPS$\downarrow$) |
| :----- | :-----------------: | :---------------: | :----------------: | :-----------------: |:-----------------:|:-----------------:|
| GNT    | 25.86 | 0.867 | 0.116 | 27.29 | 0.937 | 0.056 |
| GNT+MRVM   | **26.25** | **0.873** | **0.110** | **27.78** | **0.942** | **0.052** |

Our proposed MRVM proves to be effective in improving the generalizability for GNT as well.

[3]  Mukund Varma T, Peihao Wang, Xuxi Chen, Tianlong Chen, Subhashini Venugopalan, Zhangyang Wang. Is attention all that nerf needs?  

Besides, we'd like to clarify that *FreeNeRF and SparseNeRF as the reviewer mentioned are **not** generalizable NeRF methods* which may not be compared to our proposed approach targeting on generalizable NeRFs directly. These two methods focus on solving the few-shot reconstruction problem especially designed for *a single particular scene*, while generalizable NeRF methods use a shared network to *represent various different scenes*.

Reviewer comment (Experiments):
Can this method only handle object-centric and facing-forward scenes? In addition, can the authors provide results on the tank-and-temple datasets, where the scenes have a wider range of views and complex backgrounds.

Author response:
We currently perform experiments on the object-centric and forward-facing scenes because the previous generalizable NeRF baselines are mainly evaluated on these two kinds of scenes. For more complicated scenes like in-the-wild unbounded 360 degree scenes as in the tank-and-temple dataset, or the large-scale complex indoor scenes, most SOTA methods in the NeRF community still learn an independent NeRF per scene for satisfying rendering quality. However, we see no obstacles that may hinder our proposed method in applying for other types of scenes, and we'll leave these as a future work.

Reviewer comment (Evaluation):
As this method adds an additional branch to the baseline network, it is not as computationally efficient as the baseline methods. The computational overhead is not well discussed in the paper.

Author response:
Most of the Neural Radiance Field works adopt coarse-to-fine sampling strategy, using two independent network modules handling sampled points at coarse and fine stage respectively. We reuse the two network module branches as target and online branch, aiming to introduce the fewest additional parameters. Therefore, the only additional parameters introduced by our MRVM are the *light-weight projector and predictor*, which brings negligible additional computational overhead (only +0.2% parameters compared to the baseline model). Please refer to *Table 4* for more details. Moreover, since the projector and predictor are discarded at inference time, the inference speed is consistent with the original baseline methods.

Reviewer comment (Novelty):
Baseline generalizable NeRFs are already doing "pretraining". I think this paper's contribution is to **improve** the pretraining by adding a self-supervised masked prediction task to the baseline methods. The author may need to clarify this point in the paper.

Author response:
Yes we agree with the reviewer's comment. Thanks for the valuable suggestion and we will clarify this point in our revision.

Reviewer comment (Evaluation):
Can the authors provide more details on the computational efficiency of MRVM-NeRF?

Author response:
Please refer to **Answer 1**.

Reviewer comment (Experiments):
To predict coarse latent features, we take the masked latent feature as input to the prediction network (Equation 4). Can the $Pred^f$ function take additional inputs besides the latent feature? E.g., geometrical

Author response:
Theoretically, $Pred^f$ can take additional geometrical inputs like position **$x$** and view direction **$d$** by modifying the predictor network structure. It's worth exploring whether this operation will bring greater performance gain in the future work. Actually, the geometrical information has already been introduced at the beginning of the generalizable NeRF framework. In Section 3.1, the input latent embedding $h_i^j$ has already contained the information of position $x$ and direction $d$ $-$ in the way that $x$ and $d$ are projected into a latent embedding after positional encoding, which are then added to $f_j^j$ , $c_i^j$ to produce input vector $h_i^j$. We apologize for the lack of these details and will clarify them in our revision.