Paper: InsertNeRF: Instilling Generalizability into NeRF with HyperNet Modules (ICLR-2024)

Reviewer comment (Evaluation):
How about the computational cost and efficiency of the proposed method compared with other methods?

Author response:
Thank you for bringing this up. Computational cost and efficiency are crucial aspects in the experimental process. Therefore, we addressed this issue in Figure 4 of the initial draft. We employed bar charts to depict evaluation times and compared them with the state-of-the-art method, GNT (2023 ICLR). To further discuss this point, we additionally tested GeoNeRF(ECCV 2022) under Setting I, which evaluated at **24s under 3 inputs**, while **InsertNeRF took 11.3s**. It is evident that our approach demonstrates faster inference speeds across varying numbers of reference images, possibly attributed to our framework not to use the popular Transformer architecture. At the same time, thanks to the non-Transformer architecture, the computational cost of our approach is significantly reduced.

Reviewer comment (Reproducibility):
It is suggest to public their Source code for the readers to better reproduce the proposed method.

Author response:
We agree wholeheartedly with your comment. We promise to open-source the code after publication. Thank you for your suggestion.

Reviewer comment (Experiments):
In the experimental section we do not have experiments with the ShapeNet-based dataset (see pixelNeRF).

Author response:
Dear Reviewer pJTL,

Once again, thank you for your suggestions and assistance with our work. Regarding your points in Weaknesses-4 and Qusetion-9 about ShapeNet, while it hasn't been extensively addressed in recently works, we believe discussing this aspect is significative [1][2]. We've conducted additional experiments over the past few days:

| | Chairs |1view |Chairs  |2view|Cars  | 1view|Cars  |2view|
| -|- | -|- |- |- | -|- | -|
|Model    | PSNR      | SSIM |PSNR      | SSIM    |PSNR      | SSIM    |PSNR      | SSIM    |
| ENR|22.83     | -      |-      | - |22.26    | -      |-      | - |
| SRN|22.89     |0.89      |24.48      | 0.92 |  22.25  | 0.89      |24.84     |0.92 |
| ViewFormer[2] (ECCV 2022)|14.74      | 0.79     |17.20     |0.84     | 19.03     |0.83     |20.09     |0.85     |
| pixelNeRF[1] (CVPR 2021)|23.72     | 0.91     |26.20   |0.94     | 23.17     |0.90    |25.66 |0.94   |
|pixelNeRF+HyperNet modules|**24.51**| **0.92**|  **26.71**|**0.94**     | **24.18**    |**0.91**|**26.05**    |**0.95**|

We explore the performance of the HyperNet modules on ShapeNet under chairs and cars scenes. Our work primarily focuses on multi-views under settings I and II, making multi-layer dynamic-static aggregation strategy's validation unnecessary in few views. Therefore, we integrate the HyperNet modules into the original pixelNeRF, altering its training inputs accordingly. As shown in the table above, our modules exhibit significant improvements compared to pixelNeRF, especially in the 1-view setting. It's also evident that compared to NeRF Synthetic, LLFF, and DTU, InsertNeRF shows less improvement on ShapeNet. This might be due to the relatively simplistic appearance and geometry of ShapeNet-Scenes, and our work primarily focuses on multi-view settings as mentioned in [2].

***We have added the our experments to the Appendix. E. All changes are highlighted in BLUE font.***

[1] Yu A et al. "pixelnerf: Neural radiance fields from one or few images"

[2] Kulhánek J et al. "Viewformer: Nerf-free neural rendering from few images using transformer"

Reviewer comment (Experiments):
The model use hypenetwork so shoud be compared with hypenetworks based method like Points2NeRF or HyperNeRFGAN.

Author response:
**Q2.Compared with hypenetworks Points2NeRF or HyperNeRFGAN**  

Thank you for pointing this out. Points2NeRF focuses on training NeRF form point clouds, while HyperNeRFGAN attempts to generate a 3D-aware GAN. Points2NeRF and HyperNeRFGAN are all very forward-looking works, they directly incorporate HyperNetwork into NeRF. However, in the generalizable NeRF task, such idea is suboptimal, overlooking the characteristic of different attributes, such as volume density and color. This issue has been mentioned in the third paragraph of the Introduction section. Furthermore, they struggle to capture the relationship between the inputs (reference images) in the target's sampling process. Therefore, we propose two types of HyperNet module structures for $\mathcal{F}\_{geo}$ and $\mathcal{F}\_{app}$ and Sampling-aware Filter separately to mitigate the aforementioned two issues. 

Since Points2NeRF and HyperNeRFGAN differ from the generalized NeRF task we primarily focus on, for a fair comparison, we directly introduce HyperNetwork into NeRF (no including HyperNet modules and Sampling-aware Filters), as mentioned in Points2NeRF and HyperNeRFGAN. In fact, this experiment validating our motivation was conducted in the early experiments, as mentioned in the third paragraph of the Introduction section.
| | NeRF | Synthetic  | |  | LLFF  | |
| -| - |- |- |- |- |- |
|Model    | PSNR      | SSIM    | LPIPS     |PSNR      | SSIM    | LPIPS    |
|NeRF with HyperNetwork as Points2NeRF and HyperNeRFGAN |25.86      | 0.902      |0.081      |24.25     | 0.793     |0.177      |
|InsertNeRF|**27.57**| **0.936**|  **0.056**|**25.68**     | **0.861**    |**0.126**|

***To further reflect your comment, we have added the experiment in Table 2 and description in Introduction. As the experimental results indicate, the proposed HyperNet modules and Sampling-aware Filters significantly enhance the rendering performance of Points2NeRF and HyperNeRFGAN in GNeRF task, where HyperNetwork is directly introduced into NeRF. All changes are highlighted in BLUE font.***

Reviewer comment (Writing):
It is quite difficult to understand formula (3). F_view take as an input one point (x,d) or all points on the ray? F_sample take as an input one point (x,d) or all points on the ray?

Author response:
**Q5. It is quite difficult to understand formula (3). F_view take as an input one point (x,d) or all points on the ray? F_sample take as an input one point (x,d) or all points on the ray?**

We apologize for any inconvenience caused. Due to limited space, we simplify the description of existing generalizable NeRF processes in formula (3), which is consistent with most existing work. 

$$ \mathcal{F}\_{\text{sample}}\left(\left\\{\\mathcal{F}\_{\\text{view}}\left( \left\\{\mathbf F_n\left( \Pi_n(\mathbf r(t_i))\right)\right\\}_{n=1}^N\right)\right\\}\_{i=1}^K\right) \ \ \mapsto \ \ \mathbf (\mathbf c,\sigma)$$


In formula (3), $\mathcal{F}\_{\text{view}}$ takes **specific points from all reference images** $ \left \\{ \mathbf I\_{n}, \mathbf P\_n\right \\}\_{n=1}^N $ as input, where the specific points are obtained by projecting **One Sampled Point** $ \mathbf r(t_i)$ on the target rays onto the reference features. Specifically, given a point $ \mathbf r(t_i)$ on the target ray, formula (3) projects $ \mathbf r(t_i)$ onto the $N$ reference features through the references and target camera extrinsics $\left \\{\mathbf P\_n\right \\}\_{n=1}^N $ and $\mathbf P\_T $, obtaining the relevant positions $\left \\{\mathbf z\_n\right\\}\_{n=1}^N $ in reference features. Finally, formula (3) extracts the corresponding features $\left \\{\mathbf f_n\right \\}\_{n=1}^N $ from the reference features at $\left \\{\mathbf z_n\right \\}\_{n=1}^N $. The above process can be represented as $\left \\{\mathbf F\_n\left( \Pi\_n(\mathbf r(t\_i))\right)\right \\}\_{n=1}^N $. These features $\left \\{\mathbf f\_n\right \\}\_{n=1}^N $ from different viewpoints are used as inputs for $\mathcal{F}\_{\text{view}}$.  


After respectively processing all points $ \left\\{ \mathbf r(t\_i)\right\\}\_{i=1}^K$ along the target ray by the $\mathcal{F}\_{\text{view}}$, $\mathcal{F}\_{\text{sample}}$ takes the features obtained for each point on the ray as inputs to obtain the final rendering color. Many existing generalizable NeRF follow a similar process as formula (3). Similar representation has also been adopted by a substantial number of existing works, such as GNT[4], NeuRay[2] and GeoNeRF[5].

Reviewer comment (Writing):
It is quite difficult to understand formula (4). it looks like F_sample was change to NyperNet. What is an input to Hypernetwork and what is an input to F_sample? Such formula should be describe more carefully.

Author response:
**Q6. It is quite difficult to understand formula (4). it looks like F_sample was change to NyperNet. What is an input to Hypernetwork and what is an input to F_sample? Such formula should be describe more carefully.**

$$\mathcal{F}(\mathbf x, \mathbf d ; \mathbf \Theta, \mathbf \Omega_T) \ \ \mapsto \ \  \left ( \mathbf c, w \right )$$

$$ \ \mathbf \Omega_T = \text{HyperNet} \left ( \left\\{\mathcal{F}\_{\text{view}}\left(\left\\{\mathbf F\_n\left( \Pi\_n(\mathbf r(t\_i)) \right)\right\\}\_{n=1}^N\right)\right\\}\_{i=1}^K\right )$$


In formula (4), similar to your understanding, our process involves replacing $\mathcal{F}\_{\text{sample}}$ with HyperNet. For $\mathcal{F}\_{\text{sample}}$, it predicts the color values of rays by aggregating features from sampled points along the rays, replacing traditional volume rendering, which is typically implemented using Transformers. For HyperNet, we use scene features from $\mathcal{F}\_{\text{view}}$ to predict the parameters (or weights) $\mathbf \Omega\_T$ in the vanilla NeRF framework. And then, by incorporating the new parameters $\mathbf \Omega\_T$, we have the ability to achieve generalization following the original NeRF framework and volume rendering. This is the primary distinction between InsertNeRF and existing methods.

[2] Liu Y et al. "Neural rays for occlusion-aware image-based rendering" CVPR2022

[4]: Wang P et al. "IS ATTENTION ALL THAT NERF NEEDS?" ICLR 2023

[5]: Johari M M et al. "Geonerf: Generalizing nerf with geometry priors" CVPR2022

Reviewer comment (Presentation):
Figure 2 is completely unclear.

Author response:
**Q7. Figure 2 is completely unclear.**

We apologize for any inconvenience caused. **We have redesigned Figure 2 in the new version.** Due to space constraints, we omitted some descriptions of the entire pipeline and Fig.2 in main text, which may cause confusion about Fig.2. In the initial draft's Appendix C.2, we described the entire network's training and testing process using pseudocode, as Algorithm 1 and 2. Next, allow me to provide a detailed overview of the entire network process based on Fig. 2 ,

We depict the entire process of InsertNeRF in the Fig.2(a). In contrast to vanilla NeRF, we introduce the plug-and-play HyperNet modules. Next, we will provide a detailed description of how to construct the HyperNet modules.

1): Firstly, training scenes are sampled from the training dataset. And reference as well as target images, along with their corresponding camera parameters, are extracted from the sampled scenes. This part is described in the second line of the Algorithm 1 and the leftmost part of Fig. 2 (Novel Target View $\mathbf P_T$ and $N$ Reference Images).


$$\textbf{Sample:} \left \\{ \left \\{ \mathbf I\_{T}, \mathbf P\_T\right \\}, \left \\{ \mathbf I\_{n}, \mathbf P\_n\right \\}\_{n=1}^N \right \\} \leftarrow \mathcal{D}\_\text{TrScene}\leftarrow \mathcal{D}\_\text{Train}.$$


2): Based on $\mathbf P_T$, sample the target rays to obtain sample points $\mathbf r(t_i)$ as vanilla NeRF. Utilize U-Net to extract features $\mathbf F_{l,n}$ from the $N$ reference images as described in formula (5). Differing from existing works, the additional $\mathcal{L}\_{\text{backbone}}$ is employed for providing auxiliary supervision in this process as described in Sec. 3.3. This part is described in the Fig. 2 (b).

3): According to the sampling position $\mathbf r(t_i)$, camera parameters$\left \\{ \left \\{\mathbf P\_n\right \\}\_{n=1}^N, \mathbf P\_T\right \\}$, and reference features $\mathbf F\_{l,n}$, the features of each target sampled point can be extracted from $N$ reference features to get $\left \\{\mathbf f\_n\right \\}\_{n=1}^N $ as formula (4)'s $\mathbf F\_n()$. This part is described in the second half of Fig.2 (c).

4): Then the $\mathcal{F}\_{\text{view}}$ is employed to aggregate features from $N$ reference features. In $\mathcal{F}\_{\text{view}}$, the sampling features $\left \\{\mathbf f\_n\right \\}\_{n=1}^N $ are used as inputs to generate the scene features for each sampling position as described in Sec.3.2.2. 
This part is described in the top half of Fig.2 (c).

5): Then these scene features are introduced to Sampling-aware Filter to generate the Weights \& Bias (green and yellow block for $\mathcal{F}\_{geo}$ and $\mathcal{F}\_{app}$) and DFILM (pink block) for the HyperNet modules as described in Sec. 3.2.3. This part is described in Fig.2 (d).

6): Then multiple Weights, Bias, and DFILM are used to construct two types of HyperNet modules in  $\mathcal{F}\_{geo}$ and $\mathcal{F}\_{app}$, as Fig. 6. Finally, these HyperNet modules are inserted into the original NeRF-based framework for $\mathbf \Omega\_T$, where the original NeRF-based framework is represented by blue dashed lines. This part is described in Sec. 3.2.3, the fourth line of the Algorithm 1 and the top half of Fig.2 (a).

7): Finally, based on the sampling points along the target ray, the predicted colors and weights of the sampled points are obtained through the NeRF framework augmented with an attached HyperNet modules as $\mathcal{F}(\mathbf x, \mathbf d ; \mathbf \Theta, \mathbf \Omega\_T) \ \ \mapsto \ \  \left ( \mathbf c, w \right )$ in formula (4). The rendered colors are obtained through volume rendering, and the process is supervised using Photometric loss. Apart from the added HyperNet modules, this process remains the same as the vanilla NeRF. This part is described in Sec. 3.3 and the second half Fig.2 (a).

***To further reflect your comment, we have redesigned Figure 2 and updated the description for Figure 2. All changes are highlighted in BLUE font.***

Reviewer comment (Writing):
Authors write, "While this method can be adapted to a variety of NeRF-based systems (Sec. 4.4), we focus on its application on the vanilla NeRF in this section." and then directly built the method on formula (3) dedicated to Generalizable NeRF.

Author response:
**Q8. Authors write, "While this method can be adapted to a variety of NeRF-based systems (Sec. 4.4), we focus on its application on the vanilla NeRF in this section." and then directly built the method on formula (3) dedicated to Generalizable NeRF.**

We apologize for any inconvenience caused. In fact, our InsertNeRF is built based on $\mathcal{F}(\mathbf x, \mathbf d ; \mathbf \Theta) \ \ \mapsto \ \  \left ( \mathbf c, \sigma \right )$ (vanilla NeRF) as formula 1. With the aggregated scene features and HyperNet modules, InsertNeRF's entire process can be succinctly summarized as $\mathcal{F}(\mathbf x, \mathbf d ; \mathbf \Theta, \mathbf \Omega_T) \ \ \mapsto \ \  \left ( \mathbf c, w \right )$ in formula 4, where $ \mathbf \Omega_T$ represents the adaptive parameters form HyperNet modules. Note that formula 1 and 4 can be replaced by other NeRF-based frameworks, and we also endow them with generalizability, as demonstrated in Sec. 4.4.


We describe formula (3) merely to illustrate the pipeline of existing generalizable NeRF, which helps emphasize the distinctiveness of our method. Formula (3) and (4) share the same process regarding $\mathcal{F}\_{\text{view}}$, but the specific methods differ, as described in Sec. 3.2.2. For the generalizable NeRF task, we believe that extracting scene features from reference images, $\mathcal{F}\_{\text{view}}$, is necessary, aligning with the approach of most existing works, which prompts the network to understand the scene. However, "how to leverage scene features to build a generalizable NeRF after $\mathcal{F}\_{\text{view}}$" is a crucial aspect that needs exploration, which is also the main distinction between InsertNeRF and existing works as formula (4).

Reviewer comment (Experiments):
How difficult is to find parameters lambda_1 and lambda_2 in const function?

Author response:
**Q9. How difficult is to find parameters $\lambda\_1$ and $\lambda\_2$ in const function?**

 Thank you for your reminder and suggestions. We also believe that the values of $\lambda\_1$ and $\lambda\_2$ should be explained in the experimental details of the article. In our experiments, $\lambda\_1$ is set as 0.1 and $\lambda\_2$ is set as 1.

***We have added the description for $\lambda\_1$ and $\lambda\_2$ to the Appendix. A for experimental details. All changes are highlighted in BLUE font.***

Reviewer comment (Experiments):
In experimental section authors shud add experiments on ShapeNet data similar to pixelNeRF.

Author response:
Dear Reviewer pJTL,

Once again, thank you for your suggestions and assistance with our work. Regarding your points in Weaknesses-4 and Qusetion-9 about ShapeNet, while it hasn't been extensively addressed in recently works, we believe discussing this aspect is significative [1][2]. We've conducted additional experiments over the past few days:

| | Chairs |1view |Chairs  |2view|Cars  | 1view|Cars  |2view|
| -|- | -|- |- |- | -|- | -|
|Model    | PSNR      | SSIM |PSNR      | SSIM    |PSNR      | SSIM    |PSNR      | SSIM    |
| ENR|22.83     | -      |-      | - |22.26    | -      |-      | - |
| SRN|22.89     |0.89      |24.48      | 0.92 |  22.25  | 0.89      |24.84     |0.92 |
| ViewFormer[2] (ECCV 2022)|14.74      | 0.79     |17.20     |0.84     | 19.03     |0.83     |20.09     |0.85     |
| pixelNeRF[1] (CVPR 2021)|23.72     | 0.91     |26.20   |0.94     | 23.17     |0.90    |25.66 |0.94   |
|pixelNeRF+HyperNet modules|**24.51**| **0.92**|  **26.71**|**0.94**     | **24.18**    |**0.91**|**26.05**    |**0.95**|

We explore the performance of the HyperNet modules on ShapeNet under chairs and cars scenes. Our work primarily focuses on multi-views under settings I and II, making multi-layer dynamic-static aggregation strategy's validation unnecessary in few views. Therefore, we integrate the HyperNet modules into the original pixelNeRF, altering its training inputs accordingly. As shown in the table above, our modules exhibit significant improvements compared to pixelNeRF, especially in the 1-view setting. It's also evident that compared to NeRF Synthetic, LLFF, and DTU, InsertNeRF shows less improvement on ShapeNet. This might be due to the relatively simplistic appearance and geometry of ShapeNet-Scenes, and our work primarily focuses on multi-view settings as mentioned in [2].

***We have added the our experments to the Appendix. E. All changes are highlighted in BLUE font.***

[1] Yu A et al. "pixelnerf: Neural radiance fields from one or few images"

[2] Kulhánek J et al. "Viewformer: Nerf-free neural rendering from few images using transformer"