Paper: GML-NeRF: Gate-guided Mutual Learning Framework for Neural Rendering (ICLR-2024)

Reviewer comment (Theory):
One of the motivations of this paper is that complex scenes need more network parameters and the parameters need to be allocated in the ray dimension. In the network design, the hash table is shared. The sub-NeRFs only contain 2 small decoders. Since in Instant-NGP, the decoder only contains very few parameters than the hash table (no more than 1%), it is unclear to me why the proposed method gets very good quality improvement with a minor increase of parameters.

Author response:
Thank the reviewer for the insightful question. We conduct an ablation study in the response to reviewer MyQ8 (R1-Q3), and **the results highlight the effect of independent MLP decoders rather than feature grid**. Within the hybrid representation, the explicit part (feature grid) encodes the features of 3D spatial points, while the implicit representation encodes the ray information. As our motivation is to consider the different visibility of a target region to different views, using multiple MLP encoders with different ray specialties is reasonable and effective. Although the number of learnable parameters hardly increases, GML-NeRF achieves a more optimal capacity allocation in the ray dimension, which helps to increase the model's generalization ability. Additionally, as different rays may pass through the same region of 3D space, weight sharing for the feature grid helps to facilitate training.

Reviewer comment (Reproducibility):
For Table 3 of Importance of the ray-level allocation, when using "fusing multi-NeRFs’ outputs in the point dimension", it is unclear how the point outputs are fused. What kind of gate is used for point fusion? Is it a point-based or ray-based gate?

Author response:
(1) Ablation Study of Fusion Dimension (Table 3): In this ablation study, we explore the superiority of GMl-NeRF over the point-based multi-NeRF method. The point-based method adopts a model structure similar to GML-NeRF, incorporating a shared feature grid and a soft gate module. The difference is that the soft gate module encodes the 3D points' data and outputs a gating score for each point. The fusion process is completed on the point dimension, followed by volume rendering.

Reviewer comment (Experiments):
Since this method learns pixel-level ray allocation, this method needs to directly compare the fixed pixel-level ray allocation methods. The Mega-NeRF actually allocates rays across different sub-NeRFs although the ray-allocation is determined by 3D points. The sub-NeRFs in the Mega-NeRF are trained independently and the results are interpolated by 3D distance. The hash tables may not be shared in Mega-NeRF. However, it can still be compared by setting the proposed method as not sharing hash tables. Since the image-level Block-NeRF is compared, I think the pixel-level Mega-NeRF is also worth comparison.

Author response:
Mega-NeRF allocates rays by partitioning the intersecting points between rays and scenes. Such a method is suitable for drone scenes, where the top-down perspective allows for the approximation of ray-scene intersections by intersecting with a set horizontal plane.

However, in unstructured scenes captured by free trajectories, the intersecting points between rays and scenes cannot be determined before the training is completed, limiting the applicability of Mega-NeRF to such scenes. In contrast, GML-NeRF's learnable allocation makes it versatile and applicable to various scenes.

Since there is no reasonable implementation to determine the ray intersections before training, we adopt an alternative implementation, which employs a clustering algorithm to divide rays directly based on their origins and directions. The implementation details are demonstrated in the Appendix E. To ensure a fair comparison, the model structure of Mega-NGP is the same as the one in GML-NeRF, following the implementation of Block-NGP. We conduct a comprehensive evaluation across all datasets and add the experimental results in the Appendix E. Mega-NGP yields similar results to Block-NGP, which is less effective than our GML-NeRF.

| Method   | Metric | TAT    | 360v2  | Free Dataset | ScanNet |
|----------|:------:|:------:|:------:|:------------:|:-------:|
| Mega-NGP | PSNR   | 20.843 | 27.482 | 25.855       | 28.100  |
|          | SSIM   | 0.659  | 0.761  | 0.696        | 0.786   |
|          | LPIPS  | 0.415  | 0.311  | 0.332        | 0.437   |
| GML-NeRF | PSNR   | 21.708 | 27.870 | 26.449       | **28.805**  |
|          | SSIM   | 0.672  | 0.769  | 0.719        | **0.796**   |
|          | LPIPS  | 0.399  | 0.298  | 0.285        | **0.424**   |

Reviewer comment (Experiments):
In Table 2, what is the accuracy if we train Uniform fusion with depth mutual loss?

Author response:
Thank the reviewer for the advice. We add the results of training with uniform fusion and depth mutual loss in the Appendix D. Without gate-guided fusion, sub-NeRFs lose their ray specialties, resulting in an ineffective improvement in rendering quality. Additionally, we also find that depth mutual learning is not helpful in this case, possibly due to the low accuracy of the fused depth. 

| Method                 | Metric | M60    | Playground | Train  | Truck  | Avg    |
|------------------------|:------:|:------:|:----------:|:------:|:------:|:------:|
| uniform fusion w/o dml | PSNR   | 19.229 | 22.863     | 17.531 | 23.569 | 20.798 |
| uniform fusion w/ dml  | PSNR   | 19.005 | 22.766     | 17.532 | 23.513 | 20.704 |
| Ours.                            | PSNR   | 19.051 | 23.901     | 19.369 | 24.509 | 21.708 |

Reviewer comment (Experiments):
The results of other datasets without depth mutual loss are needed to better analyze the effects of gate and depth mutual loss.

Author response:
Thank the reviewer for the advice. Following the suggestion, we conduct the ablation studies of depth mutual learning on other datasets. The results are consistent with those presented in the main context of the paper, further proving the effect of depth mutual learning.

| Method   | 360v2  | Free Dataset | ScanNet |
|----------|:------:|:------------:|:-------:|
| w/o dml  | 27.385 | 26.443       | 28.776  |
| GML-NeRF | **27.871** | **26.449**       | **28.805**  |

Reviewer comment (Experiments):
What is the result if Switch-NGP uses all the sub-NeRF outputs weighted by gating scores? This can actually be seen as a learnable point-based allocation method aligning with the training setting of the proposed method.

Author response:
Actually, the point-level fusion in Table 3 is a combined version of Switch-NGP and soft gate module in GML-NeRF, in which the model uses all the sub-NeRF outputs weighted by gating scores and completes the fusion process in the point dimension. The results are shown below.

| Fusion-Dimension | PSNR   | SSIM  | LPIPS |
|------------------|:------:|:-----:|:-----:|
| Point-Level      | 20.796 | 0.661 | 0.413 |
| Ray-Level(Ours)  | **21.708** | **0.672** | **0.399** |

Reviewer comment (Experiments):
By the time of submission, I believe ZipNeRF paper was already out there and there was an open-source implementation of it. Gaussian Splatting was also available. Probably the author should compare against these two approaches too. For the following reasons: (a) ZipNeRF is built on Instant-NGP and is able to handle varying resolution training images which is more suitable than Instant-NGP to model relatively large (and unbounded) scenes; (b) Gaussian Splatting, by design, is able to handle the occlusion issue (mentioned by the author as a challenge in modeling complex scenes) as it's an explicit representation.

Author response:
**R2-Q1: Results of ZipNeRF and Gaussian Splatting**

Thank the reviewer for the advice regarding the comparison of GML-NeRF with the latest baselines. 

(1) We implement a ZipNeRF version of GML-NeRF, named GML-ZipNeRF, and evaluate the performance on the 360v2 dataset. The training settings are kept the same as the original paper, including the training iterations and batch size. We add the comparison results in the Appendix G, showing that integrated with GML-NeRF, ZipNeRF can also obtain performance gains. 

(2) Due to time constraints, we could not verify the effect of GML-NeRF upon the Gaussian Splatting. A possible implementation to integrate GML-NeRF and 3D Gaussian Splatting is constructing an ensemble of 3D Gaussian representations for independent rendering. Subsequent gate-guided fusion can be conducted in a similar way to this article. We leave it for future exploration.

| Method      | Metric | bicycle | bonsai | counter | garden | kitchen | room   | stump  | avg    |
|-------------|:------:|:-------:|:------:|:-------:|:------:|:-------:|:------:|:------:|:------:|
| ZipNeRF     | PSNR   | 21.019  | 33.052 | 25.982  | 24.33  | 32.843  | 34.777 | 25.406 | 28.201 |
|             | SSIM   | 0.616   | 0.962  | 0.852   | 0.758  | 0.971   | 0.951  | 0.722  | 0.833  |
| GML-ZipNeRF | PSNR   | 20.488  | 33.486 | 26.372  | 24.603 | 33.1197 | 35.795 | 25.581 | **28.492** |
|             | SSIM   | 0.601   | 0.965  | 0.859   | 0.765  | 0.9731  | 0.956  | 0.728  | **0.835**  |

As a multi-NeRF training framework, GML-NeRF is essentially orthogonal to the structure and training method of single-NeRF. In our work, for the benefit of training efficiency and its wide application, we build and validate GML-NeRF upon the Instant-NGP. Nevertheless, it can be integrated with other single-NeRF frameworks, including ZipNeRF and Gaussian Splatting, and so on.

Reviewer comment (Experiments):
I'm a bit skeptical about the claim in experiments that since MipNeRF360 is too slow, you compare against MipNeRF360 trained for one hour. Is the main claim of the paper about efficiency? I mean, you can use always bump up the number of GPUs in training to reduce the training time if that's a problem.

Author response:
**R2-Q2: Consideration of training efficiency**

Thank the reviewer for the important reminder. Considering the importance of training efficiency on the practicability of the NeRF training frameworks, we mainly choose the baselines with high training efficiency (NGP, DVGO, F2-NeRF, etc.) and other multi-NeRF training methods for comparison, while providing the results of other baselines as the extra reference (NeRF, MipNeRF360,...). The consideration for training efficiency is also the motivation for adopting hybrid representations in GML-NeRF. Using the hybrid representation, ensemble size growth only increases the number of MLPs, which brings only a moderate parameter size increase. We highlight this consideration in the structure design part of the method section in the revision.

Additionally, although our GML-NeRF implemented upon Instant-NGP may not exhibit comparable performance to MipNeRF360 trained with more than one day, it acts as a flexible multi-NeRF training framework with the potential to be integrated with various single-NeRF frameworks (NeRF, MipNeRF, etc). As the above experimental results show, implemented upon SOTA single-NeRF method ZipNeRF, an NGP-version of MipNeRF360, GML-ZipNeRF shows better rendering quality, which further proves the effectiveness of the proposed method.

Reviewer comment (Experiments):
It seems, by the end of the day, the author only uses two NeRFs (K=2) in their framework, though in the abstract the author claimed that they construct "an ensemble of sub-NeRFs". Does the method only work with K=2? If it works in more general settings with K > 2, the author should demonstrate that. Does having more sub-NeRFs help improve the rendering quality?

Author response:
**R2-Q3: Results of more sub-NeRFs**

We adopt the k=2 setting by default mainly for the sake of experiment convenience, given its lower memory footprint in training. Nevertheless, GML-NeRF can consistently obtain performance gains as the number of sub-NeRFs increases, as shown in Figure-5. 

To provide a more comprehensive demonstration of this point, we provide per-scene results of GML-NeRF with more sub-NeRFs on the ScanNet dataset in the Appendix F. As the results show, GML-NeRF with 4x sub-NeRFs performs better than the one with 2x sub-NeRFs across all scenes.

| Method    | Metric | 0046_00 | 0276_00 | 0515_00 | 0673_04 | Avg    |
|---------|:------:|:-------:|:-------:|:-------:|:-------:|:------:|
| GML-size2 | PSNR   | 29.440  | 30.871  | 29.149  | 25.759  | 28.805 |
| GML-size3 | PSNR   | 29.878  | 31.242  | 29.47   | 25.944  | 29.134 |
| GML-size4 | PSNR   | 30.018  | 31.31   | 29.679  | 26.063  | **29.268** |

Reviewer comment (Experiments):
In Figure 5, the blue curve scales the width of a NeRF. Which NeRF's width is scaled? Is it any of the baseline NeRFs shown in Table 1? And on what dataset the experiment is conducted? Is the PSNR performance averaged across one scene, multiple scenes?

Author response:
**R2-Q4: Demonstration of scalability study and the responding figure**

Thank the reviewer for pointing out several issues and questions regarding Figure 5. The feedback is important for refining the paper, and we would like to address the concerns below.

(1) Scalability results clarification: We acknowledge the omission that we only report the scalability results on scene 0046 of the ScanNet dataset rather than the average PSNR result of four scenes. We have modified it in the revision. 

(2) Hidden dimension scaling for the blue curve: The blue curve scales the hidden dim of both the geometry MLP and the color MLP while maintaining the resolution of the hash feature grid. 

(3) GML-NeRF with two independent sub-NeRFs: This implementation is different from the one of GML-NeRF, in which the difference is whether the feature grid is shared among the sub-NeRFs. The number of parameters increases significantly within the design of unshared feature grids. We have slightly modified this figure in the revision to clarify this detail.

We hope that our modification and clarification can help!

Reviewer comment (Experiments):
Also in Figure 5, I think it makes more sense to compare against the following: You scale the resolution of the feature grid used in Instant-NGP, and show how the performance of Instant-NGP scales. Since I think the key to model large scale scenes well is to leverage locality, and in Instant-NGP, you need to bump up the feature grid's resolution to better leverage locality.

Author response:
**R2-Q5: Effect of scaling up the resolution of feature grid**

Following the reviewer's suggestion, we further conduct experiments to explore the impact of increasing the number of parameters in the feature grid, and the results are incorporated into Figure-5 in the latest revision. While maintaining default values for resolution and the number of hash levels, we opted to increase the hash table size to scale up the feature grid, consistent with the test methodology employed in the original NGP paper. As the results show, although the number of parameters increases significantly, the performance gains achieved by increasing the "log2_hash_table_size" from 19 to 24 are limited, which is consistent with the results in the original paper.

| Method    | 0046_00 | 0276_00 | 0515_00 | 0673_04 | Avg    | Params/MB |
|-----------|:-------:|:-------:|:-------:|:-------:|:------:|:---------:|
| NGP(t=19) | 28.71   | 30.08   | 28.182  | 25.324  | 28.074 | 24.960    |
| NGP(t=21) | 28.888  | 30.243  | 28.405  | 25.593  | 28.282 | 91.303    |
| NGP(t=24) | 29.133  | 30.459  | 28.569  | 25.753  | 28.478 | 615.59    |
| GML-NeRF  | **29.581**  | **30.884**  | **29.172**  | **25.844**  | **28.870** | **24.981**    |

Reviewer comment (Presentation):
Figure 5 is a bit confusing. Why "GML-NeRF with two independent sub-NeRFs" does not overlap with the "2x NeRFs"? I guess these are two different approaches? But I'm really confused here. The author could do a better job to clarify this.

Author response:
**R2-Q4: Demonstration of scalability study and the responding figure**

Thank the reviewer for pointing out several issues and questions regarding Figure 5. The feedback is important for refining the paper, and we would like to address the concerns below.

(1) Scalability results clarification: We acknowledge the omission that we only report the scalability results on scene 0046 of the ScanNet dataset rather than the average PSNR result of four scenes. We have modified it in the revision. 

(2) Hidden dimension scaling for the blue curve: The blue curve scales the hidden dim of both the geometry MLP and the color MLP while maintaining the resolution of the hash feature grid. 

(3) GML-NeRF with two independent sub-NeRFs: This implementation is different from the one of GML-NeRF, in which the difference is whether the feature grid is shared among the sub-NeRFs. The number of parameters increases significantly within the design of unshared feature grids. We have slightly modified this figure in the revision to clarify this detail.

We hope that our modification and clarification can help!

Reviewer comment (Reproducibility):
Eq. (3), G(r)'s output is a K-dimensional vector? Do the elements of the K-vector sum to one or not? I guess not, as there is no such constraint imposed through losses.

Author response:
**R2-Q6: G(r) in Equ-3**

G(r) is the K-dimensional vector representing the prediction confidence of each sub-NeRF for the ray r. Since we insert a softmax normalization layer at the end of the gate module, the elements of the G(r) sum to one.

Reviewer comment (Experiments):
I have some concerns regarding the validity of the depth regularization. The fused depth is estimated by weighting depths from different subnets (which is named as sub-depth for simplification). In this regard, the sub-depth with higher weighting score is supposed to be closer to the target depth, thus the difference between them should be smaller, and the one between the sub-depth with lower weighting score and the target is large. Simply summing them together may cause the training penalize more for the sub-depth with lower weighing score, which seems less proper. The authors need to give more discussion (may with ablation study) about it.

Author response:
Thank the reviewer for the questions. To empirically analyze the effect of depth mutual learning, we conduct an ablation study on the TAT dataset and add the results to the Appendix.D following the reviewer's suggestion. When using the average of the sub-depths as the target depth (all sub-NeRFs have equal regularization strength), the rendering quality will be slightly worse (PSNR 21.419 vs 21.708). This observation highlights the pivotal role of gate-guided depth mutual learning. In the GML-NeRF framework, an ensemble of sub-NeRFs is jointly optimized via the soft gate module, a crucial component enhancing rendering consistency. Within the design of the soft gate module, all the sub-NeRFs contribute to the same ray, necessitating accurate encoding of scene geometry. Meanwhile, the sub-NeRF with a lower gating score exhibits a relatively poor specialty for the ray, resulting in the inaccuracy of geometry encoding. A larger regularization force could have a better effect in such cases. By differently penalizing sub-depths based on their respective gating scores, our approach significantly contributes to accurate scene representation and improved rendering quality. 

| Method | Metric | M60 | Playground | Train | Truck | Avg |
|:---------------:|:---------------:|:------------:|:-------------------:|:--------------:|:--------------:|:------------:|
| Equal DML       | PSNR            | 18.929       | 23.108              | 19.012         | 24.625         | 21.419       |
| Ours            | PSNR            | 19.051       | 23.901              | 19.369         | 24.509         | **21.708**       |

Reviewer comment (Experiments):
The ablation study with more subnetworks can achieve better rendering result while a smaller value (i.e., 2) is used by default in this paper. What’s the consideration behind it? I think using a larger number of subnets may introduce difficulty for training. The difficulty in training convergence may significant grow as ray capacity and the number of subnetwork increases, as there may be a great number of possible arrangements. In contrast, using prior knowledge (e.g., nearby points or pixels may highly go through the same subnet) is likely to be helpful for reducing the complexity. The authors need to give more discussion about it as well as analysis on training complexity.

Author response:
Thank the reviewer for the thoughtful questions. For the convenience of the experiments (lower memory footprint in training), we adopt the k=2 setting by default. However, it is crucial to note that GML-NeRF consistently obtains performance gains as the number of sub-NeRFs increases, as shown in Figure-5. We add per-scene results in the Appendix F for reference.
Moreover, **the training difficulty will not increase significantly with the increase in the number of sub-NeRFs**. In our experiments, we find that the model with four sub-NeRFs converges faster than the one with two sub-NeRFs while achieving better rendering quality with the same training iterations. The convergence curves are also added to the Appendix F. The ease of training convergence can be attributed to two aspects. On the one hand, the feature grid is shared among multi-NeRFs, and thus, the number of learnable parameters increases marginally. On the other hand, as the neural network is better at fitting low-frequency information, our gate module (a 4-layer MLP without sinusoidal position encoding) has implicitly incorporated "smoothness prior", leading to closer rays to be more possibly assigned closer gating scores. Such smoothness prior is similar to what the reviewer mentioned, helping to reduce the training complexity with more sub-NeRFs.

| Method | Metric | 0046_00 | 0276_00 | 0515_00 | 0673_04 | Avg |
| --- | --- | --- | --- | --- | --- | --- |
| GML-size2 | PSNR | 29.440 | 30.871 | 29.149 | 25.759 | 28.805  |
| GML-size3 | PSNR | 29.878 | 31.242 | 29.470 | 25.944 | 29.134  |
| GML-size4 | PSNR | 30.018 | 31.310 | 29.679 | 26.063 | **29.268**  |

Reviewer comment (Experiments):
For a detail in the design, is sharing the feature grid necessary? The authors mentioned that it is helpful for reducing parameters. If using different feature grids for subnets, it may introduce some problems due to the lack of global index. It would be better to give some discussion and experimental result to validate the design.

Author response:
Thank the reviewer for the questions. In GML-NeRF, the shared feature grid is a better choice than the unshared one. We conduct an ablation study on the TAT dataset and add the results to the Appendix D. The experiment results show that the model employing a shared feature grid outperforms its counterpart with multiple independent feature grids. Such superiority comes from two aspects. 

Firstly, within the hybrid representation, the feature grid is responsible for encoding features of 3D spatial points, while the MLP encoder is designed to encode view information. The crucial design of independent MLP decoders aligns with our visibility-aware motivation, thereby enhancing the view-dependent effect. Secondly, the training complexity will also increase as the trainable parameters increase. With the limited amount of training data, increasing the number of feature grids leads to poor convergence. By contrast, as different rays may pass through the same region of 3D space, weight sharing for the feature grid helps to facilitate training.

Although the model with unshared feature grids can still achieve satisfactory results, particularly in terms of the SSIM and LPIPS metrics, we still recommend using the shared feature grid scheme due to its parameter efficiency. 

| Method                  | Metric | M60    | Playground | Train  | Truck  | Avg    |
|-------------------------|:------:|:------:|:----------:|:------:|:------:|:------:|
| unshared\_feature\_grid | PSNR   | 18.765 | 22.839     | 18.958 | 24.493 | 21.264 |
| Ours-size2              | PSNR   | 19.051 | 23.901     | 19.369 | 24.509 | 21.708 |

Reviewer comment (Theory):
The authors need to discuss the limitation of the work. It is unclear that if the method can handle the scene with translucent objects. The method is built on the assumption that each ray is associated with a single confidence score, i.e., depth value. But depth estimation is naturally a problem for translucent objects.

Author response:
Thank the reviewer for the insightful suggestion.

We think one limitation of GML-NeRF is that compared with single-NeRF or multi-NeRF (with hard gating) methods, it will introduce additional inference latency, although achieving better rendering performance. When the ensemble size is large, we might need to choose several sub-NeRFs instead of using all sub-NeRFs to reduce the inference latency. In addition, in order to strike a good balance between the rendering performance and inference latency, we might need to investigate how to decide the best choice of ensemble size (i.e., $K$) for a given scene.

As for the handling of translucent objects. In our view, we think the difficulty of tackling is a problem faced by all NeRF methods, and GML-NeRF doesn't bring extra difficulty. We'd like to know if the reviewer thinks that GML-NeRF has introduced extra in rendering transparent objects than other work and look forward to further suggestions.