Paper: LeCO-NeRF: Learning Compact Occupancy for Large-scale Neural Radiance Fields (ICLR-2024)

Reviewer comment (Experiments):
The HMoE design is questionable. According to Section 4.5 HMoE, the reconstruction quality dropped with MoE (the empty expert is as large as scene experts).

Author response:
**Q1: Homogeneous MoE and Heterogeneous MoE.** We are very sorry for the misleading caption of Table 4 and the text in the Section 4.5 HMoE. We forget to mention the experiment details of this ablation study. In Table 4, both the Homo. and Heter. MoE are trained with the imbalanced gate loss and without density loss for 40K steps and than their occupancy networks are used to guide the training of Switch-NeRF. The accuracy in Paper Table 4 are the final reconstruction accuracy of the guided stage. These experiments are to show that with our HMoE and imbalanced gate loss, our method can already learn very good occupancy. The Homogeneous MoE with imbalanced gate loss **cannot** learn reasonable occupancy so if we use its occupancy network to guide the training of Switch-NeRF, many important (occupied) points will be classified as unoccupied. Therefore, the accuracy drops a lot. We have fixed this misleading components in the revised version. In our main experiments, the empty space expert is set as an Identity layer and have been shown as effective across quite different large-scale scenes. We also add experiments of different sizes of the empty space experts in Table 9 in Appendix G and the table below. In the table, Homo. is a large MLP with 7 layers. The 4-layer version can learn better occupancy wile the Identity version can gets results very close to Switch-NeRF. Therefore, from our extensive experiments, with an Identity layer as the expert space expert can learn very good results. The output of the Identity empty space expert will still go through the prediction head so the network can still be trained.


| Method    | PSNR↑ | SSIM↑ | LPIPS↓ |
|-----------|-------|-------|--------|
| Homo      | 20.23 | 0.631 | 0.506  |
| 4-layer   | 23.62 | 0.701 | 0.463  |
| Identity  | **26.30** |**0.781** | **0.383**  |

Reviewer comment (Theory):
I believe it is the density loss that enforces the small sigma for empty points. If a larger empty expert leads to failure, that means the density loss is invalid / not effective.

Author response:
**Q1: Homogeneous MoE and Heterogeneous MoE.** We are very sorry for the misleading caption of Table 4 and the text in the Section 4.5 HMoE. We forget to mention the experiment details of this ablation study. In Table 4, both the Homo. and Heter. MoE are trained with the imbalanced gate loss and without density loss for 40K steps and than their occupancy networks are used to guide the training of Switch-NeRF. The accuracy in Paper Table 4 are the final reconstruction accuracy of the guided stage. These experiments are to show that with our HMoE and imbalanced gate loss, our method can already learn very good occupancy. The Homogeneous MoE with imbalanced gate loss **cannot** learn reasonable occupancy so if we use its occupancy network to guide the training of Switch-NeRF, many important (occupied) points will be classified as unoccupied. Therefore, the accuracy drops a lot. We have fixed this misleading components in the revised version. In our main experiments, the empty space expert is set as an Identity layer and have been shown as effective across quite different large-scale scenes. We also add experiments of different sizes of the empty space experts in Table 9 in Appendix G and the table below. In the table, Homo. is a large MLP with 7 layers. The 4-layer version can learn better occupancy wile the Identity version can gets results very close to Switch-NeRF. Therefore, from our extensive experiments, with an Identity layer as the expert space expert can learn very good results. The output of the Identity empty space expert will still go through the prediction head so the network can still be trained.


| Method    | PSNR↑ | SSIM↑ | LPIPS↓ |
|-----------|-------|-------|--------|
| Homo      | 20.23 | 0.631 | 0.506  |
| 4-layer   | 23.62 | 0.701 | 0.463  |
| Identity  | **26.30** |**0.781** | **0.383**  |

Reviewer comment (Experiments):
If a larger empty expert fails, the authors need to justify how "small" the empty expert should be. It would be impractical if the HMoE is sensitive to the network design. Probably it requires different network designs when working with different scenes.

Author response:
**Q1: Homogeneous MoE and Heterogeneous MoE.** We are very sorry for the misleading caption of Table 4 and the text in the Section 4.5 HMoE. We forget to mention the experiment details of this ablation study. In Table 4, both the Homo. and Heter. MoE are trained with the imbalanced gate loss and without density loss for 40K steps and than their occupancy networks are used to guide the training of Switch-NeRF. The accuracy in Paper Table 4 are the final reconstruction accuracy of the guided stage. These experiments are to show that with our HMoE and imbalanced gate loss, our method can already learn very good occupancy. The Homogeneous MoE with imbalanced gate loss **cannot** learn reasonable occupancy so if we use its occupancy network to guide the training of Switch-NeRF, many important (occupied) points will be classified as unoccupied. Therefore, the accuracy drops a lot. We have fixed this misleading components in the revised version. In our main experiments, the empty space expert is set as an Identity layer and have been shown as effective across quite different large-scale scenes. We also add experiments of different sizes of the empty space experts in Table 9 in Appendix G and the table below. In the table, Homo. is a large MLP with 7 layers. The 4-layer version can learn better occupancy wile the Identity version can gets results very close to Switch-NeRF. Therefore, from our extensive experiments, with an Identity layer as the expert space expert can learn very good results. The output of the Identity empty space expert will still go through the prediction head so the network can still be trained.


| Method    | PSNR↑ | SSIM↑ | LPIPS↓ |
|-----------|-------|-------|--------|
| Homo      | 20.23 | 0.631 | 0.506  |
| 4-layer   | 23.62 | 0.701 | 0.463  |
| Identity  | **26.30** |**0.781** | **0.383**  |

Reviewer comment (Theory):
I am confused about the statement "we can use our frozen occupancy network to guide the sampling and training of NeRF methods". The occupancy network can be trained end-to-end according to the design. Why do we need to freeze the occupancy network and train the NeRF models again?

Author response:
**Q2: Guided training.** The main motivation of freezing the occupancy network is to decrease the computation in the training of NeRF and to sample denser samples. In the training of occupancy network, we uniformly samples 3D points along rays. And the unoccupied points still need gradient and optimization for training the occupancy which consumes more memory and computation. Since our occupancy network converges very fast, we can freeze the occupancy network to accelerate the classification of 3D points and sampling denser samples to train NeRF methods, without extra gradients and optimization for the occupancy network.

Reviewer comment (Experiments):
Unfair experiment design. Most experiments are of fixed training time, e.g., 23.8h for Table 2, 12-15h for Table 3. Indeed the HMoE gives better results with limited training time. However, we need to compare the final result with enough training time. Does the HMoE still outperform the baselines with enough training time, e.g., 40+ hours for Tables 2 & 3?

Author response:
**Q3: Training time in experiment design.** In the main experiments, we fix the same training time to clearly show the advantage of our method to get better results with the same training time. As stated in Section 4.3, the training times are aligned to the training time of S+Grid for 500k steps. We revised this part to make it more clear. We add a curve figure to show the accuracy with respect to the training time in Fig. 9 in Appendix D, as suggested by Reviewer KM5v Q3. As shown in this figure, our method enjoys better accuracy abd fast converging speed. We also add Table 8 in Appendix F to train S+Grid and S+Ours with the same time as the original Switch-NeRF, as shown in the table below and Table 8, HMoE can consitently out-performs the baselines with enough training time.



| Dataset    | Metrics | S-NeRF | S+Grid | S+Ours |
|------------|---------|--------|--------|--------|
| Rubble     | PSNR    | 24.31  | 23.94  | **24.52** |
|            | SSIM    | 0.562  | 0.538  | **0.581** |
|            | LPIPS   | 0.496  | 0.526  | **0.477** |
|            | Time    | 41.5h  | 41.5h  | 41.5h  |
| Residence  | PSNR    | 22.57  | 22.48  | **22.88** |
|            | SSIM    | 0.654  | 0.645  | **0.658** |
|            | LPIPS   | 0.457  | 0.474  | 0.454** |
|            | Time    | 43.4h  | 43.4h  | 43.4h  |

Reviewer comment (Experiments):
An obvious alternative is that the occupancy network serves as a classifier, that is: (a) Remove the empty space expert; (b) The empty points do not involve in the rendering — in other words, set σ=0 if a point is predicted as empty.

Author response:
**Q4: The importance of empty space expert.** The empty space expert is a key component to make the occupancy network trainable. The occupancy network cannot distinguish the points before the training. It will get false classification in the beginning of the training. We need the empty space expert and its prediction to backpropagate the gradients and give training signals to the occupancy network. Therefore, we need to predict the sigmas and RGBs of points in the empty space expert. The rendering loss, imbalanced gating loss and density loss all need the output of the empty space expert to train the occupancy network in a self-supervised manner. As for "An obvious alternative is that occupancy network serves as a classifier. That is: (a) Remove the empty space expert (b) The empty points do not involve in the rendering, in another word, set sigma=0 if a point is predicted as empty.", it is actually our guided stage in the paper using our frozen occupancy network to guide the training of NeRF methods. In contrast, in the stage of training occupancy network, we need the empty space expert to pass training signals.

Reviewer comment (Experiments):
The paper introduced an algorithm to handle very large scenes. However I wonder if there are intuitions that can be shared on 1) how big of the scene can the algorithm handle in theory (and in practice), and 2) how small of the scene can LeCO-NERF become effective (against existing SOTA NERF variants for 3D objects).

Author response:
**Q1: Intuitions of the scene scales.** Many thanks for this question. We describe the large-scale scenes in our experiments to give more intuitions. The Mega-NeRF dataset comprises 5 scenes each consisting of $2k$ to $6k$ images with a resolution from $4k$ to $5k$, covering urban regions from 0.15 to 1.3 $km^2$. The Block-NeRF dataset includes a street scene with $12k$ images at $1k \times 1k$ resolution, covering an Urban street of $1km$ long. These scenes are very challenging due to the huge image data and the coverage of large-scale regions. The experiments have shown that our compact occupancy network can handle these large-scale scenes effectively. From our perspective, for the small scale objects, the efficiency problem is not as critical as the large-scale scene because of the efficient representation such as the hash-encoding, small data volume, dense multi-view images and requirements of small resolution of grids. In the large-scale urban scenes, due to the large data volume and large scene range, the learning of occupancy becomes much harder and critical, and our proposed compact occupancy network can effectively solve these problems. Our newly add experiments on large-scale scnens with Instant-NGP as baseline also show that it can handle different large-scale representations (Appendix C, Table 6 and Reviewer KM5v Q4).

Reviewer comment (Presentation):
I spent considerable amount of time trying to understand the blue box with white contents of Fig.1, before learning more context about what it represents. I wonder if this visualization can be better improved by providing more explicit notations in the caption or in the main text.

Author response:
**Q2: Visualization in Fig. 1.** Many thanks for this suggestion. We add more details about the visualization in the caption of Fig. 1 and the main text.

Reviewer comment (Presentation):
In Fig 4b, with L_d, it seems that the rendering is all blank, is this a mistake?

Author response:
**Q3: Blank image in Fig 4b.** Many thanks for the question. This is not a mistake. The bottom right image of Fig 4b shows the unoccupied points with alpha. Points with small alpha will be shown as transparent. Please refer Section 4.2 Occupancy visualization for more details. Since with density loss, our method can predict more accurate occupancy, the points in this image are all with very small density and alpha. Therefore, all of the points of the bottom right image of Fig 4b are transparent when shown with alpha and thus the image is all blank. Compare with the bottom left image, the bottom right image shows the effectiveness of our density loss to separate the occupied and unoccupied points more clearly. We add more description of this image in the caption and main text in the revised version.

Reviewer comment (Reproducibility):
Will the code be available upon publication? I think this would be useful for reproducing this research work.

Author response:
**Q4: Code releasing.** We will surely release our codes upon publication to benefit the community.

Reviewer comment (Novelty):
My major concern with the paper is limited novelty and improvement over existing literature on large-scale neural rendering, as well as inadequate evaluation of related benchmarks.

Author response:
**Q1: The difference between our method and Switch-NeRF.** We would like to clarify our motivation, novelty, and our distinctive features compared to Switch-NeRF. 

In this paper, we solve a fundamentally different problem compared to Switch-NeRF in large-scale NeRF modeling. The Switch-NeRF aims to learn the decomposition of a large-scale scene for scalable training. Our paper targets learning a compact representation of a binary occupancy of a large-scale scene, which can provide effective guidance to learn a large-scale NeRF efficiently. This is the first time that this problem has been introduced for large-scale NeRF modeling.

To tackle this important and new problem of modeling the occupancy of a large-scale scene, our approach is designed based on our insights on the problem, and our contribution is three-fold:

**(i) A novel heterogeneous MoE structure with scene experts and a tiny Empty Space Expert.** This structure is designed to effectively model the high heterogeneity of occupied and unoccupied 3D scene points. 

**(ii) A novel and effective gate imbalanced loss.** If we train the HMoE with the balanced loss as used in the original Switch-NeRF that targets a homogeneous modeling of the scene, the model cannot learn the occupancy because the scene distribution is highly heterogeneous with a lot more points belonging to the empty space compared to the surface. It is thus very important to enforce a training prior to make the network distinguish the occupied and the unoccupied 3D points. The proposed gate imbalanced loss can effectively control the portion of points dispatched to the empty space expert, which is a critical design for the successful large-scale occupancy learning.

**(iii) A novel and effective density loss.** Our designed density loss can directly control the gating network according to the predicted density. It enables our occupancy network to explicitly model the important scene geometry prior, which is the density of an unoccupied scene point is much smaller than that of an occupied scene point. The density loss helps to ensure the learning of more accurate scene occupancy.

In the experiments of the main paper, our method has shown consistent advantages of accuracy, converging speed, and the GPU memory overhead over the occupancy grid method and the Switch-NeRF trained with the same time. In our newly added experiments (Table 6 in the appendix and in Q4) with the Instant-NGP as a baseline, our method also outperforms the Instant-NGP by large margins. These experiments show that our method gains consistent performance improvements compared to the occupancy grid method in various datasets and baselines, demonstrating the effectiveness of our proposed approach for learning a compact large-scale scene occupancy.

In conclusion, our paper tackles a fundamentally different problem compared to Switch-NeRF and all our three novel contributions are closely related to our insights on the new problem. Our method confirms consistent advantages over the occupancy grid method.

Reviewer comment (Novelty):
The general idea of using MoE to improve the performance of large-scale neural radiance fields has been explored by Switch-NeRF. The work is rather incremental by modifying the homogenous mixture of experts into the heterogeneous mixture of experts (HMoE).

Author response:
**Q1: The difference between our method and Switch-NeRF.** We would like to clarify our motivation, novelty, and our distinctive features compared to Switch-NeRF. 

In this paper, we solve a fundamentally different problem compared to Switch-NeRF in large-scale NeRF modeling. The Switch-NeRF aims to learn the decomposition of a large-scale scene for scalable training. Our paper targets learning a compact representation of a binary occupancy of a large-scale scene, which can provide effective guidance to learn a large-scale NeRF efficiently. This is the first time that this problem has been introduced for large-scale NeRF modeling.

To tackle this important and new problem of modeling the occupancy of a large-scale scene, our approach is designed based on our insights on the problem, and our contribution is three-fold:

**(i) A novel heterogeneous MoE structure with scene experts and a tiny Empty Space Expert.** This structure is designed to effectively model the high heterogeneity of occupied and unoccupied 3D scene points. 

**(ii) A novel and effective gate imbalanced loss.** If we train the HMoE with the balanced loss as used in the original Switch-NeRF that targets a homogeneous modeling of the scene, the model cannot learn the occupancy because the scene distribution is highly heterogeneous with a lot more points belonging to the empty space compared to the surface. It is thus very important to enforce a training prior to make the network distinguish the occupied and the unoccupied 3D points. The proposed gate imbalanced loss can effectively control the portion of points dispatched to the empty space expert, which is a critical design for the successful large-scale occupancy learning.

**(iii) A novel and effective density loss.** Our designed density loss can directly control the gating network according to the predicted density. It enables our occupancy network to explicitly model the important scene geometry prior, which is the density of an unoccupied scene point is much smaller than that of an occupied scene point. The density loss helps to ensure the learning of more accurate scene occupancy.

In the experiments of the main paper, our method has shown consistent advantages of accuracy, converging speed, and the GPU memory overhead over the occupancy grid method and the Switch-NeRF trained with the same time. In our newly added experiments (Table 6 in the appendix and in Q4) with the Instant-NGP as a baseline, our method also outperforms the Instant-NGP by large margins. These experiments show that our method gains consistent performance improvements compared to the occupancy grid method in various datasets and baselines, demonstrating the effectiveness of our proposed approach for learning a compact large-scale scene occupancy.

In conclusion, our paper tackles a fundamentally different problem compared to Switch-NeRF and all our three novel contributions are closely related to our insights on the new problem. Our method confirms consistent advantages over the occupancy grid method.

Reviewer comment (Evaluation):
The reconstruction quality, when compared to the original Switch-NeRF, seems to be only on par or marginally improved (Tab. 2, Tab. 3).

Author response:
**Q2: Reconstruction quality.** We would like to highlight our clear advantages of accuracy, converging speed, and the GPU memory overhead over the occupancy grid method and the Switch-NeRF, which have been extensively verified in our experimental results. For example, our model achieves an improvement of about 0.6 PSNR over Switch-NeRF on Sci-Art (Table 3 in the main paper) when being trained with the same time. We obtain a similar PSNR with only 8 hours training compared to Switch-NeRF using 14 hours (Fig. 9 in the main paper). Our method also uses much less memory (2.4G) than Switch-NeRF (10.5G) and the occupancy grid (5.8G) as reported in Table 3 of the main paper. In our newly added experiments (Table 6 in the appendix and also in Q4 of the rebuttal) with the Instant-NGP as the baseline, our method also outperforms the Instant-NGP by large margins, e.g., 1.07 points of PSNR on Campus and 0.85 on Building. These experiments on the different datasets and baselines demonstrate the effectiveness and the advantages of our proposed approach.

Reviewer comment (Novelty):
Improve the related work section.

Author response:
**Q6: Revision of paper.** Thanks for the suggestions about the paper revision. We added more related papers in the related work section, and added new experiments related to the hash-encoding baseline in Appendix C and Table 6 show the improvements of our method over baselines. We fixed the typo and added more details to Table 3.

Reviewer comment (Experiments):
Discuss more about how this method improves over the plain MoE/Switch-NeRF.

Author response:
**Q1: The difference between our method and Switch-NeRF.** We would like to clarify our motivation, novelty, and our distinctive features compared to Switch-NeRF. 

In this paper, we solve a fundamentally different problem compared to Switch-NeRF in large-scale NeRF modeling. The Switch-NeRF aims to learn the decomposition of a large-scale scene for scalable training. Our paper targets learning a compact representation of a binary occupancy of a large-scale scene, which can provide effective guidance to learn a large-scale NeRF efficiently. This is the first time that this problem has been introduced for large-scale NeRF modeling.

To tackle this important and new problem of modeling the occupancy of a large-scale scene, our approach is designed based on our insights on the problem, and our contribution is three-fold:

**(i) A novel heterogeneous MoE structure with scene experts and a tiny Empty Space Expert.** This structure is designed to effectively model the high heterogeneity of occupied and unoccupied 3D scene points. 

**(ii) A novel and effective gate imbalanced loss.** If we train the HMoE with the balanced loss as used in the original Switch-NeRF that targets a homogeneous modeling of the scene, the model cannot learn the occupancy because the scene distribution is highly heterogeneous with a lot more points belonging to the empty space compared to the surface. It is thus very important to enforce a training prior to make the network distinguish the occupied and the unoccupied 3D points. The proposed gate imbalanced loss can effectively control the portion of points dispatched to the empty space expert, which is a critical design for the successful large-scale occupancy learning.

**(iii) A novel and effective density loss.** Our designed density loss can directly control the gating network according to the predicted density. It enables our occupancy network to explicitly model the important scene geometry prior, which is the density of an unoccupied scene point is much smaller than that of an occupied scene point. The density loss helps to ensure the learning of more accurate scene occupancy.

In the experiments of the main paper, our method has shown consistent advantages of accuracy, converging speed, and the GPU memory overhead over the occupancy grid method and the Switch-NeRF trained with the same time. In our newly added experiments (Table 6 in the appendix and in Q4) with the Instant-NGP as a baseline, our method also outperforms the Instant-NGP by large margins. These experiments show that our method gains consistent performance improvements compared to the occupancy grid method in various datasets and baselines, demonstrating the effectiveness of our proposed approach for learning a compact large-scale scene occupancy.

In conclusion, our paper tackles a fundamentally different problem compared to Switch-NeRF and all our three novel contributions are closely related to our insights on the new problem. Our method confirms consistent advantages over the occupancy grid method.

Reviewer comment (Presentation):
Include some visualizations (of reconstructed images) for the main experiment.

Author response:
**Q5: Visualizations of reconstructed images.** Thanks for the suggestion. We provide Figure 8 in Appendix B for the visualization of the reconstructed images. We also provide a video demo in the supplementary files.

Reviewer comment (Presentation):
Include more details about Tab. 3.

Author response:
**Q3: S-NeRF\* and S-NeRF in Table 3 and a new figure of PSNRs changing with time.** We are sorry for the confusion. S-NeRF\* and S-NeRF are trained with different times. The S-NeRF denotes the Switch-NeRF trained with 500k steps. The results are obtained from the original paper of Switch-NeRF. The S-NeRF* is the Switch-NeRF trained with the same time the Switch-NeRF+Grid (S+Grid). We align the training time to S+Grid for a fair comparison. We have revised the caption of Table 3 to make it more clear. 

Thanks for the suggestion of a figure of showing PSNRs changing against the training time. We add Fig. 9 in Appendix D. It shows that our approach (S+Ours) obtains clearly better accuracy and faster converging speed.

Reviewer comment (Presentation):
Minor: Tab. 3 caption: -> MegaNeRF.

Author response:
**Q6: Revision of paper.** Thanks for the suggestions about the paper revision. We added more related papers in the related work section, and added new experiments related to the hash-encoding baseline in Appendix C and Table 6 show the improvements of our method over baselines. We fixed the typo and added more details to Table 3.