Paper: Neural Field Classifiers via Target Encoding and Classification Loss (ICLR-2024)

Reviewer comment (Experiments):
The ablation study is insufficient. It can be observed from Eq. (6) that the original regression loss is still included. Although the authors pointed out that the BCE loss is significantly larger than the regression loss, there is still curiosity about the effect of using BCE loss only. Since the BCE loss is more dominant, is the regression loss essential?

Author response:
A good question. Although the (slightly modified with ) BCE is more dominant, the regression loss is still helpful. We note that the the modification (the clip) of the classification loss is desired for the numerical stability as the integrated probability through volume rendering may be slightly greater than or equal to one. However, the clip operation is non-differentialable for some data samples where the predicted target $\hat{C}\geq 1-\epsilon$. For these samples, the classification loss can only produce zero gradient. This explains when the regression loss dominate the classification loss and why the regression loss is still usually helpful.

Reviewer comment (Experiments):
The ablation study is insufficient. It seems that without text encoding, the performance is close to the final version. Moreover, only two scenes of one dataset are used to conduct the ablation study. On average (all scenes and other datasets), is the text encoding essential, or is it optimal?

Author response:
We respectfully note that the PSNR gains in our two ablation studies are 1.76 and 1.25, and such PNSR gains can be considered to be very significant in NeRF studies. Most new NeRF methods cannot outperform the baselines by more than 1 PSNR gain. If the reviewer want to know more about recent empirical advancements of NeRF studies, please refer to recent top-conference NeRF papers ( e.g. Trivec ICCV2023).

Thus, while the classification loss itself can lead to a more significant improvement, we hold that the target encoding is still expected to be helpful in most cases. 

In fact, the regression loss (discussed in Q1-A1) and target encoding are both helpful and positive, while their effects are less significant than the (modified) BCE loss term.

Reviewer comment (Experiments):
The implementation and comparison are conducted based on methods published in 2020 and 2021. Only DVGO was published in CVPR 2022, which was also long ago. It is suggested to conduct experiments on more recent SOTA methods to evaluate the effectiveness and generality of the proposed approach.

Author response:
Thanks for the constructive suggestion. The currently used NeRF backnones are all popular and representative methods covering the tasks of static scenes, geometry reconstruction, and dynamic scenes. The experiments are comprehensive.

Some recent works proposed in 2023 (e.g. Strivec) are considered to be SOTA. In our recent supplementary experiment, Strivec-C can outperform Strivec-R by 3 PSNR gain. We are conducting more comparative experiments using SOTA methods proposed in 2023. We would like to present more empirical results of recent NeRF variants for comparing NFC and NFR in the revision.

Reviewer comment (Theory):
Besides the experiments and visualization, there is no theoretical analysis of why the proposed classification based formulation is better than the standard regression methods. It is because that the classification loss is easier to optimize or?

Author response:
We frankly admit that there is no theoretical analysis yet. In fact, nearly all NeRF methods lack formal theoretical analysis about the rendering quality.

However, according to the results in Figure 7, we may conjecture that the main advantage of NFC comes from better generalization of learned neural field models, while we also observe that optimizing the classification loss also leads to slightly better training quality. It will be a very promising future direction to explore why the classification loss may lead to better generalization.

Reviewer comment (Evaluation):
There is also little analysis on why it is more robust that the regression based methods?

Author response:
Thanks for pointing out this promising research challenge. Similarly to Q1-A1, indeed, our work and related neural field studies did not formulate theoretical understanding of NeRFs. In fact, most 3D vision-related works lack formal theoretical understanding due to the methodology complexity.

According to the intuition and existing empirical analysis, the advantages of NFC may come from multiple sources, including better training quality and better generalization. However, the formal theoretical mechanism is beyond the main scope of this work. We appreciate your suggestion and will explore the theoretical mechanism of generalization and robustness of neural field models in future.

Reviewer comment (Evaluation):
Why the proposed classification based formulation perform better than the standard regression based methods?

Author response:
We frankly admit that there is no theoretical analysis yet. In fact, nearly all NeRF methods lack formal theoretical analysis about the rendering quality.

However, according to the results in Figure 7, we may conjecture that the main advantage of NFC comes from better generalization of learned neural field models, while we also observe that optimizing the classification loss also leads to slightly better training quality. It will be a very promising future direction to explore why the classification loss may lead to better generalization.

Reviewer comment (Evaluation):
Why is more robust that regression based methods?

Author response:
Thanks for pointing out this promising research challenge. Similarly to Q1-A1, indeed, our work and related neural field studies did not formulate theoretical understanding of NeRFs. In fact, most 3D vision-related works lack formal theoretical understanding due to the methodology complexity.

According to the intuition and existing empirical analysis, the advantages of NFC may come from multiple sources, including better training quality and better generalization. However, the formal theoretical mechanism is beyond the main scope of this work. We appreciate your suggestion and will explore the theoretical mechanism of generalization and robustness of neural field models in future.

Reviewer comment (Experiments):
The qualitative results for the baselines look unreasonably poor. I would think that a vanilla NeRF model would look better on the scenes presented in the figures.

Author response:
A2: Thanks for the suggestion. We presented the qualitative results of vanilla NeRF in Figure 3 Top Row. The regression baseline look well, while the classification variant is still significantly better. We would like to present more qualitative results of vanilla NeRF in the revision.

Reviewer comment (Experiments):
How many training views are given for the Replica scenes? I couldn't find that detail in the paper. The results for regression in figure 3 look particularly poor. I would expect a crisper result for regression with a reasonable number of views.

Author response:
A3: We respectfully note that the details of dataset preprocessing including datset split of Replica are presented in Appendix A.2. We used 90% images (~90 views) for training and 10% images (~10 views) for evaluation. 

The results of NeuS-R on Replica in Table 6 actually look reasonably well, while the results of DVGO-R on Replica in Table 3 are indeed poor. As the training dataset are totally same, our results suggest that the accelerated NeRF variants like DVGO are not powerful enough complex scenes (e.g. Replica), which contain many details.

However, for both NeuS and DVGO, our classifier-based method can significantly improve the rendering quality. Thus, we believe our experiments using DVGO and NeuS together can support the advantage of our NFC over the conventional NFR.

Reviewer comment (Reproducibility):
How is sigma being output by the MLP? Is that still being regressed, and if so why not encode it as a classification problem?

Author response:
A4: A very good question. While, in principle, the density $\sigma$ can be encoded as a classification problem, we believe regressing $\sigma$ is a better choice for three reasons. First, most importantly, encoding $\sigma$ does not significantly improve the rendering quality in our early experiments. Second, the extra computational cost of encoding $\sigma$ is significantly larger than only encoding rgb, because, for example, an 8-channel $\sigma$ encoding may reqiure $8$ decoding costs for making once prediction. Thrid, some NeRF variants use two different network backbones for predicting the rgb color and the density, where encoding the density neural network may require more coding costs. In summary, the pitfalls (more costs and complexity) significantly outweigh its benefits (nearly zero performance improvement) .

Reviewer comment (Reproducibility):
How is the volume rendering part performed? Do you use the standard volume rendering equation and integrate the bits with sigma?

Author response:
A5: We follow the standard volume rendering process. The only difference is that we integrate the predicted probabilities of each bits with the density.

Reviewer comment (Novelty):
Some important reference is missing, such as [a].

Author response:
A1: Thanks for providing the reference. In fact, any existing regression-based NeRF variant could be the baseline. We would like to discuss more existing NeRF methods, including PixelNeRF.

Reviewer comment (Novelty):
[a] uses the CNN encoders to process the images. The idea of the proposed classifier looks similar to [a].

Author response:
A2: We are afraid that the reviewer may misunderstand our work or PixelNeRF. We respectfully argue that PixelNeRF and the proposed NF classification paradigm are two totally different methods. Our work does not use any CNN encoder to process the images. Our two contributions, Target Encoding and Classification Loss, are not touched by PixelNeRF. Moreover, the classification paradigm can be incorporated into the (regression-based) PixelNeRF rather than replacing.

Reviewer comment (Experiments):
The biggest problem is the compared method. Among all the experiments, DVGO is compared extensively. However, DVGO is a method designed for acceleration, not for accuracy. It is unfair to compare the accuracy with a method not targeting accuracy.

Author response:
A3: Our work actually has only one baseline method, the regression paradigm compared with our classification paradigm. We kindly argue that, besides DVGO, we also train vanilla NeRF, NeuS, and D-NeRF as the backbone methods on 14 different scenes, including static scenes, geometry reconstruction, and dynamic scenes. The experiments are still comprehensive. Particularly, when the regression-based neural field models seriously overfit training views (as Figure 7 suggests), our classification-based method can significantly improve generalization and mitigate overfitting.  


Our prior backbone is the accelerated NeRF variant also because the accelerated variant is more environment-friendly and can significantly reduce the energy costs and carbon emissions of our work. In the line of NeRF research, people actually focused more on improving the rendering quality of the accelerated NeRF variants, because the efficiency rather than the quality (for standard static scenes) is often considered the main limitation of vanilla NeRF.

Recent NeRF works, including Strivec (ICCV2023) and Zip-NeRF (ICCV2023), almost all aimed at (1) accurate and fast rendering, or (2) accurate rendering under some constrains (e.g. sparse/dynamic/corruption). Our experiments have covered these experimental settings.

Reviewer comment (Evaluation):
What is the computational cost of the proposed classifier-based methods? Please compare with DVGO regarding FLOPs and wall-clock time.

Author response:
A5: We respectfully note that we studied the extra computational costs of the classifier-based methods in Tables 7 and 8, using DVGO and NeuS for novel view synthesis and geometry reconstruction, respectively. The numerical results show that the extra computational costs of two components are marginal (only 6% for DVGO and 5% for NeuS).