Paper: Factored-NeuS: Reconstructing Surfaces, Illumination, and Materials of Possibly Glossy Objects (ICLR-2024)

Reviewer comment (Novelty):
This paper appears to be a combination of numerous prior works. From my perspective, the first stage borrows from the papers NeuS and PhySG. The second stage seems inspired by NeRFactor and IndiSG, particularly in how they learn visibility mapping from a previously trained geometry neural network. The final stage also bears similarities to IndiSG. Given these observations, I contend that the paper presents limited novelty in its three-stage training approach.

Author response:
Our first stage is fundamentally different from other work in both the implementation and the design concept behind it. The other two stages build on previous work, but they are modifications, to improve the results and therefore also contain novelty. Also, the way our system is constructed is novel. We do use and build upon components from previous work, but creating a complete and well working system and adapting existing components and combining them with our novel Stage 1 is still a significant challenge.  We provide more details below.

In the first stage, we proposed a novel surface reconstruction method. The volume rendering module is better suited to radiance reconstruction and the surface rendering module ensures higher quality specular and diffuse separation close to the surface. Both working together enable accurate and efficient reconstruction of glossy object surfaces. Previous methods consider only using volume rendering (NeuS, VolSDF) or surface rendering IDR(PhySG, IndiSG) to reconstruct geometric information, and cannot handle the reconstruction of glossy objects, as shown in Tab. 1, Fig. 3, and Appx Fig. 7.

We have indeed incorporated the concept of continuous light visibility from NeRFactor to enhance the binary light visibility of IndiSG, which predicts light visibility as either 0 or 1. We predict light visibility as 1-Σw. Existing ablation studies demonstrate that employing this approach yields better results in Tab 3 and Fig 10. Indirect lighting is currently a widely used method for lighting modeling, and we do not claim novelty in this aspect.

In the third stages, we present a specular albedo estimation method based on spherical Gaussians. Although the component is simple, the exceptional effectiveness of our way of utilizing them is evident in Tab. 2, Fig. 4, and Appx Fig. 10.

Reviewer comment (Writing):
The primary weakness is a lack of consistency with vocabulary and definitions. There is a long history of physical models of appearance, and the model the authors derive is effectively the Bi-illuminant Dichromatic Reflection model (see equations 9 and 10 of Maxwell et. al, CVPR 2008), which is built on the Dichromatic Reflection model (Shafer, 1986). The physics-based vision community has long used the terms body reflection and surface reflection to refer to what the authors are calling diffuse radiance and specular radiance. Using a phrase like "surface diffuse radiance" mixes this vocabulary and creates confusion.

Author response:
Thank you for your suggestion. Currently, the terminology is mainly aligned with the 'neural radiance field' used in NeRF for representing colors. Due to your suggestion, we realized that our definition in the main text should be improved. We change the description of diffuse radiance in the main text using “We use the term diffuse radiance to refer to the component of the reflected radiance that stems from a diffuse surface reflection”. We would like to break down the radiance measured by a pixel in the image into two components, the diffuse radiance and the specular radiance. The diffuse radiance is the component due to a diffuse surface reflection and the specular radiance includes all other non-diffuse components.  Regarding the confusion of the usage of the term “surface diffuse radiance”: we simply meant diffuse radiance (as used in other parts of the paper) that stems from the surface rendering part of stage one. We have modified this terminology in the revised manuscript. “We propose to simultaneously optimize the radiance from the volumetric rendering and the surface rendering. For surface rendering, we further split the reflected radiance into a diffuse and a specular component.”
 
These terms are also used in physically-based rendering and atmospheric science. For example “Advanced Global Illumination”, chapter 2 “The physics of light transport”.

We also checked the reference “A bi-illuminant dichromatic reflection model for understanding images”, but we think the used terminology is not common in the NerF literature. To better align our work with the most relevant previous work, we therefore prefer not to change the terminology. There are many different communities working on reconstruction, lighting, and appearance models and we understand that terminology is not always consistent between communities.

Reviewer comment (Writing):
The term "diffuse albedo" is also somewhat confusing, though albedo is commonly used (for greyscale?). Using something like body reflection color would be less confusing.

Author response:
Many literatures use diffuse albedo and albedo interchangeably. To distinguish our specular albedo, we opt for diffuse albedo. This terminology is also used by multiple other papers cited by our work. For example, see PhySG, NeRD, and IndiSG.

Reviewer comment (Experiments):
Lack of reference to NeRO [a], DIP [b]. NeRO [a] proposes a two-step approach for reconstructing the geometry and the BRDF of reflective objects with strong reflective appearances. They also introduce real/synthetic glossy dataset for evaluation. DIP [b] proposes a physics-based interreflection-aware illumination model and end-to-end learns the illumination, geometry and materials. Both are released earlier and with code available. The authors should compare the proposed method with them.

Author response:
We appreciate the reviewer's feedback regarding the lack of reference to NeRO [a] and DIP [b]. We duly incorporate references to NeRO [a] and DIP [b] in our revised manuscript. For comparison, we compared our method with the already published work NeRO. Our method is compared with NeRO on the established real datasets DTU and SK3D for evaluating glossy objects. Our findings demonstrate that NeRO performs less effectively than our approach on real glossy datasets DTU in Appx Fig. 17. NeRO not only struggles to accurately restore detailed information but also fails to address the negative impact of partial highlights on the geometry. Moreover, the presence of shadows causes NeRO to mistakenly reconstruct shadowed areas as real objects and fill them in (bricks and skull models). In addition, our quantitative evaluation of Chamfer distance for the DTU dataset performs better as presented in Appx Tab. 7. When running the SK3D dataset, similar issues were encountered as in the DTU. Even in scenarios with a simple background and straightforward geometry, NeRO still tends to lose certain details and fills in shadowed areas (flower pot model) as shown in Appx Fig. 18. In addition, our quantitative evaluation of Chamfer distance for the SK3D dataset performs better as presented in Appx Tab. 8. Furthermore, we extended our comparison to include new glossy datasets, Glossy-Blender dataset and Glossy-Real dataset in Appx Fig 19 and Appx Tab 9, where although NeRO performs better, but our method is also capable of mitigating the impact of highlights on geometry. Our method demonstrated comparable results to NeRO. The experimental results have been presented in the Appendix. DIP is a concurrent work that will be published in 3DV 2024. We added a citation, but we think the paper only establishes that DIP looks good on synthetic data and does not claim that the method can handle glossy objects.

Reviewer comment (Novelty):
Lack of novelty. The proposed method consists of three stages which combine ideas from different papers. For the first stage, they propose to perform joint volume and surface rendering, which is similar to S^3-NeRF [c], and joint appearance and BRDF modeling which follows TensoIR [d], decompose color into explicit diffuse and specular components for glossy surfaces which follows Ref-NeRF[e]. The visibility and indirect light modeling in second stage mostly follow NeRFactor [f] and IndiSG [g]. The BRDF modeling in last stage follows PhySG [h].

Author response:
We respectfully disagree with this observation. To address the degradation issue in the model training caused by the reduced sampling range during the transition from volume rendering to surface rendering in UNISURF, S^3 NeRF defines volume rendering as the integration of surface rendering colors on the ray to increase the sampled region. This approach is fundamentally different in implementation and motivation from ours. When we applied the construction method of S^3 NeRF, which involves integrating surface rendering colors into volume rendering, to modify our model structure, we found that this modeling approach did not eliminate the impact of highlights on the surface reconstruction results. The experimental quantitative and qualitative results have been shown in Appx Fig 11 in the appendix. In addition, we exhibit differences in many other aspects, such as our use of Signed Distance Function (SDF) instead of an occupancy field to represent the implicit function and determine the position of surface points.

At the first stage, the decomposition of diffuse and specular is not a true BRDF model. This is because the MLP in the first stage is used solely for predicting the components of diffuse and specular reflection, rather than predicting material properties such as albedo and roughness. The decision to directly predict colors instead of material properties in the first stage serves two purposes: reducing model complexity by focusing on the direct prediction of specular reflection color, and optimizing geometry for better reconstruction. By decomposing highlights through the network in the first stage, surfaces with specular reflections can be reconstructed more effectively, demonstrated by the presence of flower pot ablation in Fig 5, and without encountering the concavity issues observed in other methods. No prior work has utilized this method for geometry (SDF) optimization. After the completion of the first stage, where we believe the model's geometry has been optimized, the third stage focuses on predicting materials (BRDF model) based on the optimized geometry. During this process, the geometry model remains fixed and unchanged.

A distinction between our method and Ref-NeRF is that we use surface point colors for both diffuse and specular components, whereas Ref-NeRF utilizes volume rendering colors for diffuse and specular components. In our ablation study, we demonstrate that using volume rendering colors does not completely eliminate the impact of highlights on the geometric reconstruction of the flower pot. Additionally, Ref-NeRF relies on a density field, while our approach is based on an SDF field, allowing us to obtain smoother 3D shape models. (When using an SDF field, the incorporation of additional feature vectors from the SDF network output into the color prediction model is necessary.)

We have indeed incorporated the concept of continuous light visibility from NeRFactor to enhance the binary light visibility of IndiSG, which predicts light visibility as either 0 or 1. We predict light visibility as 1-Σw. Existing ablation studies in Tab 3 and Appx Fig 10 demonstrate that employing this approach yields better results. Indirect lighting is currently a widely used method for lighting modeling, and we do not claim novelty in this aspect. However, there are many possible ways to construct a complete system and it is not trivial to arrange all parts into a well working system, even if some of these parts have been used before and are mainly adapted.

In the third stage, our contribution is the introduction of specular albedo, a feature absent in PhySG. Through analysis, we discovered that incorporating specular albedo enhances the reconstruction of material properties and rendering colors.

Reviewer comment (Evaluation):
Evaluations/analysis are not thorough. For instance, the diffuse/specular component of the first stage is not visualized/analyzed. There is also no ablation study on the design of the modeling of the two components. Besides, it would be better to visualize the geometry/albedo/specular/roughness/visibility/indirect light/environment map at the same time for all the datasets. Most results in the paper are partially shown for different datasets. Visibility and indirect lights are only included in the videos of three objects without any analysis. Environment maps are only visualized in one ablation study in supplementary.

Author response:
We actually think our experiments are quite extensive, especially compared to other papers in the field. In the original submission, we presented an ablation study in Fig. 5 to demonstrate the impact of designing the two components on shape reconstruction for the first stage.
Additionally, our goal in the first stage is the reconstruction of well-defined geometric shapes for glossy objects rather than generating diffuse/specular components. Nevertheless, we showcase the decomposed results of diffuse and specular components in Appx Fig 12 in the appendix, demonstrating a reasonable outcome of the decomposition process.

Thank you for your suggestion. We have added these components for all indiSG, DTU and SK3D datasets in Appx Fig 13, Fig 14 and Fig 15 in the appendix.

In the original submission, we have included the ablation study for visibility in Appx Fig 10 and analyse the necessity of including the visibility components. In the appendix, we additionally showcase ablation experiments on indirect light in Appx Fig 16. Through analysis, it is evident that the reconstruction quality of materials and rendering deteriorates in the absence of indirect light.

Reviewer comment (Experiments):
The progressive decomposition is not end-to-end, the geometry is fixed after first stage.

Author response:
In the first stage, the decomposition of diffuse and specular components contributes to the improvement in geometric quality, as evidenced by flower pot ablation. After the training in the first stage, if the model parameters are not fixed and the subsequent 2nd and 3rd stages are executed, the geometric quality experiences a decline due to the increased complexity of model training. Not being end-to-end is a disadvantage in some sense, but we obtain better quality with our current method which is an advantage.

Reviewer comment (Writing):
Symbols used are a bit messy.

Author response:
We appreciate the reviewer's feedback on the symbols used in our manuscript. We respectfully disagree with the notion that our symbols are messy. 
We use the IndiSG paper as the starting point for our notation, but we also introduced several changes. We had been making multiple revisions to our symbol usage while drafting the manuscript, with a dedicated focus on improving clarity and coherence. We believe that the current set of symbols accurately represents the technical aspects of our work and facilitates a comprehensive understanding of the methodologies employed.
At the same time, we are fully open to further enhancing our manuscript. If the reviewer could provide specific instances or aspects of our symbol usage that they find unclear or confusing, we would be more than willing to address these concerns in a targeted manner. Such specific feedback would be invaluable in helping us refine our manuscript to better meet the needs of our audience. We are committed to making our research as clear and understandable as possible and welcome any constructive feedback to achieve this goal.

Reviewer comment (Experiments):
The datasets for evaluation should be more diverse. The lighting of SK3D dataset is not complex and the backgrounds are all black. The proposed method should also be evaluated on the Glossy-Blender dataset and Glossy-Real dataset proposed by NeRO [a] whose surfaces are more reflective.

Author response:
We presented results on the Glossy-Blender dataset and Glossy-Real dataset in Appx Fig 19 and Appx Tab 9, demonstrating comparable outcomes to NeRO. It is worth clarifying that our method performs better than NeRO on the real-world DTU and SK3D datasets.

Reviewer comment (Evaluation):
What’s the performance on objects that are diffuse or not very glossy?

Author response:
In the original submission, we presented material decomposition results for diffuse objects such as hotdog in Fig 4 and Tab 2. In the revised paper, we further showcase results for the entire DTU dataset in Appx Tab 7 and Fig 17, which includes many objects that are diffuse or not very glossy. Our method is also well-suited to handle this type of data.

Reviewer comment (Presentation):
Better visualize the pipeline and models of all stages for easier understanding.

Author response:
Thank you for your suggestion. In the revised version, we have illustrated all stages of the model in Fig.1 and Appx Fig.6.

Reviewer comment (Theory):
The proposed method chooses simplified modeling for diffuse and specular component in the first stage, where diffuse does not consider shading and specular and only consider reflection direction and normal as input of MLP. Since the geometry will be fixed in the following two stages, how would the design affect the geometry reconstruction?

Author response:
This is one of the main insights from our work. If the lighting and material model is too complex, the surface reconstruction will be unstable and overfit. If the lighting and material model is too simple, the surface reconstruction will interpret glossy reflections as bumps and grooves as shown in Fig.5 in the paper. We design a surface reconstruction method that is robust in real scenes which is a trade-off between being too complex and overly simplistic. We experimented with many different versions of simpler and more complex material and lighting models for surface reconstruction and this is the best one we found.