Paper: GNeRP: Gaussian-guided Neural Reconstruction of Reflective Objects with Noisy Polarization Priors (ICLR-2024)

Reviewer comment (Writing):
The explanation of the proposed method is unclear. What is the input? Is a polarization camera required for reconstruction? If not, is the polarization prior regressed during optimization?

Author response:
We add an illustration to  the Appendix B Figure 6. (a) to show the pipeline of training data preparation.  Please see the training data box for intuition. The inputs include polarization images (AoP and DoP), RGB images, and object masks. A polarization camera is needed for polarization image acquisition, we add the claim to the limitation discussion in Appendix E.

Reviewer comment (Evaluation):
The proposed approach neglects the effect of self-occlusions, which can be severe for complicated objects. The paper should at least include a qualitative discussion about the effects of self-occlusions.

Author response:
Thanks for pointing out self-occlusion as a potential challenge. We tested additional scenes and found the shading caused by self-occlusion is a limitation of our method, and we added a failure case in the limitation discussion shown in Figure 14.

Reviewer comment (Experiments):
The property of polarization is closely related to materials. The proposed approach relies on the intermediate output of a coordinate network and hence seems to handle only objects with a uniform material, as shown in Fig. 5.

Author response:
Thanks for the suggestion of testing scenes with mixed materials. During the discussion time, we processed the raw capture of the Camera scene in PMVIR. It’s a Sony camera with various materials. The reconstruction is shown in Figure 9. As is shown in the figure, our method reconstructs more details of small structures with different materials concurrently. However, the lack of GT mesh in the dataset makes quantitative evaluation unavailable.

Reviewer comment (Experiments):
The experiments are conducted only on relatively simple objects. It would be beneficial to show results on more complicated objects to better understand its performance.

Author response:
Thanks for suggesting testing complicated objects. In the initial submission, the Ironman scene shown in Figure 5 has detailed geometry such as the edges of armours. During the discussion time, we rendered two additional synthetic scenes (Bunny and Dragon) using Mistuba render. The results are shown in Figure 8 and Table 3. As shown in GT normal in Figure 8, The geometry of the Dragon is complicated such as claws, the head with feelers, and scales on the body. It shows our method reconstructs the geometry preserving more details while the geomrty of the other methods are over-smoothed.

Reviewer comment (Presentation):
It would be great if the authors could provide an illustration of how the polarization priors intuitively improve the performance of NeuS.

Author response:
Thank you for the suggestion on illustrations. We have illustrated the principle of how Gaussians of normals supervised by polarization improve NeuS in Figure 1. (b). We include the polarization prior in the Figure and modify the caption for clarification.

Reviewer comment (Theory):
I have some questions about Sec. 3.2.3, which I would like to understand better. They are not necessarily the weaknesses of the paper. (a) How to decide the scale factor $s$? Or do we assume the mean loss should be scale invariant and we only care its orientation?

Author response:
(a) We estimated the polarized 2D Gaussians based on unitary projected normals vectors derived from the Angle of Polarization ( the right term of Eq. 8). However, the projected normal vectors are unnecessarily unitary in the left term. The scale factor is multiplied for the equivalence of Eq. 8. The final loss gets rid of the dependent of it, so it isn’t explicitly calculated. We added a detailed description of Gaussian estimation in the Appendix A.3.

Reviewer comment (Reproducibility):
I have some questions about Sec. 3.2.3, which I would like to understand better. They are not necessarily the weaknesses of the paper. (b) Do we compute $\mu_i^{j}$ by projecting every $x_{i}^{j}$ along the ray to 2D image, i.e. $M'$ equal to $6M$?

Author response:
(b) In the paper, M and M’ are set to 6 and 4, respectively. To estimate 3D Gaussians, we sample 2 points along the ray (before and after the center) and 4 points along the rays passing adjacent pixels. Hence, when estimating 2D Gaussians in polarization images, we only query the corresponding pixels, and M’ is set to 4.

Reviewer comment (Theory):
I have some questions about Sec. 3.2.3, which I would like to understand better. They are not necessarily the weaknesses of the paper. (c) I can see that 2D convariance matrices computed from polarization and splatting can both indicate if geometry is changing rapidly in the neighborhood. But it is a bit difficult for me to understand why they should be equal. A side by side visualization of both 2D Gaussian distributions will be useful. In addition, will noisy AoP of diffuse surface cause the 2D covariance matrix from polarization to have very large eigen values?

Author response:
(c).1 The equivalence is ensured by the relation between the azimuth angle and the Angle of Polarization (AoP), shown in Figure 2 and the equation in the last paragraph of Sec. 3.1 (azimuth angle + pi/2 = AoP mod pi).  So we can recover the orientation of 2D projected normal vectors from AoP, which should align with the orientation of splatted 3D normal vectors. Therefore, covariance should be the same, too. A visualization of 2D Gaussians is Figure 4. (e). Low saturation white pixels indicate that the 2D Gaussian at that location is close to isotropic, resembling a circle. Conversely, high saturation areas indicate strong anisotropy, resembling an ellipse, with the color representing the orientation of the ellipse. This is shown in the left column of the figure.  

(c).2 The eigenvalues of covariance will be irregular in diffuse regions due to noisy AoP. Hence we introduce DoP reweighting to reduce their impact since DoP will be small in these regions, as mentioned in Sec. 3.2.3.

Reviewer comment (Writing):
I have some questions about Sec. 3.2.3, which I would like to understand better. They are not necessarily the weaknesses of the paper. (d) Typos: there is an extra $)$ in $L_{\text{mean}}$ and $L_{\text{conv}}$.

Author response:
(d)Thanks for the error pointed out. We have fixed it.

Reviewer comment (Experiments):
Results on less specular objects may be interesting. I notice that all real objects shown in the paper are quite glossy. Do we have results on less glossy objects? As shown in the paper, the AoP of diffuse surfaces can be noisy but hopefully the DoP weight can fix this issue.

Author response:
Thanks for the advice on the evaluation of diffuse scenes. We processed the raw capture of the Camera scene in PMVIR, which is a more diffuse-dominant scene. A visual comparison is shown in Appendix C.3. Since the ground truth mesh isn't provided, quantitative evaluation is unavailable. However, Figure. 9 shows that our method reconstructs more details of small structures such as buttons, knobs, and slots of the camera. This proves that our method can handle diffuse objects effectively.

Reviewer comment (Novelty):
Missing reference. One related paper is cite is: "Sparse Ellipsometry: Portable Acquisition of Polarimetric SVBRDF and Shape with Unstructured Flash Photography, Hwang et al."

Author response:
Thanks for the additional related paper mentioned. We have added it to the Related Work Section.

Reviewer comment (Presentation):
Figure 2: the polarized direction may use a different color from that of the x-axis. Zenith angle $\theta$ is mentioned but never discussed in the paper. It may be more useful to label out angle $\varphi$.

Author response:
Minor. We have changed the Figure 2 as the suggestion. The relation between AoP and the normal vector is more clear. Hope it can help you solve W1. (c).

Reviewer comment (Reproducibility):
Can we explain in more details how to compute 2D normal distribution from a polarized image?

Author response:
Q1. A detailed explanation of Gaussian calculation is added to the Appendix A.3. 2D Gaussians of polarization images are introduced in the second paragraph.

Reviewer comment (Experiments):
Can we further verify if 3D Gaussian distribution is necessary? If we only compute the normal following standard volume ray tracing and then use reweighted $L_{\text{mean}}$ loss, will the results be similar or worse?

Author response:
Q2. We think the question is the same as W2.

Reviewer comment (Evaluation):
The proposed method only works for glossy objects that exhibit clear polarization information.

Author response:
To validate the effectiveness of our method on diffuse objects, we processed the raw capture of the Camera scene in PMVIR. A visual comparison is shown in Appendix C.3. Since the ground truth mesh isn't provided, quantitative evaluation is unavailable. However, Fig. 9 shows that our method reconstructs more details of small structures such as buttons, knobs, and slots of the camera. This proves that our method can handle diffuse objects effectively.

Reviewer comment (Experiments):
The proposed method comes with many hyperparameters which are hard for users to set in practice. Parameters like alpha and beta typically vary between 0.1 and 1, contingent on the intricacy of the geometry. Regrettably, the authors did not carry out any experiments to assess the optimal selection of these parameters, potentially hindering the method's practical implementation and performance.

Author response:
The setting of hyper-parameters is relatively fixed in our method. For clarification, we rearrange the notation of the weights of loss functions in Eq. 9 and add a description of the settings in Appendix D.3. The only hyper-parameter that needs to be tuned is alpha. We tested different choices of alpha on the Bunny scene, as shown in Appendix D.3.2, to facilitate the practical implementation of the method.

Reviewer comment (Experiments):
Only the results of a few objects are shown in the experiments.

Author response:
In the initial submission, we showed reconstructed meshes of the Owl, Black Vase, Cat, and Vase scenes in the Appendix. Moreover, we rendered two additional synthetic scenes (Bunny and Dragon)  with ground truth normals during the discussion time, and qualitative and quantitative comparisons are shown in Appendix C.2, along with an additional evaluation metric, Mean Angular Error (MAE) of normals.

Reviewer comment (Writing):
There are typographical errors and improper terms, which are described in the questions.

Author response:
Thank you for pointing out the errors. We have fixed them.

Reviewer comment (Writing):
In Section 2.4, the authors stated that "the learned 3D Gaussians imply the anisotropic normals distribution of 3D points and capture more details of surface geometry." Could the authors elaborate on the description or provide some references? Does it mean the learned 3D Gaussian is anisotropic? And the anisotropicity results in more details?

Author response:
Yes, as parameterized by the covariance matrix, the learned Gaussians are anisotropic. Normal vector can be seen as the mean of 3D Gaussians, and changes of normal vectors within the neighborhood are captured by covariance of Gaussians. Therefore, we claim that the 3D Gaussians capture more details. We have modified the description in Sec. 2.4.

Reviewer comment (Experiments):
The ablation study could include an examination of the impact of alpha and beta. Additionally, the authors did not provide a clear rationale for why they included only two objects in Table 2 while incorporating four objects in Table 1. The ablation study of the other two objects is not supporting?

Author response:
As mentioned in W2, the required experiments are listed in Appendix D.3.2. Due to the limitation of computational resources, ablation studies were only done on two of the objects in Table 1. We plan to complete them when the computational resources become available. We’ll post them if the results are available during the discussion.

Reviewer comment (Theory):
In the last paragraph of Section 3.2.3, the authors stated "Intuitively, the first term measures the complexity of the geometry, while the second term reveals the specific geometric shape." Could the authors elaborate on this description?

Author response:
The first term supervises the eigenvalues of the covariance matrix, and the second term supervises the eigenvectors of the covariance matrix. Similar to PCA techniques, the covariance matrix determines the magnitude and direction of normal changes through eigenvalues and eigenvectors. If the local shape is like a plane, normals will change smoothly in all directions, and the difference between eigenvalues will be small, resulting in the Anisotropy (defined as the ratio of eigenvalues in the paper) approaching 1. If there are some details like edges, normals tend to change abruptly and exhibit directionality, which is represented by eigenvectors. We have modified the description in Sec. 3.2.3.