Paper: Fast Inverse Rendering by Unified Voxelization of Scene Representation (ICLR-2024)

Reviewer comment (Novelty):
The explicit scene representation of UniVoxel is very similar to Voxurf, except that it additionally predicts various properties of object materials in space. It is difficult to evaluate if adapting the semantic field during model training is novel in this paper.

Author response:
**[A1.1]:** Thanks for your valuable comment. We agree that some previous works like Voxurf have explored explicit scene representation, but this is not the main contribution of our method. We emphasize that our contributions are twofold. First, in contrast to previous methods that only use voxel grid to model geometry or appearance, we design a unified framework of scene representation, which allows for efficient learning of geometry, materials and illumination in a unified manner.   
**[A1.2]:** Second and more importantly, as also mentioned by reviewer Ft2v, we propose to leverage Spherical Gaussians (SG) to model the local incident light fields, which is capable of modeling the joint effects of direct lighting, indirect lighting and light visibility without the need of expensive multi-bounce ray tracing. The SG parameters of surface points can be predicted directly through voxel grid and lightweight MLP networks, which enables seamless integration of illumination modeling into the unified voxelization framework, thus significantly improving training efficiency.  
**[A1.3]:** We conduct experiments on Shiny Blender dataset [2] and the results are shown in **Table 4** and **Figure 9** of the appendix. We also highlight the comparison results with Voxurf in the following table, demonstrating that our method achieves better geometry quality than Voxurf.  
| Methods  | Normal MAE ↓ |
------------ | :----:
| Voxurf  | 18.110 |
| Ours | 9.292 |

Reviewer comment (Evaluation):
UniVoxel cannot recover Envmap compared to other related works, which may limit its application scenarios.

Author response:
**[A2.1]:** Thank you for your valuable comment. In theory, it is possible to filter the incident light map of all surface points to recover the environment map, but more sophisticated design is required to eliminate the influence of indirect lighting.  
**[A2.2]:** Although our UniVoxel, by utilizing SG to model the local incident light field, cannot directly acquire the environment map, it achieves comparable accuracy to the state-of-the-art methods while having higher training efficiency (about 40 $\times$ faster than MII [3] as shown in **Table 1** of the paper), which is the main advantage of our approach.  
**[A2.3]:** Environment map and incident light field are two different technical approaches for representing illumination. The former can recover the environment lighting of the scene, but the obtained lighting is not sufficiently good for direct application. On the other hand, the latter has higher training efficiency (18 minutes vs 58 minutes as shown in **Table 2** of the paper), although it cannot directly obtain the environment light. In the future, we will explore how to combine the advantages of these two methods.

Reviewer comment (Evaluation):
From the experimental results, as shown in Figure 3, there is no obvious advantage in predicting the effect of materials, and the results of relighting are not very prominent. As shown in Figure 5, it seems that albedo did not decompose successfully and retained the light and dark shadows of the scene itself.

Author response:
**[A3.1]:** Thank you for your meticulous comment. We acknowledge that our UniVoxel does not have significant advantages in terms of performance compared to other methods. However, as shown in **Figure 3** of the paper, our UniVoxel is able to recover more high-frequency details on the albedo maps. Additionally, our UniVoxel achieves comparable reconstruction quality to the state-of-the-art methods, with significantly higher training efficiency, which is the main advantage of our method.  
**[A3.2]:** Actually, we can control the shading component on the albedo maps by balancing the weight of the SG smoothness loss $L_{sg}$. In **Figure 11** of the appendix, we have demonstrated that the light and dark shadows on the albedo maps can be eliminated by reducing the weight of $L_{sg}$. Due to the complexity of lighting condition in outdoor scenes, the constraints on illumination should be relaxed to achieve more realistic materials.

Reviewer comment (Experiments):
Can you provide a video comparing the training speed with other methods? As demonstrated by the author of Plenoxels (Plenoxels vs. NeRF)  ref: https://alexyu.net/plenoxels/

Author response:
**[A5]:** Yes, we have included the video in the supplementary materials. It can be observed that when the training of our UniVoxel is almost completed, MII [3] is still in the process of recovering geometry. While the normal and albedo estimated by TensoIR [4] have rough outlines, their detailed parts still need several hours to optimize.

Reviewer comment (Experiments):
A question about the explicit voxelization of scene representation is that can the explicit voxelization of scene representation model reflective surfaces such as **metal or mirror**? Will the proposed method work better on those objects than NeRF?

Author response:
**[A4]:** Yes, we have conducted experiments on the challenging Shiny Blender dataset [2]. The quantitative results in **Table 4** of the appendix show that our UniVoxel achieves better geometry quality compared to other NeRF-like methods. The visualization in **Figure 10** of the appendix also demonstrates that our UniVoxel is capable of reconstructing more realistic materials such as metal.

Reviewer comment (Novelty):
The paper can include more related works, e.g.,  
- Yang, Wenqi, et al. "Ps-nerf: Neural inverse rendering for multi-view photometric stereo." ECCV, 2022.   
- Wang, Zian, et al. "Neural Fields meet Explicit Geometric Representations for Inverse Rendering of Urban Scenes." CVPR, 2023.   
- Mai, Alexander, et al. "Neural Microfacet Fields for Inverse Rendering." ICCV. 2023.  
- Zhang, Youjia, et al. "NeMF: Inverse Volume Rendering with Neural Microflake Field." ICCV. 2023.

Author response:
**[A6]:** Thanks for pointing this out. We have added the reference of these works in the related work section.

Reviewer comment (Novelty):
The paper's innovative contribution is nuanced, as it applies a latent volumetric representation to enhance efficiency in inverse rendering—a concept that has been extensively studied. Although it offers incremental advancements, the overall novelty is tempered by similarities to existing methods, such as those utilizing MLPs. Additionally, the application of Spherical Gaussians (SGs) in the context of lighting is previously detailed in works like PhySG, suggesting the paper's approach is not entirely unprecedented.

Author response:
**[A1.1]:** Thank you for your meticulous comment. We agree that volumetric representation has been studied in recent works, but this is not the main contribution of our method. We emphasize that our contributions are twofold. First, in contrast to previous methods that only use voxel grid to model geometry or appearance, we design a unified framework of scene representation, which allows for efficient learning of geometry, materials and illumination in a unified manner. Second and more importantly, as also mentioned by Reviewer Ft2v, we propose to leverage Spherical Gaussians (SG) to model the local incident light fields.  
**[A1.2]:** Although some works like PhySG [1] has explored the application of SG in the context of lighting, they directly represent the entire scene’s environment map with SG, requiring expensive multi-bounce ray tracing. In contrast, we utilize SG to model the local incident light radiance, which is capable of modeling the joint effects of direct lighting, indirect lighting and light visibility. And the SG parameters of surface points can be predicted directly through voxel grid and lightweight MLP networks, which enables seamless integration of illumination modeling into the unified voxelization framework, thus significantly improving training efficiency.  
**[A1.3]** We have reorganized the introduction section of the paper to clarify these major contributions.

Reviewer comment (Novelty):
While the paper acknowledges PhySG, it does not sufficiently recognize the prior exploration of SGs for modeling incident light. A more explicit acknowledgment would strengthen the paper by correctly attributing the origins of this idea.

Author response:
**[A2]:** Thanks for your suggestion. We have added more descriptions about PhySG in section 2.1 and section 3.4 of the main paper, and explained the differences between our approach and it.

Reviewer comment (Novelty):
The utilization of environment map-based lighting is a well-trodden area in research, evidenced by efforts like Nvidia’s nvdiffrec-mc. The paper's critique of environment map-based lighting in favor of SGs may not be entirely justified, and the simplifications made in visibility reasoning warrant a closer examination. The unconventional modeling of the environment map using 128 Spherical Gaussians in this paper departs from traditional methods and calls into question the asserted superiority of SG over environment maps. Therefore, claims regarding the advantages of SG should be made with greater technical precision and careful comparison.

Author response:
**[A3.1]:** Thanks for your valuable comment. Following your suggestion, we have adopted the conventional modeling of environment map, representing it as a learnable parameter of size 256x512x3. As shown in the following table, it obtains better reconstruction quality than using mixture of SG to represent environment map, and achieves comparable results to our method, but it also leads to much longer training time.  
**[A3.2]:** Environment map and incident light field are two different techniques for representing illumination. Methods based on environment map can usually achieve better reconstruction quality, but they require more training time due to the need of multi-bounce ray tracing. On the other hand, methods based on incident light field usually can achieve higher training efficiency. In the future, we will explore how to combine the advantages of both methods and obtain high-quality reconstruction effects with shorter training time.  
| Methods |  MVS PSNR ↑ | Albedo PSNR ↑ | Roughness MSE ↓ | Relighting PSNR ↑ |  Training Time ↓ |
 ------------ | :----: | :----: | :----: | :----: | :----: |
| Mixture of 128 SG  | 34.185 | 27.368 | 0.012 | 27.446 | 58 minutes |
| 256x512x3 learnable map | 35.042 | 29.369 | 0.010 | **29.592** | 2 hours |
| Ours  | **36.232** | **29.933** | **0.007** | 29.445 | **18 minutes** |

Reviewer comment (Evaluation):
The paper would benefit from an explanation of the observed floating noise in TensoIR's results, providing clarity on whether this is an artifact of the algorithm, a limitation of the model, or an issue with the dataset or experimental setup used.

Author response:
**[A4]:** Thank you for your insightful suggestion. TensoIR has achieved impressive results on the MII synthetic dataset [3], but its performance in real-world scenes is poor and may have floating noise. The main reason is that the environment in each view of the real-world scenes is not completely static, making it difficult to handle complex lighting with a single environment map. One solution is to use a separate environment map for each view, but this would significantly increase training time. Our approach only requires introducing the view embedding as an additional input to the illumination model, allowing us to effectively handle complex real-world lighting without affecting training efficiency.

Reviewer comment (Experiments):
The inclusion of experiments on the challenging shiny NeRF-synthetic dataset could significantly enhance the paper's empirical foundation. Successful results on this dataset would serve as a robust testament to the algorithm's capabilities.

Author response:
**[A5.1]:** Following your suggestion, we conducted experiments on the Shiny Blender dataset [4]. The quantitative and qualitative results of the estimated normal maps are shown in **Table 4** and **Figure 9** of the appendix. It can be observed that our approach achieves better geometry quality compared to our methods, especially in the specular surfaces.  
**[A5.2]:** We also present the qualitative comparison of geometry, materials and illumination on the Shiny Blender dataset in **Figure 10** of the appendix. Although our method does not recover the details of illumination, it can be able to predict realistic albedo and roughness, while TensoIR fails to reconstruct materials in the specular surfaces and bakes the lighting into the albedo maps.

Reviewer comment (Experiments):
It is important for the paper to detail the distinctions between the environment map used in this study and that employed by Nvdiffrec, particularly since the paper touts the superiority of Spherical Gaussians (SG) over environment maps. A clear comparison would illustrate the specific contributions and advantages of the proposed method.

Author response:
**[A3.1]:** Thanks for your valuable comment. Following your suggestion, we have adopted the conventional modeling of environment map, representing it as a learnable parameter of size 256x512x3. As shown in the following table, it obtains better reconstruction quality than using mixture of SG to represent environment map, and achieves comparable results to our method, but it also leads to much longer training time.  
**[A3.2]:** Environment map and incident light field are two different techniques for representing illumination. Methods based on environment map can usually achieve better reconstruction quality, but they require more training time due to the need of multi-bounce ray tracing. On the other hand, methods based on incident light field usually can achieve higher training efficiency. In the future, we will explore how to combine the advantages of both methods and obtain high-quality reconstruction effects with shorter training time.  
| Methods |  MVS PSNR ↑ | Albedo PSNR ↑ | Roughness MSE ↓ | Relighting PSNR ↑ |  Training Time ↓ |
 ------------ | :----: | :----: | :----: | :----: | :----: |
| Mixture of 128 SG  | 34.185 | 27.368 | 0.012 | 27.446 | 58 minutes |
| 256x512x3 learnable map | 35.042 | 29.369 | 0.010 | **29.592** | 2 hours |
| Ours  | **36.232** | **29.933** | **0.007** | 29.445 | **18 minutes** |

Reviewer comment (Novelty):
My major concern is that the utilization of feature grids for more efficient training (the major claim) is not a novel insight. For instance, in InstantNGP, it has been demonstrated that the combination of dense hashable grids and shallower MLPs significantly enhances efficiency. Furthermore, the memory concerns highlighted in this submission can potentially be alleviated by adopting more efficient data structures for the grid, such as hashable grids as employed in InstantNGP.

Author response:
**[A1.1]** Thank you for your valuable comment. We agree that some prior works have explored the use of feature grids to improve training efficiency, but this is not the main contribution of our method. We emphasize that our contributions are twofold. First, in contrast to previous methods that only use voxel grid to model geometry or appearance, we design a unified framework of scene representation, which allows for efficient learning of geometry, materials and illumination jointly.  
**[A1.2]** Second and more importantly, as also mentioned by Reviewer Ft2v, we propose to leverage Spherical Gaussians (SG) to model the local incident light fields, which is capable of modeling the joint effects of direct lighting, indirect lighting and light visibility without the need of expensive multi-bounce ray tracing. The SG parameters of surface points can be predicted directly through voxel grid and lightweight MLP networks, which enables seamless integration of illumination modeling into the unified voxelization framework, thus significantly improving training efficiency.  
**[A1.3]** We have reorganized the introduction section of the paper to clarify these major contributions.

**[A2]** Thanks for your construstive suggestion. The volumetric representation of our method is not in conflict with the hashable grids proposed in InstantNGP [1], and it is easy to apply it to our framework. We plan to implement it in the future.

Reviewer comment (Evaluation):
The observed performance improvement is incremental and does not surpass the performance achieved by previous methods in some cases. For instance, a notable limitation of the proposed method is that it often results in albedo estimates that include shading components as shown in Fig.5 first row, in contrast to TenSor RT.

Author response:
**[A3]:** Thanks for your meticulous comment. Actually, we can control the shading component on the albedo maps by balancing the weight of the SG smoothness loss $L_{sg}$. In **Figure 11** of the appendix, we have demonstrated that the shading components can be eliminated by reducing the weight of $L_{sg}$. Due to the complexity of lighting condition in outdoor scenes, the constraints on illumination should be relaxed to achieve more realistic materials.

Reviewer comment (Experiments):
The assertion that SH cannot effectively model higher-frequency illumination may not be conclusive, as it is based on testing only up to order-3 SH. Increasing the order of SH could potentially address this limitation. To provide a more compelling evaluation, it is advisable to ensure an equal number of parameters when comparing SG and SH representations.

Author response:
**[A4]:** Thanks for your valuable commet. Following your suggestion, we use a higher order SH representation of illumination (order-4 with 75 parameters). For a fair comparison, we also adopt the SG with 12 Gaussian lobes (72 parameters) for lighting modeling. The results on the MII synthetic dataset are presented in the following table. It can be observed that using higher order SH leads to some improvement in reconstruction performance, but using fewer parameters, SG performs better than SH, which demonstrates the effectiveness of SG in lighting modeling.  
| Methods  |  MVS PSNR ↑ | Albedo PSNR ↑ | Roughness MSE ↓ | Relighting PSNR ↑ |
 :------------: | :----: | :----: | :----: | :----: 
| SH (order-3 with 48 parameters)  | 35.328 | 29.185 | 0.020 | 28.981 |
| SH (order-4 with 75 parameters)  | 35.344 | 29.200 | 0.016 | 28.813 | 
| SG (12 lobes with 72 parameters)  | **36.177** | **29.746** | **0.008** | **29.148** |

Reviewer comment (Reproducibility):
Details missing: It is not mentioned clearly how the global illumination and self-occlusion are handled during relighting. Please see questions below for more detailed questions.

Author response:
**[A5]:** Thanks for your meticulous comment. We have added the relighting procedure in the Section B of the appendix. As the incident light field obtained from previous training is not applicable to the new illumination, we adopt a similar procedure as previous works like TensoIR [2] and NerFactor [3], where we compute light visibility using Eq.7 in the main paper and consider only direct lighting.

Reviewer comment (Experiments):
For SG-based illumination representation, how many parameters are used for representing the Gaussians? Is it comparable to the SH counterpart?

Author response:
**[A4]:** Thanks for your valuable commet. Following your suggestion, we use a higher order SH representation of illumination (order-4 with 75 parameters). For a fair comparison, we also adopt the SG with 12 Gaussian lobes (72 parameters) for lighting modeling. The results on the MII synthetic dataset are presented in the following table. It can be observed that using higher order SH leads to some improvement in reconstruction performance, but using fewer parameters, SG performs better than SH, which demonstrates the effectiveness of SG in lighting modeling.  
| Methods  |  MVS PSNR ↑ | Albedo PSNR ↑ | Roughness MSE ↓ | Relighting PSNR ↑ |
 :------------: | :----: | :----: | :----: | :----: 
| SH (order-3 with 48 parameters)  | 35.328 | 29.185 | 0.020 | 28.981 |
| SH (order-4 with 75 parameters)  | 35.344 | 29.200 | 0.016 | 28.813 | 
| SG (12 lobes with 72 parameters)  | **36.177** | **29.746** | **0.008** | **29.148** |

Reviewer comment (Novelty):
More focused on novelty. Several recent methods use feature volume plus MLP to jointly reconstruct materials and geometry, such as TensorIR and NeuralPBIR. The true difference of the proposed method and previous works is the local illumination model that can accelerate the optimization. Therefore, I feel authors can emphasize this more. For example, the introduction gives me the impression that this method is faster because it uses a voxel-based scene representation, which has been adopted by many previous works, but instead it probably makes more sense to emphasize the illumination model.

Author response:
**[A1]:** Thanks for insightful suggestion. Following the suggestion, we have rephrased the introduction section and emphasize more on the novelty regarding the proposed illumination model.

Reviewer comment (Experiments):
Geometry reconstruction. I feel one experiment that is missing is the geometry quality, especially compared to TensoIR. While it is mentioned in section 4.2, it is difficult to tell the differences. I am curious how different lighting representations will impact the geometry quality, especially for the highly specular surfaces and concave regions.

Author response:
**[A2.1]:** Thanks for your suggestion. Due to the lack of ground truth normal in MII dataset, we conducted additional experiments on the Shiny Blender dataset [1], and the quantitative results are shown in **Table 4** of the appendix. We also highlight the comparison results with TensoIR in the following table, showing that our method achieves more accurate geometry quality.  
**[A2.2]:** In **Figure 9** of the appendix, we demonstrate qualitative comparisons with other methods and different illumination models. It can be seen that our SG-based local illumination model produces higher geometry quality on specular surfaces compared to using environment maps. Additionally, using SH and NeILF as illumination models can achieve comparable geometry quality to ours. However, the quality of material recovered using our illumination model is superior to others, and the training efficiency is also higher, as demonstrated in **Table 2** of the paper.  
| Methods  | Normal MAE ↓ |
:------------: | :----:
| TensoIR  | 49.747 |
| Ours | 9.292 |

Reviewer comment (Novelty):
Reference. One concurrent work that can be discussed in the paper is the NeuralPBIR (Sun et al.), which accelerates the reconstruction process by precomputing visibility and GI from NeRF so that material reconstruction can be done through local optimization. The idea of avoiding expensive ray tracing is related, while I feel the local illumination model proposed here is more different from previous works.

Author response:
**[A3]:** Thanks for pointing us to the interesting work. We have added the reference of Neural-PBIR in the related work section.

Reviewer comment (Reproducibility):
Missing details. One detail I did not find in the paper is after predicting the SG parameters, how will you render the appearance? Are you going to sample rays uniformly, with importance sampling, or just compute the integral analytically? Please remind me if I missed this part in the paper.

Author response:
**[A4]:** We discuss how to render the appearance in the implementation details (Section B) of the appendix: we utilize Fibonacci sphere sampling method over the half sphere to sample incident lights for each surface point.

Reviewer comment (Reproducibility):
About limitation in Sec. F. When you say the proposed method needs more GPU memory, how much more GPU memory does it require?

Author response:
**[A6]:** We present the batch size and required GPU memory for training each method on the MII synthetic dataset in the following table. It can be seen that our method does not significantly exceed the GPU memory of other methods, thanks to our efficient implementation. However, for larger-scale scenes, we would need to increase the resolution of voxelization and also require more GPU memory. Nevertheless, this can be addressed by the hashable grids proposed in InstantNGP [2]. And we plan to implement it in the future.  
| Methods  | batch size | GPU memory |
:------------: | :----: | :----:
| TensoIR [3]  | 4096 | ~12 GB  |
| MII [4]     | 1024 | ~14 GB  |
| Ours     | 8192 | ~19 GB  |

Reviewer comment (Presentation):
In Figure 13, the color of the normal maps from different methods are very different. I wonder if that is caused by any visualization issue, like a different way to turn normals into RGB images?

Author response:
**[A7]:** Thank you for pointing this out. The way Nvdiffrec and Nvdiffrec-mc convert normal maps into RGB images is very different from other methods. We are working on it and will address this issue in the final version.