Paper: MATLABER: Material-Aware Text-to-3D via LAtent BRDF auto-EncodeR (ICLR-2024)

Reviewer comment (Writing):
I believe the paper could benefit from a better motivation. In particular, it is not clear why separating shading from reflectance is difficult and which approaches exists for this problem. Further, the lambertian assumption and its limitations require better explanation, to help the reader understand what are the challenges that this paper addresses and why the solution is valuable. I suggest looking into and referring the reader to "A Survey on Intrinsic Images: Delving Deep Into Lambert and Beyond, Garces et al. IJCV 2022) for a contextualization of this problem.

Author response:
A: Thanks for your suggestion, and we will refine our manuscript based on your comments. It's worth noting that the mentioned IJCV paper is more related to inverse rendering (a reconstruction task), which is conceptually different from text-to-3D (a generation task). In text-to-3D, only a text input provides loose semantic constraints about the shading and the reflectance, leading to a lot of ambiguities and uncertainties that simply applying inverse rendering techniques can not resolve.

Reviewer comment (Novelty):
The related work analysis is somewhat limited. First, I think this paper requires a more in-depth analysis of radiance field representations (At least Gaussian Splatting should be mentioned). Second, more recent work on generative models for material estimation should be included (ControlMat, UMat, SurfaceNet, etc.). Importantly, work on BRDF compression should also be analyzed (eg the work by Gilles Rainer et al. on neural BRDFs and BTF compression). Recent work on text-to-3D is missing (SMPLitex, HumanNorm), although these may be concurrent and thus not applicable to this submission. Finally, I think that this paper is also missing an analysis of illumination representations, as the authors only test basic environment maps, but other approximations (point lights, spherical gaussian, neural illumination approximations, spherical harmonics, etc) also exist and I believe should be mentioned.

Author response:
A: As discussed in the **Clarification**, our focus of this paper is on the material side rather than the representation side, thus discussing works on 3D representations in the related work and comparing different 3D representations may deviate from our focus. As mentioned in Section 3.4, we followed previous studies and used **DMTet, not Mip-NeRF**, as our 3D representation. Finally, we will revise our manuscript to include possible related work on material estimation, BRDF compression, and illumination representations.

Reviewer comment (Reproducibility):
There are important details missing, particularly in terms of computational cost.

Author response:
A: We will clarify this in the revision. Also, code and model to reproduce our results will be released. For hyperparameters of different losses, we borrow some part of the default setting of Neural-PIL and fine-tune them to ensure four losses decrease simultaneously.  For the computational cost, the optimization of each object will take around 40 minutes (20 minutes for geometry and another 20 minutes for appearance) on 4 A100 GPUs, and about 30G memory is used in each GPU. Finally, we follow standard practice in text-to-3D (e.g. Fantasia3D) to select initial geometry, which is ruled by the configuration file of each 3D object to be optimized. This part is not investigated in-depth as it is not our main focus.

Reviewer comment (Presentation):
There is no analysis of limitations or suggestions for future work. I believe these should be included.

Author response:
A: The discussion of limitations is included in the supplementary.

Reviewer comment (Experiments):
How much of the capabilities of this method are linked with the radiance field representation that was chosen? That is, why was MIP-NeRF chosen and what would happen if other model was selected instead?

Author response:
A: As discussed in the **Clarification**, our focus of this paper is on the material side rather than the representation side, thus discussing works on 3D representations in the related work and comparing different 3D representations may deviate from our focus. As mentioned in Section 3.4, we followed previous studies and used **DMTet, not Mip-NeRF**, as our 3D representation. Finally, we will revise our manuscript to include possible related work on material estimation, BRDF compression, and illumination representations.

Reviewer comment (Theory):
Why was Cook-Torrance chosen as the material model? What would happen if a more complex or a simpler model was used instead? This material model, in the form explained in the paper, does not model anisotropy, among other reflectance properties. This limits the generality of the materials and objects that can be generated with them.

Author response:
A: As mentioned, our work is the first one to introduce material prior to the task of text-to-3D. Being the first attempt, we choose Cook-Torrance as the material model and TwoShotBRDF as the source of our material prior, as they cover a large portion of possible materials, and we believe they are sufficient to verify the effectiveness and importance of involving such a material prior to text-to-3D. Our experiments also demonstrated this. Specifically, the Cook-Torrance model is a widely used material model in many commercial softwares such as UE4 and academic works. TwoShotBRDF is a large-scale BRDF dataset containing most of the commonly observed materials. In terms of fibrous objects, both its geometry and material are too challenging for current text-to-3D. Involving other material models, as well as adding more datasets to enhance the material prior is worth exploring, which is left as future work. Same as the handlement fibrous objects. We will clarify this in the revision.

Reviewer comment (Experiments):
Why was the TwoShotBRDF dataset used? There are plenty of other datasets of SVBRDFs available, of higher resolutions and with a different diversity of material classes. I am wondering how many of the limitations of the method (eg it struggles with metallic objects) are due to the dataset choice.

Author response:
A: As mentioned, our work is the first one to introduce material prior to the task of text-to-3D. Being the first attempt, we choose Cook-Torrance as the material model and TwoShotBRDF as the source of our material prior, as they cover a large portion of possible materials, and we believe they are sufficient to verify the effectiveness and importance of involving such a material prior to text-to-3D. Our experiments also demonstrated this. Specifically, the Cook-Torrance model is a widely used material model in many commercial softwares such as UE4 and academic works. TwoShotBRDF is a large-scale BRDF dataset containing most of the commonly observed materials. In terms of fibrous objects, both its geometry and material are too challenging for current text-to-3D. Involving other material models, as well as adding more datasets to enhance the material prior is worth exploring, which is left as future work. Same as the handlement fibrous objects. We will clarify this in the revision.

Reviewer comment (Experiments):
How were the hyperparameters of the different losses selected? What are their impact?

Author response:
A: We will clarify this in the revision. Also, code and model to reproduce our results will be released. For hyperparameters of different losses, we borrow some part of the default setting of Neural-PIL and fine-tune them to ensure four losses decrease simultaneously.  For the computational cost, the optimization of each object will take around 40 minutes (20 minutes for geometry and another 20 minutes for appearance) on 4 A100 GPUs, and about 30G memory is used in each GPU. Finally, we follow standard practice in text-to-3D (e.g. Fantasia3D) to select initial geometry, which is ruled by the configuration file of each 3D object to be optimized. This part is not investigated in-depth as it is not our main focus.

Reviewer comment (Evaluation):
In section 4, could the authors provide an in-depth analysis of the computational cost of each part of the method? I think it would be interesting to see timings, memory usage and FLOPs.

Author response:
A: We will clarify this in the revision. Also, code and model to reproduce our results will be released. For hyperparameters of different losses, we borrow some part of the default setting of Neural-PIL and fine-tune them to ensure four losses decrease simultaneously.  For the computational cost, the optimization of each object will take around 40 minutes (20 minutes for geometry and another 20 minutes for appearance) on 4 A100 GPUs, and about 30G memory is used in each GPU. Finally, we follow standard practice in text-to-3D (e.g. Fantasia3D) to select initial geometry, which is ruled by the configuration file of each 3D object to be optimized. This part is not investigated in-depth as it is not our main focus.

Reviewer comment (Reproducibility):
The authors mention that "we initialize DMTet with either a 3D ellipsoid, a 3D cylinder, ...". How is this selected? Is it automatic?

Author response:
A: We will clarify this in the revision. Also, code and model to reproduce our results will be released. For hyperparameters of different losses, we borrow some part of the default setting of Neural-PIL and fine-tune them to ensure four losses decrease simultaneously.  For the computational cost, the optimization of each object will take around 40 minutes (20 minutes for geometry and another 20 minutes for appearance) on 4 A100 GPUs, and about 30G memory is used in each GPU. Finally, we follow standard practice in text-to-3D (e.g. Fantasia3D) to select initial geometry, which is ruled by the configuration file of each 3D object to be optimized. This part is not investigated in-depth as it is not our main focus.

Reviewer comment (Evaluation):
What are the demographics of the user study? Are they instructed in any meaningful way? Can the authors provide a more detailed description of the test that each participant undertook? I am not convinced that it is fair to ask random people to measure the "disentanglement" of a text-to-3D generative model, as this is very hard to evaluate even for experts or automatic metrics.

Author response:
A: For the dimension of "disentanglement", we will present the shaded images of generated 3D objects and the corresponding diffuse map to the participants, and ask them whether the diffuse map contains weird and unexpected bright lights or shadows.

Reviewer comment (Evaluation):
How correlated are the generated objects with their materials? For example, if prompted for a "goblet", does it always generate metallic objects or are more variations allowed? How well does it generate implausible objects (eg "a wool goblet" or a "ceramic pizza", etc)?

Author response:
A:  Thanks for your suggestion. Our method is capable of generating implausible objects like "a wool goblet" and "a golden ice cream". We update our manuscript and include these results in Appendix Figure 8 (supplementary). More cases will be included later.

Reviewer comment (Experiments):
How does this method handle fibrous objects (eg a fleece fabric or a knitted teddy bear)?

Author response:
A: As mentioned, our work is the first one to introduce material prior to the task of text-to-3D. Being the first attempt, we choose Cook-Torrance as the material model and TwoShotBRDF as the source of our material prior, as they cover a large portion of possible materials, and we believe they are sufficient to verify the effectiveness and importance of involving such a material prior to text-to-3D. Our experiments also demonstrated this. Specifically, the Cook-Torrance model is a widely used material model in many commercial softwares such as UE4 and academic works. TwoShotBRDF is a large-scale BRDF dataset containing most of the commonly observed materials. In terms of fibrous objects, both its geometry and material are too challenging for current text-to-3D. Involving other material models, as well as adding more datasets to enhance the material prior is worth exploring, which is left as future work. Same as the handlement fibrous objects. We will clarify this in the revision.

Reviewer comment (Experiments):
I suggest the authors include the illumination used in every render, particularly when showing results of relighting. It is unclear which type of illumination was used. Given the images that are shown, my guess is that very diffuse illumination was used to render the objects, which makes me wonder how well these 3D objects look on more directional illumination.

Author response:
A: Thanks for your suggestion, and we will include the illumination in the revision.

Reviewer comment (Experiments):
Could the material MLP be used to generate different set of BRDF parameters from different material models (eg Phong, complex Cook-Torrance, etc.) from the same latent space? If so, how would it impact the final results?

Author response:
A: It is an interesting direction to explore in the future. It's possible to learn different latent spaces or a shared latent space for different material models, and subsequently use a set of MLPs to generate multiple sets of BRDF parameters.

Reviewer comment (Experiments):
The semantic-aware material regularization is not well evaluated in my eyes. Many previous works have proposed techniques to regularize the material values, e.g., Munkberg et al., 2022. To me, if the author claimed L_mat is their main contribution, more comparisons to previous techniques are expected. However, the paper only compares their method to the w/o L_mat baseline.

Author response:
A: In the last sentence of the method section, we state that our proposed regularization $\mathcal{L}_\text{mat}$ is more effective than the material smoothness regularization proposed in prior works (Zhang et al 2021, Munkberg et al, 2022). As requested, we include a comparison between different regularizations in Appendix Figure 7 (supplementary). More cases will be updated in a few days. From the case we can see, that text-to-3D requires not only two **geometrically close points** to have similar materials but also two **semantically close points** to have similar materials. Therefore previous material smoothness regularization is not sufficient. This is also an important challenge of text-to-3D that differentiates it from inverse rendering.

Reviewer comment (Evaluation):
I find that the paper randomly samples the environment maps from a pool of collections and randomly rotates the map during training. I think such a multi-light setup can reduce the ambiguity in the inverse rendering process a lot. However, as shown in Figure 8, the reconstructed albedo is disentangled without the BRDF prior and the L_mat. Can the author provide some insights on these unexpected results?

Author response:
A: This is exactly a specific example of challenges mentioned in Q1 that differentiate text-to-3D from inverse rendering. The task of text-to-3D only has a text input that provides loose semantic constraints to regularize generated materials via the SDS loss. The SDS essentially maximizes the similarity between the text input and images rendered from the target 3D content, and we empirically found some tiny bright lights and shadows in the rendered images can easily fool the SDS loss for a higher similarity. To summarize, although our material prior provides more information to help generate proper materials, and experiments demonstrate its effectiveness, the task of text-to-3D is very challenging and some ambiguities need further investigation to resolve.