Paper: Large-Vocabulary 3D Diffusion Model with Transformer (ICLR-2024)

Reviewer comment (Reproducibility):
The paper lacks the introduction of the whole pipeline which links each module together and also the training pipeline which demonstrate which module is pre-trained and which is trained in each step. I am still confused how Step 1 and 2 are related, especially how the fitted tri-plane are used, or is it only the tri-plane decoder is used.

Author response:
[a] **Introduction of our pipeline**. Thanks for your careful advice. Note that in the sampling process, Step 1 aims to generate the triplane via the trained diffusion model while Step 2 focuses on rendering those generated triplanes via the trained triplane decoder. The fitted triplanes are only used to train the diffusion model. The trained decoder will used in fitting the triplanes (training) and rendering the generated triplanes (sampling).

We have revised our paper and added a detailed introduction of our workflow in the Appendix. Section 1.2 as follows:

Training: The training process consists of two steps. In the **step I. triplane fitting**, the objective is to obtain the diverse triplane features and robust triplane decoder. Therefore, for clarification, we divide **step I** into two subtasks: **step I-I** training shared decoder and **step I-II** optimizing triplanes from diverse 3D objects. 
To maintain the robustness of the decoder, we adopt around 20 percent diverse and high-quality objects for optimizing the shared decoder in the **step I-I**. 
Then, in **step I-II**, we adopt the trained decoder with frozen parameters to merely fit the triplanes. Finally, we can use those trained triplanes as the ground truth to train the 3D-aware transformer-based diffusion model in **step II**. 

Sampling: Similar to the training process, sampling the 3D content from DiffTF has two individual steps: 1) using a **trained diffusion model** to denoise latent noise into triplane features, and 2) adopting the **trained triplane decoder** to decode the implicit features into the final 3D content.

Reviewer comment (Evaluation):
The improvement over GAN-based method is not obvious when the texture is on. They all seem realistic.

Author response:
[b] **Comparison with GAN-based methods**. Although GAN-based methods can generate a series of 3D objects, as we have shown in our paper, they are not skilled in generating both the details of objects and rich semantical information. The semantic information of those 3D contents is ambiguous, posing huge challenges in categorizing those objects from a human perspective. Therefore, there is a significant gap between the generated objects of GAN-based methods with real-world objects.
In contrast to GAN-based approaches, our method can handle the challenges effectively relying on the novel 3D-aware modules. Benefiting from the generalized 3D prior and specialized feature, the 3D objects generated by our method encapsulate rich semantic information and reasonable details, making them more similar to real-life objects.

Reviewer comment (Theory):
The time complexity should be analyzed, with comparisons over prior methods.

Author response:
[c] **Analysis of complexity**. Thanks for your kind suggestion. We have provided the related information in the new version of the Appendix (Table. 3). Without the most efficient structure, our DiffTF is capable of generating a 3D object within 30 seconds. Additionally, to accelerate the speed of sampling further, we can adopt the DDIM with 50 diffusion steps. Adopting this setting, DiffTF can generate a 3D object within 9s.

Methods|FLOPs|Parameters
---|---|---
NFD w/texture |250.78G|124.01M
DiffRF|778.54G|251.29M
**DiffTF**|514.84G|734.40M

We sincerely appreciate your great efforts in reviewing this paper. Your constructive advice and valuable comments really help improve our paper. Please let us know if you have follow-up concerns. We sincerely hope you can consider our reply in your assessment, and we can further address unclear explanations and remaining concerns if any.

Reviewer comment (Evaluation):
Upon the efforts of enhancing 3D-awareness of the generation, the rendered results still suffer from noisy geometry and texture details. It is not well-discussed what could be the source of these limitations.

Author response:
[b] **More discussion on limitation**. We thank you for bringing this issue to our attention. We contend that one potential reason is the unoptimized noise scheduling scheme. According to prior work in image diffusion[1], adopting an appropriate noise scheduling scheme can improve the generative performance. Choosing the most proper noise scheduling scheme can boost the performance of DiffTF further. 

[1] Chen T. On the importance of noise scheduling for diffusion models[J]. arXiv preprint arXiv:2301.10972, 2023.

We sincerely appreciate your great efforts in reviewing this paper. Your constructive advice and valuable comments really help improve our paper. Please let us know if you have follow-up concerns. We sincerely hope you can consider our reply in your assessment, and we can further address unclear explanations and remaining concerns if any.

Reviewer comment (Presentation):
The method section shows that this is a two-stage model; but Fig. 2 seems to suggest that this is a single-stage model. The authors might want to make Fig. 2 clearer.

Author response:
[a] **Workflow of DiffTF**. Thanks for your careful advice. We have revised the Fig. 2 and the caption to make it more legible. The top of the new figure represents the first step while the bottom represents the second one.

Reviewer comment (Presentation):
For figure 4, since the generation results on OmniObject3D are class-conditioned, would it make sense to arrange the results in a specific order of classes in order for the readers to easily compare different methods?

Author response:
[b] **Comparison in a specific order**. Thanks for bringing this issue to our attention. According to your suggestion, we have added the additional visualization involving all class-conditional generative methods in the Appendix (Fig. 11). Note that EG3D does not support the class-conditional generation.

Reviewer comment (Reproducibility):
I have to dig into the appendix to find the architecture of the shared triplane decoder; there’s no reference to appendix in section 3.1. What’s worse, even looking at the appendix, I’m still unable to find out the width of the MLP layers for the triplane decoder.

Author response:
[c] **Details of the share-weight decoder**. We thank you for your kind advice. We have added the reference to the Appendix in Section 3.1. Detailed information about our triplane decoder has been added in the Appendix (Fig.2). The size of MLP can also be found in this figure. Please refer to it.

Reviewer comment (Presentation):
I find it a bit hard to imagine what exactly the architectures are for the different variants in the ablation “Studies of 3D-aware transformer modules”. The authors might want to elaborate more in the appendix, or better draw figures to the ablated architectures.

Author response:
[d] **Architectures in ablation studies**. We appreciate your detailed suggestion. For clarification, we have included a figure to illustrate the different structures used in our ablation studies in Appendix (Fig. 7)

Reviewer comment (Reproducibility):
Section 4.3 mentioned about the importance of triplane normalization. But this is never mentioned in the method section. In fact, the authors might want to show a histogram of the unnormalized triplane values in the appendix.

Author response:
[e] **Distribution of fitted triplanes**. Thank you for your insightful comment. We have added a histogram in the Appendix (Fig.13) to show the distribution of triplane values. As we mentioned in our paper, the range of -10 to 10 is too large for optimizing the diffusion model. Therefore, we normalize the triplane values to the range of -1 to 1.

Reviewer comment (Reproducibility):
What is the prediction objective of DiffTF’s diffusion model? Is it \epsilon-prediction, x_0 prediction or v-prediction?

Author response:
[f] **Prediction objective of DiffTF**. In the Appendix. Section 3.1, we report that we use \epsilon as the prediction objective of DiffTF.

Reviewer comment (Reproducibility):
What’s the size of the shared triplane decoder?

Author response:
[c] **Details of the share-weight decoder**. We thank you for your kind advice. We have added the reference to the Appendix in Section 3.1. Detailed information about our triplane decoder has been added in the Appendix (Fig.2). The size of MLP can also be found in this figure. Please refer to it.

Reviewer comment (Experiments):
For the 3D-aware transformer part, as the triplane resolution is only 16x16 and the patch size is 2, this means that there’re only 8x8=64 tokens on each plane, and 64x3=192 tokens in total. Why not just do self-attention over all these 192 tokens, instead of using the cross-plane attention? Would this simpler architecture also work? In fact, looking at figure 3(b), such simple self-attention is already used in the 3D-aware encoder/decoder.

Author response:
[g] **Cross-plane attention in DiffTF**. In this paper, we try to extract the generalized 3D prior from diverse objects for raising the performance of large-vocabulary generation. Therefore, the novel cross-plane attention should focus on the interdependencies among planes rather than the specialized feature mainly introduced by self-attention. 

Furthermore, the comparison of Ori-TF and CP-TF in our ablation studies (Table. 3) proves that the combination of cross-plane attention and self-attention (FID: 25.36) is better than the pure self-attention modules (FID: 52.35).

Reviewer comment (Experiments):
How does the proposed cross-plane attention compared against Rodin’s 3D-aware convolution? Does it make sense to compare with this work qualitatively and quantitatively, as the cross-plane attention seems a major contribution of this work?  
Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion

Author response:
[h] **Comparison with Rodin**. We appreciate your valuable advice. We have added the comparison with Rodin in the Appendix (Table. 1). As shown in the following table, our 3D-aware models achieve an impressive improvement in generative performance compared with 3D-aware convolution in Rodin. 

Methods|FID|KID(%)|COV(%)|MMD(‰)|
---|---|---|---|---
Rodin|84.55|4.5|33.79|11.47
**DiffTF**|**25.36**|**0.8**|**43.57**|**6.64**

Reviewer comment (Experiments):
It seems that the model is trained per-category on shapenet cars, chairs, airplanes? Why not train a class-conditioned generator just like what has been done on the OmniObject3d data?

Author response:
[i] **Class-conditional generation on ShapeNet**. Note that most prior works, such as NFD, evaluate the generative methods on a single category. For fairness of evaluation, we adopt the same setting. 
Furthermore, we have added the comparison of the class-conditional model trained on three categories of ShapeNet in the Appendix (Table. 4). The comparison between our single-category model and multi-categories model in the following Table shows that our method can extend to multi-categories without bringing significant drop in performance.

Category|Training Setting|Method|FID|KID(%)|COV(%)|MMD(‰)
---|---|---|---|---|---|---
Car|Single-category|DiffTF|36.68|1.6|53.25|2.57
Car|Multi-category|DiffTF|42.40|1.9|48.27|3.07
Plane|Single-category|DiffTF|14.46|0.8|45.68|2.58
Plane|Multi-category|DiffTF|19.92|1.1|41.02|2.91
Chair|Single-category|DiffTF|35.16|1.1|39.42|5.97
Chair|Multi-category|DiffTF|39.74|1.3|35.14|6.38