Paper: Compact Text-to-SDF via Latent Modeling (ICLR-2024)

Reviewer comment (Evaluation):
An SDF-based representation can only represent the geometry of the object without appearance, which limits the usage of the generated shape.

Author response:
We acknowledge this limitation. It's important to note that the primary focus of this paper is on lightweight 3D geometry generation. The current prevalence of SDF-based representations is primarily associated with detailed 3D geometry. We have the option to incorporate texture onto the 3D mesh, drawing from existing texture synthesis research [4] or utilizing texture generation products [5].

[4] Text2Tex: Text-driven Texture Synthesis via Diffusion Models, 2023

[5] MeshyAI, https://twitter.com/jimmygunawanapp/status/1724969804040389036

Reviewer comment (Experiments):
The categories of generated objects demonstrated in the experiment are very limited. Is it possible to generate more flexible objects (for example, hamburger)?

Author response:
We seriously consider the point you mentioned in the rebuttal. To reconstruct a hamburger, we train our model on a small subset of Objaverse-Cap3D pairs [1,2]. Please refer to: https://drive.google.com/file/d/1CnzMd5MOngOxYHo_27suHdQCmDKOV_O5/view?usp=sharing, which showcases a variety of hamburgers, each with different numbers and types, demonstrating our model's ability to generalize compositionally (to some levels).

In the submission, we provided more results at Appendix C. Additionally, we used the trained model in the submission to generate more results from diverse furniture categories. Please refer to https://drive.google.com/file/d/1wM9mHzUehW_FYB8ITVUZaB8Byx8JI16O/view?usp=sharing.

Reviewer comment (Evaluation):
The results of image-conditioned generation in Fig.1 seem not so similar to the input image.

Author response:
We currently have very limited image-3D pairs on the Pix3D dataset. This situation could be alleviated by using Objaverse dataset. Please allow us to further improve this point later due to computation constraints.

Reviewer comment (Novelty):
Limited technical novelty: The proposed method follows a two-stage approach: Firstly, fitting the DeepSDF latent code and subsequently training conditional diffusion models in the latent space. This latent diffusion paradigm has been standard and extensively studied. The novelty of the paper appears more in the introduction and improvement of minor techniques, like 1) fitting DeepSDF with Gaussian perturbations, and 2) incorporating hard example mining, inspired by Curriculum DeepSDF. Overall, there's a noticeable lack of fresh technical insights.

Author response:
> Limited technical novelty: The proposed method follows a two-stage approach: Firstly, fitting the DeepSDF latent code and subsequently training conditional diffusion models in the latent space. This latent diffusion paradigm has been standard and extensively studied. The novelty of the paper appears more in the introduction and improvement of minor techniques, like 1) fitting DeepSDF with Gaussian perturbations, and 2) incorporating hard example mining, inspired by Curriculum DeepSDF. Overall, there's a noticeable lack of fresh technical insights.

We would like to point out that the introduction of Gaussian perturbation in the latent SDF modeling and diffusion model has not been previously explored. Also, Curriculum DeepSDF did not investigate a hard example mining mechanism. 

We view our contribution as tailored latent diffusion for compact text-to-3D modeling: (1) the choice of SDF representation offers superior efficiency in detailed shape representation and ray tracing compared to the NeRF; (2) the integration of the DeepSDF objective with the diffusion objective via properly introducing Gaussian noise improves the performance; (3) a comprehensive examination of DeepSDF design choices further boost final performance. 

All of the above represent a non-trivial endeavor that contributes to the achievement of our ultimate solution for compactness.

Reviewer comment (Novelty):
Recent state-of-the-art methods, such as DiffRF (CVPR 2023), 3DShape2Vecset (SIGGRAPH 2023), and Shap-E (arXiv), have also leveraged the idea of performing diffusion in the latent space of NeRF/SDF/occupancy. This paper lacks an in-depth comparison and discussion concerning these methods.

Author response:
> Recent state-of-the-art methods, such as DiffRF (CVPR 2023), 3DShape2Vecset (SIGGRAPH 2023), and Shap-E (arXiv), have also leveraged the idea of performing diffusion in the latent space of NeRF/SDF/occupancy. This paper lacks an in-depth comparison and discussion concerning these methods. 

Incorporating your valuable suggestions, we have included a comparative analysis with Shap-E and 3DShape2Vecset in the table below. As 3DShape2Vecset did not originally support text-conditional generation, we mainly compare its model details. (We tried to extend 3DShape2Vecset, `kl_d512_m512_l8` and `kl_d512_m512_l8_d24_edm`, to support text-conditional generation, but failed to do so. It seems finetuning their model costs a lot of GPU memory, and we can barely support batch size 1 backpropagation.)

------------
|                | Latent Length | Encoder Parameters | Decoder Parameters | Speeds (s) |
| -------------- | ------------- | ------------------ | ------------------ | ---------- |
| 3DShape2Vecset | 4,096         | 106,116,097        | 164,218,368        | 11.16      |
| Shap-E         | 1,048,576     | 443,771,357        | 315,692,032        | 16.18      |
| Ours           | 256           | 3,564,502          | 3,904,640          | 1.87       |
------------


Based on the table, our proposed model features significantly fewer parameters in both the encoder and diffusion model when compared to 3DShape2Vecset and Shap-E.
Besides, 3DShape2Vecset employs a fixed predefined resolution (its Section 5.3) and limited input resolution, i.e. 2048 points ([ref1](https://github.com/1zb/3DShape2VecSet/blob/bedbd1091664be8e2409d580c4e1630df1a37c89/main_class_cond.py#L39), [ref2](https://github.com/1zb/3DShape2VecSet/blob/bedbd1091664be8e2409d580c4e1630df1a37c89/main_ae.py#L36)), while our proposed approach offers the distinct advantage of unlimited query resolution. 

Unfortunately, we were unable to include DiffRF in this comparison as it has not released its source code. We welcome any additional suggestions or feedback you may have in this regard.

Reviewer comment (Experiments):
The ablation study seems to miss experiments that demonstrate the role of hard example mining.

Author response:
> The ablation study seems to miss experiments that demonstrate the role of hard example mining.

In the submission (09/28/2023), we included the related ablation in Table 4 and further fine-grained results in Table 5. They are both in the appendix.

Reviewer comment (Experiments):
The claim on page 8 regarding the lightweight framework outperforming larger models like Shap-E and Point-E ("A comparison with Point-E and Shap-E reveals that a lightweight framework can outperform these larger models with training over limited high-quality 3D-text data in the specific geometrical-text- guided 3D shape generation task.") seems unsubstantiated. To strengthen this claim, the paper should consider re-training or fine-tuning Shap-E/Point-E on the same dataset and comparing results.

Author response:
> The claim on page 8 regarding the lightweight framework outperforming larger models like Shap-E and Point-E ("A comparison with Point-E and Shap-E reveals that a lightweight framework can outperform these larger models with training over limited high-quality 3D-text data in the specific geometrical-text- guided 3D shape generation task.") seems unsubstantiated. To strengthen this claim, the paper should consider re-training or fine-tuning Shap-E/Point-E on the same dataset and comparing results.

Following your suggestions, we re-training Shap-E/Point-E on the same dataset. Here’s our anonymized training codes: https://drive.google.com/drive/folders/1m-qJEbHGVqB-rvp3WjeMkUnuxfY2RHxb?usp=sharing. Some details are listed below.

For shap-E: we leveraged the pre-trained encoder, specifically [this transmitter](https://github.com/openai/shap-e/tree/main/shap_e/models/transmitter), and retrain the latent diffusion model from scratch. We opted for the use of a pre-trained encoder because training an encoder from scratch with our data proved to be unfeasible in achieving satisfactory performance.

For point-E, we retrain the base diffusion model (producing 1,024 points), and use the pre-trained upsampling diffusion model (1024 points -> 4096 points) and the pre-trained point-to-sdf model (the regression-based model mentioned in Section 4.5 of point-E paper).

Qualitative results are here: https://drive.google.com/file/d/1wfrBTroGIa84e2d4H8r3EiVKPJN1PCVf/view?usp=sharing. It shows the point-E model failed to predict desirable objects with smooth surface even with pre-trained upsampling and point-to-sdf models. Shap-E model get improvements by train from scratch along with the pre-trained encoder. 


The table below demonstrates that Shap-E (re-train) achieves improvements over Shap-E (pre-train) while maintaining competitive performance compared to Compact-DiffSDF, despite having 100 times more parameter, `(443771357+315692032)/(3564502+3904640)=101.68013796`.

---
| ShapeGlot | FID⬇️ | MMD⬇️ | TMD⬆️ |
| --------- |  ---- | ---- | ---- |
| Shap-E (pre-train)    |  92.1 | 11.45 | 2.07 |
| Shap-E (re-train)    |  78.5 | 6.89 | 3.04 |
| Ours      |  73.4 | 6.42 | 2.87 |
----



It's important to highlight that the original training datasets for Shap-E and Point-E are substantially larger, ranging from 100 to 1,000 times the size of our dataset. Given this considerable difference in data scale, it's essential to acknowledge that fine-tuning with pre-trained weights may not result in a fair comparison.

Reviewer comment (Experiments):
In page 3, Section 3.2, the paper devises its method for obtaining text embeddings. Why not utilize pre-trained language or multimodal models?

Author response:
> In page 3, Section 3.2, the paper devises its method for obtaining text embeddings. Why not utilize pre-trained language or multimodal models?

For the sake of compactness, we did not leverage pre-trained language/multimodal models. We employed [the CLIP model](https://github.com/openai/shap-e/blob/main/shap_e/models/generation/pretrained_clip.py), specifically ViT-L/14, as used in Shap-E, to obtain text embeddings. It improved our performance over ShapeGlot by 5.2 on FID and 0.71 on MMD. However, this improvement results in a significant increase in parameters as shown in the below table.

------------------------
|                                                              | total       | clip.model vs. text_emd | clip_embed vs. text_pos_emb |
| ------------------------------------------------------------ | ----------- | ----------------------- | --------------------------- |
| [CLIP Model](https://github.com/openai/shap-e/blob/main/shap_e/models/generation/pretrained_clip.py) | 427,813,377 | 427,616,513             | 196,864                     |
| Ours                                                         | 12,845,056  | 12,713,984              | 131,072                     |
-------------------------

The specific implementation of our current text-embedding modules and the one use CLIP are listed below. Based on the above parameter-Table, we can tell our current implementation has $0.03 \times$ parameters of the CLIP model (`12845056/427813377=0.0300249`). 

Thank you for raising this up, and we are happy to include this discussion in the revision. 


```
## our current text modeling
## text are tokenized via BPE (no parameters)
self.dim = 256
self.text_seq_len = 512
self.num_text_tokens = 49664 ##vocabulary size
self.text_emb = nn.Embedding(self.num_text_tokens, self.dim)
self.text_pos_emb = nn.Embedding(self.text_seq_len, self.dim)

## CLIP modeling (https://github.com/openai/shap-e/blob/main/shap_e/models/generation/pretrained_clip.py)
self.clip = FrozenImageCLIP(torch.device('cuda' if torch.cuda.is_available() else 'cpu'))
self.clip_embed = nn.Linear(
    self.clip.feature_dim, self.dim, 
)
```

Reviewer comment (Experiments):
How does the proposed model perform when subjected to "out-of-distribution" or more creatively constructed text prompts?

Author response:
> How does the proposed model perform when subjected to "out-of-distribution" or more creatively constructed text prompts?

In our submission, we have included the top-2 nearest neighbor results in Appendix Figure 10, specifically for cases where the text prompts fall outside the training distribution. However, it's worth noting that for highly imaginative text prompts, such as one describing `a table with 100 legs`, the current model may struggle to generate accurate responses.

Reviewer comment (Novelty):
Most strategies introduced in Sec. 3.3 lack enough novelty.

Author response:
Thank you for bringing these relevant works to our attention! We will incorporate a discussion of these works into our revision and provide proper citations. 

We've explored the use of Gaussian noise for 3D shape generation, as demonstrated in DeepSDF (equation (1) in our paper). Notably, we've extended this approach to better align with the diffusion objective, as outlined in equation (4) in our paper, which is new. Additionally, it's important to highlight that the effective integration of hard example mining and latent-code-based SDF reconstruction is both new and vital to our ultimate solution for compactness.

Reviewer comment (Novelty):
Augmenting the latent space with Gaussian noise to improve the 3D shape generation quality of an implicit decoder is not new. Example prior work:  
[a] Sanghi et al. CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation. CVPR 2022.  
[b] Park et al. Deepsdf: Learning continuous signed distance functions for shape representation. CVPR 2019.

Author response:
Thank you for bringing these relevant works to our attention! We will incorporate a discussion of these works into our revision and provide proper citations. 

We've explored the use of Gaussian noise for 3D shape generation, as demonstrated in DeepSDF (equation (1) in our paper). Notably, we've extended this approach to better align with the diffusion objective, as outlined in equation (4) in our paper, which is new. Additionally, it's important to highlight that the effective integration of hard example mining and latent-code-based SDF reconstruction is both new and vital to our ultimate solution for compactness.

Reviewer comment (Novelty):
Using the top-k highest losses for hard example mining is not new. Please consider citing related work and discussing the difference. Example prior work:  
[a] Fan et al. Learning with Average Top-k Loss. NeurIPS 2016.  
[b] Yu et al. Loss Rank Mining: A General Hard Example Mining Method for Real-time Detectors. IJCNN 2018.

Author response:
Thank you for bringing these relevant works to our attention! We will incorporate a discussion of these works into our revision and provide proper citations. 

We've explored the use of Gaussian noise for 3D shape generation, as demonstrated in DeepSDF (equation (1) in our paper). Notably, we've extended this approach to better align with the diffusion objective, as outlined in equation (4) in our paper, which is new. Additionally, it's important to highlight that the effective integration of hard example mining and latent-code-based SDF reconstruction is both new and vital to our ultimate solution for compactness.

Reviewer comment (Experiments):
The ABO dataset used in the paper contains multiple categories other than chairs and tables. The paper has only shown results of shape reconstruction on multiple categories and text-to-3D generation on chair-like & table. It would be much better to help understand the performance if the authors could also include some results of text-to-3D generation on other non-chair-like/table categories, e.g., lamp, dresser, etc., (although the quality might be a bit worse than chairs/tables due to fewer training examples).

Author response:
Thanks, we’ve generated more results on other categories during the rebuttal! We trained a model with 3D-text pairs from a small subset from Objaverse-Cap3D datasets [1,2]. Please refer to https://drive.google.com/file/d/1-GYKMrnuqFQD3J1XkTJt8j0zkxbyF5KB/view?usp=sharing.

[1]Objaverse: A universe of annotated 3d objects, CVPR23

[2]Scalable 3D Captioning with Pretrained Models, NeurIPS23