Paper: TextField3D: Towards Enhancing Open-Vocabulary 3D Generation with Noisy Text Fields (ICLR-2024)

Reviewer comment (Experiments):
I would suggest also testing with the DreamFusion testing list, which contains over 400 prompts.

Author response:
We reviewed the original paper of DreamFusion and didn't find the testing list with over 400 prompts. The paper mentions that "*we use the 153 prompts from the object-centric COCO validation subset of Dream Fields*", which is exactly the set we used in our paper. However, we find a prompt list in a public repo [threestudio](https://github.com/threestudio-project/threestudio/blob/main/load/prompt_library.json), which includes 415 prompts. We evaluate the CLIP R-Precision in this test set. Since we haven't surveyed any method that is evaluated in this set, we only compare our method with Shap-E. As shown in Table R3, although it is a harder test set, our method still significantly outperforms Shap-E.

Table R3. CLIP R-Precision results on a test list with over 400 prompts.
|CLIP R-Precision|ViT-B/32|ViT-L/14|
|:----:|:----:|:----:|
|Shap-E (300M)|13.98|19.51|
|Ours, 1-shot|19.04|24.33|
|Ours, 9-shot|37.83|42.41|

Reviewer comment (Evaluation):
For the visual results, I would like to see more prompts from the DreamField or DreamFusion test sets, which include more challenging and complex examples like concept-mixing. It would help us better evaluate the open-vocabulary ability of the proposed method. Right now, the showcasing prompts only contain one simple concept and can be easily found in Objaverse training set. I understand that concept mixing is super hard given such limited training data. But at least, it is nice to see some failure cases and analysis on the failure modes.

Author response:
We select some complicated prompts from DreamField/DreamFusion evaluation sets and compare our results with Shap-E in Figure 15 (in the updated Appendix). Our results basically contain more detailed textures and more reasonable geometries when compared with Shap-E, exhibiting stronger open-vocabulary capability. As the reviewer mentioned in Weakness 4, concept mixing is super hard given such limited training data. 
Our method can understand most of the descriptions in the given prompts. While we acknowledge that the quality of our generation results is still not satisfactory enough. The generation quality compromises as the descriptions are increased. 
For example, the *giraffe* in Figure 15 is not as complete as the one generated in Figure 11. And our method is limited in imagination. As for the *chair that looks like an avocado*, our generated result simply combines the shape of a chair with the color of an avocado. Nonetheless, it is crucial to acknowledge the substantial contributions our work has made in advancing open-vocabulary text-to-3D with limited data.
We will add an additional discussion in the limitation part of our revised manuscript.

Reviewer comment (Experiments):
Shape Diversity is limited by the training dataset. Quoting from the original paper - "With limited 3D data, can we train a real-time generator that is equivalent to V-L optimized methods", I think this is impossible considering the method is training with a relatively small scale 3D shape dataset compared with VLMs. VL-optimized methods clearly can generate synthesized imagined shapes, like a chair with the shape of an avocado, but given the qualitative examples that the authors provide, this method seems not able to generate imagined shapes. Even though I don't think this is a major drawback of the proposed method, I think this claim is faulty.

Author response:
We thank the reviewer for the constructive comments.
### Q1. Diversity Claim
We acknowledge the reviewer's point regarding the challenge of achieving shape diversity with a smaller scale 3D shape dataset in comparison to 2D diffusion models. The term "equivalent" was perhaps optimistic, and we have amended our language to more accurately reflect our findings.In Table 1 of the main text, our CLIP R-Precision results significantly outperform previous 3D supervised methods and even get close to V-L pre-trained methods. 
The results indicate that our method exhibits its potential towards open-vocabulary generation. Accordingly, we revise the expression here as: *With limited data, can we train a real-time 3D generator with the potential towards open-vocabulary content creation?*

Reviewer comment (Experiments):
Open-vocabulary capability - The open-vocabulary capability is the major claim of this paper. However, in the qualitative experiment section, the paper only provides very simple prompts, like category names, adjective nouns, or a phrase with two nouns. I think these simple phrases are not complicated enough to prove the open-vocabulary capability of the method, especially considering the method is training with complicated enough captions generated by BLIP-2 or MiniGPT4. I hope the authors can provide more results from complicated text prompts as in the captions generated.

Author response:
We thank the reviewer for the constructive comments.
### Q2. Open-Vocabulary Capability
We select some complicated prompts from DreamField/DreamFusion evaluation sets and compare our results with Shap-E in Figure 15 (in the updated Appendix). Our results basically contain more detailed textures and more reasonable geometries when compared with Shap-E, exhibiting stronger open-vocabulary capability. As Reviewer 9TYY mentioned in Weakness 4, concept mixing is super hard given such limited training data. Our method can understand most of the descriptions in the given prompts. While we acknowledge that the quality of our generation results is still not satisfactory enough. The generation quality compromises as the descriptions are increased. For example, the *giraffe* in Figure 15 is not as complete as the one generated in Figure 11. And our method is limited in imagination. As for the *chair that looks like an avocado*, our generated result simply combines the shape of a chair with the color of an avocado. Nonetheless, it is crucial to acknowledge the substantial contributions our work has made in advancing open-vocabulary text-to-3D with limited data. We will add an additional discussion in the limitation part of our revised manuscript.

Reviewer comment (Experiments):
View-invariant experiments. The paper claims that the NTFBind model produces a view-invariant feature, but the experiment provided is not strong enough to prove the point. The experiment uses image features across views and image features across ShapeNet categories to prove that image features across views are more reassembled than features across categories. I think this setting is not strong and persuasive enough. A better experiment setup would be comparing features across views with features across instances in the same category. See CLIP-NeRF[1] Figure 2 for more details.

Author response:
We thank the reviewer for the constructive comments.
### Q3. View-Invariant Experiments
It is reasonable to compare features in the same category. We randomly select 50 instances in each category and calculate the average retrieval precision of all 55 categories in Table R1. Our view-invariant features still present an obvious improvement against vanilla CLIP features. Note that comparing different instances in the same categories is a more challenging task, as different instances in the same view sometimes share more similar content than the same instance in different views. We will update the results in our ablation study. Thanks for the suggestion.

Table R1. Comparison of view-invariant and vanilla CLIP features across instances in the same category. $r$ is the retrieval precision with CLIP features and $\widetilde{r}$ is the retrieval precision with our view-invariant features.
|Image Source|$r$|$\widetilde{r}$|
|:----:|:----:|:----:|
|Ground-Truth|45.60%|59.04%|
|Generated|44.84%|57.34%|

Reviewer comment (Experiments):
Comparison. Though the paper compared with VL optimized methods and 3D supervised methods, it didn't directly compare with the TAPS3D, which has the same training setting as the proposed method. Both of them methods use an image captioning model to augment text prompts and work on a 3D-generated model. Though the paper provides an ablation study that replaces the major component to the TAPS3D component, I wonder if it will outperform the TAPS3D original method.

Author response:
We thank the reviewer for the constructive comments.
### Q4. Comparison with TAPS3D
The reason why not compare TAPS3D is that it is a fine-tuning method based on single-category GET3D pre-training. In the implementation of the original TAPS3D, a pre-trained category-wise GET3D checkpoint (e.g., car checkpoint, table checkpoint, and so on) is first loaded to the generator. The generator is then fine-tuned by the supervision of a fixed discriminator. Considering that GET3D hasn't been pre-trained in an open-vocabulary setting, it is hard to evaluate the original TAPS3D in our benchmark.

Nevertheless, we still provide a comparison based on their official code. We load the checkpoint of our pre-trained GET3D (Table 2 in the main text) and further finetune the generator with TAPS3D. As shown in Table R2, its FID and CLIP-Score (ViT-B/32) results are uncompetitive to our method and even below the version we reproduced (named as (b) in the main text). We think the results are reasonable, as the performance of fine-tuned TAPS3D strongly depends on the pre-trained GET3D. We have demonstrated in Table 2 of the main text that GET3D can hardly generate high-quality 3D results in a multi-category dataset. As a result, TAPS3D is not an appropriate approach to open-vocabulary generation.

Furthermore, since TAPS3D hasn't released the training captions and finetuned checkpoints, we cannot even make a comparison on single-category generation with them. We notice that SDFusion has released its text-to-shape checkpoint, which is trained on a table-chair dataset. Therefore, we update Figure 10 in Appendix, further evaluating the original SDFusion with recursive text prompts. As shown in Figure 10, SDFusion is a shape-only generator with low generation resolution. Besides, it is not sensitive to long sentences, failing to generate curved legs in the last prompt.

Table R2. Performance of the original TAPS3D. The results of GET3D and (b) are from Table 2 in the main text.
|Method|FID|CLIP-Score|
|:----:|:----:|:----:|
|GET3D|41.66|-|
|TAPS3D|35.12|28.38|
|(b)|29.02|29.80|
|Ours|26.94|30.35|

Reviewer comment (Experiments):
For noisy text latent code, the paper uses a learned noise, which is referred to as "dynamic noise". I wonder how the method performs with a non-dynamic noise, which could set \sigma to a static number.

Author response:
### Q5. Dynamic Noise
We have tried this non-dynamic noise before. We set $\sigma$ to a static number 0.016, which is exactly the setting in [A]. We trained the non-dynamic noise in ShapeNet55 dataset with BLIP-2 captions, which is evaluated with 27.64 in FID and 30.06 in CLIP-Score (ViT-B/32). The results are not competitive to our dynamic noise (26.94 in FID and 30.35 in CLIP-Score). We will add the results to Appendix.

Reviewer comment (Reproducibility):
Textured Mesh Generator. Are they training from scratch, or training from the pre-trained GET3D?

Author response:
### Q7. Textured Mesh Generator
We train the generator from scratch. We aim to train a unified generator for multiple categories, while GET3D aims to train category-wise generators. Loading its pre-trained model is not beneficial for our training.

Reviewer comment (Writing):
Silhouette loss. The silhouette loss in equations (4) and (5) is not introduced clearly. For a first-time reader, it might be a little confusing.

Author response:
### Q8. Silhouette Loss
Thanks for the suggestion. We will provide an additional explanation in our revision that the silhouette loss is followed from GET3D.

Reviewer comment (Experiments):
My main concern is the potential overfitting problem. In Figure 12 and 13, certain prompts (e.g., "A beer can", "A wooden crate", and "A cardboard box with graffiti") generate unusually detailed outputs, showing a much higher level of details than others. Based on my experience, the training dataset likely contains very similar examples. I am interested in understanding how the authors have addressed and evaluated the risk of overfitting associated with their method.

Author response:
Overfitting in generative models is a complicated issue that is worth investigating. In [A], overfitting is defined as "*containing an object (either in the foreground or background) that appears identically in a training image, neglecting minor variations in appearance that could result from data augmentation*". They detect and analyze content replication in the diffusion synthesis results, concluding that most of the generated images do not contain significant copied content, but a non-trivial amount of copying does occur. 

Following [A], we detect the most similar object to our generated *beer can* among the Objaverse dataset, which is shown on the left of Figure 14 in the updated Appendix. 
Our generated *beer can* only shares a similar rectangular text box with it. 
Similar to the conclusion in [A], our text box is a regional copy, rather than a significant content replication.
The same phenomenon can also be observed in the graffiti of the bus in Figure 15, which is similar to the graffiti on the cardboard you mentioned.
Therefore, those unusually detailed outputs may be a possible regional copy but are not exactly the overfitting problem. 

To some extent, our NTFGen module is beneficial for alleviating overfitting, as the injected noise enlarges the text input to a range of objects. 
As shown in Figure 14 (in the updated Appendix), we provide 9-shot examples of our generated *beer cans* and *vases*. Our results present various textures. Few of them share a similar color or texture. 
We will update the discussion of overfitting in the Appendix accordingly.

Reviewer comment (Experiments):
Can you provide some more examples of the 9-shot experiments?

Author response:
Our 9-shot experiment is to generate an object nine times and choose the best generation based on CLIP similarity. We include this experimental setting because our generation process is significantly faster than other methods. As an example of visualization results, we update nine beer cans and vases in Figure 14. Our generation results vary greatly in geometry and texture, presenting our diversity in a given text prompt. In the 9-shot experiment, we chose the most similar rendered image (the first one, based on CLIP similarity) for calculating R-Precision.