Paper: Re-imagine the Negative Prompt Algorithm for 2D/3D Diffusion (ICLR-2024)

Reviewer comment (Evaluation):
The results part lacks quantitative fidelity results like FID, clip similarity, and user preference.

Author response:
Dear Reviewer 6H1W,

Thank you for your thorough review and positive feedback on our paper. We appreciate your recognition of our novel approach in solving the Janus problem using a novel negative prompt algorithm.

We acknowledge your suggestion regarding the inclusion of quantitative fidelity results like FID, CLIP similarity, and user preference. We agree that these metrics will enhance our paper's robustness and will include them in our revised submission.

Thank you again for your support and positive comments on the value of our work!!

Reviewer comment (Experiments):
The proposed method seems to be applicable in many tasks where gradients are mixed during inference (See [1] for an example). But the authors only focus on alleviating the Janus problem in 3D generation. I think it would be a stronger paper if more applications are included.

Author response:
**1-Expanding the Scope Beyond 3D Generation:**

We appreciate your emphasis on the broader potential applications of our Perp-Neg methodology. Initially, our focus was predominantly on 3D generation due to its profound impact and the specific challenges presented by the Janus problem in score distillation-based methods. Recognizing the value of your suggestion, we will extend our research to encompass additional applications, such as segmentation mapping and guided bounding box generation, and leave the rest for future work.

Reviewer comment (Experiments):
The number of qualitative results of 3D generation is too small. All I can find is Figure 2, 7, 20, which contain 9 examples with overlapping prompts (e.g. lions). The Janus problem is very common in score distillation based generation methods, so more such comparisons with a diverse set of prompts should be presented.

Author response:
**2-More 3D Experiment**

 We thank you for pointing out the need for a wider array of examples. Initially, we presented results from 12 prompts, totaling 240 experiments. Based on your valuable feedback, we expanded our dataset to include 10 additional prompts, each tested with and without Perp-Neg. The diversity and increased number of experiments strengthen our findings, as detailed in the revised Table.

Please refer to the response provided to reviewer vLLW question 2nd.

Reviewer comment (Experiments):
I think this method is interesting, but why do you focus so much on the Janus problem of 3D generation, instead of trying to demonstrate the method's effectiveness on more applications?

Author response:
**1-Expanding the Scope Beyond 3D Generation:**

We appreciate your emphasis on the broader potential applications of our Perp-Neg methodology. Initially, our focus was predominantly on 3D generation due to its profound impact and the specific challenges presented by the Janus problem in score distillation-based methods. Recognizing the value of your suggestion, we will extend our research to encompass additional applications, such as segmentation mapping and guided bounding box generation, and leave the rest for future work.

Reviewer comment (Theory):
The assumption of this method is based on the pependicular gradient sampling, but this design is somewhat heuristic, and have not been proved in the paper.

Author response:
**1-Regarding Perpendicular Gradient Sampling Theory:**

We acknowledge your concern regarding the lack of formal mathematical proof for our perp-neg method. While a formal mathematical proof is not presented, we believe our geometric reasoning and mathematical motivations, detailed in sections 2.2.1 and 2.2.2, provide a solid foundation for our approach. We appreciate your pointing out this limitation and will consider adding a more rigorous theoretical underpinning in future work.

Reviewer comment (Evaluation):
Weakness 1 caused the effectiveness of this method on text-to-3D is limited actually. Janus problems almost cannot be mitigated according to implementation in threestudio. [1]

Author response:
**2-Effectiveness of Perp-Neg in alleviating the Janus Problem - Threestudio:**

We acknowledge your concern about our method's limited effectiveness in text-to-3D as highlighted in the Threestudio implementation.

It is essential to clarify that the Threestudio version is a derivative of our original algorithm and does not mirror our methodology or results. In our experiments, we observed a significant reduction in the Janus problem, as presented in the appendix. 

In our experiments, comprising 240 trials (120 with Perp-Neg and 120 without), we observed a significant mitigation of the Janus problem. The following table, which we plan to include in the revised manuscript, details our findings:

| Algorithm                     | Perp-Neg Usage    | Acceptance Rate | Total Trials | Total Prompts |
|-------------------------------|-------------------|-----------------|--------------|---------------|
| Stable-Diffusion DreamFusion  | With Perp-Neg     | 20.24%          | 84           | 6             |
| Stable-Diffusion DreamFusion  | Without Perp-Neg  | 7.14%           | 84           | 6             |
| DeepFloyd-IF DreamFusion      | With Perp-Neg     | 50.00%          | 36           | 6             |
| DeepFloyd-IF DreamFusion      | Without Perp-Neg  | 2.78%           | 36           | 6             |

Overall, our method with Perp-Neg demonstrated a 29.17% improvement over the non-Perp-Neg approach (5.83%). These results are presented in the original manuscript with 12 different prompts in total. We have mentioned the success rate for each prompt in the paper.


It's crucial to note that while we haven't entirely solved the Janus problem, our method offers a significant advancement in addressing it. We will clarify this in the paper.

Following your suggestion and other reviewers, we will also include an additional 120 experiment results on text-to-3D experiments in the appendix to further substantiate our claims. The table of the new results is presented in the following:

| Prompt                                                             | % Success with Perp-Neg | % Success without Perp-Neg |
|:-------------------------------------------------------------------|------------------------:|---------------------------:|
| Michelangelo style statue of an astronaut                          |                 50.00%  |                    16.67% |
| A blue poison-dart frog sitting on a water lily                    |                 66.67%  |                    33.33% |
| A humanoid robot using a laptop                                    |                 50.00%  |                    16.67% |
| A kangaroo sitting on a bench playing the accordion                |                 50.00%  |                     0.00% |
| A rabbit cutting grass with a lawnmower                            |                 66.67%  |                    33.33% |
| A teddy bear pushing a shopping cart full of fruits and vegetables |                 50.00%  |                     0.00% |
| A high quality photo of a classic silver muscle car                |                 83.33%  |                    83.33% |
| A DSLR photo of a yellow duck                                      |                 33.33%  |                     0.00% |
| A photo of a horse walking                                         |                 33.33%  |                     0.00% |
| A chimpanzee dressed as a football player                          |                 33.33%  |                    16.67% |