Paper: Negative-prompt Inversion: Fast Image Inversion for Editing with Text-guided Diffusion Models (ICLR-2024)

Reviewer comment (Theory):
I find the core assumptions of the paper difficult to understand and I think more justification/verification is needed (see questions below).

Author response:
## W1, Q1
We have revised the justification regarding the assumption in Appendix A.1. In the revised version, we have argued after Proposition 2, that the “velocity field”, which the model is supposed to learn, is continuous in $z$ and $t>0$. It implies that, even though the equality $ \epsilon_{\theta}(z_t,t,C)=\epsilon_{\theta}(z_{t-1},t-1,C) $ does not hold exactly, one can expect that it holds approximately once one assume $ |\alpha_t - \alpha_{t-1}| $ is small.

The reviewer’s concern that the accuracy of the approximation will worsen as the sampling step width increases is reflected in the experimental results in figure 4 left, which show that taking a smaller sampling step width improves the quality of reconstruction.

Reviewer comment (Experiments):
The experiments could be more thorough, especially on editing. How does the performance of DDIM inversion scale with $w$? If I understand correctly it is only shown for $w=1$. Moreover, editing performance should be evaluated on the same benchmark as Mokady et. al (2023) (Table 2. in Mokady et. al).

Author response:
## W2
We appreciate the reviewer for the thoughtful proposal. As the reviewer assumed, we have shown the results of DDIM inversion ($w=1$) followed by DDIM sampling ($w=7.5$) in the manuscript. One can also consider a method of DDIM inversion ($w=7.5$) followed by DDIM sampling ($w=7.5$), but this led to very bad results.

Regarding the proposal to evaluate on the same benchmark as Mokady et al. (2023), we did not conduct that comparison since we could not identify the setting of editing and could not replicate the results of Table 2 in Mokady et al. (2023).

Reviewer comment (Theory):
Why would the predicted noise in adjacent steps equal (Eq. 6)? This would completely undermine the idea of iterative noising/denoising and would be very inaccurate for fairly large steps (such as N=50). How is the approximation impacted by the number of steps?

Author response:
## W1, Q1
We have revised the justification regarding the assumption in Appendix A.1. In the revised version, we have argued after Proposition 2, that the “velocity field”, which the model is supposed to learn, is continuous in $z$ and $t>0$. It implies that, even though the equality $ \epsilon_{\theta}(z_t,t,C)=\epsilon_{\theta}(z_{t-1},t-1,C) $ does not hold exactly, one can expect that it holds approximately once one assume $ |\alpha_t - \alpha_{t-1}| $ is small.

The reviewer’s concern that the accuracy of the approximation will worsen as the sampling step width increases is reflected in the experimental results in figure 4 left, which show that taking a smaller sampling step width improves the quality of reconstruction.

Reviewer comment (Evaluation):
The experiments would be better presented in a way that compares inversion/editing performance *given a fixed time budget* in order to highlight the efficiency of the algorithm.

Author response:
## Q3
We appreciate the reviewer for this advice. We have added comparison of LPIPS when the computation time is limited to less than 30 seconds in Appendix C.1, figure 7. Considering realistic waiting time, our method outperformed null-text inversion in the reconstruction quality.

Reviewer comment (Writing):
I don't quite understand why the method is called negative-prompt inversion. Negative prompting commonly refers to a description of features in an image we do not want to generate in the context of text-conditioned diffusion models.

Author response:
## Q4
In text-guided image generation, if a negative-prompt is given, it is input to the model instead of the null-text in CFG calculation. As a result, the generated image diverges from the negative-prompt. Since our inversion takes a similar process in which uses a text prompt instead of the null-text, we named it “negative-prompt inversion”.

Reviewer comment (Evaluation):
More detailed discussions and analyses on the computational cost and memory usage should be provided since the authors claim them as one of the main contributions.

Author response:
We thank the reviewer for this comment. The term of computational cost represents the computational time, whose results have been shown in figure 4. In memory usage, our method uses half the memory of null-text inversion. We have added this description at the end of Section 4.2.

Reviewer comment (Presentation):
It is advisable to avoid too many detailed discussions on the relevant studies in the Introduction section, which can be moved to the Related Work section. Also, the general idea of the proposed method should be briefly discussed in the Introduction. The previous version is a bit vague.

Author response:
We appropriate the reviewer for this comment. We have removed expressions that were considered redundant and clarified the general idea of our method in the third paragraph of the Introduction section.

Reviewer comment (Reproducibility):
Will the authors release all the code, models, and data to verify the soundness and ensure the reproducibility of this work?

Author response:
We have included our code in supplementary materials for the evaluation purpose by the reviewers, as described in the reproducibility statement of the manuscript. Additionally, we will publicly release our code if our paper is accepted.

Reviewer comment (Writing):
Line 2 of Abstract: Image editing not only changes the style of the image. Sometimes it involves certain semantics or geometry changes.

Author response:
We thank the reviewer for pointing this out. According to the author guide, we cannot change the abstract except in the camera-ready version, so we are planning to change the corresponding part, “changing the style” to “changing the subject and the style”, if our paper is accepted.

Reviewer comment (Reproducibility):
The authors mentioned that by parallelizing and optimizing the program, there is potential to further accelerate their, where even real-time processing would be possible. The reviewer is interested in how to parallelize the program since reverse diffusion is an iterative process.

Author response:
We thank the reviewer for this question. We have revised  the description regarding parallelization in the conclusion section. We assumed parallelization to mean processing several frames simultaneously using multiple GPUs, which improves not the latency but the throughput, not parallelizing the reverse diffusion process which is iterative as the reviewer assumed.

Reviewer comment (Experiments):
**Lack of comparison with existing baselines** The paper could benefit from a more comprehensive comparison with existing baseline methods. Specifically, it does not compare the proposed Negative Prompt Inversion with recent image editing methods based on prompt interpolation, such as Imagic or UniTune. This comparison would help assess the relative strengths and weaknesses of the proposed method in the context of image editing. Moreover, the proposed work as well as the Null-text inversion are also very close to the prompt tuning inversion paper and could also be compared to it.

Author response:
We thank the reviewer for this comment. As the reviewer suggested, we have added results of Imagic for comparison, which shows that our method also outperformed Imagic. We also tried UniTune and prompt tuning inversion, but we failed to replicate the respective papers’ results since the official codes are not released.

Reviewer comment (Evaluation):
**Mixed performances** The paper mentions that the image editing performances of Negative Prompt Inversion are below DDIM with CFG. This observation raises questions about the practical utility of the proposed method.

Author response:
CLIP score calculates the similarity between an image and its prompt, and it does not regard how an edited image is similar to the original one. By summarizing the evaluation both via LPIPS and CLIP score, we believe that the editing performance of our method is not below DDIM with CFG.

Reviewer comment (Experiments):
Are the edited images provided in the supplementary Fig. 8, second column, obtained with DDIM + CFG or just DDIM? In the former case, the results seem in disagreement with the CLIP score shown in Table 1. In the latter case, editing results with DDIM + CFG could be provided here.

Author response:
As the reviewer pointed out, it was difficult to tell what the description of “DDIM Inv” represents. The description “DDIM Inv” represents DDIM inversion followed by DDIM + CFG. We have aligned the description in the manuscript. Regarding the disagreement with the CLIP score, we think it as follows: It is true that DDIM inversion fails to generate snow in the fourth row of figure 9, but the other descriptions for a Doberman are correct, which could be leading not to low the CLIP score. In fact, the CLIP score of the image is $27.25$, which is higher than the mean score.

Reviewer comment (Evaluation):
The evaluation is done with only 100 COCO images, which is quite small. Moreover, only objective metrics are provided. Human subjective evaluation should also be provided since it is more reliable to judge the quality, which is quite easy to do considering the data set is small.

Author response:
We appreciate the reviewer for this comment. We evaluated our method by using 100 images in the same manner as in the original paper of Null-text Inversion (Mokady et al. CVPR2023).
Regarding human subjective evaluation, we did not conduct it since it was difficult for us to prepare it and to collect dozens of subjects during the short discussion period. We think that several editing images in the manuscript including those in appendices should be helpful for judging our editing quality.

Reviewer comment (Evaluation):
The authors claimed some limitations about can not reconstruct faces well. It will be helpful to provide some failure face cases (and failure cases beyond faces if there are).

Author response:
We thank the reviewer for this comment. We have provided some failure cases, including an instance of face reconstruction, in Appendix C.4, figure 12. Please refer to this section.