Paper: PnP Inversion: Boosting Diffusion-based Editing with 3 Lines of Code (ICLR-2024)

Reviewer comment (Evaluation):
In the column PnP of Fig.1, PnP doesn't do a good job in preserving content, sometimes texture and shape is hallucinated. Using direct inversion can correct them. But why not background color in the third row?

Author response:
Since Plug-and-Play (PnP) does not have a background protection mechanism but performs editing globally, it often fails to preserve the background effectively. Although our method can improve essential content preservation, it is highly related to the editing model (The $DDIM$ _ $Forward_{Editing}$ _ $_{Model}$) base. Our proposed method aims to maximize the efficacy of existing editing rather than instituting fundamental changes. As evidenced in Figure 1, our approach successfully amends specific background areas, marked in red, in line with our objectives. The further advancement of the performance of these editing methods is designated as an area for future research, as detailed in Section G of the main paper.

Reviewer comment (Evaluation):
Following Fig.2, "...This results in a learned latent with a discernible gap between $z_0^{''}$ and the original $z_0$...". So how does the optimized $z_0^{''}$ deviates from original distribution? It's not quite clear how deviation happens, what does it look like, and why it negatively affects performances. Could authors provide concrete examples?

Author response:
The deviation of pure DDIM Inversion and DDIM Forward is shown in Figure 4 (b) and Figure 8 (b). The deviation would completely distort the original image content, making it hard to perform essential content preservation.

Reviewer comment (Writing):
The writing in the method part is hard to follow. I would suggest authors use meaningful subscripts to denote source latent, source prompt forward latent, and other patents instead of $z_0, z_t', z_{t}''$.

Author response:
We sincerely thank the reviewer for pointing out a better writing approach for articulating and explaining our methods. We have revised the denotations as follows: 

- We use $z_{0}^{src}$ to denote the source latent.
- We use $z_{t}^{I}$ to denote DDIM **i**nversion latent. Specifically, we use $z_{t}^{I}$ to denote the nonexistent ideal DDIM inversion latent, and use $z_{t}^{I_{p}}$ to denote the perturbed DDIM inversion latent.
- We use $z_{t}^{F}$ to denote DDIM **f**orward latent. Specifically, we use $z_{t}^{F_s}$ to denote the DDIM forward latent with **s**ource prompt without guidance scale, and use $z_{t}^{F_t}$ to denote the DDIM forward latent with **t**arget prompt without guidance scale. To further distinguish the DDIM forward process with guidance scale, we use $z_{t}^{F_{sg}}$ to denote the DDIM forward latent with **s**ource prompt with guidance scale, and use $z_{t}^{F_{tg}}$ to denote the DDIM forward latent with **t**arget prompt with guidance scale.

We also revised some other denotations:

- We use $C^{src}$ to denote source prompt embedding (or embedding of null for some editing methods), and use $C^{tgt}$ to denote target prompt embedding.
- We use $d_{t}^{rec}$ to denote the difference between the perturbed DDIM inversion latent $z_{t}^{I_{p}}$ and the DDIM forward latent of the source prompt $z_{t}^{F_{sg}}$.

Reviewer comment (Presentation):
What is the meaning of step 1, 2 in Figure 2?

Author response:
The step denotes the order of steps in the editing process.

Specifically, for techniques such as Null-Text Inversion, StyleDiffusion, and Negative-Prompt Inversion, the process involves initially inverting $z_{0}^{src}$ to $z_{1}^{I_p}$ (step 1) followed by inverting it to $z_{2}^{I_p}$ (step 2). Subsequently, editing on $z_{2}^{I_p}$ is executed using a two-branch diffusion model to compute the denoised latent $z_{1}^{F_{sg}}$ and $z_{1}^{F_{tg}}$(step 3), and eventually $z_{0}^{F_{sg}}$ and $z_{0}^{F_{tg}}$(step 4).

For our method, steps 1 and 2 mirror those in Null-Text Inversion, StyleDiffusion, and Negative-Prompt Inversion. Post these steps, editing on $z_{2}^{I_p}$ is conducted using the two-branch diffusion model to determine the denoised latent $z_{1}^{F_{sg}}$ and $z_{1}^{F_{tg}}$(step 3), followed by step 4, which involves adding differences to rectify deviations. Steps 5 and 6 are repetitions of steps 3 and 4. Notably, steps 4 and 6, introduced by our method, are denoted by a solid circle. A comprehensive explanation of these steps is provided in the main paper.

Reviewer comment (Writing):
In the algorithm1, what is the meaning of argument $[C^{src}, C^{tgt}]$ in the DDIM_Forward function call? Is it suggesting that the forward function is called twice, with one of the calls on $C^{src}$ and the other on $C^{tgt}$? If so, why line 9 function call has only one output?

Author response:
We apologize for the lack of clarity and unclear denotations of our proposed algorithm in the historical version. We have revised the algorithm and polished section 4.2 according to your constructive advice:

- **The bracket notations:** The bracket notations denote the batch concatenation for the inputs and outputs involved in DDIM Inversion and DDIM Forward processes instead of calling a function twice. Since existing editing methods simultaneously process the source prompt $C^{src}$ and target prompt $C^{tgt}$, with a batch size of 2, to ensure essential content preservation and editing by integrating information between the source and target batches. In adherence to these established methods, we write the algorithm with an input concatenated in batch dimension. We apologize for not specifying it clearly in the previous version of our manuscript and have added the corresponding clarification in the revised version.
- **The misalignment of batch size in input and output:** We thank the reviewer for pointing out the misalignment. We have revised our manuscript and ensured the batch size is maintained consistently in input and output.

Additionally, we have updated the algorithm in response to constructive feedback from other reviewers. For detailed information, please refer to the revised version of the paper.

Reviewer comment (Reproducibility):
I am confused about how the source branch interacts with the target branch. It seems that the $z_{t}^{tgt}$ is only updated using $z_{t+1}^{tgt}, C^{tgt}$ without any source branch information. Could authors clarify line 9 in the algorithm? Is it related to the implementation of $DDIMForward_{Editing_Model}$?

Author response:
The interaction is related to the implementation of $DDIM\underline{}Forward_{Editing\underline{}Model}$. We have provided a detailed review of how previous editing methods perform the source and target branches interaction in the related work. For more information, please refer to Section A in the revised version.

To illustrate this interaction, we utilize the Prompt-to-Prompt method as an example. Prompt-to-Prompt uses the cross-attention map in the source branch to enhance or replace the ones in the target branch. The selection of the cross-attention map is based on the correspondence between the cross-attention map and the text prompt. For instance, in transforming an image of a cat to a dog, the method involves using the source prompt 'a cat' and the target prompt 'a dog'. Here, the cross-attention map associated with 'dog' is exchanged with that of 'cat' in the source branch, thereby infusing the edit with relevant information. Concurrently, other attention maps in the source branch are maintained as is, ensuring the preservation of essential content. 

This also explains why an accurate source branch latent would be important for essential content preservation during the editing. This is because the feature (such as attention map) in the source branch will be further used in target branch editing.

Reviewer comment (Experiments):
What's the number of images for each editing type in the PIE-Bench creation? As far as I understand, most of the editing types are local region editing, e.g., change object, add object, and large region editing is limited to style change only. I wonder if the dataset is mostly local editing images.

Author response:
We have clarified the number of images in each editing type in the revised Section B, listed as follows:

0. random editing written by volunteers (140 images)

1. change object (80 images)

2. add object (80 images)

3. delete object (80 images)

4. change object content (40 images)

5. change object pose (40 images)

6. change object color (40 images)

7. change object material (40 images)

8. change background (80 images)

9. change image style (80 images)

We distribute the editing into several classes: random (type 0, 140 images), object-related (type 1-3, 240 images), object attribute change (type 4-7, 160 images), and large region edit (type 8-9, 160 images). Thus, PIE-Bench covers a wider range of editing categories instead of only local editing.

Reviewer comment (Evaluation):
What is the input text for CLIP similarity evaluation in **Whole** and **Edit**? As far as I understand, the target prompt in the PIE-Bench is a full description of the target image instead of the specification of the local region.

Author response:
- **The input text of CLIP Similarity evaluation:** We directly use the target prompt to calculate both CLIP Similarity in Whole and Edit. It is possible that the masked image is not completely aligned with the target prompt for Edit Region CLIP Similarity. However, since most mask regions in PIE-Bench take up around half of the image size, our visualization results show alignment between the Edit Region CLIP Similarity and masked image. Moreover, the same variation trend of Edit Region CLIP Similarity, Whole Image CLIP Similarity, and Human Evaluation Metrics may also demonstrate the reasonableness of using this metric.
While acknowledging the logical basis of this metric, we concur with the identified potential risks associated with its use. In response, we have decided to switch it to human evaluation metrics in the final version of our work.

Reviewer comment (Evaluation):
The CLIP Similarity for different methods seems very close in Table 1, 4, and 8. Is it possible the CLIP similarity cannot distinguish the editing quality? Could authors include BLIP similarity or human evaluation to make evaluation more comprehensive?

Author response:
- **Include more metrics to make the evaluation more comprehensive:** Since the editing is expected to only perform in a part of the image and not make huge changes in the image structure. It is true the CLIP score varies modestly among different editing methods. However, we hold the view that CLIP score is still possible to distinguish the editing quality since it is aligned with visualization results, shown in the visualization of the main paper.
To make the evaluation more comprehensive, we added two human evaluation metrics to rate the editing quality. Due to the limited time, we randomly selected 50 images and let two volunteers rate 1-10 source-editing image pairs on two dimensions: editing image quality and alignment with the target prompt. Both two volunteers rated all 50 images, and we averaged their scores to formulate the table below. The Table shows the same compared methods as in Table 1 of the main paper. For the editing method Prompt-to-Prompt (P2P), we compare four different inversion methods: DDIM Inversion (DDIM), Null-Text Inversion (NT), Negative-Prompt Inversion (NP), and StyleDiffusion. For editing methods MasaCtrl, Pix2Pix-Zero (P2P-Zero), and Plug-and-Play (PnP), we compare with DDIM Inversion (DDIM). Results show that the human evaluation metrics show the same tendency with CLIP Similarity scores.
    
    
    | Method | Whole Image CLIP Similarity  | Editing Image Quality | Alignment with Target Prompt | Average of Human Evaluation Metrics |
    | --- | --- | --- | --- | --- |
    | DDIM+P2P | 25.01 | 1.74 | 5.68 | 3.71 |
    | NT+P2P | 24.75 | 6.32 | 5.19 | 5.76 |
    | NP+P2P | 24.61 | 5.19 | 4.65 | 4.92 |
    | StyleD+P2P | 24.78 | 6.21 | 5.15 | 5.68 |
    | Ours+P2P | 25.02 | 6.50 | 5.52 | 6.01 |
    | DDIM+MasaCtrl | 23.96 | 4.54 | 3.88 | 4.21 |
    | Ours+MasaCtrl | 24.38 | 4.61 | 4.01 | 4.31 |
    | DDIM+P2P-Zero | 22.80 | 2.86 | 3.01 | 2.94 |
    | Ours+P2P-Zero | 23.31 | 4.33 | 4.20 | 4.27 |
    | DDIM+Plug-and-Play | 25.41 | 6.39 | 5.60 | 6.00 |
    | Ours+Plug-and-Play | 25.41 | 6.56 | 5.61 | 6.09 |
    
    Table 2. Comparing our methods with other inversion techniques across various editing methods on Whole Image CLIP Similarity and Human Evaluation Metrics.
    
    We will further evaluate the human evaluation metrics on the whole benchmark with 700 images and report the value in the final version.

Reviewer comment (Presentation):
The content of Figure 3 seems to be largely irrelevant to the content of the paper. While the detailed explanation and organization of previous works are appreciated, Figure 3 does not play a significant role in aiding understanding. It is suggested that Figure 5 from the supplementary material be included in the main text instead.

Author response:
We would like to express our sincere thanks to the reviewer for the constructive feedback on the structure of our paper. We fully agree that Figure 5 is more relevant to the content of the paper in comparison with Figure 3 and have accordingly updated the revised manuscript to reflect this change. But we still want to explain why we put substantial effort into introducing previous works. This is because our key insight is the disentangling of two branches, which is ignored by previous works. Based on this insight, we review previous research and dig into the approaches they employed to preserve essential content and facilitate editing. Then, we analyze how to disentangle the two branches in previous works to combine them with our proposed inversion method better. In the revised version, we have put the main part of the related work and Figure 3 into the supplementary file. Moreover, we added Figure 5 to the main text with more explanation.

However, we apologize that due to the page limit, it is hard for us to put a detailed explanation of evaluation metrics into the main text. In this paper, we do not introduce new evaluation metrics but explore existing metrics suitable for the task and incorporate them into our benchmark. We posit that a comprehensive explanation of these metrics in the supplementary files suffices and should not detract significantly from the reader's comprehension.

Reviewer comment (Evaluation):
A more detailed explanation of the benchmarks and evaluation metrics in the main body of the text would be beneficial. This information is considered one of the major contributions of the paper, yet it is not present in the main text.

Author response:
We would like to express our sincere thanks to the reviewer for the constructive feedback on the structure of our paper. We fully agree that Figure 5 is more relevant to the content of the paper in comparison with Figure 3 and have accordingly updated the revised manuscript to reflect this change. But we still want to explain why we put substantial effort into introducing previous works. This is because our key insight is the disentangling of two branches, which is ignored by previous works. Based on this insight, we review previous research and dig into the approaches they employed to preserve essential content and facilitate editing. Then, we analyze how to disentangle the two branches in previous works to combine them with our proposed inversion method better. In the revised version, we have put the main part of the related work and Figure 3 into the supplementary file. Moreover, we added Figure 5 to the main text with more explanation.

However, we apologize that due to the page limit, it is hard for us to put a detailed explanation of evaluation metrics into the main text. In this paper, we do not introduce new evaluation metrics but explore existing metrics suitable for the task and incorporate them into our benchmark. We posit that a comprehensive explanation of these metrics in the supplementary files suffices and should not detract significantly from the reader's comprehension.

Reviewer comment (Writing):
Based on the reviewer’s understanding from the code, the paper stores all the z_t values generated during DDIM inversion, uses them to calculate a small editing direction at each step, and then reflects this in the z^tgt used to generate the actual results. This algorithm feels somewhat similar to the approach used in CycleDiffusion [https://arxiv.org/abs/2210.05559]. A clearer explanation of the algorithm would greatly assist in correcting this understanding.

Author response:
The comparison to CycleDiffusion: CycleDiffusion directly calculates an offset between the latent generated by the source prompt and the target prompt of the current time step t in the DDIM Forward process, then adds the offset to the latent embedding of time step t-1. This is quite similar to the methods used in [3][4][5], which use the difference between the source and target prompts to perform image editing. However, our methods are distinguished from their approaches in several aspects: (1) Our method targets editing methods that involve detailed editing operations such as attention map fusion and latent fusion. These editing methods usually perform more satisfying editing results than [3][4][5] and CycleDiffusion. (2) Our method is a plug-and-play improvement for diffusion inversion targeted for two-branch prompt-based editing methods, instead of an editing method. (3) Our method largely improves the essential content preservation and editability by two-branch disentangling, while [3][4][5] and CycleDiffusion directly use a one-branch approach to perform editing.
Nevertheless, we are thankful to the reviewer for pointing out the missing citation for CycleDiffusion. We add citations for [6][7] to our related work.

[3] General Image-to-Image Translation with One-Shot Image Guidance

[4] SINE: SINgle Image Editing with Text-to-Image Diffusion Models

[5] SEGA: Instructing Diffusion using Semantic Dimensions

[6] Unifying Diffusion Models' Latent Space, with Applications to CycleDiffusion and Guidance

[7] A Latent Space of Stochastic Diffusion Models for Zero-Shot Image Editing and Guidance

Reviewer comment (Evaluation):
The benchmark is divided into 10 categories, but scores for each category are not reported. Scores for the category involving pose changes are of particular interest, as most of the proposed methodologies might struggle with changing poses. A discussion and reporting of scores on this matter would be appreciated, at least in the supplementary material.

Author response:
In fact, we have already provided the results and analysis of our method added to Prompt-to-Prompt on different editing categories in supplementary file E.6. We agree that a discussion and reporting of scores on various editing types would be helpful for a better understanding of the paper. Thus, we additionally add each category's result of MasaCtrl in supplementary file E.6. 

Different editing types perform differently in each category. For instance, Prompt-to-Prompt excels in material alterations, resulting in an overall superior performance in category 7. Conversely, MasaCtrl **demonstrates proficiency in modifying object poses** using the mutual self-attention mechanism, leading to relatively better results in the respective category 5. The detailed results and analysis can be found in the supplementary file.

Reviewer comment (Presentation):
Regarding Figure 4, it is disappointing that the only result shown with the proposed method applied is Ours+P2P, although additional results are available in the supplementary material.

Author response:
The target of Figure 4 is to provide a comparison of different inversion methods and our inversion method. For a fair comparison, all inversion methods are combined with Prompt-to-Prompt. Additional visualization results of our method combined with different editing techniques can be found in Figure 1 and supplementary files from Figure 8 to Figure 13. Due to the page limit, most visualization results are in the supplementary files.

Reviewer comment (Writing):
The method section 4.2 is not very clear to me, especially the bracket notations in the algorithm box. It would be helpful to explain lines 3, 7-9 in more detail.

Author response:
**Answer1:** 

We apologize for the lack of clarity in the explanation of our proposed algorithm in the historical versions and polish section 4.2 according to your constructive advice:

- **The bracket notations:** The bracket notations denote the batch concatenation for the inputs and outputs involved in DDIM Inversion and DDIM Forward processes. Since existing editing methods simultaneously process the source prompt $C^{src}$ and target prompt $C^{tgt}$, with a batch size of 2, to ensure essential content preservation and editing by integrating information between the source and target batches. In adherence to these established methods, we write the algorithm with an input concatenated in batch dimension. We apologize for not specifying it clearly in the previous version of our manuscript and have added the corresponding clarification in the revised version.
- **Explain lines 3, 7-9 in more detail:** We have reformulated the algorithm to enhance its readability, adopting a new style of presentation. The revised algorithm and added explanation can be found in the edited rebuttal revision. Specifically, our method achieves editing performance enhancement in 3 steps: (1) we first compute the perturbed latent in DDIM forward; (2) we then store the difference between the original latent in the inversion process and the perturbed latent in the forward process with input condition $C^{src}$; (3) we finally add the difference to the source/reconstruction branch in existing editing model.

We express our gratitude for the reviewer's suggestions, which have significantly improved the quality and readability of the paper.

Reviewer comment (Novelty):
It might be worth adding discussion and comparison of a related but concurrent work [1].

Author response:
**Answer2:**

We regret the initial oversight of not providing discussion and comparison with the AIDI method introduced in the current literature. The method under consideration, known as Accelerated Iterative Diffusion Inversion (AIDI), employs an iterative procedure to find a fixed-point solution for the ideal diffusion latent, as depicted by the dashed circle in Figure 2. 

It is pertinent to note that AIDI is distinct from and complementary to our approach; while AIDI concentrates on refining DDIM Inversion, our methodology is aimed at the DDIM Forward correction. We have conducted additional experiments to both compare and combine our method with AIDI. The results have been incorporated into the supplementary material in Section E.2 since AIDI reports proficiency in reconstruction. Due to the absence of an official AIDI implementation, we adapted the Prompt-to-Prompt (P2P) codebase to include AIDI with the iteration parameters set to 5 and 20, which is denoted as AIDI(5) and AIDI(20).

| Method | Structure Distance ↓ | PSNR ↑ | LPIPS ↓ | MSE ↓ | SSIM ↑ | $\mathrm{CLIPSimilarity}_{\mathrm{Whole}}$ ↑ | $\mathrm{CLIP Similarity}_{\mathrm{Edited}}$ ↑ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| AIDI(5)+P2P | 12.19 | 26.96 | 57.92 | 39.82 | 84.17 | 24.96 | 22.01 |
| AIDI(20)+P2P | 12.16 | 27.01 | 56.39 | 36.90 | 84.27 | 24.92 | 22.02 |
| Ours+P2P | 11.65 | 27.22 | 54.55 | 32.86 | **84.76** | **25.02** | **22.10** |
| Ours+AIDI(5)+P2P | **11.54** | **27.26** | **54.54** | **32.78** | 84.69 | **25.02** | 22.09 |

Table 1. Results of comparing and combining our method with AIDI.

The table above illustrates that our method outperforms AIDI in terms of structural distance, background preservation, and editability. Additionally, the integration of AIDI into our method results in further enhancements to performance. This improvement substantiates the orthogonality and potential synergistic relationship between our approach and AIDI.

Moreover, given that AIDI necessitates an iterative process to locate the fixed-point solution, it is inherently more time-consuming than our proposed method, even when the iteration parameter is limited to 5. Our method achieves an average inversion time of 16.15 seconds, compared to 21.39 seconds for AIDI. These results have been updated in the main body of the paper.

Reviewer comment (Writing):
The name "direct inversion" clashes with another existing work [2], which might cause ambiguous.

Author response:
**Answer3:**

We thank the reviewer for highlighting the potential for ambiguity with the name ‘Direct Inversion’ used in existing work. To avoid confusion and maintain our methodology's distinctiveness, we have renamed our approach to 'PnP Inversion.' We have updated our manuscript accordingly to reflect this change.

Reviewer comment (Writing):
Typo: Algorithm 1, Part I: "Invert" z_0^{src}; sec 4.2, "optimization-based" inversion.

Author response:
**Answer4:** 

We appreciate the reviewer's careful reading of our manuscript and the assistance in identifying typographical errors. We have thoroughly reviewed the document and corrected the typos that were pointed out.