Paper: DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models (ICLR-2024)

Reviewer comment (Writing):
In the Abstract, the sentence "... they often lack the ability to precisely edit the generated or real images." should be toned down to "interactively" as "precisely" might have a broad meaning (e.g., precise in terms of pose, shape, etc.). In a broader meaning of precision, existing works can also achieve "precise" image editing (e.g., ControlNet [1]).

Author response:
# W2. Writing

Thank you for these constructive suggestions. We have reviewed the relevant sections and made the necessary modifications.

Reviewer comment (Writing):
In Section 2.3, the sentence "However, text-guided image editing is coarse." would benefit from an added explanation of why "coarse" is a bad thing.

Author response:
# Q1. Compare with self-guidance

Yes, comparing with self-guidance is necessary. Since self-guidance was not open-sourced before the ICLR submission, our original paper did not include a performance comparison with it. Our main difference lies in modeling editing operations. Self-guidance calculates the similarity between the text (corresponding to the object to be edited) and image features, selects regions with high similarity as editing areas, and then applies editing operations. Therefore, the attention in self-guidance is used to locate editing operations. However, this text-guided image editing is coarse. For example: (1) Text can only locate content at the object level, and more accurate local content cannot be expressed in text, making it impossible to be edited with self-guidance. (2) In multi-object scenes, a single text corresponds to multiple objects, making it impossible to use self-guidance to edit a specific object. (3) Text can not provide effective content consistency, leading to deviations between the editing results and the original image. In contrast, our method models editing operations based on the correspondence of intermediate features in the pre-trained diffusion model, which can achieve pixel-to-pixel correspondence, and the editing area is manually defined. Therefore, our method can achieve fine-grained image editing, such as content dragging, which is not supported by self-guidance. In our method, attention is used to inject content consistency guidance, resulting in better consistency between the edited results and the original image.

In Sec.4.1 of the updated version, we add the comparison between our method against Self-Guidance in object moving and appearance replacing. We add more comparisons in Appendix.8. These results demonstrate that our method has higher editing accuracy and content consistency.

Reviewer comment (Writing):
From Section 2 through Section 3.1, the paper uses $x_{T}$, then suddenly in Section 3.2 it uses $z_{T}$. As far as understood, the authors intend to use Latent Diffusion (Stable Diffusion), which is $z_{T}$. The authors should revise Section 2.1 (and other related parts in Sections 2–3) to ensure that $z_{T}$ is mentioned earlier for coherence.

Author response:
Thank you for your efforts and positive rating. In Sec. 3.2, we update a description that SD is a latent diffusion model (LDM), as well as the relationship between $x_t$ and $z_t$.

Reviewer comment (Novelty):
The problem targeted by this paper is not clear. Therefore, it is unclear why diffusion feature matching, drag-style editing, memory bank, and visual cross-attention strategy were introduced, giving an incremental feel.

Author response:
# W1. The problems targeted in this paper.

- drag-style editing is the target task

In this paper, we want to build a general framework to complete drag-style image editing, e.g., object moving, resizing, pasting, content dragging, and appearance replacing. This has not been achieved before.

- feature matching builds the energy guidance as an editing tool without training

In this paper, we choose energy guidance in diffusion models to achieve image editing. Energy guidance is the second term in Eq.2 in our original paper, which combines the generation target y with the diffusion process as q(y|$x_t$). Therefore, the task becomes designing a suitable energy function [1] to measure the distance between the current state x_t and the editing target y. The first generation control design in energy guidance is to control the category of the generation result. It trains a classifier for $x_t$ to measure the distance between the current state $x_t$ and the target category y. However, this type of method requires additional training, and due to the influence of noise, the classification of $x_t$ is challenging. In this paper, we propose taking advantage of the strong semantics in the intermediate features of pre-trained stable diffusion, which transforms the editing target y into the feature matching between the editing starting point and the target point. Our approach eliminates training cost while achieving good editing results.

[1] Diffusion models beat GANs on image synthesis

- memory bank is a container to store information for visual cross-attention

Although feature matching can achieve editing operations, the final result can not maintain consistency with the original image, such as in unedited areas. To maintain consistency, we use the memory bank to collect intermediate information during the image inversion and inject it into the editing process through visual cross-attention. This allows the editing process to fully capture the original image information. In Fig.9 of the original paper, we demonstrate the important role of visual cross-attention in maintaining content consistency.

Therefore, these components are indispensable for our editing framework.

Reviewer comment (Experiments):
If the paper is focused on the problem of drag-style image manipulation, more experimental results should be presented. For example, it is unclear why the FID score is higher compared to DragGAN. There is no related ablation study for that part.

Author response:
# W2. FID compared with DragGAN

This is determined by the base model. DragGAN is designed based on the pre-trained StyleGAN, while our method is based on the pre-trained Stable Diffusion. The generative capability of stable diffusion is significantly better than that of StyleGAN. The results in Fig.8 in the original paper also show that the generative capability of GAN models is limited. For unaligned images, GAN models can produce severe degradation. Even in aligned images, GANs cannot generate a good background, as shown in DragGAN’s results in our paper. In the new version of the paper, we added more discussion on DragGAN's robustness in Appendix.7.

Reviewer comment (Evaluation):
Despite the introduction of the visual cross-attention strategy, it feels like that the identity or content of the image is not sufficiently preserved.

Author response:
# W3. Identity by visual cross-attention

We need to clarify that our method does not require training. Compared to the latest diffusion-based image editing method, DragDiffusion, our method also has performance advantages without the need for training LORA for each image like DragDiffusion. We will continue to improve content consistency in future work.

Reviewer comment (Experiments):
Just as selecting the feature layer or combining information from the layer is important, I know that at which diffusion time the guidance is given also has a significant impact. Were there any related experiments on this?

Author response:
# W4.&Q1. Ablation study of hyper-parameters and guidance time 

Thank you for this useful suggestion. We add an ablation study on the weights ($w_e$, $w_c$) of the image editing ($E_{edit}$) and content consistency ($E_{content}$) energy functions in Appendix 4. Results show that $E_{edit}$ can complete the editing operation with a small $w_e$, and the influence of $w_c$ on $E_{edit}$ is relatively small. Therefore, in our design, we apply a larger weight to $E_{content}$ to better maintain content consistency while achieving the editing goals.

The ablation study on when to introduce gradient guidance shows that it is important to introduce guidance in the early sampling stages. As the sampling iterations progress, the effect of the guidance information becomes weaker. Moreover, introducing guidance information in more time steps helps improve the texture details. Finally, we introduce guidance information in the first 30 steps to balance performance and complexity.

Reviewer comment (Experiments):
The clarity of how the memory bank is meaningful is not evident in this draft. As the memory bank is proposed as a contribution, the authors should provide a more comprehensive ablation study, including both quantitative and qualitative analysis.

Author response:
We need to clarify that the memory bank is a container we designed to store intermediate information ($z_t$, $K_t$, $V_t$) during the DDIM inversion process. These intermediate outputs will be reused to guide the subsequent editing process. In Fig.9 of the original paper, we demonstrated the importance of $K_t$ and $V_t$ (i.e., visual cross-attention) in the memory bank. We also showed the importance of generating content consistency gradients $E_{content}$ through the stored $z_t$. This intermediate information can also be generated in other ways, such as a separate generation branch. But this would result in additional computation cost. In Appendix.2 of the original paper, we conducted an ablation study on the efficiency of the memory bank. However, the connection between these two ablation parts is not tight enough. We added a brief description of Appendix.2 in the ablation study section to highlight the contribution of the memory bank.

Reviewer comment (Experiments):
How the energy design makes it works is not clear, the authors should provide more details numerical studies.

Author response:
Energy guidance is the second term in Eq.2 in our original paper, which combines the generation target y with the diffusion process as q(y|$x_t$). It has been rigorously proven in [1] for generation control in the diffusion models. To make it work, we need to design an energy function to measure the distance between the current state $x_t$ and the editing target y. The first generation control design in energy guidance is to control the category of the generation result. It trains a classifier for $x_t$ to measure the distance between the current state $x_t$ and the target category y. However, this type of method requires additional training, and due to the influence of noise, the classification on $x_t$ is challenging. In this paper, we propose taking advantage of the strong semantics of the intermediate features in pre-trained stable diffusion to transform the editing target y into the feature similarity between the editing starting point and the target point. Our approach eliminates training cost while achieving good editing results. In Fig. 4, we investigate how to make the energy guidance work better. In Fig. 5, we demonstrate how to combine different energy functions to achieve editing goals. In addition, we update more ablation studies in Appendix.3 and Appendix.4, showing the impact of more components on the working of energy guidance.

To make the gradient guidance process clearer, we visualize the gradients generated by the content consistency ($E_{content}$) and editing ($E_{edit}$) energy functions at different time steps in Appendix.5. It presents a gradually converging process, and the gradient generated by each energy function mainly focuses on their respective target areas.

[1] Diffusion models beat GANs on image synthesis

Reviewer comment (Evaluation):
The inference time is too slow, approximately 15.93 in Table 1, which makes the solution incomparable with dragGAN.

Author response:
The higher inference complexity is determined by the base model. DragGAN is designed based on StyleGAN, while our method is based on Stable Diffusion. GAN models only require one inference to output results, while diffusion requires multiple iterations, resulting in a higher time complexity. However, the generative capability of the diffusion models is significantly better than that of the GANs. For example, in terms of robustness, DragGAN requires a specific trained model when editing different categories of images, and the input image cannot deviate from the distribution of GAN's training set. In Fig.8 of the original paper, we show that DragGAN's editing results have severe degradation without image alignment. In the new version of the paper, we added more discussion on DragGAN's robustness in Appendix 7. As can be seen, the GAN-based editing method fails in complex scenarios. In contrast, our method is based on the diffusion model with high robustness to edit general images. It is worth noting that the complexity of our method in the preparation stage is significantly lower than that of DragGAN (3.62s vs 52.40s in Tab.1).

Most importantly, our DragonDiffusion is a general image editing method that can accomplish a range of image editing tasks, e.g., object moving, resizing, pasting, content dragging, and appearance replacing. In contrast, DragGAN is limited to content dragging.

Reviewer comment (Experiments):
The ablation is a bit incomplete. E.g. it will be nice to see some ablations on the usefulness of S_global.

Author response:
In our method, $S_{global}$ controls the appearance of a certain area to be consistent with a reference area. It is used in the inpainting of the object moving task and the appearance control in the appearance replacing task. Indeed, the previous version lacked an experiment to show the effectiveness of $S_{global}$. We add an ablation study for $S_{global}$ in Appendix.3, demonstrating its appearance control in object moving and appearance replacing.

Reviewer comment (Experiments):
Some flickering still happens in the no-change areas, e.g. clouds in the sun example and background in the apple example. If this is because of the balance between different losses, some ablation could be very helpful.

Author response:
Yes, there is a trade-off between different energy functions. The energy functions in our method mainly include $E_{edit}$ and $E_{content}$, designed for editing and content consistency, respectively. We add an ablation study for the weights ($w_e$, $w_c$) of these two functions in Appendix.4. Results show that $E_{edit}$ can complete the editing operation with a small w_e, and the influence of $w_c$ on $E_{edit}$ is relatively small. Therefore, in our design, we apply a larger weight to $E_{content}$ to better maintain content consistency while achieving the editing goals. In our demo video, we directly connect the results of different editing operations into a video, which may amplify some subtle jitter.

Reviewer comment (Evaluation):
The identity preservation also seems a bit off, e.g. patterns of the apple. However, I think it is a relatively minor issue as other works also cannot completely fix this issue.

Author response:
Yes, some details may have some deviations. Our method does not require training, and maintaining high consistency is indeed a challenge. We will continue to improve in future work.

Reviewer comment (Experiments):
First of all, the paper proposes to use gradient guidance sampling for a bunch of tasks, but the paper writing and the experiments mainly focus on dragging-based editing. This will narrow down the scope of the paper quite a lot. It is suggested to formulate the paper as a general solution and equally treat multiple tasks.

Author response:
Thank you for this suggestion to help us highlight the contributions of this paper. A minor clarification is that Self-Guidance was not open-sourced before the ICLR submission, so the initial version did not include a comparison with it. In our new version, we divide Section 4.1 into two parts: content dragging comparison and comparisons on other tasks. In the updated part, we demonstrate our method against Self-Guidance and Paint-by-example in object moving, appearance replacing, and object pasting tasks. We add more comparisons in Appendix.8. 

Moreover, we further discuss the differences between our method and text-guided editing methods in Appendix.9 by comparing with Null-text inversion, InstructPix2Pix, and Self-Guidance. 

- Compare with Self-Guidance. Self-Guidance is a text-guided editing method. Due to the coarse-grained correspondence between text and image content, it can only achieve object-level editing and is difficult to achieve editing objectives in some complex scenes or multi-object scenarios. In addition, Self-Guidance lacks consistency constraints, which leads to deviations between the editing results and the original image. In comparison, the impact of text on our method is minimal, as shown in Appendix.4. It shows that the editing results produced by different texts (relevant or irrelevant) are close and meet the editing requirements.
- Compare with Paint-by-example. Paint-by-example is a learnable object pasting method. We must admit that such learnable methods can produce more natural pasting effects. However, encoding the object image into multiple tokens can disrupt the object's identity, leading to changes in the identity of the editing results. In contrast, our method adds guidance at each diffusion step, resulting in better identity consistency in the editing results.
- Compare with other text-guided methods (e.g., Null-text inversion, InstructPix2Pix). Although these methods can also achieve object modulation, the editing results have no reference and randomly dependent on text description. In addition, similar to Self-Guidance, text has difficulty achieving a good correspondence with images in complex scenes and multi-object scenarios, leading to editing failures. 

It is worth noting that most of these compared methods can only accomplish one task, while we can complete these tasks with a single framework. Through these comparisons, we want to demonstrate that our method is a general fine-grained image editing framework with promising performance.

Another related change is that we remove the original Application section and present the application visualization as the teaser on the first page to reduce redundant content.

Reviewer comment (Experiments):
Also, the experiment is not thorough enough. It is suggested to conduct comparisons on other tasks besides dragging edit. For example, for object pasting, it is suggested to compare against the work "paint by example". For appearance replacement, it is suggested to compare against "Diffusion Self-Guidance for Controllable Image Generation" and "Null-text Inversion".

Author response:
Thank you for this suggestion to help us highlight the contributions of this paper. A minor clarification is that Self-Guidance was not open-sourced before the ICLR submission, so the initial version did not include a comparison with it. In our new version, we divide Section 4.1 into two parts: content dragging comparison and comparisons on other tasks. In the updated part, we demonstrate our method against Self-Guidance and Paint-by-example in object moving, appearance replacing, and object pasting tasks. We add more comparisons in Appendix.8. 

Moreover, we further discuss the differences between our method and text-guided editing methods in Appendix.9 by comparing with Null-text inversion, InstructPix2Pix, and Self-Guidance. 

- Compare with Self-Guidance. Self-Guidance is a text-guided editing method. Due to the coarse-grained correspondence between text and image content, it can only achieve object-level editing and is difficult to achieve editing objectives in some complex scenes or multi-object scenarios. In addition, Self-Guidance lacks consistency constraints, which leads to deviations between the editing results and the original image. In comparison, the impact of text on our method is minimal, as shown in Appendix.4. It shows that the editing results produced by different texts (relevant or irrelevant) are close and meet the editing requirements.
- Compare with Paint-by-example. Paint-by-example is a learnable object pasting method. We must admit that such learnable methods can produce more natural pasting effects. However, encoding the object image into multiple tokens can disrupt the object's identity, leading to changes in the identity of the editing results. In contrast, our method adds guidance at each diffusion step, resulting in better identity consistency in the editing results.
- Compare with other text-guided methods (e.g., Null-text inversion, InstructPix2Pix). Although these methods can also achieve object modulation, the editing results have no reference and randomly dependent on text description. In addition, similar to Self-Guidance, text has difficulty achieving a good correspondence with images in complex scenes and multi-object scenarios, leading to editing failures. 

It is worth noting that most of these compared methods can only accomplish one task, while we can complete these tasks with a single framework. Through these comparisons, we want to demonstrate that our method is a general fine-grained image editing framework with promising performance.

Another related change is that we remove the original Application section and present the application visualization as the teaser on the first page to reduce redundant content.

Reviewer comment (Writing):
There are some typos in the paper. For example, Equation 2 is not correct.

Author response:
Thanks for the careful review. We have corrected Equation 2.

Reviewer comment (Evaluation):
There is no limitation analysis for the proposed method.

Author response:
Thanks for this suggestion. We add a discussion of limitations in Appendix.12. It mainly includes that there is still room to further improve the inference speed, and there are some hyperparameters in our training-free pipeline that can affect the editing results.