Paper: Object-Aware Inversion and Reassembly for Image Editing (ICLR-2024)

Reviewer comment (Evaluation):
One major drawback of the method is the search speed. Since the inversion steps of each editing part should be comprehensively searched, this may pose a challenge when editing for multiple regions. It would be interesting to conduct a coarse-to-fine search strategy for speedup. Moreover, in the paper, it is suggested to report the image editing speed for different methods since some other baselines, like prompt-to-prompt, can edit images in a feed-forward manner.

Author response:
__Q1: It is suggested to report the image editing speed.__

A1: Our image editing speed table has been incorporated into Major Response Q1.

Reviewer comment (Writing):
Some writing parts can be improved for better clarity. For example, what's the detailed formulation of min-max normalization? It is suggested to better rephrase the term "concept mismatch".

Author response:
__Q2: Some writing parts can be improved for better clarity.__

A2: We comprehensively revise the paper to enhance clarity in expression.

Reviewer comment (Experiments):
Since the re-inversion step is always set to 20% of the total steps, the non-edit regions will inevitably be affected. Such subtle change is particularly apparent for faces and small objects.

Author response:
__Q3: Since the re-inversion step is always set to 20% of the total steps, the non-edit regions will inevitably be affected.__

A4: Thank you for bringing out this issue. Throughout our experiments, we observed that the impact of the number of re-inversion steps is trivial, thus we fix it as 20% of the total steps. However, better results should be achieved with a more careful hyperparameter tuning.

Reviewer comment (Experiments):
The proposed method determines the starting point in the latent space to deviate the semantics of the objects. I think this search strategy is orthogonal to other editing methods that manipulate the latent based on the textural prompt. Hence, I wonder whether the proposed method is compatible with other editing methods, like null inversion. I would like to hear the authors' feedback regarding this.

Author response:
__Q4: Is the proposed method compatible with other editing methods?__

A4: Our search metric can be combined with other inversion-based image editing methods to enhance their performance. Taking Null-text Inversion (NTI) as an example, the search metric can improve NTI in multiple ways. We introduce the compatible method in Appendix A.8 (Figure 13) and supplemented it with experiments. In Appendix A.8, we supplement experiments using the search metric to find the hyperparameters for Null-text Inversion. Additionally, we provide detailed explanations of various compatible methods.

Reviewer comment (Evaluation):
The method introduces significant computational overhead. For each input image, it has to run a search for optimal denoising steps for each edit pair. The search process can be very time consuming, as it requires denoising from each inversion step, plus a metric calculation step. The overall amount of computation for a single image editing task can thus be very large.

Author response:
__Q1: The search process can be very time consuming.__

A1: We supplement the time overhead experiments and summarize our acceleration methods in the Major Response Q1.

Reviewer comment (Evaluation):
It relies on SAM to do fine-grained segmentation of the input image to localize the object, and then do localized editing followed by resembling. However, it has some limitations. For example, the segmentation may not work well for small object. In addition, using SAM constrained the edit to object-level, while the applicability of more global change is questionable (e.g., change the style of an image from spring to winter, change the background to Mars, etc.). It may not be able to change the location of an object.

Author response:
__Q2.1: It relies on SAM to do fine-grained segmentation.__

A2.1: 
- SAM is not mandatory. Our method is complementary to various mask generation schemes, such as attention map-based mask extraction methods that do not require introducing additional models. These methods were employed by notable works such as DiffEdit [a] and MasaCtrl [b].


- In the revised paper, we provide a comparison of different mask generators in Appendix A.7 (Figure 12). From the figure, it can be observed that our method is highly robust to different mask generators.

[a] MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing, ICCV 2023.

[b] Diffedit: Diffusion-based semantic image editing with mask guidance, ICLR 2023.

__Q2.2: The applicability of more global change is questionable (e.g., change the style of an image from spring to winter, change the background to Mars, etc.).__

A2.2: 
- We are applicable to global change and we have extensively demonstrated these results in our initial submission. For example, in the main text, Column h of Figure 4, Figure 6, and Figure 16 with tasks (b, k), (b, m), (c, l), (c, m), (d, m), as well as Figure 17 with tasks (a, h), (c, f), (d, f), (d, h), (e, g), (e, h), (e, i), all involve background editing. In Figure 16, tasks (c, k), (f, j), and in Figure 17, tasks (c, g), (c, i) all include style editing.
- Background can be considered as a generalized object for editing. In tasks that involve changing the style, alterations can be made on different objects separately, and then our reassembly strategy can be utilized to achieve global interaction.

__Q2.3: It may not be able to change the location of an object.__

A2.3: 
- Given an input image $I$ and a target prompt $P$, image editing's goal is to generate a new image $O$ that complies with $P$ and preserves the structure and semantic layout of $I$. The task of changing the location of an object is not under the current setting. More details about the editing setting are thoroughly explained in the first paragraph of Section 3 of our paper and have also been discussed in literature, such as Prompt-to-Prompt [c] and Plug-and-Play [d].

[c] Prompt-to-Prompt Image Editing with Cross Attention Control, ICLR 2023.

[d] Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation, CVPR 2023.

Reviewer comment (Experiments):
Related to the above point, the competitive method, such as null-text inversion, does not involve segmentation and localization, therefore it is a bit unfair to compare with them, as in principle they can only benefit from the segmentation and localization. This is independent from the main innovation of this paper, which is searching for optimal inversion step.

Author response:
__Q3: Related to the above point, the competitive method, such as null-text inversion, does not involve segmentation and localization, therefore it is a bit unfair to compare with them, as in principle, they can only benefit from segmentation and localization. This is independent from the main innovation of this paper, which is searching for optimal inversion step.__

A3: 
- Null-text Inversion (NTI) is a training-based approach that introduces learnable null-text embeddings to address DDIM Inversion's inability to accurately reconstruct the original image. It needs to be combined with other editing methods to achieve image editing, so whether NTI needs a mask depends on the editing method with which it is combined.
- Our method is tightly coupled with mask generation and is completely training-free. We employ an additional mask generator to disentangle objects, enabling object-aware editing by separating individual entities. 
- We combine NTI with Prompt-to-Prompt (P2P) as a comparative method by using Grounded-SAM as the mask generator in P2P, which is the same as our method. The experimental results presented in Appendix A.10 (Figure 15) indicate that our training-free method outperforms the NTI+P2P counterpart which requires training. Moreover, we have also compared our method with other mask-based editing methods, and stable diffusion inpainting, in Section 4.1 of the initial submission. 

[e] Null-text Inversion for Editing Real Images using Guided Diffusion Models, CVPR 2023.

Reviewer comment (Experiments):
In [a], it is mentioned that the inversion step can be considered as a fixed hyperparameter. However, the author only presents the results of the optimal inversion steps for example images, without demonstrating the overall distribution trend of the optimal inversion step across the entire dataset. Moreover, what are the distinct characteristics of editing pairs with larger and smaller optimal inversion steps respectively?

Author response:
__Q1.1: What is the overall distribution trend of the optimal inversion step across the entire dataset?__

A1.1: We collect 200 editing pairs from our multi-object dataset and employ the search metric to identify the optimal inversion step for each editing pair. The distribution of the optimal inversion steps is presented in Appendix A.9 (Figure 14). According to the figure, the majority of the optimal inversion steps occur within the range of 25 and 45.

__Q1.2: What are the distinct characteristics of editing pairs with larger and smaller optimal inversion steps respectively?__

A1.2: We notice that when modifying backgrounds or objects with substantial changes in shape, such as the sky or the ground, larger optimal inversion steps are required. On the other hand, situations involving objects and targets with similar shapes generally require smaller inversion steps. Appendix A.9 (Figure 14) displays several cases within these scenarios.

Reviewer comment (Experiments):
In the reassembly strategy of OIR discussed in this paper, both reassembly step and re-inversion are mentioned. The paper indicates that both methods can smooth the edges of images and enable global information interaction. Re-inversion first inverts the spliced latent and then denoises it, essentially increasing the denoise step. Is it possible to replace the operation of re-inversion by increasing the reassembly step?

Author response:
__Q2: Is it possible to replace the operation of re-inversion by increasing the reassembly step?__

A2:
The re-inversion can not be replaced by increasing reassembly steps for the following reasons. 
- The reassembly steps cannot be increased arbitrarily. As detailed in Section 3, it must be smaller than the smallest optimal inversion step in all editing pairs, to preserve the original features of non-editing regions. This limits the ability to improve editing results by changing the reassembly step. By contrast, the number of re-inversion steps is free from such limitations.
- Re-inversion improves interactions between various regions. Because re-inversion can extend the denoise steps, it enhances the global information exchange between various regions, leading to more realistic and natural outcomes.

Reviewer comment (Experiments):
To determine the optimal inversion steps for image editing, we need to inverse the image to all steps and then edit them accordingly. It is time-consuming and not automatic.

Author response:
__Q1: The method is time-consuming and not automatic.__

A1: We supplement the time overhead experiments and summarize our acceleration methods in the Major Response Q1.

Reviewer comment (Novelty):
The two contributions are more like the engineering stuff. However, admittedly, they do bring a lot of performance gain.

Author response:
__Q2: The two contributions are more like engineering stuff.__

A2: We identify a significant challenge in multi-object image editing and highlight that different editing pairs require distinct optimal inversion steps. Our two contributions, including a new search metric and an object-aware inversion and reassembly process, are specifically designed to enable object-level fine-grained control, remarkably outperforming previous SOTA methods. More details can be found in Major Response Q2.