Paper: Noise Map Guidance: Inversion with Spatial Context for Real Image Editing (ICLR-2024)

Reviewer comment (Presentation):
Figure 2 seems confusing to me. Since this paper repeatedly mentioned that NMG address the challenges in Null-text Inversion, I think it would be nice to compare NMG to Null-text Inversion in this Figure, and demonstrate how NMG outperforms Null-text Inversion. In addition, it would be nice to add the corresponding text descriptions in the Figure. *E.g.*, it's unclear whether the caption indicate to change the blue fire hydrant to a red one or it's just some deviations during the reconstruction.

Author response:
Thank you for insightful feedback on Figure 2. We acknowledge the necessity of clearly demonstrating how our NMG approach effectively addresses the challenges associated with NTI. In response to suggestions, we have revised Figure 2 for enhanced clarity. In the updated Figure 2(a), we illustrate the common divergence issues encountered in naive reconstruction with an unaligned reverse process. This part of the figure highlights how such reconstructions often fail due to a deviation from the original inversion path, necessitating an alignment to achieve reliable reconstruction, which motivates our work. Figure 2(b) then delves into the alignment process via NTI, which optimizes the null-text embedding to reduce the disparity between the inversion and reconstruction paths. However, NTI's reliance on a single vector of the null-text embedding for alignment lead to challenges in preserving the spatial context of the input image due to having no spatial dimension of the null-text embedding. In contrast, as shown in Figure 2(c), our NMG offers a novel approach to alignment. Our method corrects the reconstruction path by conditioning it on the discrepancies between the inversion and reconstruction paths. NMG utilizes a noise map for this process, which inherently contains a richer spatial context than that of the null text embedding. This direct use of the noise map in NMG allows for more effective preservation of the spatial context during the editing process.

Reviewer comment (Experiments):
Limitations and future work should be discussed. For example, whether NMG can address the relationship change task like SGC-Net [1]? Whether NMG can achieve various non-rigid image editing tasks like Imagic [2]? It seems that NMG can achieve some non-rigid editing tasks such as viewpoint alternation or face expression modification. However, spatial information between the output and input seems consistent in the majority parts, from my view. Thus, it would be great to see experiments exploring whether NMG can perform other operations (with more obvious spatial information change) such as "from a tiger" to "a jumping/running tiger".

Author response:
Thank you for the question regarding the capabilities of NMG. We acknowledge the importance of discussing our work's limitations and potential future directions.

**[W2-1]**
We have appended the limitation section in Section B, Appendix of the revised paper. As detailed in Section B, our current methodology is primarily designed to enhance inversion-based editing methods, focusing on improving spatial context during the editing process. However, NMG faces challenges integrating with methods like SGC-Net, which operates on a text-guided diffusion model for relationship change tasks; thus, not compatible with our current method. This is due to SGC-Net's deviation from the inversion-based editing paradigm, and it is non-trivial to design the integration. As an alternative, we attempted relationship change tasks using MasaCtrl in Figure 9(a) in the  Appendix of the revised paper. However, Figure 9(a) depicts the ineffectiveness of this approach, as MasaCtrl is not inherently designed for relationship change tasks. Consequently, NMG fails to conduct relationship change tasks.

**[W2-2]**
We have conducted additional edited results of the non-rigid image editing task in Figure 8(b) in the revised paper. Figure 8(b) shows that NMG effectively conducts edits with pronounced spatial information changes, like standing to sitting and running to standing.

Reviewer comment (Experiments):
The visualization results depict many recurring stylization outcomes, such as the "oil painting style" and "Van Gogh style". It would be beneficial for the paper to exhibit a broader variety of more challenging editing instances.

Author response:
We have newly appended additional stylization results in Figure 8(a) and a description to Section A.4 in the Appendix of the revised paper. In this figure, we present additional results that showcase NMG's capabilities in handling a wide range of artistic styles, extending well beyond the initially noted painting stylizations. Specifically, the figure includes examples of editing outcomes in styles like anime and mosaic. This diversity in style adaptation illustrates the flexibility and robustness of NMG in accommodating various artistic expressions and editing challenges.

Reviewer comment (Experiments):
The user study could benefit from providing more accurate directives in its questions; if it pertains to local editing, for instance, the question should contemplate including "evaluate original preservation in unedited areas”.

Author response:
We agree that a precise evaluation framework is crucial for assessing the performance of our method. To this end, we have conducted an additional user study and have detailed the methodology and results in Section A.4 and Table 3 in the Appendix of the revised paper. This study is designed with a particular focus on evaluating the preservation of integrity in unedited areas during local editing tasks and the retention of overall structure in global editing scenarios. As evidenced in Table 3, the outcomes affirm that NMG successfully maintains unedited regions during local editing tasks and preserves the overarching structure in global edits, underlining its effectiveness in diverse editing contexts.

Reviewer comment (Experiments):
How about the editing performance of adding or deleting elements in the images?

Author response:
We have added additional results in Figure 9(c) in the revised paper. NMG can eliminate objects from an image, such as removing a person. However, it is essential to note that while NMG can successfully execute such deletions, it is currently limited in its specificity when selecting individual targets for removal. Similarly, NMG faces challenges in adhering to detailed locational instructions when adding new objects to an image. Although NMG provides reliable editing for general tasks, its reliance on text for image editing introduces difficulties in executing detailed spatial changes. We have added this discussion in Section B of the revised paper as a limitation discussion.

Reviewer comment (Evaluation):
While both global and local editing in the ProxNPI paper seem promising, the editing capability in this paper doesn't appear as effective. For instance, in Figure 3, the second row shows an edited result with a clear boundary between two types of backgrounds. In the third row, under "Van Gogh," the overall style seems to have undergone minimal change. Can you provide an explanation for these observations?

Author response:
We have conducted an additional comparison with ProxNPI, illustrated in Figure 8(d) and thoroughly discussed in Section A.4 in the revised paper. This comparison reveals that NMG achieves results on par with those showcased in the ProxNPI study. However, it is essential to note that ProxNPI does not extensively explore various editing scenarios, such as contextual alterations or stylization. In contrast, NMG has demonstrated its proficiency across a wide range of editing scenarios. 

Additionally, as elaborated in Section 4.1, ProxNPI faces limitations due to its dependence on inversion guidance. NMG, free from such constraints, exhibits greater versatility and effectiveness in various image editing tasks, underscoring its broader applicability and utility in the field of image editing.

Note that ProxNPI was an unpublished arxiv work at the time of our submission, and our work was independently developed in parallel(i.e., concurrent work). While ProxNPI may be a noteworthy project, we may believe it'd be unfair to us to be discounted by ProxNPI with a single perspective. Our research offers unique insights and contributions that stand on their own merit and we respectfully ask an unbiased evaluation by the reviewer, considering this matter.