Paper: Guiding Instruction-based Image Editing via Multimodal Large Language Models (ICLR-2024)

Reviewer comment (Novelty):
While certain aspects of the work might appear less novel, its practical effectiveness compensates for this.

Author response:
We thank reviewer TXUn for acknowledging our practical effectiveness.

Reviewer comment (Presentation):
There are typographical errors. Specifically, "Methods" in Tables 1 and 2 should be placed at the top.

Author response:
Thanks for the debugging! We have updated all Tables in our revision.

Reviewer comment (Experiments):
The reviewer is curious about the method's potential in handling tasks like modifying an object's color, as in "L-CAD: Language-based Colorization with Any-level Descriptions using Diffusion Priors, NeurIPS 2023."

Author response:
We attempt to perform such transfers in Sec. A (Transferring Image Texture/Color/Emotion). For texture, we follow CLVA [1] and adopt the style prompt "*make the whole image as texture [ins]*". InsPix2Pix can only do limited transfer, but MGIE shows clear visual attributes (e.g., "*orange*" or "*pinkish*") as well as the complex "*colorful circular round*". We perform fine-grained color manipulation, including "*glasses frame*" or "*hair*". However, the baseline even alters the whole color. For global colorization [2], both InsPix2Pix and our MGIE cannot present appealing results, which indicates the need for fine-tuning. Transferring the emotion is more challenging as the model has to perceive the latent semantics. We are able to illustrate the visual concept of "*bright day*" or "*chaotic and confused*" as the beach in the early morning or the gloomy street at night. MGIE can also transform from the cozy snowy day into suspenseful and thrilling through "*nightmare and scared*". Although exhibiting promising potential, it still requests more profound texture/emotion perception for each specific goal. We leave them as future research for creative visual editing [3].

+ [1] (ECCV'22) Language-Driven Artistic Style Transfer
+ [2] (NeurIPS'23) L-CAD: Language-based Colorization with Any-level Descriptions using Diffusion Priors
+ [3] (ICCV'23) Affective Image Filter: Reflecting Emotions from Text to Images

Reviewer comment (Experiments):
The reviewer is curious about the method's potential in handling tasks like interpreting user-provided emotions, as in "Affective Image Filter: Reflecting Emotions from Text to Images, ICCV 2023."

Author response:
We attempt to perform such transfers in Sec. A (Transferring Image Texture/Color/Emotion). For texture, we follow CLVA [1] and adopt the style prompt "*make the whole image as texture [ins]*". InsPix2Pix can only do limited transfer, but MGIE shows clear visual attributes (e.g., "*orange*" or "*pinkish*") as well as the complex "*colorful circular round*". We perform fine-grained color manipulation, including "*glasses frame*" or "*hair*". However, the baseline even alters the whole color. For global colorization [2], both InsPix2Pix and our MGIE cannot present appealing results, which indicates the need for fine-tuning. Transferring the emotion is more challenging as the model has to perceive the latent semantics. We are able to illustrate the visual concept of "*bright day*" or "*chaotic and confused*" as the beach in the early morning or the gloomy street at night. MGIE can also transform from the cozy snowy day into suspenseful and thrilling through "*nightmare and scared*". Although exhibiting promising potential, it still requests more profound texture/emotion perception for each specific goal. We leave them as future research for creative visual editing [3].

+ [1] (ECCV'22) Language-Driven Artistic Style Transfer
+ [2] (NeurIPS'23) L-CAD: Language-based Colorization with Any-level Descriptions using Diffusion Priors
+ [3] (ICCV'23) Affective Image Filter: Reflecting Emotions from Text to Images

Reviewer comment (Experiments):
Including comparisons or referencing the mentioned studies could further enrich the paper's depth and context.

Author response:
We attempt to perform such transfers in Sec. A (Transferring Image Texture/Color/Emotion). For texture, we follow CLVA [1] and adopt the style prompt "*make the whole image as texture [ins]*". InsPix2Pix can only do limited transfer, but MGIE shows clear visual attributes (e.g., "*orange*" or "*pinkish*") as well as the complex "*colorful circular round*". We perform fine-grained color manipulation, including "*glasses frame*" or "*hair*". However, the baseline even alters the whole color. For global colorization [2], both InsPix2Pix and our MGIE cannot present appealing results, which indicates the need for fine-tuning. Transferring the emotion is more challenging as the model has to perceive the latent semantics. We are able to illustrate the visual concept of "*bright day*" or "*chaotic and confused*" as the beach in the early morning or the gloomy street at night. MGIE can also transform from the cozy snowy day into suspenseful and thrilling through "*nightmare and scared*". Although exhibiting promising potential, it still requests more profound texture/emotion perception for each specific goal. We leave them as future research for creative visual editing [3].

+ [1] (ECCV'22) Language-Driven Artistic Style Transfer
+ [2] (NeurIPS'23) L-CAD: Language-based Colorization with Any-level Descriptions using Diffusion Priors
+ [3] (ICCV'23) Affective Image Filter: Reflecting Emotions from Text to Images

Reviewer comment (Experiments):
The performance is limited by the dataset utilized for training, especially for learning effective instruction and editing performance. Will this approach complete the editing operation which is not appeared in the training data?

Author response:
We first want to clarify that MGIE is not limited by the training dataset. In IPr2Pr, there is no removal or photo optimization. We have appended such cases in Sec. A (Unseen Editing Operation). InsPix2Pix has failed due to the shortage of training examples. In contrast, our MGIE is able to handle such editing via the visual-aware derivation of MLLM. We can accurately remove "*the boy in the red shirt*" or "*lighten out the yellow tone*", which demonstrates better generalizability for unseen operations. More qualitative comparisons can be found on our project website (https://mllm-ie.github.io).

Reviewer comment (Evaluation):
The editing effects are not ideal, for example, the woman in the background of Fig. 1 is removed, while there are still some residuary artifacts in the background.

Author response:
We also observe these artifacts. Though the editing results are not perfect, MGIE can effectively remove the target, compared to the failed InsPix2Pix. Perhaps a step-by-step pipeline (e.g., removal followed by inpainting) could deal with such issues.

Reviewer comment (Experiments):
Can this approach add new objects into the image?

Author response:
We have appended such examples in Sec. A (Adding New Object). MGIE also supports adding new objects that are not present in the input and placing them in reasonable positions. For example, the "*hat*" is put on the girl's head, and the "*river*" is added along with the grass. More surprisingly, the appended "*fireworks*" further makes the beach colorful, which drives the night scene coherent and visually appealing.

Reviewer comment (Experiments):
What are the details of the user study’s participants? Like their ages, educations, and the time for taking part in the user study.

Author response:
We have updated the details in Sec. B (Human Evaluation). We sample 100 examples (25 for each dataset) to conduct our human evaluation. Each task is assigned 3 annotators, who rank across baselines and our MGIE, to avoid potential bias. We require workers to have a 97% approval rate and over 500 approved tasks to ensure quality. The worker is awarded $5 for each task (5 examples) and takes 21 minutes on average to complete.

Reviewer comment (Novelty):
Leveraging multi-modal language-models provides an intuitively promising avenue when striving to follow image-manipulation intentions defined by only terse human text instruction. Good evidence is provided that the presented strategy goes some way to tackling the observed shortcomings; however, the technical contributions (size, sufficiency) can be regarded as moderate. The overarching system makes use of an array of pre-existing components; however, the particular implementation that facilitates component concatenations can be regarded as somewhat novel.

Author response:
We first want to highlight that our deriving human instructions into informative descriptions is a brand-new attempt, which yields surprising enhancements. Though our MGIE is a joint framework with the MLLM and the diffusion model, we want to emphasize that our contribution includes:
+ an end-to-end pipeline to learn/incorporate explicit yet concise guidance;
+ an investigation on how to utilize expressive instructions (Table 3);
+ a comprehensive automatic/human study covering diverse editing aspects.

All of these are valuable contributions to controllable image editing research.

Reviewer comment (Presentation):
Authors may wish to update to an ICLR24 template (c.f. ICLR23).

Author response:
Thanks for the notification! We have updated the ICLR 2024 template as well as "Instruction Prompt-to-Prompt" for IPr2Pr in our revision. We also add the caption (overview architectures of baselines) to Figure 4.

Reviewer comment (Writing):
On pp.2: suggest explicitly expanding the acronym 'IPr2Pr' on first use (presumably Instruction Prompt-2-Prompt?).

Author response:
Thanks for the notification! We have updated the ICLR 2024 template as well as "Instruction Prompt-to-Prompt" for IPr2Pr in our revision. We also add the caption (overview architectures of baselines) to Figure 4.

Reviewer comment (Presentation):
The model schematic image found between Figure 3 and Figure 4 is unnumbered. It is unclear if this is by design. It is suggested that the 'Baselines' paragraph might serve as suitable caption content.

Author response:
Thanks for the notification! We have updated the ICLR 2024 template as well as "Instruction Prompt-to-Prompt" for IPr2Pr in our revision. We also add the caption (overview architectures of baselines) to Figure 4.

Reviewer comment (Reproducibility):
Can the authors commit to a full source code release? The submission opted to stay silent on this issue. Code release would be of clear benefit to the community, aiding reproducibility, public probing of method robustness, consistency, and range of model abilities. This will undoubtedly increase the value of the contributions. If impossible, the web page might be extended to allow public inference-time testing.

Author response:
Yes. We will release the code/checkpoints after the review process. We also attach a copy of our codebase in the appendix.

Reviewer comment (Writing):
The property of 'elaborate descriptions' is posed as something one wishes to evade in natural commands, and yet 'explicit yet detailed guidance' is conversely noted as a sought-after notion. Do the authors suggest that the former pertains largely to image space, in a similar fashion to e.g. regional masks? Concrete examples of the considered 'elaborate descriptions' may help to clarify this point.

Author response:
Sorry for the misunderstanding. Those previous methods (e.g., Text2Live [1] and Null-Inv [2]) require human-provided descriptions, which are labor-intensive and not intuitive, compared to edit instructions. For example, to manipulate an image of "*a girl is walking at the beach*". The user needs to specify "*a girl with a hat is walking at the beach*", instead of the more straightforward "*give her a hat*". In contrast, our MGIE can automatically derive the concrete guidance "*the girl will wear a hat and walk at the beach*", where elaborate descriptions are no longer necessary in this scenario. This can significantly enhance the accessibility of the edit instruction.
+ [1] (ECCV'22)Text2LIVE: Text-Driven Layered Image and Video Editing
+ [2] (CVPR'23) Null-text Inversion for Editing Real Images using Guided Diffusion Models

Reviewer comment (Writing):
How is the phrase 'reasonable image editing' to be defined?

Author response:
We call it "*reasonable*" since our model can follow the reasoning from the MLLM. MGIE leverages the derived expressive instruction to perform related image manipulation, where the guidance from the MLLM can offer a reasonable editing goal through powerful language modeling. For example, "*make it more healthy*" for a pizza can be perceived as "*need more vegetables*"; "*add fireworks*" will alter not only the sky but also the reflection on the ground. We calculate the CLIP-Score between edited results and their input/expressive instructions. Our MGIE achieves better alignment than the baseline and shows an even higher correlation with expressive instructions. This indicates that we indeed follow the MLLM and benefit from its reasoning capability. Looking ahead, one potential direction is to adopt the recent GPT-4V [1] to evaluate the actual reasonableness.

| Method | CLIP-S | EVR | GIER | MA5k | MagicBrush |
| :-- | :-- | :-- | :-- | :-- | :-- |
| InsPix2Pix | input instruction | 21.19 | 20.36 | 20.31 | 22.58 |
| MGIE | input instruction | 23.03 | 21.74 | 21.89 | 24.09 |
| | expressive instruction | 28.33 | 28.25 | 27.69 | 28.37 |

+ [1] (arXiv'23) The Dawn of LMMs: Preliminary Explorations with GPT-4V(vision)

Reviewer comment (Presentation):
Minor: Small icons (flame, blue cube) tagged to components in the system schematic (Fig.2) presumably represent learnable and frozen model components, respectively. It is suggested to make this key explicit.

Author response:
Thanks for the nice suggestion! We have added it to the caption of Figure 2.

Reviewer comment (Novelty):
My main concern with this work is the limited novelty. The crux of the innovation is the use of a multimodal language model instead of a CLIP model in the instructpix2pix setting. The second main innovation is the introduction of [IMG] tokens, which are processed by a transformer head to generate conditioning embedding for a LDM model. This approach is very strongly inspired from Koh et al. 2023, where they train a language model to generate image tokens, which are then transformed via a transformer architecture, and used as conditioning for stable diffusion based generation. The source of inspiration should be credited more clearly (the main reference in the method section simply mentions using a similar feature extractor architecture).

Author response:
Thank you for pointing this out! Our MGIE is motivated by GILL [1], and we have made the citation clear in our revision. Despite adopting a similar framework, we are the first to edit an existing image via the MLLM+Diffusion pipeline, rather than creating a new but distinct one. We also want to highlight our deriving human instructions into informative descriptions is a novel attempt at controllable image editing. For the experiments, we investigate the best way to utilize our derived expressive guidance and conduct a comprehensive study of diverse editing aspects, all of which are valuable contributions to this research field.

+ [1] (NeurIPS'23) Generating Images with Multimodal Language Models

Reviewer comment (Presentation):
Figure 2 is mentioned at the beginning of chapter 3.2 and shows a MLLM* model without explanation, while this model is only introduced (in a footnote) in a later paragraph.

Author response:
Thanks for the constructive feedback! We have appended the detailed experimental setup in Sec. B (Edit Head / Editing Loss). For the MLLM, we first want to clarify that our framework is model-agnostic and compatible with various architectures (as long as they can take visual inputs and provide responses through language modeling). As we are using LLaVA [1] in our implementation, we have made it clearer in Eq. 1. We also added the explanatory footnote of MLLM* to the same page as Figure 2. 
+ [1] (NeurIPS'23) Visual Instruction Tuning

Reviewer comment (Experiments):
While the evaluation is detailed with experiments carried out on many datasets, state of the art references are limited. The only pre-existing work that authors compare to is instructpix2pix, while the LGIE baseline is an overly poor baseline, which is expected to perform worse than any model with vision-language mappings (e.g., CLIP) in a lot of settings. It does not make a lot of sense to ask a pure language model to hallucinate detailed descriptions of an unseen image.

Author response:
We have involved the results of description-based baselines (TEXT2Live [1] and Null Text Inversion [2]) in Sec. A (Comparison to Description-based Baselines).  We leverage GIT [3] to caption the input image as its input description and ChatGPT to merge the edit instruction as the goal description via the prompt "*Combine two sentences A: [description] and B: [instruction] into a single sentence. The output should be at most similar to sentence A*". For instance, "*a girl is walking at the beach*" and "*give her a hat*" will be merged into "*a girl with a hat is walking at the beach*". For MagicBrush, we directly apply their released descriptions instead. Text2LIVE and Null-Inv only yield feasible results on the traditional L1 distance but are obviously inferior to our MGIE on semantic-level evaluations (e.g., lower CVS), which supports that they cannot present concrete editing results and carry out goal descriptions well. On the other hand, both count on inference optimization (CLIP alignment and DDIM inversion), which takes more than 200 seconds (vs. ours 9.2 seconds) for each editing task.

| Method | EVR | | | GIER | | | MA5k | | | MagicBrush | | | |
| :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- |
| | **L1↓** | **DINO↑** | **CVS↑** | **L1↓** | **SSIM↑** | **CVS↑** | **L1↓** | **SSIM↑** | **LPIPS↓** | **L1↓** | **DINO↑** | **CVS↑** | **CTS↑** |
| Text2LIVE | 0.169 | 66.19 | 78.22 | **0.126** | 58.32 | 79.32 | 0.165 | 57.62 | 0.342 | **0.071** | **83.35** | 89.71 | 23.59 |
| Null-Inv | 0.174 | 69.24 | 78.35 | 0.149 | 58.24 | 82.33 | 0.179 | 61.36 | 0.335 | 0.073 | 81.72 | 87.24 | 27.62 |
| InsPix2Pix | 0.189 | 67.82 | 81.38 | 0.144 | 57.51 | 86.63 | 0.176 | 58.92 | 0.359 | 0.101 | 71.46 | 85.22 | 29.34 |
| MGIE | **0.163** | **71.49** | **81.73** | 0.135 | **59.24** | **88.59** | **0.133** | **66.25** | **0.298** | 0.082 | 82.22 | **91.14** | **30.40** |

+ [1] (ECCV'22) Text2LIVE: Text-Driven Layered Image and Video Editing
+ [2] (CVPR'23) Null-text Inversion for Editing Real Images using Guided Diffusion Models
+ [3] (TMLR'22) GIT: A Generative Image-to-text Transformer for Vision and Language

Reviewer comment (Experiments):
Evaluations on prompt-based editing methods are available on the magic brush dataset, and could provide additional context. For consistency, instructions could easily be converted to a prompt using an LLM.

Author response:
We have involved the results of description-based baselines (TEXT2Live [1] and Null Text Inversion [2]) in Sec. A (Comparison to Description-based Baselines).  We leverage GIT [3] to caption the input image as its input description and ChatGPT to merge the edit instruction as the goal description via the prompt "*Combine two sentences A: [description] and B: [instruction] into a single sentence. The output should be at most similar to sentence A*". For instance, "*a girl is walking at the beach*" and "*give her a hat*" will be merged into "*a girl with a hat is walking at the beach*". For MagicBrush, we directly apply their released descriptions instead. Text2LIVE and Null-Inv only yield feasible results on the traditional L1 distance but are obviously inferior to our MGIE on semantic-level evaluations (e.g., lower CVS), which supports that they cannot present concrete editing results and carry out goal descriptions well. On the other hand, both count on inference optimization (CLIP alignment and DDIM inversion), which takes more than 200 seconds (vs. ours 9.2 seconds) for each editing task.

| Method | EVR | | | GIER | | | MA5k | | | MagicBrush | | | |
| :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- |
| | **L1↓** | **DINO↑** | **CVS↑** | **L1↓** | **SSIM↑** | **CVS↑** | **L1↓** | **SSIM↑** | **LPIPS↓** | **L1↓** | **DINO↑** | **CVS↑** | **CTS↑** |
| Text2LIVE | 0.169 | 66.19 | 78.22 | **0.126** | 58.32 | 79.32 | 0.165 | 57.62 | 0.342 | **0.071** | **83.35** | 89.71 | 23.59 |
| Null-Inv | 0.174 | 69.24 | 78.35 | 0.149 | 58.24 | 82.33 | 0.179 | 61.36 | 0.335 | 0.073 | 81.72 | 87.24 | 27.62 |
| InsPix2Pix | 0.189 | 67.82 | 81.38 | 0.144 | 57.51 | 86.63 | 0.176 | 58.92 | 0.359 | 0.101 | 71.46 | 85.22 | 29.34 |
| MGIE | **0.163** | **71.49** | **81.73** | 0.135 | **59.24** | **88.59** | **0.133** | **66.25** | **0.298** | 0.082 | 82.22 | **91.14** | **30.40** |

+ [1] (ECCV'22) Text2LIVE: Text-Driven Layered Image and Video Editing
+ [2] (CVPR'23) Null-text Inversion for Editing Real Images using Guided Diffusion Models
+ [3] (TMLR'22) GIT: A Generative Image-to-text Transformer for Vision and Language

Reviewer comment (Evaluation):
Image editing performance can be influenced by the random seed. Results in tables 1-2 compared to results reported in Zhang et al. (MagicBrush) show that there can be a noticeable difference for some metrics (e.g., 70→74 for DINO score on instructpix2pix). Several of these evaluations show performance scores that are relatively close; have authors investigated how consistent these rankings are across multiple seeds?

Author response:
Thanks for the nice suggestion! We have updated the empirical results in our revision, where all evaluations are averaged from 5 random seeds. The statistics variance are shown below. The rankings and observations are still consistent with our previous conclusions.

| Method | EVR | | | GIER | | | MA5k | | | MagicBrush | | | |
| :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- |
| **Zero-shot** | **L1↓** | **DINO↑** | **CVS↑** | **L1↓** | **SSIM↑** | **CVS↑** | **L1↓** | **SSIM↑** | **LPIPS↓** | **L1↓** | **DINO↑** | **CVS↑** | **CTS↑** |
| InsPix2Pix | 0.189±0.013 | 67.82±3.09 | 81.38±3.58 | 0.144±0.007 | 57.51±1.51 | 86.63±1.44 | 0.176±0.005 | 58.92±1.15 | 0.359±0.071 | 0.101±0.009 | 71.46±3.12 | 85.22±3.59 | 29.34±0.30 |
| LGIE | **0.159**±0.017 | 69.71±2.89 | **82.04**±1.66 | 0.152±0.014 | 56.86±2.37 | 86.99±2.14 | 0.144±0.017 | 64.60±1.88 | 0.327±0.006 | 0.084±0.041 | 80.90±2.07 | 88.87±1.79 | 30.10±0.29 |
| MGIE | 0.163±0.008 | **71.49**±3.83 | 81.73±2.99 | **0.135**±0.011 | **59.24**±0.72 | **88.59**±0.75 | **0.133**±0.018 | **66.25**±2.41 | **0.298**±0.053 | **0.082**±0.016 | **82.22**±2.70 | **91.14**±4.06 | **30.40**±0.31 |

Reviewer comment (Evaluation):
Are ground truth images (post edit) available for all datasets? Why were these specific sets of evaluation metrics chosen? Why not, e.g., measure image quality using FID?

Author response:
Yes. All ground-truth goal images are available in EVR, GIER, MA5k, and MagicBrush. For Photoshop-style modification (EVR and GIER), we adopt semantic-level visual similarity DINO and CVS. Since there are many photo optimization cases in GIER, we also consider SSIM. MA5k aims to adjust the contrast/brightness/saturation, where DINO and CVS cannot clearly tell the difference; Hence we leverage the widely-used style distance LPIPS. As we have the ground-truth description in MagicBrush, we follow the additional CTS to evaluate the alignment between goal captions and edited images.

> **Q) FID score**

We calculate the FID score between ground-truth and edited images in Sec. A (Evaluating Image Editing via FID). However, the differences are all pretty limited. Since most editing results still resemble the original input images, it is difficult for FID to determine their authenticity. These results indicate that FID is insufficient to compare the quality of image editing.

| Method | Zero-shot | | | | Fine-tuned | | | |
| :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- |
| | **EVR** | **GIER** | **MA5k** | **MagicBrush** | **EVR** | **GIER** | **MA5k** | **MagicBrush** |
| InsPix2Pix | **6.19** | **5.61** | 5.91 | 5.69 | **5.31** | **5.31** | **5.30** | 5.64 |
| LGIE | 6.67 | 5.69 | 5.80 | **5.31** | 5.32 | 5.42 | 5.59 | 5.48 |
| MGIE | 6.45 | 5.64 | **5.48** | 5.61 | 5.53 | 5.59 | 5.41 | **5.42** |