Paper: Benchmarking Diffusion Based Text-Guided Image Editing Methods (ICLR-2024)

Reviewer comment (Experiments):
the proposed benchmark only includes real images from MSCOCO, a missing evaluation on common use case is editing on synthetic generated images, which I think should be added to the benchmark

Author response:
We thank the reviewer for these ideas - they are interesting! The scope of our paper was to develop an evaluation pipeline to evaluate edits on real images. Although our pipeline could plausibly be extended to evaluating edits in synthetically generated images, we did not explicitly consider these in our work. We also note that expanding the size of the evaluation dataset can significantly increase the evaluation time, so our choice of dataset size was carefully considered.  For this reason, we consider an extension to synthetically generated images as future work.

Reviewer comment (Evaluation):
the automatic evaluation pipeline cannot capture hallucinations, i.e. Figure 1 object addition, Dreambooth added a plausible wine glass next to the pizza, but the original content does not preserve well, and Pix2Pix is the opposite

Author response:
For the reviewer’s second point on understanding if the original content is preserved or not, we highlight that the automatic evaluation does contain computing the DINO scores in conjunction with the scores from OwL-ViT. These DINO scores are proxies for the third question in human-study which captures if the original image content is preserved well or not. We have updated Sec. (3.1) to better reflect this. Indeed, we find that it displays a strong correlation with human-study as well (see Sec. 4.3 and Sec. (D)). We have made these points more clear in the main paper and provided additional guidelines in the form of a checklist (see updated Sec. (5)) on how our framework can be used to evaluate text-guided editing methods.

Reviewer comment (Evaluation):
what are your thoughts on the practical usage of the benchmark? does achieving a high score translate into a better editing method in practice or does it just mean it is relatively better than other methods?

Author response:
Current text-guided image editing methods predominantly hinge on a singular metric, the CLIP-Score. Additionally, there is no single standardized dataset that covers a diverse array of edit operations (e.g. object-addition or changing the location of an object) which further compounds the evaluation challenge. Existing works often resort to human studies, yet the templates employed vary across papers, making fair comparison difficult.
Our paper addresses these challenges by presenting a comprehensive standardized package, inclusive of a dataset and two evaluation schemes, designed to capture different facets of text-guided image editing methods. Our benchmark's evaluations offer richer insights than CLIP-Scores, probing methods across various dimensions of edits. Moreover, our human-study template can seamlessly adapt to future editing methods without the reliance on edited images from preceding methods.
Our benchmark can be employed as a practical checklist or guide for evaluating text-guided editing methods. We expound on these points in the main paper (refer to Sec. 5) to clarify how EditVal can be practically utilized. In essence, we propose the following use of EditVal as a checklist:
(i) Dataset Utilization: Incorporate the 648 edit-operations from our dataset along with a new image editing method.
(ii) Editing Accuracy Evaluation:
(a) Employ Owl-ViT rules in the automatic evaluation pipeline to obtain scores for the six edit-types.
(b)Use our standardized human-study template to acquire scores for the seven edit-types.
(iii) Context Preservation Assessment: Utilize DINO and FID to gauge the efficacy of the method in preserving image context.
(iv) Compile a comprehensive report integrating information from Steps 1, 2, and 3 to comprehend the effectiveness of the editing method in fine-grained detail.

We contend that relying on a single score is inadequate, as exemplified by Fig. 5, where SDE-Edit outperforms Dreambooth in object-addition but lags behind in object-replacement. A singular score fails to unveil such nuanced details about a method, underscoring the importance of our proposed comprehensive benchmark report for a more thorough understanding of editing method efficacy.
We want to highlight another important point in terms of practical usage : We have open-sourced our code at https://github.com/deep-ml-research/editval_code, will maintain a leaderboard for different edit-types at https://deep-ml-research.github.io/editval/ and our dataset is extremely easy to use with a COCO-dataloader used in conjunction with a json file (as shown in Sec. (B)).

Reviewer comment (Evaluation):
I do have concerns on the practical usefulness of the benchmark, especially the automatic evaluation pipeline, hopefully the authors can address them as mention in the weakness

Author response:
Current text-guided image editing methods predominantly hinge on a singular metric, the CLIP-Score. Additionally, there is no single standardized dataset that covers a diverse array of edit operations (e.g. object-addition or changing the location of an object) which further compounds the evaluation challenge. Existing works often resort to human studies, yet the templates employed vary across papers, making fair comparison difficult.
Our paper addresses these challenges by presenting a comprehensive standardized package, inclusive of a dataset and two evaluation schemes, designed to capture different facets of text-guided image editing methods. Our benchmark's evaluations offer richer insights than CLIP-Scores, probing methods across various dimensions of edits. Moreover, our human-study template can seamlessly adapt to future editing methods without the reliance on edited images from preceding methods.
Our benchmark can be employed as a practical checklist or guide for evaluating text-guided editing methods. We expound on these points in the main paper (refer to Sec. 5) to clarify how EditVal can be practically utilized. In essence, we propose the following use of EditVal as a checklist:
(i) Dataset Utilization: Incorporate the 648 edit-operations from our dataset along with a new image editing method.
(ii) Editing Accuracy Evaluation:
(a) Employ Owl-ViT rules in the automatic evaluation pipeline to obtain scores for the six edit-types.
(b)Use our standardized human-study template to acquire scores for the seven edit-types.
(iii) Context Preservation Assessment: Utilize DINO and FID to gauge the efficacy of the method in preserving image context.
(iv) Compile a comprehensive report integrating information from Steps 1, 2, and 3 to comprehend the effectiveness of the editing method in fine-grained detail.

We contend that relying on a single score is inadequate, as exemplified by Fig. 5, where SDE-Edit outperforms Dreambooth in object-addition but lags behind in object-replacement. A singular score fails to unveil such nuanced details about a method, underscoring the importance of our proposed comprehensive benchmark report for a more thorough understanding of editing method efficacy.
We want to highlight another important point in terms of practical usage : We have open-sourced our code at https://github.com/deep-ml-research/editval_code, will maintain a leaderboard for different edit-types at https://deep-ml-research.github.io/editval/ and our dataset is extremely easy to use with a COCO-dataloader used in conjunction with a json file (as shown in Sec. (B)).

Reviewer comment (Evaluation):
Only 6 out of 13 edits are supported by the automated evaluation pipeline.

Author response:
**Only 6 out of 13 edits are supported.** : The reviewer rightly points out that our OWL-ViT pipeline is used to evaluate only 6 out of the 13 edit types. This is because these are the edit types for which the pipeline produces scores that are strongly correlated with human scores. For the remaining 7 edit types, the OWL-ViT pipeline, similar to CLIP produces scores that are only weakly correlated with human scores (see updated Sec. (K)). We did not want to throw the baby out with the bathwater, so we proposed the OWL-based pipeline as a proxy for a human study for the first 6 edit types. We recommend our standardized human template to evaluate the remaining 7 edit types which does not require using edited images from earlier methods to evaluate a given editing method. We note that this standardized template is a contribution in and of itself, and can be used by future editing methods to compare to earlier methods.
While this does not present a ‘unified’ evaluation pipeline, we note that the OWL-based pipeline offers a more fine-grained metric compared to CLIP Scores (see Sec. (J)). .

While we would have liked to also contribute an automated way of evaluating the remaining 7 edits types, we have left this for future work - perhaps a future extension of EditVal!

We have also updated Sec. (5) in the paper to provide guidelines/a checklist for how to evaluate an image editing method on EditVal and how to interpret the results.

Reviewer comment (Evaluation):
The pipeline can only check the object presence, location and size. For example, it cannot recognize if a cup stands naturally on the bench for “add a cup on the bench”. These aspects make this evaluator quite vulnerable.

Author response:
**The pipeline can only check the object presence, location and size. For example, it cannot recognize if a cup stands naturally on the bench for “add a cup on the bench”. These aspects make this evaluator quite vulnerable.** : We recognize that our current pipeline has a limitation in determining whether a cup is naturally placed on a bench. We consider this issue as part of the larger challenge of assessing text-to-image generation methods across all conceivable scenarios. Our work represents a step forward in evaluating these methods within a specific subset of scenarios -  For e.g.,(i)  if a new object is added (object-addition), then our pipeline checks if both the old and the new object is present; (ii) If an object is altered (alter-parts), then the alteration is within that given object, leveraging predicted bounding boxes.  

We consider our automated evaluation pipeline to be a significant step from CLIP Scores, which are the current standard for evaluating text-to-image models and provide little fine-grained understanding of where models are failing and succeeding. As the evaluation of generative models continues to evolve, we anticipate that criteria addressing issues such as the natural placement of objects can be smoothly incorporated into future versions of our framework.

Reviewer comment (Evaluation):
The pipeline does not evaluate image fidelity and image-context preservation. One still needs FID and DINO/CLIP scores which are not specifically designed for editing evaluation.

Author response:
**It does not evaluate image fidelity and image-context preservation. One still needs FID and DINO/CLIP scores which are not specifically designed for editing evaluation.** : The reviewer correctly notes that our pipeline uses FID and DINO scores to evaluate image fidelity and image-context preservation (which correspond to the second and third question in our human study). Even though these metrics are not specifically designed for editing evaluation, we find that they are strongly correlated with our human study scores (see Sec. (4.3) and Sec. (D) in the appendix). We have updated the text in Sec. (3) to emphasize that our automated pipeline uses FID/DINO metrics to check image fidelity and context preservation, so that there is no confusion.

Reviewer comment (Experiments):
The reviewer suggests that it might be reasonable to finetune some visual language model on the collected human scores to obtain something similar to ImageReward but for the editing tasks, and consider combining it with the proposed detector-based algorithms.

Author response:
**Probably, it would be reasonable to finetune some visual language model on the collected human scores and obtain something similar to ImageReward[1] but for the editing tasks. In addition, one can consider combining it with the proposed detector-based algorithms** : ImageReward [1] method focuses on collecting 9k samples annotated on image-quality as well as overall text-alignment, and utilizes this data for better finetuning of T2I diffusion models. However, as explored in the limitations of ImageReward [1], even these many samples may not be enough for learning an accurate reward model. To follow the same trajectory as ImageReward, we need extensive ground truth annotations on a much larger scale for each editing type and question type across all the given methods for finetuning a vision-language-based reward model. This necessitates us to have extensive human annotations with minimal annotation noise; further in the case of a novel editing type, we need to annotate and retrain the reward model again for this unseen editing type as well. Therefore, we defer this extension to future works.
The primary contribution of EditVal lies in proposing a new evaluation framework which integrates the evaluations of editing methods on multiple editing types of varying complexity. Any future editing method can easily be benchmarked on these editing types, and new editing types can be coded as directed in EditVal's open-source code.

Reviewer comment (Presentation):
One big limitation of the paper is its poor presentation. It is very crowded, with a lot of vspace adjustments, making the reading experience uncomfortable. Several facts are repeated numerous times, notably editval’s description and main contributions, while key elements are left for the reader to find in the appendix (e.g. the choice of the COCO dataset as source of images).

Author response:
**One big limitation of the paper is its poor presentation. It is very crowded, with a lot of vspace adjustments, making the reading experience uncomfortable. Several facts are repeated numerous times, notably editval’s description and main contributions, while key elements are left for the reader to find in the appendix (e.g. the choice of the COCO dataset as source of images)** : 
We apologize that the paper’s presentation felt crowded - we had many results to share! To address this, we have:
(i) Updated Sec. (3) (especially 3.1) to be more readable with additional context to remove ambiguity. 
(ii) Removed the \vspace adjustments from all the sections to make the paper more presentable. 
(iii) Removed irrelevant information from 3.1 to the Appendix, to only highlight the main parts of the data-collection and annotation process.   (iv) We have also clarified the definition of EditVal in the introduction and other parts of the paper (Sec. 3 in particular). Specifically, we consider EditVal as a benchmark for evaluating text-guided editing methods. EditVal, therefore, includes all the pieces needed to evaluate a method - the data and its annotations, the automated evaluation pipeline (for 6 edit-types) and the templates for the standardized human-study (for 7 edit-types). (v) In addition, we have added a new section (see Sec. (5)) with guidelines on how to use it, to remove any confusion. 

We hope these changes collectively make the paper more readable and presentable!

Reviewer comment (Writing):
The definition of editVal is not clear. In certain parts of the paper it is described as the data, edits and automated eval, and in others (e.g. the introduction) it includes the human study as well.

Author response:
(iv) We have also clarified the definition of EditVal in the introduction and other parts of the paper (Sec. 3 in particular). Specifically, we consider EditVal as a benchmark for evaluating text-guided editing methods. EditVal, therefore, includes all the pieces needed to evaluate a method - the data and its annotations, the automated evaluation pipeline (for 6 edit-types) and the templates for the standardized human-study (for 7 edit-types).

Reviewer comment (Evaluation):
The paper is presenting many contributions — building a benchmark, analyzing pre-existing methods through a human study, developing an automated evaluation metric, and comparing this metric’s performance to user preferences. It is impossible to address all thoroughly and clearly, leading to a lack of in-depth discussion and exploration.

Author response:
**The paper is presenting many contributions : building a benchmark, analyzing pre-existing methods through a human study, developing an automated evaluation metric and comparing this metric’s performance to user preferences. It is impossible to address all thoroughly and clearly, leading to a lack of in depth discussion and exploration. For example, the user study can provide ....** : We thank the reviewer for raising this point. We did consider splitting our contributions into multiple papers, but felt that we could not present a new benchmark without thoroughly validating it (through the automated and human-based assessment of pre-existing methods). We therefore relied on the appendix, with deeper explorations of the models and findings included there. To remedy this, we have moved some of the important points of our work to the main paper such as : (i) Better describing the dataset and its filtering process in updated Sec. (3.1); (ii) Reorganizing the presentation of results in Sec. (4.2). (iii) Provide guidelines on how to use EditVal in a new Sec. 5.

Reviewer comment (Experiments):
The user study could provide a lot of insights over the different editing methods’ behaviours, but only the top 4 methods (selected according to an unknown criterion) are briefly discussed in section 4.2.

Author response:
**How did author select the top 4 performing methods to show in Figure 4? Was a global score computed, or a ranking across editing types and question type?** : We select the top 4 performing methods by computing the median score for the first question type related to "editing quality" for each editing method and aggregate it across all the 13 editing types. Based on this score, we choose the top 4 editing methods and normalize the annotation score of each question ({0,1,2,3}) to a scale of 0-1 and represent our results in Figure 4. We are happy to answer, if the reviewer has any more questions on this!

Reviewer comment (Presentation):
The work might be more impactful if split into multiple papers or submitted to a venue where more space is available.

Author response:
**The paper is presenting many contributions : building a benchmark, analyzing pre-existing methods through a human study, developing an automated evaluation metric and comparing this metric’s performance to user preferences. It is impossible to address all thoroughly and clearly, leading to a lack of in depth discussion and exploration. For example, the user study can provide ....** : We thank the reviewer for raising this point. We did consider splitting our contributions into multiple papers, but felt that we could not present a new benchmark without thoroughly validating it (through the automated and human-based assessment of pre-existing methods). We therefore relied on the appendix, with deeper explorations of the models and findings included there. To remedy this, we have moved some of the important points of our work to the main paper such as : (i) Better describing the dataset and its filtering process in updated Sec. (3.1); (ii) Reorganizing the presentation of results in Sec. (4.2). (iii) Provide guidelines on how to use EditVal in a new Sec. 5.

Reviewer comment (Evaluation):
Authors overclaim their contributions by presenting their automated pipeline as an alternative to the human study. However, their detection-based strategy can only evaluate a subset of editing tasks, and only provides a binary success/fail output.

Author response:
In response to ‘overclaiming contribution that automated pipeline is an alternative to a human study’ -  In almost all cases, an automated pipeline (e.g. using OWL-ViT) cannot be used as a full replacement for a human study. As the reviewer notes, these pipelines often only give simplified outputs (e.g. binary success/fail) and cannot be used to evaluate more complex edit types (e.g. image quality). These simplified outputs however, can still provide useful signals for understanding model performance. So rather than throwing the baby out with the bathwater, EditVal leverages these for 6 of the edit types, and complements them with other metrics (e.g. FID, DINO) and a human study to evaluate the remaining 7 edit types.

Reviewer comment (Evaluation):
Additional editing criteria such as content preservation and image quality cannot be measured with the proposed detection-based strategy, leading the authors to revert to standard metrics (FID, DINO scores) to complete their evaluation process.

Author response:
In response to ‘overclaiming contribution that automated pipeline is an alternative to a human study’ -  In almost all cases, an automated pipeline (e.g. using OWL-ViT) cannot be used as a full replacement for a human study. As the reviewer notes, these pipelines often only give simplified outputs (e.g. binary success/fail) and cannot be used to evaluate more complex edit types (e.g. image quality). These simplified outputs however, can still provide useful signals for understanding model performance. So rather than throwing the baby out with the bathwater, EditVal leverages these for 6 of the edit types, and complements them with other metrics (e.g. FID, DINO) and a human study to evaluate the remaining 7 edit types.

Reviewer comment (Evaluation):
It is not clear whether these two additional metrics (FID, DINO) are part of the automated evaluation pipeline.

Author response:
We consider these other metrics as part of the full automated pipeline, however, note that we have slightly conflated some terminology. We have updated the manuscript in Sec. (3.2.2)  to refer to the OWL-ViT based pipeline as the ‘detector-based automated pipeline’ and the DINO/FID metrics as the context-preservation pipeline. These two parts in conjunction make the automated pipeline in our paper.

Reviewer comment (Novelty):
The related work section is too limited. An important topic of the paper is the lack of reliable evaluation metrics for editing tasks, yet pre-existing works are not reviewed (besides CLIP).

Author response:
**The related work section is too limited. An important topic of the paper is the lack of reliable evaluation metrics for editing tasks, yet pre-existing works are not reviewed (besides CLIP). For examples, authors do not discuss recent techniques such as ImageReward (Xu et al, Neurips 2022...**: We thank the reviewer for bringing these works to our attention. During the project, we only consider published works and to our knowledge, ImageReward was still under review at NeurIPS ‘23 just before ICLR submissions. However, we have added and discussed these papers to our related works, and hope that future works will benchmark them against other methods on the EditVal benchmark. 

We note that ImageReward can evaluate the image editing quality of a method via  assessing the overall image quality and text alignment. However, our evaluation is multi-dimensional and granular in nature as we generate scores based on the  edit type for each edited image. Therefore, there is no straightforward way of comparing ImageReward and our automated detection pipeline scores; further to rank/compare the editing methods, a simple score averaging across edit-types would be a misleading comparison.

In Sec. (J) - Appendix, we compare the detection-based metric to other metrics (e.g., CLIP-Score which is still widely used to evaluate text-guided image editing methods). We have added new results here, showing that the CLIP-Score metric has low correlation with the human-study annotations for spatial edits, therefore highlighting that OwL-ViT is indeed a better choice than CLIP-Score.

Reviewer comment (Novelty):
The paper does not discuss recent techniques such as ImageReward (Xu et al., NeurIPS 2023) or Pickscore (Kirstain et al., 2023).

Author response:
**The related work section is too limited. An important topic of the paper is the lack of reliable evaluation metrics for editing tasks, yet pre-existing works are not reviewed (besides CLIP). For examples, authors do not discuss recent techniques such as ImageReward (Xu et al, Neurips 2022...**: We thank the reviewer for bringing these works to our attention. During the project, we only consider published works and to our knowledge, ImageReward was still under review at NeurIPS ‘23 just before ICLR submissions. However, we have added and discussed these papers to our related works, and hope that future works will benchmark them against other methods on the EditVal benchmark. 

We note that ImageReward can evaluate the image editing quality of a method via  assessing the overall image quality and text alignment. However, our evaluation is multi-dimensional and granular in nature as we generate scores based on the  edit type for each edited image. Therefore, there is no straightforward way of comparing ImageReward and our automated detection pipeline scores; further to rank/compare the editing methods, a simple score averaging across edit-types would be a misleading comparison.

In Sec. (J) - Appendix, we compare the detection-based metric to other metrics (e.g., CLIP-Score which is still widely used to evaluate text-guided image editing methods). We have added new results here, showing that the CLIP-Score metric has low correlation with the human-study annotations for spatial edits, therefore highlighting that OwL-ViT is indeed a better choice than CLIP-Score.

Reviewer comment (Evaluation):
Although the authors claim a strong correlation between their human study and their proposed metric, the DINO scores show a much stronger correlation (Figure 10) than the owl-vit based metric (Figure 6).

Author response:
**It should also be noted that authors claim a strong correlation between their human study and their proposed metric, yet DINO scores show a much stronger correlation (fig 10), than the owl-vit based metric (fig. 6). This further highlights the importance of comparing the proposed technique to pre-existing works..**: The reviewer raises a relevant point. We note that Fig (6) compares OwL-ViT’s scores with human scores from only the first question in the human-study template. This question specifically asks if the edit is successful or not based on the text prompt. On the other hand, the correlations with DINO scores in Fig (10) should be compared with the third question in the human-study template which asks if the image context is preserved or not. 

Given that both these metrics are comparable to different parts of the human study, they are not directly comparable against each other. We have emphasized these points more in the updated version of our paper in Sec. (3.2.2).

Reviewer comment (Experiments):
The stronger correlation of DINO compared to the proposed owl-vit based metric further highlights the importance of comparing the proposed technique to pre-existing works.

Author response:
**It should also be noted that authors claim a strong correlation between their human study and their proposed metric, yet DINO scores show a much stronger correlation (fig 10), than the owl-vit based metric (fig. 6). This further highlights the importance of comparing the proposed technique to pre-existing works..**: The reviewer raises a relevant point. We note that Fig (6) compares OwL-ViT’s scores with human scores from only the first question in the human-study template. This question specifically asks if the edit is successful or not based on the text prompt. On the other hand, the correlations with DINO scores in Fig (10) should be compared with the third question in the human-study template which asks if the image context is preserved or not. 

Given that both these metrics are comparable to different parts of the human study, they are not directly comparable against each other. We have emphasized these points more in the updated version of our paper in Sec. (3.2.2).

Reviewer comment (Reproducibility):
There is a need for more detailed information regarding the implementation of the image editing methods. For example, methods such as Null-Text and SINE are not based on instructions. It is confusing that instruction-based editing evaluations are applied to them.

Author response:
**There is a need for more detailed information regarding the implementation of the image editing methods. For example, methods such as Null-Text and SINE, are not based on instructions. It is confusing that instruction-based editing evaluations are applied to them** : We emphasize that, when utilizing Null-Text and SINE, we abstain from employing explicit instructions to generate edited images. We have provided details of our instruction-free prompts in Section I and Table 3-Appendix, and will aim to make these more prominent in the final manuscript. Specifically, the prompts are constructed by focusing on the object in the image (e.g., a bench) and its corresponding editing operation (e.g., object-addition). We ensure that the prompts share a similar nature across models, comprising the object and descriptive text indicating the editing operation (e.g., "A brown bench" when altering the bench color to 'brown'). However, there are instances where the prompt may be adjusted based on the specific editing method; for example, for Dreambooth, which necessitates a [V]* token, the curated prompt might be 'A brown [V]*bench'.
In the case of instruction-based approaches like Instruct-Pix2Pix, we adhere to a generic instruction template. For instance, if the task involves adding a new object alongside an existing one, the prompt would be 'Add a <new-object> to the <object>'. These details are also updated in Sec. (I).