Paper: ImagenHub: Standardizing the evaluation of conditional image generation models (ICLR-2024)

Reviewer comment (Evaluation):
I think the paper would have been stronger if the authors would have directly compared the human evaluations with existing automatic evaluation metrics. It would be very interesting to know the correlations.

Author response:
We have compared the correlations in Table 8: Metrics correlation. (Appendix A1)

Reviewer comment (Evaluation):
Ideally we would have computable metrics which correlate high with human evaluations. The paper does not explain very well what the main problems are of existing metrics. It would also be interesting to see what parts of human evaluation are missed by currently used metrics.

Author response:
The computable metrics all hold different assumptions which only contribute part of the aspects in human evaluation. This makes a low correlation with human evaluations. For example, CLIPscore only measures the compatibility of image-caption pairs. But there are more aspects from the human perspective of semantic consistency. In editing tasks there are the level of editing, in subject-driven tasks it has to measure whether the subject aligns. Those are still an open problem in the field. Thus we found a low correlation for CLIP score across all tasks in Table 8. On the other hand, perceptual quality metrics like FID focused on visual quality and LPIPS evaluates the distance between image patches. While they are good at detecting distortion and artifacts, From the human perspective of perceptual quality, the overall natural sense of whole image is often considered.

Reviewer comment (Reproducibility):
It remains unclear how a third party with a new method could make use of this benchmark since it is based on human raters. It would be nice if there is some rater-training guide which would allow other researchers to also evaluate their method on the proposed benchmark. There are no safeguards for the maintenance of the benchmark by the authors.

Author response:
There are two ways to evaluate their methods on ImagenHub. The preferred option is that the third party will send a pull request to our GitHub repo to add their model inference interface. We will use their code to perform standaridized evaluation and then publish our leaderboard.
The less preferred option is that the third party will follow our human judgment guide and hire their own raters to do it. Our guidelines and examples are in Appendix A2 and A3. We also added Appendix A7, A8, and A9 to show how a third party with a new method could make use of the benchmark. Links are hidden due to the Anonymous policy.

Reviewer comment (Evaluation):
I found the discoveries and insights not especially surprising. They were often more based on looking at results than referring to the human evaluation rates.

Author response:
We think there are some interesting insights: (1) our rigorous eval shows that the existing methods despite their popularity are actually doing very poorly on several generation tasks. There is still a large room for improvement, (2) we went through the claims in previous papers and found that 17% of them are ill-posed due to rigorous human evaluation, (3) we found that besides T2I and subject-driven generation, the automatic metrics are having extremely low correlation with human raters. It indicates that we need to develop better metrics for those tasks.

Reviewer comment (Novelty):
The technical contribution is still a bit shallow and the possible usage for future model evaluation is unclear.

Author response:
Our main technical contribution focused on the inference library and the unified inference and evaluation protocols. Although we have already released the codebase on GitHub, we cannot mention the links due to the Anonymous policy. In this revised version, we added Appendix A7, A8, and A9 to show how a third party and researchers can benefit from our work and easily extend it.

Reviewer comment (Experiments):
It would be better if the authors can involve more top-performing image generation approaches in this comparisons (though some of them may not be opensource), like MidJourney or DALLE-3 (in a later revision of the paper).

Author response:
We have gathered and evaluated the generation results of MidJourney and DALLE-3 on our ImagenHub benchmark. Please refer to B1 in our new paper version. It turns out that DALLE-3 and Mijourney are extremely good, outperforming the other methods by an absolute overall score of 10-20%.

Reviewer comment (Evaluation):
Some previous works (like T2I CompBench) have also proposed some evaluation metrics for benchmarking conditional image generation models (related to the Semantic Consistency part in this paper). It would be better to discuss and see if such metrics result in similar trends compared to the human-based evaluation.

Author response:
T2I CompBench’s proposed metric is specifically for Text-To-Image task. They cannot be used across all the ImagenHub defined tasks. For example, UniDet (from T2I CompBench) does not work on Subject-Driven Image Editing and Generation tasks. Our paper focused on evaluation metrics that apply to most of the tasks.

Reviewer comment (Reproducibility):
How much human labor is required to build this benchmark (in annotator x hour)?

Author response:
* The current benchmark (total of 30 models) was built by 24 annotators in a total of an estimated 150 hours. Each of the annotators conducted evaluations on 3-4 sets (or more) of images.  
* We required an annotator to rate one set of images in one go.  
* For the time used in evaluating one set of images (around 150-200 images). The mean time reported is 1 hour 40 minutes.

Reviewer comment (Evaluation):
The paper states that 83% of the published results are consistent with the ranking, and the presented evaluation results often validate the results of published works. Where does this 83% come from?

Author response:
(w1a) The paper states that 83% of the published results are consistent with the ranking ... Where does this 83% come from?

We validated the claims in Section 5.1. We compared the rankings of human evaluation and automatic rankings to the published works. 25 models aligned with our results thus 25/30 ~= 83%.

Reviewer comment (Evaluation):
The paper states that 83% of the published results are consistent with the ranking, and the presented evaluation results often validate the results of published works. What about the other 17%? What kind of results from published work is not consistent with the presented work? Is it due to limitations of the presented paper or wrong claims by published work?

Author response:
(w1b) ... What about the other 17%? What kind of results from published work is not consistent with the presented work? Is it due to limitations of the presented paper or wrong claims by published work?

We mentioned the works that were not consistent with the presented work in Section 5.1. We compared the rankings of human evaluation and automatic rankings to the published works. There is a total of 5/30 ~= 17%. The 5 of them are Pix2PixZero, DreamEdit, UniControl, DiffEdit, and BLIP-Diffusion. The inconsistency is mainly due to their human evaluation not being conducted rigorously enough. For example, some of the papers only use single raters for each instance and do not inter-rate agreement. Some of the papers heavily tune the hyper-parameters on the instance level, which is not allowed in our setting where we only allow fixed hyper-parameters.

Reviewer comment (Evaluation):
It would be beneficial to incorporate more automatic measures such as the commonly used FID or detection-based scores to evaluate spatial fidelity, object recognizability, as well as counts of objects.

Author response:
(w2) It would be beneficial to incorporate more automatic measure such as the commonly used FID or detection based scores to evaluate spatial fidelity, object recognizability as well as counts of objects.

We did not have FID score because FID score requires a large set of reference images to compute. For most tasks, our reference set is very small and the FID score might not be statistically significant enough. 
Per the request, we have added KID and FID scores in our revised paper (Appendix A1). Our current benchmark does not have enough data for detection-based scores.

Reviewer comment (Evaluation):
It could be interesting to see an analysis on the costs and time needed for such a unified evaluation protocol given that the method relies on human raters.

Author response:
(w3) It could be interesting to see an analysis of the costs and time needed for such a unified evaluation protocol given that the method relies on human raters.

* For the time used in evaluating one set of images (around 150-200 images). The mean time reported is 1 hour 40 minutes.
* We paid $25 (£20) for each set of images.

Reviewer comment (Evaluation):
Is it possible to analyze drift of user ratings over time? In other words, how much is the rating influenced by the experience/exposure of a rater to the evaluation platform?

Author response:
(q1) Is it possible to analyze drift of user ratings over time? In other words, how much is the rating influenced by the experience/exposure of a rater to the evaluation platform?

That is an interesting problem. We believe that tracking the ratings of each rater across different time points might help analyze the drift of user ratings. From our observation, the user rating is rather often affected by the leniency, and the leniency can be driven by multiple factors such as screen resolution, the rater’s eyesight condition, and mood etc.. However, we also observed that does not create a large impact on the overall mean rating due to our multiple raters policy.

Reviewer comment (Writing):
Typos here and there:  
One of the most popular task —> tasks  [page 1]  
We found that evaluation results from the published papers from are generally [page 3]  
These methods rely on the statistics on an InceptionNet pre-trained on the ImageNet dataset. [page 4]  
A limitation in this work is the reliance on human raters, which is not only expensive and time-consuming. [page 9]

Author response:
We now fixed the mentioned typos and improved the grammar. Thanks for pointing them out!

Reviewer comment (Presentation):
Fig 2 -> what does y axis show? No label.

Author response:
The y-axis in Figure 2 is the overall human evaluation score $O = \sqrt{SC\times PQ}$ in [0.0,1.0] (equation explained in Section 4.1). We have updated Figure 2 in the revised version.

Reviewer comment (Experiments):
ImagenHub Inference Library is a great job. How did you ensure that the best parameter setting is chosen to generate the best results for each model?

Author response:
We used the suggested parameters contained in the official code repositories of each model's work (mentioned in Section 5). We compare our implemented model outputs against their original officially released model outputs. We ensure everything is aligned. We believe the parameters released by the original authors are already well-tuned.

Reviewer comment (Presentation):
What is the last row of Fig 3 showing?!

Author response:
The last row of Figure 3 shows an OpenPose-conditioned Image Generation. In control-guided image generation tasks, the condition can be canny edges, depth maps, openpose, etc. Figure 19 shows more examples of this task.

Reviewer comment (Theory):
In Eq. 1, why is min used? Isn’t min too stringent here? Why not mean instead!?

Author response:
* Choosing min for this purpose emphasizes the importance of meeting all criteria without exception.  
* Using mean would also increase the complexity of human user rating and introduce a misalignment across tasks, as different tasks contain a different number of conditions c_i.

Reviewer comment (Presentation):
Why are there multiple error bars for each condition? Not clear.

Author response:
Each error bar in a color represents a model in the specific task. The idea of Figure 4 is to show the model performance distribution in each task.

Reviewer comment (Evaluation):
“We assigned 3 raters for each model and computed the SC score, PQ score, and Overall human score”. 3 subjects is really not that many here and this makes the results less reliable.

Author response:
We found that 3 raters achieved high inter-rater reliability and adding more raters will not create a large impact on the mean score (refer to A6 in Appendix).

Reviewer comment (Presentation):
What is the keyword column in Table 3?

Author response:
The keyword column lists the distinctive features, techniques, or unique selling points of the model.

Reviewer comment (Presentation):
How is the overall column computed in Table 4?

Author response:
The overall column is the overall human evaluation score $O = \sqrt{SC\times PQ}$ (equation explained in Section 4.1).

Reviewer comment (Experiments):
No information about the subjects and biases etc. are given.

Author response:
We added the demographic information of annotators in A6.