VIES core: Towards Explainable Metrics for Conditional Image Synthesis Evaluation (2024)

Abstract: In the rapidly advancing field of conditional image generation research, challenges such as limited explainability lie in effectively evaluating the performance and capabilities of various models. This paper introduces VIEScore, a Visual Instruction-guided Explainable metric for evaluating any conditional image generation tasks. VIEScore leverages general knowledge from Multimodal Large Language Models (MLLMs) as the backbone and does not require training or fine-tuning. We evaluate VIEScore on seven prominent tasks in conditional image tasks and found: (1) VIEScore (GPT4-o) achieves a high Spearman correlation of 0.4 with human evaluations, while the human-to-human correlation is 0.45. (2) VIEScore (with open-source MLLM) is significantly weaker than GPT-4o and GPT-4v in evaluating synthetic images. (3) VIEScore achieves a correlation on par with human ratings in the generation tasks but struggles in editing tasks. With these results, we believe VIEScore shows its great potential to replace human judges in evaluating image synthesis tasks.

## introduction diffusion models have become a focal point in ai research for image synthesis. over the past year, several new models @xcite @xcite have been introduced to enhance control over image generation. however, comprehensively evaluating ai-synthesized images remains a challenging and unresolved issue. while metrics like lpips @xcite , clip-score @xcite , and dreamsim @xcite were proposed, they have certain limitations: (1) these metrics are agnostic the end task, which can fail to measure the desired aspects of the generated images, (2) the score is opaque with limited explainability. these limitations heavily restrict their effectiveness in assessing conditional image generation. some research work @xcite @xcite @xcite relied on human-driven evaluation methods. while humans excel at understanding and interpreting visual content, such methods in the context are facing challenges such as scalability limits and preference subjectivity issues. this reliance on human judgment highlights the need for more uniform evaluation methods in the field. to solve the mentioned issues, we formulate the problem definition with our desired properties, as presented in equation 1. the function f takes an instruction i, a synthesized image o, and c * which is a set of conditions (e.g. style, subject, background, cannyedge, etc). the score function should produce the intermediate rationale in the form of natural language before generating the final score according to the prompt instruction i: f vie (i, o, c * ) = (rationale, score) the function f can be any multimodal large language model (mllm) such as @xcite and llava @xcite , which can take input images to generate human-like text responses. unlike the automatic metrics, mllm can receive human instructions and produce rationale. with such motivation, we introduce viescore (visual instruction-guided explainable score), a framework to assess synthetic images in different conditional image generation tasks. viescore has multiple advantages compared to auto-metrics and human evaluation. it includes: task awareness. existing metrics were often designed to measure a certain aspect of generated images. for example, lpips measures the perceptual similarity of a pair of images, while clip-score measures the text alignment of one single image. as a consequence, these metrics cannot be adapted to evaluate other tasks. viescore acts as a silver bullet to tackle all conditional image generation evaluation processes due to its instruction-guiding property. it can be carefully adjusted with different instruction requirements. explainability. the existing metrics normally output a single float-point score, which cannot offer detailed insights into the 'rationale' behind its evaluations. such a score makes it difficult to interpret the decisions from the metric output. instead, viescore can offer the rationale in the form of natural languages to help humans understand the reasoning process. as depicted in figure 1 , the rationale can significantly improve the trustworthiness of viescore. while the ultimate goal is to derive an mllm that can rate images like humans, in this paper we also explore how well mllms can assess synthetic images compared to human evaluation and present insights and challenges on state-of-the-art mllms towards human evaluators, as shown in figure 2 . ## method during the experiment, we select 29 models evaluated in imagenhub @xcite to compare the correlations with human ratings. see appendix b for the listed models. ## artifacts all datasets and models are publicly accessible for academic use, and the official openai api is available for academic purposes. ## computational experiments all open-source model experiments were conducted on an nvidia rtx a6000 gpu. approximately 500 us dollars were spent on an openai api call for gpt-4v and gpt-4o experiments. ## acknowledgement we thank kai zhang, yujie lu, and tianle li for the discussion. we also thank xueguang ma and tianle li for sharing the quotas of gpt-4v and gpt-4o api calls from openai.