Paper: SelfEval: Leveraging the discriminative nature of generative models for evaluation (ICLR-2024)

Reviewer comment (Novelty):
W1. Novelty. The motivation of their method is reminiscent of the CLIP-R-Precision of Park et al. (2021), where the goal is to pick the right caption among distractors. Although the authors used the likelihood estimations to get a score for text-image matching, it can be seen as an ad-hoc method to replace the cosine similarity in CLIP. Since we cannot control the pretrained models where one is CLIP and the other one is diffusion models, the authors need to study the effectiveness of the proposed method extensively. The limitation of the CLIP score was also explicitly explored in MID [2].

Author response:
- We point out some important distinctions of our work from CLIP R-Precision. CLIP R-precision measures the text retrieval performance given a **generated** image whereas in our case, we use the **ground truth** image-text pairs for evaluation. Unlike such embedding-based metrics, our evaluation is absolute, since models are always evaluated against ground truth. This along with the splits introduced in our work, provide a diagnostic tool for analyzing generative models. Another key takeaway and novelty of SelfEval is the fact that a generative model can be used to evaluate itself. To the best of our knowledge, this capability has never been shown before and differentiates our work from other evaluation metrics proposed in literature. 

- Limitations of CLIP were highlighted in MID [2] but their analysis is limited and doesn’t show clearly how this affects the final evaluation. In our work, we show the limitations of CLIP’s text understanding and how using different CLIP models affects the actual evaluation.

Reviewer comment (Novelty):
W2. State-of-the-art comparison. For the text-image faithfulness, you could compare the state-of-the-art performance with DALL-Eval [1], MID [2], LLMScore [3], and VPEval [4]. The paper failed to cite them. Please allocate the dedicated paragraph in Sec. 2 to include them and compare the related works.

Author response:
- We believe [4] was made public AFTER the ICLR paper submission deadline. Except method [2], none of the other works are published in literature. Moreover, methods [1-3] use external models (like CLIP) for evaluation and as mentioned throughout our paper, using external models requires additional error analysis to prevent any biases. With SelfEval, we aim to eliminate any external model for evaluating text faithfulness of generative models and we believe it is unfair to compare SelfEval against these methods. We will add a separate section in the related works to discuss these methods.

Reviewer comment (Experiments):
As the authors said, the formulation can extend to any conditional diffusion model. However, the current analysis and experiments only demonstrate that the method is effective for text-to-image generation tasks with diffusion models.

Author response:
Using Eq. 4-9 (in the main paper)  one can compute the likelihood of the data $x$ given conditioning $c$.  Throughout the derivation we do not make any assumptions on the shape/format of the conditioning variable $c$. We chose to proceed with text-to-image generation because evaluating the performance of the model on the image-text matching task is correlated with its generation capabilities. If the reviewer has specific conditioning variables or downstream tasks in mind, please let us know and we can provide a specific and detailed answer.

Reviewer comment (Evaluation):
Considering the method only evaluates the text-faithfulness without image quality in text-to-image generation tasks, the application scope will be very limited. Therefore, the contribution is not overwhelming and might not be enough for this conference, in my opinion.

Author response:
We believe that proper evaluation metrics are necessary to make progress in any field. Given the exponential progress in generative modeling, there is a need for developing strong evaluation metrics. We argue that research on evaluation metrics is not narrow but rather necessary to propel the advancement of generative models. Moreover, [1]  proposed CLIP score, widely used for evaluating text faithfulness while [2] proposed FVD a metric to measure quality of generated videos, both of which are widely used by the generative community making such research very valuable.

[1] Jack Hessel et al. CLIPScore: A Reference-free Evaluation Metric for Image Captioning. ”EMNLP (2021)”.

[2] Unterthiner, Thomas et al. “FVD: A new Metric for Video Generation.” DGS@ICLR (2019)

Reviewer comment (Experiments):
The experiments are also limited. Since the proposed method is an evaluation metric, trying it to evaluate other frameworks, such as VAEs and GANs, is also necessary. If the method can only be applied to diffusion models, it will be less valuable.

Author response:
We re-iterate that the contribution of our work is broadly two fold. 1) Generative models can themselves be used to evaluate their generations and 2) a principled approach to achieving this with text-to-image diffusion models and repurposing discriminative ground truth image-text pairs. We believe that 1) is an important contribution with far reaching consequences and is not specific to diffusion models. Generative models can inherently be used to evaluate themselves using Bayes’ Rule and this applies to any future developments in this area. We agree with the reviewer that showing results on additional models is beneficial but after an extensive review we fail to obtain any text-to-image GAN or VAE models which are publicly available making it impossible to evaluate using this approach (see common response above). Instead we chose four popular models, with enough details in their respective papers for re-implementation, from two families of diffusion models to validate our hypothesis. We urge the reviewer to provide specific details (papers, codes, checkpoints) of good text-to-image generative models we can include in our analysis.

Reviewer comment (Experiments):
Although diffusion models have become very popular recently, this framework has not become the only generative model. Meanwhile, other frameworks also perform well in text-to-image generation tasks. Therefore, the method has a limited application scope. The authors should try other conditional generation tasks rather than only text-to-image generation tasks, or try applying the method to other generative models.

Author response:
Using Eq. 4-9 (in the main paper)  one can compute the likelihood of the data $x$ given conditioning $c$.  Throughout the derivation we do not make any assumptions on the shape/format of the conditioning variable $c$. We chose to proceed with text-to-image generation because evaluating the performance of the model on the image-text matching task is correlated with its generation capabilities. If the reviewer has specific conditioning variables or downstream tasks in mind, please let us know and we can provide a specific and detailed answer.

We re-iterate that the contribution of our work is broadly two fold. 1) Generative models can themselves be used to evaluate their generations and 2) a principled approach to achieving this with text-to-image diffusion models and repurposing discriminative ground truth image-text pairs. We believe that 1) is an important contribution with far reaching consequences and is not specific to diffusion models. Generative models can inherently be used to evaluate themselves using Bayes’ Rule and this applies to any future developments in this area. We agree with the reviewer that showing results on additional models is beneficial but after an extensive review we fail to obtain any text-to-image GAN or VAE models which are publicly available making it impossible to evaluate using this approach (see common response above). Instead we chose four popular models, with enough details in their respective papers for re-implementation, from two families of diffusion models to validate our hypothesis. We urge the reviewer to provide specific details (papers, codes, checkpoints) of good text-to-image generative models we can include in our analysis.

Reviewer comment (Experiments):
The performance of SELFEVAL is inherently linked to the effectiveness of the generative model itself, which could limit its reliability if the generative model has weaknesses in certain areas. From this perspective, more generative models should be evaluated.

Author response:
We agree with the reviewer that the limitations of the evaluating model affects the performance of the evaluation itself but as repeatedly mentioned throughout the paper, this is an issue only when an external model is used. SelfEval uses the generative model itself for evaluation. For example, if the generative model is weak in “counting”, then as shown in the paper, the scores for this task are lower. We show that this is the case with four popular text-to-image models and request the reviewer to provide the details (paper, code and checkpoints) of other generative models they deem useful.

Reviewer comment (Evaluation):
The authors don't discuss the potential limitations of their proposed method in detail.

Author response:
SelfEval relies on the sampling of the generative model to compute the scores. So the limitations of the sampling process of a generative model affect SelfEval. Unlike other black-box evaluation methods which only require the generations from the model, SelfEval requires the model definition and its checkpoints for evaluation. We can provide more details if necessary and will add this discussion to the next draft.

Reviewer comment (Experiments):
Can this method be employed in GAN, VAE, and flow-based generative models?

Author response:
Any likelihood-based generative model (i.e. ones that model the data distribution $p_{\text{data}}(x)$ , like VAEs) can be used with SelfEval. For implicit generative models, like GANs, methods like GAN inversion [1] can be used to obtain likelihood scores from them as described in [2]. After an extensive survey of existing methods, we failed to obtain public repositories of  text-to-image GANs or VAEs with checkpoints. We request the reviewer to point us to relevant text-to-image generative models (With code and checkpoint) for further experiments.

Reviewer comment (Experiments):
Including a broader range of comparative studies with current methods for assessing the quality of text-to-image generative models could strengthen your arguments.

Author response:
Dear Reviewer,  
We perform additional experiments comparing SelfEval with MID[1] as suggested by Reviewer M8yh. MID uses a CLIP model to compute mutual information divergence and we observe that it suffers from the same drawbacks as CLIP-Score (Fig. 2 in the main paper). We show the results in Table-1 below. The second and third column in Table-1 compute MID using ViT-B/32 and ViT-L/14 backbones respectively. Similar to CLIP score, depending on the evaluation model used, LDM-CLIP (with ViT-L/14 text encoder) ranks higher/lower than LDM-CLIP (with ViT-H/14 text encoder). We believe this issue exists regardless of the metric as long as it relies on external models. For more details refer to the review posted to Reviewer M8yh.  

Table-1: Drawback of MID  
|             Model               |  ViT-B/32 | ViT-L/14  |  
|-----------------------------|-------------|-------------|  
| LDM-CLIP (ViT-L/14) |    27.77    |     25.70   |     
| LDM-CLIP (ViT-H/14) |    29.25    |    23.53    |   

Please take a look at our posted rebuttal and additional discussions and provide us with more details. We believe this discussion can convince the reviewer to improve their score.  

[1] Kim, J.-H. et al. (2022). Mutual Information Divergence: A Unified Metric for Multimodal Generative Models. Advances in Neural Information Processing Systems 35.

Reviewer comment (Experiments):
How different text encoders impact the performance of SELFEVAL.

Author response:
We show the effect of different text encoders in Sec. 4.2 of the main paper.