Paper: Attribute Based Interpretable Evaluation Metrics for Generative Models (ICLR-2024)

Reviewer comment (Experiments):
In 5.2, the authors intend to highlight the need of PaD over SaD. However, they do not actually compare them, with no results for SaD to be found in this section.

Author response:
We support the necessity of PaD over SaD by presenting an additional experiment as follows.
1. We curate three sets of images sampled from CelebA, with the identical marginal distribution of individual attributes.
  **A**. Randomly sampled with a restriction: only *men* wear eyeglasses.
  **B**. Randomly sampled with a restriction: only *men* wear eyeglasses (Identical to set A).
  **C**. Randomly sampled with a restriction: only *women* wear eyeglasses (Corrupted correlation).
1. We measure SaD and PaD of **B** and **C** from **A**, i.e., SaD(**A**, **B**), SaD(**A**, **C**), PaD(**A**, **B**), and PaD(**A**, **C**).
1. Observe whether SaD captures difference due to the corrupted correlation between gender and eyeglasses.

Attribute distributions of set **A** and **B** 
|       |  |  |     |
|:-----:|:------------:|:------------:|:---:|
|       | eyeglasses O | eyeglasses X |  marginals   |
|  man  |    **0.1**   |      0.4     | 0.5 |
| woman |       0      |      0.5     | 0.5 |
|   marginals    |      0.1    |      0.9    | 1.0 |


Attribute distribution of set **C** 
|       |  |  |     |
|:-----:|:------------:|:------------:|:-----:|
|       | eyeglasses O | eyeglasses X |  marginals   |
|  man  |       0      |      0.5     | 0.5 |
| woman |    **0.1**   |      0.4    | 0.5 |
|   marginals    | 0.1         | 0.9         | 1.0 |

Note that the marginals are the same across **A**, **B**, and **C**. Hence we expect SaD to be consistent. Furthermore, we expect PaD to be different because the correlation between gender and eyeglasses is corrupted.

|                      |  |        |  |
|:--------------------:|:----:|:--------:|:-------------------:|
|                      |  SaD |     PaD   | PaD(man&eyeglasses) |
| between sets **A** and **B** | 1.52 |   6.14   |        5.68      |
| between sets **A** and **C** | 1.51 |  **9.60** |      **251.34**     |



Indeed, SaD(**A**, **B**) $\simeq$ SaD(**A**, **C**) while PaD(**A**, **B**) << PaD(**A**, **C**). In addition, PaD correctly reveals the most influential pair of attributes (man & eyeglasses) by 26$\times$ higher PaD than the mean PaD.

This highlights the effectiveness of PaD in capturing errors in pairwise relations, and the necessity of employing PaD for comprehensive model analysis.

*PaD is a new evaluation metric, using the novel concept of paired attributes and regarding their relationships, and is also interpretable.* In our humble opinion, this is of great interest to the ICLR community.

We added full SaD and PaD in Figure S12. Please refer to the revised version.
We will incorporate this discussion into the main paper of the final camera-ready version.

Reviewer comment (Evaluation):
The approach would be, by nature, sensitive to the quality of the attribute detector, and only attributes that are consistently visually detectable should be used, since the metrics would be a mixture of the quality of the generator and that of the detector.

Author response:
This is the characteristics shared by most existing metrics that use feature extractor including FID, precision, recall, LPIPS, etc. We respectfully suggest that designing metrics without dependency is an orthogonal research direction.

Reviewer comment (Writing):
The paper needs to be improved in terms of writing and structure.

Author response:
Thank you for helping us improve readability. Changes are summarized below.

> Single-attribute Divergence (SaD) measures the divergence regarding PDFs of a single attribute. Paired-attribute Divergence (PaD) measures the divergence regard- ing joint PDFs of a pair of attributes. *They provide which attributes the models struggle*. 

$\rightarrow$ Single-attribute Divergence (SaD) *reveals the attributes that are generated excessively or insufficiently by measuring* the divergence of PDFs of individual attributes. Paired-attribute Divergence (PaD) *reveals such pairs of attributes by measuring* the divergence of joint PDFs of pairs of attributes. 


> *all SaD worst-rank attributes have negative mean differences that mean SDs tend to omit some objects*

$\rightarrow$ *mean difference of attribute strengths are below zero. It implies that SDs tend to omit some objects rather than over-generating them.*


> *We infer that the color-related attributes are the inferiority of DMs.*

$\rightarrow$ *It implies that DMs fall short on modeling color-related attributes.*


Besides, we have revised writings that were difficult to comprehend. Please refer to the revised version.

Reviewer comment (Writing):
With respect to the writing, there are many sentences that need some improvement. Just some examples:  
- “They provide which attributes the models struggle.”  
- “all SaD top-rank attributes have negative mean differences that mean SDs tend to omit some objects”  
- “We infer that the color-related attributes are the inferiority of DMs”

Author response:
Thank you for helping us improve readability. Changes are summarized below.

> Single-attribute Divergence (SaD) measures the divergence regarding PDFs of a single attribute. Paired-attribute Divergence (PaD) measures the divergence regard- ing joint PDFs of a pair of attributes. *They provide which attributes the models struggle*. 

$\rightarrow$ Single-attribute Divergence (SaD) *reveals the attributes that are generated excessively or insufficiently by measuring* the divergence of PDFs of individual attributes. Paired-attribute Divergence (PaD) *reveals such pairs of attributes by measuring* the divergence of joint PDFs of pairs of attributes. 


> *all SaD worst-rank attributes have negative mean differences that mean SDs tend to omit some objects*

$\rightarrow$ *mean difference of attribute strengths are below zero. It implies that SDs tend to omit some objects rather than over-generating them.*


> *We infer that the color-related attributes are the inferiority of DMs.*

$\rightarrow$ *It implies that DMs fall short on modeling color-related attributes.*


Besides, we have revised writings that were difficult to comprehend. Please refer to the revised version.

Reviewer comment (Presentation):
The mean difference (Eq 7) seems to be an element of the methodology but appears in the experiments section.

Author response:
Thank you for the great suggestion. We moved equation for mean difference to methology section 4.1.

Reviewer comment (Presentation):
Many of the figures (like Fig 4a) display text that is impossible to read due to its size (and when zooming in it actually becomes pixelated).

Author response:
Thanks for great suggestion. In response, we revised Fig 4a, showcasing only the most influential attribute/attribute pairs contributing to SaD and PaD. Besides, the labels of the peripheral attributes are removed from Figure 4a, S7, and S8. We have also included complete figures for SaD and PaD (Figure S9, Figure S10),  allowing for the full readability of all text labels.

Reviewer comment (Evaluation):
Some of the comparisons could be more comprehensive. For instance, Table 4 shows no other metrics than the proposed ones.

Author response:
In Table 4, other metrics are irrelevant because they have nothing to do with attributes. The message of Table 4 is that Stable Diffusions omit or over-generate specific attributes, and these differences vary across the versions. Nevertheless, we report them below.


Table 4: SaD and PaD of different versions of Stable Diffusion


|        | FID    | FID_CLIP | Precision | Recall    | Density   | Coverage  |
|--------|--------|----------|-----------|-----------|-----------|-----------|
| SDv1.5 | **14.748** | **8.902**    | **0.707** | 0.962     | **0.567** | **0.869** |
| SDv2.1 | 15.203 | 10.659   | 0.663     | **0.969** | 0.414     | 0.672     |

Reviewer comment (Evaluation):
My major concern with the proposed metrics is with respect to resolution. How does the difference in absolute values translate to actually seeing a difference in the generations. For example, in [1] authors identify, with the help of human evaluation, that the resolution of FVD is 50, i.e. a human rater can tell the difference between generations of two models if their corresponding FVD scores differ by atleast 50 points.

Author response:
Thank you for the nice suggestion.

Resolution of FVD can be measured by a few samples because most users have common sense for high quality video. On the other hand, SaD and PaD require users to assess the distribution of attributes in the training set and the generated set. As it is prohibitive, to make the problem easier, we will ask users to rank the four sets of generated images from LDM-20/-50/-100/-200 which are used in Table 2. We expect the users to be able to tell the difference between sets with large SaD/PaD gaps and vice versa.


|  | SaD | PaD |
| -------- | -------- | -------- |
| LDM_20steps     | 21.07     | 50.06     |
| LDM_50steps     | 10.42     | 25.36     |
| LDM_100steps     | 11.91     | 27.26     |
| LDM_200steps     | 14.04     | 30.71     |

We will share the results as soon as the user study is completed.

Reviewer comment (Evaluation):
How does SaD or PaD correlate with human evaluation? Do humans agree that the attributes identified by SaD and PaD are indeed misrepresented in the generations of the model?

Author response:
Thank you for the nice suggestion. We will conduct two user studies with toy experiments as follows.

With the training set having a medium *smile*, we will ask users to rank three sets with strong, medium, and no smile. We expect the rank to agree to SaD.

With the training set having a strong positive correlation between *man* and *smile*, we will ask users to rank three sets with strong positive, no, and strong negative correlations. We expect the rank to agree to PaD.

We will share the results as soon as the user study is completed.

Reviewer comment (Evaluation):
Do these metrics correlate well with any existing evaluation metrics?

Author response:
Frechet Inception Distance[FID][FID_clip] shows a consistent trend with our metrics across most generative models, while other metrics[Precision&Recall][Density&Coverage] showed no correlations with proposed metrics in Experiment 5.2.

We found notable differences in PaD with FIDs, particularly revealing significant penalties for [ProjectedGAN]. We attribute ProjectedGAN's inferior performance in PaD to its ignorance to attribute correlations, generating 'babies with beards'. 

[FID] Heusel et al., Gans trained by a two time-scale update rule converge to a local nash equilibrium, NeurIPS, 2017

[FID_CLIP] Kynkäänniemi et al., The Role of ImageNet Classes in Fréchet Inception Distance, ICLR, 2023

[Precision&Recall] Sajjadi et al., Assessing generative models via precision and recall. NeurIPS, 2018

[Density&Coverage] Naeem et al., Reliable fidelity and diversity metrics for generative models, ICML, 2020

Reviewer comment (Evaluation):
Do these metrics work with any other features than CLIP features? For example, for textures, do features from DINO/SAM (get reference features from some texture dataset and compare to features of generated images) work as well or is the joint embedding of CLIP necessary for the success of SaD and PaD.

Author response:
DINO/SAM with texture datasets would work but they would measure the strength of textures instead of attributes. It is a natural extension of our metrics when a user wants to evaluate the distribution of textures in the generated images. This generalization is a virtue of our metric. Using multi-modal feature extractors such as [ImageBind] also could be an interesting research topic. We added these discussion in the revised version.

[ImageBind] ImageBind_ One Embedding Space To Bind Them All, Rohit Girdhar et al., CVPR2023, https://imagebind.metademolab.com/

Reviewer comment (Evaluation):
Can these metrics be used to measure quality and faithfulness of text to image generative models? If so, how would one go about that?

Author response:
_1) We are afraid but our metrics are not for measuring quality. 2) Heterogeneous CLIP score better measures faithfulness to the text than the original CLIP. 3) For the text-to-image task, SaD and PaD measure the divergence of the generated images from the training images regarding the attributes in the text prompts. Rather than the text prompts, SaD and PaD consider the training images as the reference.

Reviewer comment (Novelty):
The overall contribution is incremental, though the research motivation is fairly clear and reasonable.

Author response:
We sincerely appreciate recognizing our motivation. 
We respectfully suggest that *it is desirable to resolve problems using simple solutions because they are easy to adopt in future research*. Especially, *our solutions are helpful for the research community and practitioners* as follows:

- Heterogeneous CLIPScore indicates the presence of attributes in a given image, while CLIPScore does not.
- Single and Paired attribute Divergence (SaD and PaD) reveal the problematic attributes, while existing metrics do not.

Reviewer comment (Novelty):
The proposed SaD and PaD are straightforward to measure the divergences of single and paired attributes.

Author response:
We sincerely appreciate recognizing our motivation. 
We respectfully suggest that *it is desirable to resolve problems using simple solutions because they are easy to adopt in future research*. Especially, *our solutions are helpful for the research community and practitioners* as follows:

- Heterogeneous CLIPScore indicates the presence of attributes in a given image, while CLIPScore does not.
- Single and Paired attribute Divergence (SaD and PaD) reveal the problematic attributes, while existing metrics do not.

Reviewer comment (Evaluation):
For interpretability, the interpretability comes from the attributes, which are obtained from annotation or large models, so the interpretability of the evaluation metrics is limited by the set of attributes.

Author response:
Thank you for the constructive feedback. Indeed the interpretability of our metrics depends on the choice of attributes. As different users want different capabilities of the generative models, our metrics provide *customizable* measures that capture attributes designated by users. It would be beneficial to extend our work with different encoders for different modalities other than textual attributes as stated by reviewer B3SF. We added this discussion in the revised version.

Reviewer comment (Evaluation):
Interpretation based on attributes is only one possible solution, which may be not complete or accurate to characterize the capability of generative models.

Author response:
Thank you for the constructive feedback. Indeed the interpretability of our metrics depends on the choice of attributes. As different users want different capabilities of the generative models, our metrics provide *customizable* measures that capture attributes designated by users. It would be beneficial to extend our work with different encoders for different modalities other than textual attributes as stated by reviewer B3SF. We added this discussion in the revised version.

Reviewer comment (Novelty):
The attributes selection methods are somewhat simple, providing little insight from the process.

Author response:
Large models or annotations are merely options to ease users' burden. If a user cares about fairness, one can choose the target attributes by oneself. *Resolving the bias of annotations or large models is an orthogonal research topic.* We respectfully suggest that such bias should not be a reason for rejecting this paper. Nevertheless, we agree that we should be careful not to be biased in choosing the target attributes. This discussion is in Appendix A.1. and we added it in the main paper as well in the revised version.

Reviewer comment (Evaluation):
There is a concern that the attributes might be biased due to biased annotations or large models.

Author response:
Large models or annotations are merely options to ease users' burden. If a user cares about fairness, one can choose the target attributes by oneself. *Resolving the bias of annotations or large models is an orthogonal research topic.* We respectfully suggest that such bias should not be a reason for rejecting this paper. Nevertheless, we agree that we should be careful not to be biased in choosing the target attributes. This discussion is in Appendix A.1. and we added it in the main paper as well in the revised version.