Paper: PRIME: Prioritizing Interpretability in Failure Mode Extraction (ICLR-2024)

Reviewer comment (Novelty):
The authors mention comparisons with other studies (e.g., Eyuboglu et al., 2022; Jain et al., 2022) in the text but do not elaborate on the strengths and weaknesses of these studies and how this study compares favorably to them. It is suggested that the authors provide a more detailed explanation in this regard.

Author response:
3rd paragraph In the introduction includes a brief comparison of the framework of our method and existing work. In section 4.3 and Appendix A.10, we compare our results with two prominent existing methods [1, 2] (state-of-the-art methods in failure mode detection and explanation with natural language). Our comparison evaluates the quality of generated captions. We will add a section to the Appendix covering more details about the way those methods work.

The reason we state that our comparison is indeed in favor of existing methods is that they utilize CLIP to find text descriptions that maximize similarity score to images inside failure modes and we use CLIPScore (same metric) for comparison between different methods. Our method does not include CLIPScore in finding descriptions but we improve over existing work even in that metric.

Reviewer comment (Reproducibility):
This paper mentions the impact of using different hyperparameter values on the results when discussing generalizability, but does not explicitly state the specific range and selection methods for these hyperparameter values. It is suggested that the authors provide detailed explanations in the paper regarding this so that readers can better understand and apply these methods.

Author response:
Regarding the hyperparameters in our approach, we note that all of those hyperparameters have intuitive definitions, enabling a user to easily calibrate PRIME toward their specific preferences. We will add a more detailed section in the Appendix for these hyperparameters in the updated draft. Here are descriptions of hyperparameters in PRIME:
+ Parameter $a$ controls a tradeoff between the difficulty and the quantity of detected failure modes. For example, selecting a high value of $a$ results in failure modes that are more difficult but fewer in number. Figure 10 in the Appendix shows this tradeoff.
+ Parameter $s$ determines the minimum number of required images inside a group to be detected as a failure mode. Small groups may not be reliable, thus, we filter them out. Larger value for $s$ results in more reliable and generalizable failure modes. The choice of $s$ also depends on the number of samples within the dataset. For larger datasets, we can assume that different subpopulations are sufficiently represented in the dataset, thus, a larger value for $s$ can be used. We refer to Figure 10 in the Appendix for observing the effect of $s$.
+ $l$ determines the maximum number of tags we consider for combination. In datasets we considered, a combination of more than $5$ tags wouldn’t result in groups with at least $s$ images, thus, we set $l \leq 4$ in our experiments. The choice of $l$ depends on the dataset and tagging model. 
+ $b_i$ refers to the degree of necessity for tags inside the failure mode. The current choice of $b_i$s is only a sample choice. We wanted the appearance of each tag to have a significant impact on the difficulty of detected failure modes.

Reviewer comment (Experiments):
Lack of comparison with state-of-the-art: The paper does not compare the proposed method with the current state-of-the-art methods for failure mode detection and description, making it difficult to assess its advancement. It is recommended that the authors perform a quantitative comparison with the latest methods.

Author response:
3rd paragraph In the introduction includes a brief comparison of the framework of our method and existing work. In section 4.3 and Appendix A.10, we compare our results with two prominent existing methods [1, 2] (state-of-the-art methods in failure mode detection and explanation with natural language). Our comparison evaluates the quality of generated captions. We will add a section to the Appendix covering more details about the way those methods work.

The reason we state that our comparison is indeed in favor of existing methods is that they utilize CLIP to find text descriptions that maximize similarity score to images inside failure modes and we use CLIPScore (same metric) for comparison between different methods. Our method does not include CLIPScore in finding descriptions but we improve over existing work even in that metric.

Reviewer comment (Experiments):
The method seems highly reliant on being able to find all relevant tags first. Failure modes caused by concepts not in the tag set $T_c$ would remain undetected. Given that low frequency tags are filtered out, this could miss potentially important but less frequently occurring failure modes, such as spurious correlations. Broadly, there seems to be a tradeoff involved in choosing the size of the tag set -- larger the set, more the failure modes that can be caught, but also more computationally expensive the method is.

Author response:
> The method seems highly reliant on being able to find all relevant tags first. Failure modes caused by concepts not in the tag set would remain undetected.

We first refer the reviewer to G1 for tags inside $T_c$. Rare tags are removed to avoid any noisy or irrelevant ones in the algorithm. We agree with the reviewer that this can lead to missing some specific hard subpopulations but low-frequency tags represent subpopulations that do not include enough samples and in PRIME we ignore small subpopulations because they might not be reliable.

Reviewer comment (Evaluation):
PRIME relies on a pre-trained tagging model (in this case, RAM) to provide a set of tags for each image. However, relying on a pre-trained model may have several limitations: (a) for example, when considering the "fox+white+zoo" group in Figure 1, RAM may fail to tag a large number of images in the dataset that are in a "zoo" (false negatives), or include images that are not taken in a zoo in the group (false positives); (b) the tags output by RAM are likely not comprehensive, as there may be objects RAM is worse at recognizing due to their absence in the pretraining data; (c) the present draft is unclear on how many tags are sampled for each image in this first part, and how the tags are sampled (e.g., do you only look at the 100 "most likely" tags for each image?); (d) all evaluation experiments in Section 5 appear to assume that the tags generated by RAM identify groups of images that "match" them, but without further evidence this assumption is unfounded. The reviewer suggests small-scale evaluations reporting the precision/recall of the RAM tagger, potentially comparing its tagging performance on classes with existing metadata labels.

Author response:
### Understanding limitations of relying on a pre-trained tagging model
> As a simple example, when we consider the "fox+white+zoo" group in Figure 1 ...

We first refer the reviewer to G1, where we affirm RAM's reliability via large scale evaluations from their original paper and a small scale human study we conducted ourselves validating RAM's performance on the datasets we used.

Secondly, it is worth noting that other state-of-the-art methods also suffer from the same issues, e.g., it is possible that some images with the same semantic attributes are not assigned to the detected clusters (false negatives), or some samples that might be different from other images in the cluster are assigned to that (false positives). Furthermore, this also happens when finding the text that maximizes similarity to images inside the group, the text may not cover all common attributes within images inside the cluster while covering some attributes that might be present in images outside of the failure mode.

This is why we measure similarity, specificity, and coherency in Section 4.3 across different methods and observe improvements in our method. This shows that PRIME, even though it is not perfect and has these limitations as the reviewer pointed out, can be more promising.

> The tags output by RAM also are likely not comprehensive ...

We refer the reviewer to G1.

> I may be missing something but the present draft is also unclear on how many tags ...

We use the default set of hyperparameters for RAM that extracts tags from images. Number of tags that RAM generates is not fixed and depends on the image. Number of tags range from 5 to 20.

> the authors can perform any kind of small-scale evaluations where they report the precision/recall of the RAM tagger.

We refer the reviewer to G1.

Reviewer comment (Experiments):
Each group is defined as the intersection of a set of tags that are all present in that group's images, but there is ambiguity in how to interpret the tags. While example groups in Figures 1 and 2 have clear interpretations, in general the ambiguity of modifiers, attributes, and propositions (relationships between multiple objects) may lead to misinterpretation. For example, for the group "fox + white," which the authors interpret as foxes with white fur, it may actually include any photo with "white" anywhere in it (e.g., an orange fox in snow). Leaving this point unaddressed may mislead users of PRIME to draw incorrect conclusions.

Author response:
### Clarify how the "tags" should be interpreted (moving from tags to descriptive captions)
We appreciate the reviewer for pointing out this limitation. Interpreting some of the tags (especially attributes) might be a bit misleading. We will amend the text to avoid the confusion that the tag “white” in fox class means “fox with white fur”. In fact, our results show that at the intersection of some tags (not always related to the main object in the image), there might be the model’s failures.

> I found your G2 response to be clarifying (I hope you include this information in your updated manuscript), but it still doesn't fully address my concern ...

We appreciate the reviewer for pointing this out. In the new version, we added this discussion that interpreting some of the tags is challenging as they may not correspond to the main object in the image. We have also amended the text to avoid any misunderstanding.

Reviewer comment (Experiments):
More detail is needed in the main text about some parts of the experimental methodology to ensure the examples in the paper are representative of PRIME's performance rather than cherry-picked. Specifically: (a) clarify exactly how ChatGPT was used to move from sets of tags to semantically meaningful captions in Section 4.2, and whether any qualitative verification was done to ensure these captions "match" their corresponding image groups; (b) clarify how the images visualized for each group in Figures 1, 3, and 5 were selected – whether randomly or purposely chosen – as this relates to concerns in Weakness #1.

Author response:
### Clarify experimental methodology
> Specifically, how exactly did you "use ChatGPT" to move from sets of tags ...

We refer the reviewer to G2.

> How did you select the images that you're visualizing for each of the groups in Figures 1, 3, and 5 ... 

Images are randomly selected from the group of images representing those tags. This is the case not only for those Figures but also other additional images in the Appendix.

Reviewer comment (Writing):
Make clear in the introduction that the method does not rely on human-labeled tags, but rather uses a pre-trained model.

Author response:
> Make clear in the introduction that your method doesn't rely on human-labeled tags, but rather uses a pre-trained model, 

Sure, we’ll explicitly mention that in the updated draft.

Reviewer comment (Writing):
nit (p2): revise "images who match group of tags identified as a failure mode" to "the images that match a failure mode".

Author response:
> nit (p2): "images who match group of tags identified as a failure mode" => "the images that match a failure mode"...

Thanks for pointing this out, we will fix this issue in the updated draft.

Reviewer comment (Writing):
nit (Section 4): rather than saying "difficulty (of the detected failure modes)", explicitly use terms like "accuracy" or "performance gap", as "difficulty" is vague.

Author response:
> nit (Section 4): rather than saying "difficulty (of the detected failure modes)", explicitly use terms like "accuracy" or "performance gap", as "difficulty" is vague, nit (p2): "images who match group of tags identified as a failure mode" => "the images that match a failure mode"...

Thanks for pointing this out, we will fix this issue in the updated draft.

Reviewer comment (Novelty):
Consider adding two references to the related work section that illustrate the difficulty of generating high-quality text descriptions for error discovery: (1) Appendix C of AdaVision [https://arxiv.org/abs/2212.02774], and (2) the user study in Johnson et al. [https://arxiv.org/abs/2306.08167].

Author response:
> Two helpful references to consider adding to your related work section …

Thanks for pointing out these references, they are indeed relevant and aligned with our point of view in this paper. we will add them to the updated draft.

Reviewer comment (Experiments):
The method relies heavily on an auxiliary model (like Recognize Anything Model). Although this makes an important tool for describing failure modes, everything that the proposed method can do is bottlenecked by this auxiliary model's capability. For example, the authors present this method as a tool to describe "failure modes in trained image classification models". However, what happens when the image classification model is trained on a domain-specific dataset like chest X-Rays? How can we expect the auxiliary general purpose model to provide tags for this dataset? This is a practical paper but we do not see ablation studies with other general purpose auxiliary models (apart from RAM), or other experiments with domain-specific image classification models.

Author response:
> The method relies heavily on an auxiliary model … can be bottlenecked by this auxiliary model's capability.

We refer the reviewer to G1.

> what happens when the image classification model is trained on a domain-specific dataset like chest X-Rays?

As mentioned in G1, tagging models will be developed for specific domains. As a result, PRIME’s framework can be applied in those domains as well. We note that this is the case in existing work where a fine-tuned version of CLIP on specific domains is needed for those methods to be effective. We appreciate the reviewer for pointing this out, we will add this discussion to the updated draft.

---

We greatly thank the reviewer for reevaluating our work and improving the score. Here are some comments regarding your concerns.

> It is unreasonable to assume that tagging models can be developed for specific domains where the both data and annotators are scarce.

We note that as other vision-language models (such as CLIP) have been  developed for specific domains, e.g., ConVIRT (CLIP for medical data), tagging models can also be developed for them. For example, one can finetune RAM with image-caption datasets from specific domains. Overall, we propose PRIME as a general framework for detecting and explaining failure modes and we acknowledge that its performance is affected by the auxiliary model used for tagging. More advanced tagging model can leads to improvement in PRIME’s performance.

> So there will be instances when there are common mistakes of the same kind but we as humans can't interpret them. These are the actual situations when we need help of such methods to tell us what's going wrong.

We note that in this line of research, the goal is to explain failure modes in human-understandable terms. So, the focus is to cover and explain failure modes that can be described by humans. We agree with the reviewer that there definitely exists some uninterpretable patterns or situations that the model fails but they are in the scope of our paper.

Reviewer comment (Reproducibility):
The tag generation process requires a threshold hyper-parameter. Similarly, there a lot of hyper-parameters in the proposed method's setup like $b_i$, $a$, $s$, $l$, etc. Some of them look more sensitive than others. However, no insights are given on how they are selected apart from saying "we generally" use these values.

Author response:
> The tag generation process requires a threshold hyperparameter. Similarly, there are a lot of hyper-parameters in the proposed method's setup like $s$, $a$, ...

We used the default hyperparameters of RAM. Regarding the hyperparameters in our approach, we note that all of those hyperparameters have intuitive definitions, enabling a user to easily calibrate PRIME toward their specific preferences. We will add a more detailed section in the Appendix for these hyperparameters in the updated draft. Here are descriptions of hyperparameters in PRIME:
+ Parameter $a$ controls a tradeoff between the difficulty and the quantity of detected failure modes. For example, selecting a high value of $a$ results in failure modes that are more difficult but fewer in number. Figure 10 in the Appendix shows this tradeoff.
+ Parameter $s$ determines the minimum number of required images inside a group to be detected as a failure mode. Small groups may not be reliable, thus, we filter them out. Larger value for $s$ results in more reliable and generalizable failure modes. The choice of $s$ also depends on the number of samples within the dataset. For larger datasets, we can assume that different subpopulations are sufficiently represented in the dataset, thus, a larger value for $s$ can be used. We refer to Figure 10 in the Appendix for observing the effect of $s$.
+ $l$ determines the maximum number of tags we consider for combination. In datasets we considered, a combination of more than $5$ tags wouldn’t result in groups with at least $s$ images, thus, we set $l \leq 4$ in our experiments. The choice of $l$ depends on the dataset and tagging model. 
+ $b_i$ refers to the degree of necessity for tags inside the failure mode. The current choice of $b_i$s is only a sample choice. We wanted the appearance of each tag to have a significant impact on the difficulty of detected failure modes.

Reviewer comment (Evaluation):
In 4.1 paragraph 1, the authors say that $D$ and $D'$ are from the same distribution, and hence the accuracy drop should translate. This is not what generalization means. Generalization applies to the scenario when unseen data is from a shifted distribution. As the authors clearly mention that the unseen dataset is drawn from the same distribution (and not explicitly curated in some way like selective sampling), this dismisses all claims of generalization. The generalization on generated data is an interesting experiment but the fact that authors fine-tuned the generated model on the training data simply nullifies all claims of generalization.

Author response:
> the authors say that  $D$ and $D’$ are from the same distribution …

We note that in failure mode extraction, we do not intend to consider the performance of the classifier on out-of-distribution datasets. In principle, “generalization” refers to the model's ability to adapt properly to new, previously unseen data, drawn from the same distribution as the one used to create the model. By generalization, we showed that failure modes we extract from training data can generalize on unseen (test) data. In other words, hard subpopulations that are extracted from training data, represent hard subpopulations on test data. This measures the reliability of PRIME.

Regarding the experiments about image generation, we want to emphasize that we need to finetune the model on the Living17 dataset to generate images that come from the distribution that the classifier can classify. Otherwise, if generated images are out of distribution, the classifier does not have a good performance for any generated image, making the drop in accuracy for generated images from a hard supbopulation vacuous. Now, we do observe that the classifier performs well on easy subpopulations and loses its accuracy over harder ones.

Reviewer comment (Evaluation):
Regarding the quality of descriptions metrics: Firstly, I have no idea what the AUROC metric means at all. Moreover, these "three" metrics are again based on another auxiliary model. It suffers from all the same issues as mentioned in point 1. The authors don't talk about this potential pitfall. Additionally, this evaluation goes through multiple models to come up with embeddings to calculate a distance metric. First, the tags are generated using RAM. After the failure mode tags are identified, they are passed through a language model to generate captions. Then, the caption is passed through another foundation model to generate text embeddings while the original image is passed through the same foundation model to generate image embeddings. I don't see how we could make any definitive claim about these metrics (which don't seem to be very different from the corresponding values of DOMINO).

Author response:
>  Moreover, these "three" metrics are again based on another auxiliary model.

For evaluating the quality of descriptions, as discussed in the last paragraph of the introduction, it is hard to scalably evaluate, thus, we need to utilize vision-language models (CLIPScore) as a proxy to evaluate them. This is standard practice as other researchers also use foundation models (CLIPScore, BertScore, etc) for large-scale evaluations.

Furthermore, existing methods are *designed to maximize CLIP similarity* of the description of a failure mode to images inside that. We used this metric to show the advantage of our method, as we improve over existing results in similarity, specificity, and coherency despite the fact previous methods *optimize similarity metric by design*.

> Firstly, I have no idea what the AUROC metric means at all.

AUROC measures the specificity of descriptions to images inside the failure modes. For each failure mode, we take the similarity score of the description and images inside that. We also sample some images outside of the failure mode and take the similarity score to the description. Images outside should have smaller similarities than images inside the failure mode. This is why we report AUROC between these two groups to measure the specificity. As mentioned in G2, we generate descriptions in a bag-of-word manner, not with language models.

> which don't seem to be very different from the corresponding values of DOMINO

We note that we have improved mainly in specificity and coherency while slightly improving in similarity score. This comes even though DOMINO uses CLIP and finds text that maximizes this similarity.

Reviewer comment (Evaluation):
Regarding the similarity between representation space and semantic space: This analysis is confusing and potentially flawed. The representation space may carry nuanced information which is not captured by the tags. For example, a common tag which says "water" may have very different representations depending on whether it is ocean blue water or river green water. Therefore, it can not be expected that closeness in representation space should closely align with closeness in semantic space. The semantic space is formed after (potentially several) layers of abstraction on the representation space which is supposed to store all fine-grained details. The fact that table 5 still shows a significant number of overlapping tags in nearby images is something that I would expect to begin with. The images that share maximum number of tags might have very different subtle differences that make them distant in the representation space.

Author response:
> Regarding the similarity between representation space and semantic space

We agree with the reviewer that representation space is indeed practical, useful, and informative. We agree that representation space is not equivalent to semantic space. However, when we need to *assign human-understandable descriptions* to failure modes by considering a cluster in representation space, there must be semantic similarities between those images in a way that we can describe them with *language tags and tokens*. As a result, if images inside a cluster in proximity in the latent space do not share any common semantic attribute, it is hard to describe them in natural language which is the goal of this line of research.

We refer to Figure 11 for some visualization of images that are within a cluster detected by DOMINO but they may not share semantic (human-understandable) concepts, thus, making it harder to describe those clusters with human language. We wanted to highlight the point that the quality of description degrades for clustering-based approaches, hence, we propose another direction that sounds promising.

We thank the reviewer for all the detailed points. We will include all the above limitations in a new section for the updated version of our paper.