Paper: Holmex: Human-Guided Spurious Correlation Detection and Black-box Model Fixing (ICLR-2024)

Reviewer comment (Reproducibility):
It is unclear how the human is involved in the guiding of spurious correlation detection and model fixing. It seems that according to the algorithm listing 1, the output of the algorithm is presented to humans but it is unclear how humans guide the process as the title of the paper suggests.

Author response:
Algorithm 1 presents a list like Figure 10 in the Appendix to humans, then humans will check if there are spurious correlations. After humans decide which concept is a spurious correlation, the fixing method in Figure 4 can be applied.

Reviewer comment (Experiments):
How scalable is the proposed method?

Author response:
We applied our method on a large dataset of 40k images (the Celeb-A task), and the running time of the whole method is within 10 mins.

Reviewer comment (Evaluation):
Is accuracy the main metric used in evaluation experiments?

Author response:
Yes, it is.

Reviewer comment (Experiments):
Why are the experimental results mainly focused on showing the advantage for ensemble models?

Author response:
People may think that Ensembling is very similar to Transfer Editing as they both do arithmetic among different models and can be helpful for OOD settings. Therefore, we conduct experiments to show the difference between the two methods as well as how we can combine the two methods together to further improve the performance.

Reviewer comment (Experiments):
There is a concern that a simple alternative solution might suffice for the problem of irrelevant concepts with high similarity. Following the equation on the bottom of Page 4, one could compute 
$
P(y = c \mid z) = \frac{\exp(t_c^\top z / T) }{\sum_{c' \in C} \exp(t^\top_{c'} z / T )}
$
where $T$ is temperature in softmax, $c$ is one concept, and $\mathcal{C}$ is the set of all concepts, and then choose a low temperature to reduce the similarity among different concepts.

Author response:
**Q1: Why not use softmax with temperature to address the problem of irrelevant concepts with high similarity?**

We didn't understand what you meant by using temperature to address the issue. The irrelevant concepts with high cosine similarity will be an issue when we represent the concept with this raw text embedding and capture the concept strength by inner-product with raw text embedding with image embeddings. It is not an issue in the original CLIP classifier. We calculate the similarity by cosine-similarity, its range is from -1 to 1. If using cosine-similarity divided by $T$, then its range will change from $-1/T$ to $1/T$. The similarity does not depend on the absolute magnitude. It depends on the relative magnitude.

Reviewer comment (Theory):
For the “Transfer difference of logits” (step 2 in Section 5.1, page 7), different models (i.e., white-box and black-box) can have different scales in logits. Although the paper includes a discussion of “The scale of logits” on page 7, the reviewer’s question about this issue remains unanswered.

Author response:
**Q2: Do you assume that white-box and black-box models share a similar logit scale?**

There is no free lunch for the black-box model fixing. 
Consider a black-box model that just randomly outputs results. No method could cure this black-box model, and the only thing we can do is ignoring its outputs. One important intuition is that a normal model should output a proper probability distribution over class labels. For example, if there are 80\% rainy samples and 20\% sunny samples for the same given input $x$ in the training set, then a normal model will output a probability distribution similar to $(0.8, 0.2)$. Since the mapping between probabilities and logits is bijective (if we ignore the constant shift), we can derive that the logits are roughly $(1.375+constant, 0+constant)$ based on the probabilities $(0.8, 0.2)$. Thus, a normal model's logits can be roughly decided by the proper probabilities. As a result, if the model does not output probabilities too different from the proper ones, applying $y_b + \Delta$ should keep logits on the right scale.

Reviewer comment (Experiments):
The proposed method is only compared with a limited number of methods. For spurious correlation detection, the comparison is only with the PCBM method and its variants. There is no comparison with DISC (Wu et al., 2023), another concept-based method. For spurious correlation mitigation, many methods, particularly those that do not rely on concept vectors [1,2,7-9], are not included in the comparisons.

Author response:
**Questions about Comparison Methods**

We discuss the difference between DISC and our method, why we cannot compare it under our setting, and why we raise our setting. Those discussions are in the comment reply of comparing to DISC. We add a new section in Appendix H to illustrate the difference between our method and the others. We highly recommend reading those tables in Appendix H for a better understanding. As for those non-concept-based methods, we cannot make a fair and reasonable setting, neither. We use the example of salience-map-based methods to illustrate the difficulty and please refer to our reply to review NL8r for this part.

Reviewer comment (Novelty):
The paper does not cite several works in this domain, including: Salient ImageNet: How to detect spurious correlations in deep learning? ICLR 2022; Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations. ICLR 2023; and Wilds: A benchmark of in-the-wild distribution shifts. PMLR, 2021.

Author response:
We have cited all the mentioned papers in the revision paper, and we added a new section in Appendix H to illustrate the difference between our method and the others. We believe reading those tables in Appendix H can help better understand the difference. For the task setting in Wilds, we discuss the difference in the common reply about DISC. Below, we use the salience-map-based method as an example to show its limitations and why we cannot conduct the comparison. 
We have done thorough research on the prior work and those discussions about the salience map can be also found in related work.

Reviewer comment (Experiments):
Salient ImageNet provides a scalable methodology for identifying spurious correlations at scale, but the paper does not include any comparison with that method, and no citation is provided.

Author response:
We have cited all the mentioned papers in the revision paper, and we added a new section in Appendix H to illustrate the difference between our method and the others. We believe reading those tables in Appendix H can help better understand the difference. For the task setting in Wilds, we discuss the difference in the common reply about DISC. Below, we use the salience-map-based method as an example to show its limitations and why we cannot conduct the comparison. 
We have done thorough research on the prior work and those discussions about the salience map can be also found in related work.

**Limitation of salience map and why we cannot compare with it**

\noindent
The first limitation of the salience map is that it cannot bring high-level interpretations. Take the fire truck in Canberra as an example. Since the fire trucks in Canberra are in yellow, `yellow' becomes a spurious correlation when we apply the model to other cities. If we use a salience map for interpretation, it might focus on the fire truck's body. However, we cannot tell the detailed reason why the model pays attention to it. The reason might be the color, texture, or other things that make the model see the fire truck's body. Thus, it is hard to evaluate if it is correct when a salience map shows the truck body. Another limitation is that the salience map only interprets a single image instead of a class. In contrast, our method can provide class-level interpretations. Those differences make the comparison between our method and the salience map not appropriate in a fair and reasonable setting.

Reviewer comment (Novelty):
The paper does not cite the latter paper, which provides a method for robustifying against spurious correlations.

Author response:
We have cited all the mentioned papers in the revision paper, and we added a new section in Appendix H to illustrate the difference between our method and the others. We believe reading those tables in Appendix H can help better understand the difference. For the task setting in Wilds, we discuss the difference in the common reply about DISC. Below, we use the salience-map-based method as an example to show its limitations and why we cannot conduct the comparison. 
We have done thorough research on the prior work and those discussions about the salience map can be also found in related work.

Reviewer comment (Novelty):
The absence of these citations provides strong evidence that the paper is written without a thorough research of the prior work.

Author response:
We have cited all the mentioned papers in the revision paper, and we added a new section in Appendix H to illustrate the difference between our method and the others. We believe reading those tables in Appendix H can help better understand the difference. For the task setting in Wilds, we discuss the difference in the common reply about DISC. Below, we use the salience-map-based method as an example to show its limitations and why we cannot conduct the comparison. 
We have done thorough research on the prior work and those discussions about the salience map can be also found in related work.

Reviewer comment (Experiments):
**Too many moving parts**: While the scheme presented is modular. As it stands, there are several decisions that need to be gotten right for the overall scheme to work. Here is what I mean: 1) it looks like the traditional concept vectors (derived from model embeddings) are ineffective, so we need a modified version, 2) One needs a background word, 3) One needs to train a linear classifier to estimate correlations, 4) One needs to train two separate linear classifiers again to do editing for each spurious concept that you want to remove. This means that if you have 20 concepts to remove, then you would be training 40 linear classifiers to remove the effect of these 20 spurious concepts for that label alone. If any of the steps that I have listed does not work, then the entire scheme does not work.

Author response:
**Question in weakness: If we have 20 concepts to remove**

For this case, you can still train 2 white-box models, one contains those 20 concepts and one does not. Thus, you don't need to train 40 models.

Reviewer comment (Experiments):
**Over reliance on CLIP**: I think the dependence on CLIP in this work is quite worrisome. I think the CLIP embeddings are effective probably because the CLIP dataset is quite large, so those embeddings don't suffer from the issues the authors noticed. For example, imagine that you wanted to now fix a model that solely relies on CLIP embeddings as its classifier, then I assume the approach here would be ineffective?

Author response:
**Q2: Reliance on CLIP**

Comparing the equation at the end of page 4, the raw embedding replaces $t_{other}$ by a zero vector. We believe the zero vector is not as appropriate as the background vector since CLIP is not trained in that manner. In the case of satellite images where CLIP is not useful, we suggest using some related datasets or vision-language contrastive models to obtain meaningful concept vectors. So when CLIP does not work well, we just resort to other methods to obtain an off-the-shelf model. We want to emphasize again that in our paper we focus on how to extract concepts from off-the-shelf models.

Reviewer comment (Novelty):
How do you think the transfer editing approach here relates to the task vectors approach? See: Editing models with Task Arithmetic, and Task Arithmetic in the Tangent Space. It seems like you could avoid training two white-box models by adopting the task arithmetic editing approaches in the above papers.

Author response:
**Q1: Transfer editing vs Task Arithmetic**

One similarity between transfer editing and task arithmetic is that they both do some meaningful arithmetic combination. One major difference between transfer editing and task arithmetic is that transfer editing does arithmetic among different models while task arithmetic does arithmetic among different weights of the same model. We think task arithmetic also has a potential in model fixing, but it may have two drawbacks. First, we need extra datasets with annotated concepts. For example, to cure the model 'cat with keyboard', we need two datasets 'dog with keyboard' and 'dog without keyboard'. Second, we need access to the weights of black-box models. Those two requirements are drawbacks of task arithmetic since sometimes fine-tuning or re-training a black-box model can be costly, and accessing the model weights is prohibited for some API-based black-box models. However, transfer editing does not need any extra datasets, and as long as we can access the final output probabilities of black-box models, we can recover their logits without knowing their weights. Thus, from this point of view, transfer editing is very different from task arithmetic.

Reviewer comment (Theory):
What is the justification for why the concept vectors from raw embeddings does not work? What if I have a model that just uses the clip embedding itself for classification, but the clip embedding has spurious correlations too? Is this approach just inheriting the limitations of clip representations? What about if I have satellite images or a setting where CLIP is not useful?

Author response:
**Q2: Reliance on CLIP**

Comparing the equation at the end of page 4, the raw embedding replaces $t_{other}$ by a zero vector. We believe the zero vector is not as appropriate as the background vector since CLIP is not trained in that manner. In the case of satellite images where CLIP is not useful, we suggest using some related datasets or vision-language contrastive models to obtain meaningful concept vectors. So when CLIP does not work well, we just resort to other methods to obtain an off-the-shelf model. We want to emphasize again that in our paper we focus on how to extract concepts from off-the-shelf models.

Reviewer comment (Experiments):
Ensembling: Did you test ensembling alone in Table 3? It would be interesting to see whether simply ensembling recovers the performance gains that you see in that table. I ask this because ensembling has been shown to give OOD benefits.

Author response:
**Q3: More ensembling results**

Yes, you can find those results in Table 22 and Table 23 in Appendix.