Paper: Estimation of Concept Explanations Should be Uncertainty Aware (ICLR-2024)

Reviewer comment (Experiments):
The usage of CLIP seems essential, yet it is not involved in the synthetic experiments.

Author response:
Sorry for any confusion. We would like to point out that U-ACE uses CLIP to encode concepts in all the experiments including synthetic. As described in Section 3.1 and Proposition 1, all our experiments, including synthetic experiments, used text descriptions to encode concepts using the multi-modal CLIP model.

Reviewer comment (Writing):
TYPO: Shouldn't it be $\mathcal{D}_c^{(k)}$ in the expected value instead of $\mathcal{D}_c^{(k)}$?

Author response:
Sorry, looks like your question has a typing mistake, we believe you may be asking $\mathcal{D}_c^{(k)}$ instead of $\mathcal{D}$. If so, we do not see any typo on that line in Algorithm 1. We estimate the expectation over the probe-dataset ($\mathcal{D}$) and not over the concept dataset ($\mathcal{D}_c^{(k)}$). U-ACE assumes $\mathcal{D}_c^{(k)}$ is not available (please see inputs of U-ACE shown in Algorithm 1). $\mathcal{D}_c^{(k)}$ were introduced only to explain how previous concept explanation methods such as TCAV estimate concept activation vectors.

Reviewer comment (Experiments):
The limitation subsection highlights that such models are very dependent on the probe-dataset, is U-ACE robust to that issue?

Author response:
Yes, we demonstrated the relative robustness of U-ACE to probe-dataset in Table 2 (“Dataset shift” of Section 4.3) and Figure 3 (“Unreliability due to dataset shift.” in Section 4.1).

Reviewer comment (Theory):
$W_c \epsilon = 0$ means that either $W_c=0$ or $\epsilon$ is in the kernel of $W_c$. The second case is even more likely than in the previous point. If you don't justify that, you can't unroll your sequence of implications.

Author response:
Sorry, there could be some misunderstanding. We did not assume anything about the kernel or the matrix. We estimated $W_c$ that keeps the value of $W_c\epsilon$ low while also approximating the logits well (please see Alg. 1). U-ACE estimated $W_c=0$ only when all the concepts are irrelevant (please see Appendix F.1 and Proposition 3 for an example). In every other case, the estimated $W_c\neq 0$, which is why we see meaningful explanations such as in Figure 6.

Reviewer comment (Reproducibility):
Could you be more precise regarding the order of magnitude of $\lambda$? Should it be large or small?

Author response:
If $\lambda$ is very low the constraint $W_c\epsilon\approx 0$ is satisfied but the approximation error can be large. Whereas if the $\lambda$ is very high then the unconstrained regression may lead to better approximation but model is not unaware of uncertainty, i.e. $\|W_c\epsilon\|$ is likely large. 
$\lambda$ therefore must be neither very small nor very large. We found the best value of $\lambda$ through maximum likelihood estimation (MLE) on training data as described in Appendix B. Therefore the best value of $\lambda$ varied per dataset and was set automatically by MLE.

Reviewer comment (Theory):
"prior/posterior _on_ weights": the preposition 'on' is not very clear. In any case, could you give more details about how you obtained Equation 1?

Author response:
By prior/posterior on weights, we meant $\Pr(\vec{w})$ and $\Pr(\vec{w}\mid \mathcal{D})$ (where $\vec{w}$ according to the notation on page 3 is a row of $W_c$). Please see Slide 10 of https://www.utstat.toronto.edu/~rsalakhu/sta4273/notes/Lecture2.pdf#page=10 for how Equation (1) is derived, note that $m_0=0, S_0^{-1}=\lambda^{-1}\text{diag}(\epsilon\epsilon^T)$, $\Phi=C_X$, Y as defined above Equation 1. We also added the full derivation to Appendix A.1 for the sake of completeness.

Reviewer comment (Writing):
What is "cos-sim"?

Author response:
Thanks for pointing it out, we made it more explicit. We used “cos-sim” as a short form for cosine similarity (https://en.wikipedia.org/wiki/Cosine_similarity).

Reviewer comment (Writing):
What exactly is $\alpha_k$?

Author response:
The concept specific value $\alpha_k$ is a measure of uncertainty due to lack of knowledge  and due to ambiguity. We added text to Section 3.1 to make the definition clear.

Reviewer comment (Experiments):
How big is $N$? It is usually pretty large, so how do you optimize something with let's say 10K dimensions?

Author response:
N is the number of examples in the probe-dataset and can be very large. But in our experiments, the maximum value of N was only about 10,000 (on the Broden dataset). We pre-computed $e(w_k, g, \mathcal{D})$ and computed $e(v, f, \mathcal{D})$ at every step, and fully loaded both the vectors of dimension N into memory. Thanks for raising this fair point, we may need some special handling for very very large N. We will comment on the same in the revised version.

Reviewer comment (Writing):
What is $\theta_k$?

Author response:
$cos(\theta_k)$ as defined in Proposition 1 is the cosine-similarity between the embedding of the image (x) and the embedding of the concept k ($T_k$).

Reviewer comment (Theory):
The proof relies heavily on the statement: "If the examples in D are diversely distributed without any systematic bias, $A^T A$ is proportional to the identity matrix, meaning the basis of G and W are effectively the same." Can you justify or prove that? Without the point, the proof is not valid, hence the rest neither.

Author response:
There may have been some confusion. $A^TA$ is of size $S\times S$ where S is the rank of G. For $A^TA$ to be proportional to an identity matrix, we only require that the examples in the probe-dataset are such that $[g(x_1), g(x_2), …]$ are uniformly spread over the subspace of dimension S. We demonstrate this with an example.
```
def makeA(N, S):
	return np.array([[np.random.uniform(-1, 1) for s in range(S)] for n in range(N)])
def makeA2(N, S):
	return np.array([[np.random.uniform(-s-1, s+1) for s in range(S)] for n in range(N)])
>>> A = makeA(1000, 2)
>>> A.transpose() @ A
[[334.4,   8.6],
[  8.6, 338.7]])
>>> A = makeA2(1000, 2)
>>> A.transpose() @ A
[[ 341.4,   24.3],
 [  24.3, 1277.0]]
```

For the second matrix from makeA2, the first dimension is sampled from Unif[-1, 1] while the second from Unif[-2, 2]. Since all the dimensions are not identically distributed, it has systematic biases which prevented $A^TA$ (of makeA2) from being proportional to a diagonal matrix. 

We understand that our assumption is not guaranteed to hold since it depends on the diversity of the probe-dataset. Yet, the estimates were found to be empirically very effective as observed implicitly from the high quality of explanations. Moreover, direct evaluation of uncertainty, presented in Appendix G.1, further confirm the empirical effectiveness of U-ACE.

Reviewer comment (Theory):
How do you go from $cos(w, g(x)) = cos(\theta)cos(\alpha_k ) ± sin(\theta)sin(\alpha_k )$ to $m(x)=cos(\theta)cos(\alpha_k )$ and $s(x)=sin(\theta)sin(\alpha_k )$? How do you know which summant is $m$ and the other is $s$?

Author response:
By definition of m(x), s(x), concept activations lie in the range of $m(x)\pm s(x)$. The proof of Proposition 1 finds the range of concept activations as $cos(\theta)cos(\alpha_k)\pm sin(\theta)sin(\alpha_k)$. The result (of $m(x)=cos(\theta)cos(\alpha_k), s(x)=sin(\theta)sin(\alpha_k)$) follows from directly reading off from the estimated range.

Reviewer comment (Reproducibility):
If $Y$ is computed only for label $y$ of the for-loop, it should be indicated.

Author response:
Thanks for noticing. We corrected it in the latest draft.

Reviewer comment (Writing):
What is $u_k$ and $\sigma_k$?

Author response:
They are the mean and scale respectively of the normal distribution from which the concept activation vectors (w_k) are sampled.

Reviewer comment (Writing):
What is a _vanilla_ linear estimator?

Author response:
We meant ordinary least squares (OLS) by vanilla linear estimator. We corrected the phrase in the latest draft to avoid ambiguity.

Reviewer comment (Theory):
"the probability that at least of the K-1 random concepts is estimated to be more important than the relevant concept is $1-\prod \Phi( \dfrac{||u_k||}{\sigma_k ||w||} )$". How do you get this formula?

Author response:
We modified the text to introduce Corollary 1 and presented the derivation in the proof of Corollary 1. Please see the revised draft.

Reviewer comment (Writing):
"the CDF of standard normal", some words are missing here.

Author response:
We meant $\Phi$ is the Cumulative Distribution Function of standard normal distribution.

Reviewer comment (Reproducibility):
"pretrained CLIP model that is publicly available for download." Can you provide a link or reference?

Author response:
Sure, https://github.com/openai/CLIP/. We also added it in the latest draft.

Reviewer comment (Reproducibility):
I understood from Section 3, that U-ACE is built upon a text embedding. How is it learned here?

Author response:
U-ACE uses text description of a concept to encode a concept through a pretrained multi-modal model such as CLIP. But U-ACE itself does not train any text embeddings.

Reviewer comment (Presentation):
In general, the most significant weakness of this paper was presentation and clarity. The results are quite interesting and relevant to work on concept based explanations. Nevertheless, the paper was quite hard to follow in places and could do with a bit of work to more clearly express the main takeaways and experimental evaluations. In the current form, the paper is quite hard to follow.

Author response:
We thank the reviewer for their suggestions. We have rewritten the hard-to-parse sections, filled in missing details and presented all the derivations in the Appendix (Please see general response). We also acted on the suggestion made by the reviewer for Section 4.1.

Reviewer comment (Presentation):
The presentation of the experimental results could be improved. In 4.1, the "Unreliability due to misspecified concept set" subsection feels a bit odd in the flow of the paper, because it serves as a pointer to the appendix, perhaps it would be better if this is incorporated into another paragraph to keep the flow of the paper clearer.

Author response:
We thank the reviewer for their suggestions. We have rewritten the hard-to-parse sections, filled in missing details and presented all the derivations in the Appendix (Please see general response). We also acted on the suggestion made by the reviewer for Section 4.1.

Reviewer comment (Novelty):
Very Important: This paper attempts to address two topics at once, and its novelty is severely undercut by existing work in both directions. At least, I say it attempts to address two topics because that is how it is pitched and evaluated, although the technical novelty of the paper is to incorporate uncertainty into concept estimation (and it only uses existing approaches to *unsupervised* concept estimation). Anyway, the paper comes on the heels of a number of works on unsupervised concept detection, including https://openreview.net/pdf?id=iOlYmD1PtC8, https://arxiv.org/pdf/2304.09707.pdf, and https://arxiv.org/pdf/2309.08600.pdf. I believe all of these works appeared within the last six months, so they are concurrent work. However, perhaps more importantly, the unsupervised concept detection in this paper uses strictly existing methods based on CLIP, so there is not novelty on the supervision side. On the concept estimation side, there is older work on the subject, e.g. this 2021 paper https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf, in addition to the concurrent 2023 citation in the submission (Probabilistic Concept Bottleneck Models). The 2021 Slack et al paper focuses on linear feature attribution wrt input features rather than concept vectors, but rest of the subject is the same and this paper also introduces a Bayesian model for handling uncertainty in the attribution. Or if this problem setting is not too similar to the one in this paper, what is the difficulty in uncertainty for the vector v_k in Sec. 3.1, which is the concept vector, using any number of standard uncertainty measurements (whether frequentist or Bayesian)?

Author response:
Sorry, there may have been some misinterpretation. We would like to clarify that U-ACE assumes that the concepts are supplied and we did not propose a method for concept discovery (although it is an important problem in itself). As explained in Section 2, the list of concepts $\mathcal{C}=\{c_1, c_2, \dots, c_K\}$ are supplied along with their text descriptions: $\{T_1, T_2,\dots, T_K\}$.  

> On the concept estimation side, there is older work… what is the difficulty in uncertainty for the vector v_k in Sec. 3.1, which is the concept vector, using any number of standard uncertainty measurements (whether frequentist or Bayesian)?

Thanks for raising this issue.  To further motivate the need and effectiveness of uncertainty estimation of U-ACE (described in Section 3.1), we contrasted U-ACE with uncertainty estimated using MC sampling or distributional parameter fitting (similar to ProbCBM as the reviewer suggested). The results can be found in Appendix G.1 of the revised draft. We find that U-ACE estimated uncertainty is high-quality as well as compute-efficient. Thanks for pointing to Slack et.al. work, although their methods are not trivially applicable to our problem we will mention the paper in the related work and highlight the common intent.

Reviewer comment (Theory):
Important: The prior for the regression model is not well motivated. First, since the regression weights are filtered with a threshold for zeroing small weights, it’s clear that a spike-and-slab prior or another sparsity-encouraging prior would have been more appropriate. But the paper aims to encourage weights to be orthogonal to a noise vector. What is this noise vector? It’s a *mean confidence interval vector* judging by Fig. 1 which shows activations as m(x) +/- s(x). This implies that the weights are supposed to be orthogonal to the upper bound of a per-data-point confidence interval, averaged across the data. Maybe I’m misunderstanding something, but at the moment, I have no idea why this would be a good prior for the model.

Author response:
> … sparsity-encouraging prior would have been more appropriate. But the paper aims to encourage weights to be orthogonal to a noise vector. What is this noise vector? … I have no idea why this would be a good prior for the model.

Sorry for any confusion. We have two motives for the linear estimator $W_c$ (which is returned as the explanation): (1) should be robust to error in concept activations, or in other words, the linear estimator should not rely on concepts with large noise/uncertainty (3rd para of Section 3), (2) should be sparse for easy interpretation. We enforce (1) by imposing a prior such that the samples are almost orthogonal to the noise vector. Please see Equation 1 and text above it for more details about the noise vector ($\epsilon$: a vector of size K: the number of concepts) and justification on why it is a good prior for enforcing the requirement (1) above. Our prior is not designed to enforce sparsity (2); we sparsify weights post-hoc as explained towards the end of Section 3 and before Section 3.1. Further in Appendix G.2, we evaluated the role of prior in producing quality explanations.

Reviewer comment (Experiments):
Important: Fair comparison with baselines: How was Y-CBM tuned, and what exactly is the final different with U-ACE? Is it just the prior? Is the technical contribution here mainly the prior? How would an L1 or another sparsity inducing prior affect Y-CBM performance in Table 1?

Author response:
> Fair comparison with baselines: How was Y-CBM tuned, and what exactly is the final different with U-ACE? … Is the technical contribution here mainly the prior? How would an L1 or another sparsity inducing prior affect Y-CBM performance in Table 1?


Major difference between Y-CBM and U-ACE is that U-ACE estimates the linear weights using Bayesian regression with prior as described in Section 3 while Y-CBM estimates the same using lasso regression. Since the prior on U-ACE’s linear weights is a function of uncertainty in concept activations, it is aware of uncertainty. 

Estimation of explanations using Bayesian regression with an uncertainty-aware prior is one the crucial contributions of the paper. We wish to highlight that the contribution of our work also lies in demonstrating the role of uncertainty in concept explanations, and in estimation of uncertainty in concept activations.  

To address your concerns more concretely, we did the following: (a) Appendix G.3 presents results for Y-CBM and O-CBM for varying values of the regularization strength, and still perform worse than U-ACE (without any hyperparam tuning), (b) Appendix G.2 contains comparison with Bayesian regression without the uncertainty aware prior (of U-ACE). The experiments paint a clear importance of the U-ACE’s prior.

Reviewer comment (Experiments):
Of some importance: It doesn’t make sense to ask what the importance of green is when there are no green images in the probe set (Fig3 middle) or what the importance of red is when the images are all green. Just because CLIP registration enables one to ask this question does not make it meaningful, and a practicioner should notice that their probe set contains no example of green before trying to estimate the importance of green. The same applies for the “fruit” example. The quality of concept estimation is heavily bottlenecked by probe set quality, as noted in this paper, but it has to be the practitioner’s responsibility at some point to check the quality and diversity of the probe set before trying to estimate concept importance for concepts that do not appear at all in the set.

Author response:
> …It doesn’t make sense to ask what the importance of green is when there are no green images in the probe set (Fig3 middle) or what the importance of red is when the images are all green. … a practicioner should notice that their probe set contains no example of green before …. 

Thanks for your question. We agree that choosing probe-datasets carefully can solve the problem partially as was also suggested by Ramaswamy et.al.: “The choice of the probe dataset has a profound impact on explanations…. Hence, probe datasets should be chosen with caution.” However, when the number of labels, concepts and examples in the probe dataset are in the order of thousands, manually checking for presence of each concept with each label over all the examples can be impractical. Which is why we believe (among other reasons) that a concept explanation must be equipped with confidence scores on their estimate of importance, which insulates from misinterpreting importance scores of missing concepts. Our simulated study with the colored dataset is inspired by the difficulty in carefully picking the probe-dataset or concept set when operating at scale.

Reviewer comment (Theory):
If the method uses the last hidden layer, isn’t the method basically replacing the final hidden state with an estimated concept feature vector and replacing the final linear layer with a learned linear layer that tries to approximate the original model’s outputs? This is odd because if the concept vectors are directions in the model latent space, then one would think that a concept attribution explanation is explainable merely by computing the projection of the model’s linear layer with the concept vector, and there is no need to recreate a feature vector m(x) for each datapoint (which uses the learned concept vector v only in small part) and learn a new linear model to estimate concept importance.

Author response:
> …because if the concept vectors are directions in the model latent space, then one would think that a concept attribution explanation is explainable merely by computing the projection of the model’s linear layer with the concept vector… learn a new linear model to estimate concept importance.

True that “then one would think that a concept attribution explanation is explainable merely by computing the projection of the model’s linear layer with the concept vector”, but such a point estimate is bound to be noisy. m(x) is simply the mean over many such projections (using several samples of concept activation vector).

Reviewer comment (Evaluation):
So what is the source of uncertainty that U-ACE accounts for? Data uncertainty? Model uncertainty for the deep learned model? If it’s data uncertainity, why can’t TCAV be equipped with a simple data uncertainty measurement? If the TCAV concept attribution score is a binary proportion, then a binomial probability confidence interval would be very easy to use for it, and much more intuitive than s(x).

Author response:
> So what is the source of uncertainty that U-ACE accounts for?... why can’t TCAV be equipped with a simple data uncertainty measurement? …

We answered the sources of uncertainty in our response above.    

TCAV is already equipped with a simple uncertainty measurement to distinguish a truly important concept from a random concept. TCAV computes m(x) and s(x) of concept activations by simply training multiple concept activation vectors. Yet, TCAV estimated explanations are noisy as seen in Table 1 likely because simple measurement of uncertainty through MC sampling may fail to capture epistemic uncertainty as can be observed from the results of Appendix G.1.

Reviewer comment (Evaluation):
“Simple estimates explanation using concept annotations and therefore its explanation must be the closest to the ground-truth” — What would the ground-truth look like for the experiment in 4.3?

Author response:
> “Simple estimates explanation using concept annotations and therefore its explanation must be the closest to the ground-truth” — What would the ground-truth look like for the experiment in 4.3?

We noticed that concept annotations do not mark all the concepts present in an example, which is understandable given the number of concepts: 720. Since “Simple” is trained on ground-truth concept annotations, it may miss attributing importance to the concepts that are systematically missed by an annotator. We expect therefore that the most salient concepts identified by “Simple” are a subset of truly salient concepts. Nevertheless, “Simple” trained on concept annotations is the closest explanation to ground-truth, which we use to evaluate other methods.

Reviewer comment (Writing):
typo: UNCERTAINITY

Author response:
> typo: UNCERTAINITY
> typo: while other methods the importance

Thank you, we fixed these in the revision.

Reviewer comment (Writing):
typo: while other methods the importance

Author response:
> typo: UNCERTAINITY
> typo: while other methods the importance

Thank you, we fixed these in the revision.

Reviewer comment (Evaluation):
“Since the co-occurrence probability of U with car class goes from 1, 0.5 to 0, we expect the importance score of U should change from positive to negative as we move right” — It doesn’t change to negative in the graph, although it does get lower

Author response:
> “Since the co-occurrence probability of U with car class goes from 1, 0.5 to 0, we expect the importance score of U should change from positive to negative as we move right” — It doesn’t change to negative in the graph, although it does get lower

Thanks for bringing it up. The importance score does go negative but since we visualized in Figure 5 the rank-normalised concept importance scores (as explained on page 6), it can only be positive. We fixed the text accordingly.

Reviewer comment (Experiments):
**Why focus on the post-hoc CBM setting?**: I understand the justification that it is difficult to get annotations for all of the training set. This is true and a known limitation, but I think the authors missed a chance to at least demonstrate the importance of the uncertainty estimation part of their work. Right now, I think the authors choose the model difficult setting, the post-hoc CBM. One way to show the effectiveness of this scheme is to first demonstrate in the setting where you have all training set annotations. Although I understand that when you use a small concept set to convert a black-box to CBM model, the challenges of dataset shift and incomplete concept set are exacerbated. For me, it would have been easier to digest the method independent of post-hoc CBMs. This is more of a suggestion.

Author response:
> Why focus on the post-hoc CBM setting?

We request the reviewer to please rephrase or elaborate? We did not understand the concern. We have concept annotation on full training data for all our settings. However, only TCAV and Simple have access to them. Y-CBM, O-CBM and U-ACE did not use the available concept annotations in any way. 

Regarding "chance to at least demonstrate the importance of the uncertainty estimation part of their work.",
Appendix G.1 of the revised draft demonstrates the quality of uncertainty estimated by U-ACE.

Reviewer comment (Theory):
**Implications in Section 3**: Here the authors make a series of implications to arrive at distribution of weight vectors. I don't see how that holds clearly, especially the second implication. Why does a high probability bound on the dot product of the weight and noise imply that? What is the distributional assumption on the noise? Also, can the authors explain how they arrive at eqn 1? It looks like a gaussian identity, but I want to clarify.

Author response:
> Implications in Section 3: … Why does a high probability bound on the dot product of the weight and noise imply that? What is the distributional assumption on the noise? Also, can the authors explain how they arrive at eqn 1? 

Thanks for pointing out these issues. We added more detail in the main paper, added detailed derivations in the appendix to improve the readability of Section 3. We are happy to resolve any unresolved concerns.  

We did not make distributional assumption on the noise and fully derived the Eqn (1) in Appendix A.1.

Reviewer comment (Experiments):
**Sparsifying the weights**: Instead of picking the threshold by hand, why can't the authors impose a lasso penalty here as is done for the simple baseline?

Author response:
> Sparsifying the weights: Instead of picking the threshold by hand, why can't the authors impose a lasso penalty here as is done for the simple baseline?

Depending on the choice of prior, we may not get a clean analytical form for the posterior on the weights as shown in Equation (1). Although our post-hoc sparsification of weights is not fully Bayesian and introduced an additional hyperparam, it worked satisfactorily for all our experiments.

Reviewer comment (Writing):
**Section 3.1 is confusing**: Until here, there was nothing about multimodal functions/CLIP in the paper. But all of a sudden, it is incorporated to compute the mean and error function. I think you could do away with the CLIP discussion in that section and just call CLIP/Multimodal function an embedding function. Based on that description, I still don't understand how the mean and errors are computed. Proposition 1 discusses this, but why is this? It seems like this section adopts the notation of Oikarinen et al. (2023). I would've thought that the procedure here can give an error estimate for any generic function that of the input to the concept activation? Is there a way to abstract away the specific form of the concept activation of Oikarinen et al. (2023) in this formulation? Stated differently, what is $\vec{m}(x)$, and $\vec{s}(x)$ for a standard CBM?

Author response:
> Section 3.1 is confusing: … just call CLIP/Multimodal function an embedding function. 

We agree. We have revised the draft to emphasize the generality of our method to any multi-modal model. 

>…I … don't understand how the mean and errors are computed….

We greatly appreciate your concern. In Section 3.1 of the revised draft, we have added intuition and filled in the missing details. Could you please take a look and let us know? 

> Is there a way to abstract away the specific form of the concept activation of Oikarinen et al. (2023) in this formulation?  

Thanks for the great observation, we agree. The role of Oikarinen et al. (2023) is only to inform us about how much information about the kth concept is encoded in the representation layer of the model-to-be-explained, which we denote by $cos(\alpha_k)$. We can use any method for estimation of $v_k$, we just used Oikarinen et al. (2023). In the revised draft, we changed the writing to abstract out specific form of estimation: Oikarinen et al. (2023)

Reviewer comment (Experiments):
**The problem with CLIP**: The formulation here ties in with CLIP intimately. It is clear that if you want to use something like this for say proteins, sequences, or other modalities, then CLIP will not help you. You'd want the equivalent of CLIP for the domain you are working with.

Author response:
> The problem with CLIP: The formulation here ties in with CLIP intimately. …

As the reviewer pointed out in their previous concern. U-ACE has very little to do with how and where the image and text embeddings: $g, g_{text}$ come from. Although our evaluation setup used CLIP, any other pretrained multi-modal model can be plugged-in very easily.  We understand that CLIP is not suitable in biomedical applications, but we can simply replace CLIP with an appropriately pretrained multi-modal model.

Reviewer comment (Writing):
The text is definitely too dense in parts (especially Section 3). Some paragraphs feel rushed and would enjoy a rewrite. There are also some typos (I mentioned a few below).

Author response:
Thanks for pointing out these issues, all the minor issues are now resolved in the revised draft. We have also rewritten parts of Section 2, 3 to improve clarity and readability.

Reviewer comment (Theory):
My understanding is that the implementation of m(x) and s(x) is model specific -- that is, the equation explains how to derive these quantities for image-text multimodal systems. So, while the general setup is model-agnostic, the specific algorithm is likely not.

Author response:
Our implementation is not specific to any model. We only assume access to the representation layer and the logits of the model-to-be-explained. We did not make any specific assumptions about the multi-modal model. CLIP can be easily replaced with any other pretrained multi-modal model in a plug-and-play manner. 

A pretrained multi-modal model defines g, $g_{text}$ of Section 3.1 and any new model-to-be-explained must define ‘f’ (a mapping from the input to the representation and logit). The rest of the details must follow for computation of m(x), s(x) as described in Proposition 1. 

We have revised the draft to emphasize the generality of our proposal to any (neural-network-based) model and any multi-modal model. 

Please let us know if we have misinterpreted your question or you found our response not convincing.

Reviewer comment (Theory):
p 3: It wouldn't hurt to clarify the steps in the mathematical derivation. Also, why is s(x) being averaged over? Why can't it be used as-is? If $\beta$ is the inverse variance of noise in observations, why is it being optimized? Section 3 should be unpacked to facilitate understanding. Right now it is unnecessarily opaque.

Author response:
We did it for the ease of analysis. We get a clean closed form if we average over s(x).  

Thanks for raising this issue. We added the following passage to the paper. We could directly set the inverse of $\beta$ approximately 0 since there is no noise on the observations: Y. Instead of setting $\beta$ to an arbitrary large value, we observed better explanations when we allowed the tuning algorithm to find a value of $\beta, \lambda$ to balance the evidence and noise.

Thank you very much for pointing out the problems. We have rewritten Section 3 to improve its readability.