Original: C ONTRASTIVE C ORPUS A TTRIBUTION FOR E XPLAINING R EPRESENTATIONS
Revised: C ONTRASTIVE C ORPUS A TTRIBUTION FOR E XPLAINING R EPRESENTATIONS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Despite the widespread use of unsupervised models, very few methods are designed to explain them.
Revised: Despite the widespread use of unsupervised models, very few methods are designed to explain them.

Original: Most explanation methods explain a scalar model output.
Revised: Most explanation methods explain a scalar model output.

Original: However, unsupervised models output representation vectors, the elements of which are not good candidates to explain because they lack semantic meaning.
Revised: However, unsupervised models output representation vectors, the elements of which are not good candidates to explain because they lack semantic meaning.

Original: To bridge this gap, recent works defined a scalar explanation output: a dot product-based similarity in the representation space to the sample being explained (i.e., an explicand).
Revised: To bridge this gap, recent works defined a scalar explanation output: a dot product-based similarity in the representation space to the sample being explained (i.e., an explicand).

Original: Although this enabled explanations of unsupervised models, the interpretation of this approach can still be opaque because similarity to the explicand’s representation may not be meaningful to humans.
Revised: Although this enabled explanations of unsupervised models, the interpretation of this approach can still be opaque because similarity to the explicand’s representation may not be meaningful to humans.

Original: To address this, we propose contrastive corpus similarity , a novel and semantically meaningful scalar explanation output based on a reference corpus and a contrasting foil set of samples.
Revised: To address this, we propose contrastive corpus similarity , a novel and semantically meaningful scalar explanation output based on a reference corpus and a contrasting foil set of samples.

Original: We demonstrate that contrastive corpus similarity is compatible with many post-hoc feature attribution methods to generate COntrastive COrpus Attributions (COCOA) and quantitatively verify that features important to the corpus are identified.
Revised: We demonstrate that contrastive corpus similarity is compatible with many post-hoc feature attribution methods to generate COntrastive COrpus Attributions (COCOA) and quantitatively verify that features important to the corpus are identified.

Original: We showcase the utility of COCOA in two ways: (i) we draw insights by explaining augmentations of the same image in a contrastive learning setting (SimCLR); and (ii) we perform zero-shot object localization by explaining the similarity of image representations to jointly learned text representations (CLIP).
Revised: We showcase the utility of COCOA in two ways: (i) we draw insights by explaining augmentations of the same image in a contrastive learning setting (SimCLR); and (ii) we perform zero-shot object localization by explaining the similarity of image representations to jointly learned text representations (CLIP).

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Machine learning models based on deep neural networks are increasingly used in a diverse set of tasks including chess (Silver et al., 2018), protein folding (Jumper et al., 2021), and language translation (Jean et al., 2014).
Revised: Machine learning models based on deep neural networks are increasingly used in a diverse set of tasks including chess (Silver et al., 2018), protein folding (Jumper et al., 2021), and language translation (Jean et al., 2014).

Original: The majority of neural networks have many parameters, which impedes humans from understanding them (Lipton, 2018).
Revised: The majority of neural networks have many parameters, which impede humans from understanding them (Lipton, 2018).

Original: To address this, many tools have been developed to understand supervised models in terms of their prediction (Lundberg & Lee, 2017; Wachter et al., 2017).
Revised: To address this, many tools have been developed to understand supervised models in terms of their prediction (Lundberg & Lee, 2017; Wachter et al., 2017).

Original: In this supervised setting, the model maps features to labels ( f : X → Y ), and explanations aim to understand the model’s prediction of a label of interest.
Revised: In this supervised setting, the model maps features to labels ( f : X → Y ), and explanations aim to understand the model’s prediction of a label of interest.

Original: These explanations are interpretable, because the label of interest (e.g., mortality, an image class) is meaningful to humans (Figure 1a).
Revised: These explanations are interpretable, because the label of interest (e.g., mortality, an image class) is meaningful to humans (Figure 1a).

Original: In contrast, models trained in unsupervised settings map features to representations ( f : X → H ).
Revised: In contrast, models trained in unsupervised settings map features to representations ( f : X → H ).

Original: Existing supervised explanation methods can be applied to understand an individual element ( h i ) in the representation space, but such explanations are not useful to humans unless h i has a natural semantic meaning.
Revised: Existing supervised explanation methods can be applied to understand an individual element ( h i ) in the representation space, but such explanations are not useful to humans unless h i has a natural semantic meaning.

Original: Unfortunately, the meaning of individual elements in the representation space is unknown in general.
Revised: Unfortunately, the meaning of individual elements in the representation space is unknown in general.

Original: One possible solution is to enforce representations to have semantic meaning as in Koh et al.
Revised: One possible solution is to enforce representations to have semantic meaning as in Koh et al.

Original: but this approach requires concept labels for every single training sample, which is typically impractical.
Revised: but this approach requires concept labels for every single training sample, which is typically impractical.

Original: Another solution is to enforce learned representations to be disentangled as in Tran et al.
Revised: Another solution is to enforce learned representations to be disentangled as in Tran et al.

Original: and then manually identify semantically meaningful elements to explain, but this approach is not post-hoc and requires potentially undesirable modifications to the training process.
Revised: and then manually identify semantically meaningful elements to explain, but this approach is not post-hoc and requires potentially undesirable modifications to the training process.

Original: Related work.
Revised: Related work.

Original: Rather than explain a single element in the representation, approaches based on explaining the representation as a whole have recently been proposed, including RELAX (Wickstrøm et al., 2021) and label-free feature importance (Crabb´e & van der Schaar, 2022)
Revised: Rather than explain a single element in the representation, approaches based on explaining the representation as a whole have recently been proposed, including RELAX (Wickstrøm et al., 2021) and label-free feature importance (Crabb´e & van der Schaar, 2022)

Original: (Figure 1b)
Revised: (Figure 1b)

Original: (additional related work in Appendix A).
Revised: (additional related work in Appendix A).

Original: These approaches both aim to identify features in the explicand (the sample to explain) that, when removed, point the altered representation away from the explicand’s original representation.
Revised: These approaches both aim to identify features in the explicand (the sample to explain) that, when removed, point the altered representation away from the explicand’s original representation.

Original: Although RELAX and label-free feature importance successfully extend existing explanation techniques to an unsupervised setting, they have two major limitations.
Revised: Although RELAX and label-free feature importance successfully extend existing explanation techniques to an unsupervised setting, they have two major limitations.

Original: First, they only consider similarity to the explicand’s representation; however, there are a variety of other meaningful questions that can be asked by examining similarity to other samples’ representations.
Revised: First, they only consider similarity to the explicand’s representation; however, there are a variety of other meaningful questions that can be asked by examining similarity to other samples’ representations.

Original: Examples include asking, “Why is my explicand similar to dog images?”
Revised: Examples include asking, “Why is my explicand similar to dog images?”

Original: or “How is my rotation augmented image similar to my original image?”.
Revised: or “How is my rotation augmented image similar to my original image?”.

Original: Second, RELAX and label-free importance find features which increase similarity to the explicand in representation space from any direction, but in practice some of these directions may not be meaningful.
Revised: Second, RELAX and label-free importance find features which increase similarity to the explicand in representation space from any direction, but in practice some of these directions may not be meaningful.

Original: Instead, just as human perception often explains by comparing against a contrastive counterpart (i.e., foil ) (Kahneman & Miller, 1986; Lipton, 1990; Miller, 2019), we may wish to find features that move toward the explicand relative to an explicit “foil”.
Revised: Instead, just as human perception often explains by comparing against a contrastive counterpart (i.e., foil ) (Kahneman & Miller, 1986; Lipton, 1990; Miller, 2019), we may wish to find features that move toward the explicand relative to an explicit “foil”.

Original: As an example, RELAX and label-free importance may identify features which increase similarity to a dog explicand image relative to other dog images or even cat images; however, they may also identify features which increase similarity relative to noise in the representation space corresponding to unmeaningful out-of-distribution samples.
Revised: As an example, RELAX and label-free importance may identify features which increase similarity to a dog explicand image relative to other dog images or even cat images; however, they may also identify features which increase similarity relative to noise in the representation space corresponding to unmeaningful out-of-distribution samples.

Original: In contrast, we can use foil samples to ask specific questions such as, “What features increase similarity to my explicand relative to cat images?”.
Revised: In contrast, we can use foil samples to ask specific questions such as, “What features increase similarity to my explicand relative to cat images?”.

Original: Contribution.
Revised: Contribution.

Original: (1) To address the limitations of prior works on explaining unsupervised models, we introduce COntrastive COrpus Attribution (COCOA) , which allows users to choose corpus and foil samples in order to ask, “What features make my explicand’s representation similar to my corpus, but dissimilar to my foil?”
Revised: (1) To address the limitations of prior works on explaining unsupervised models, we introduce COntrastive COrpus Attribution (COCOA) , which allows users to choose corpus and foil samples in order to ask, “What features make my explicand’s representation similar to my corpus, but dissimilar to my foil?”

Original: (Figure 1c).
Revised: (Figure 1c).

Original: (2) We apply COCOA to representations learned by a self-supervised contrastive learning model and observe class-preserving features in image augmentations.
Revised: (2) We apply COCOA to representations learned by a self-supervised contrastive learning model and observe class-preserving features in image augmentations.

Original: (3) We perform object localization by explaining a mixed modality model with COCOA.
Revised: (3) We perform object localization by explaining a mixed modality model with COCOA.

Original: Motivation.
Revised: Motivation.

Original: Unsupervised models are prevalent and can learn effective representations for downstream classification tasks.
Revised: Unsupervised models are prevalent and can learn effective representations for downstream classification tasks.

Original: Notable examples include contrastive learning (Chen et al., 2020) and self-supervised learning (Grill et al., 2020).
Revised: Notable examples include contrastive learning (Chen et al., 2020) and self-supervised learning (Grill et al., 2020).

Original: Despite their widespread use and applicability, unsupervised models are largely opaque.
Revised: Despite their widespread use and applicability, unsupervised models are largely opaque.

Original: Explaining them can help researchers understand and therefore better develop and compare representation learning methods (Wickstrøm et al., 2021; Crabb ´ e & van der Schaar, 2022).
Revised: Explaining them can help researchers understand and therefore better develop and compare representation learning methods (Wickstrøm et al., 2021; Crabb´e & van der Schaar, 2022).

Original: In deployment, explanations can help users better monitor and debug these models (Bhatt et al., 2020).
Revised: In deployment, explanations can help users better monitor and debug these models (Bhatt et al., 2020).

Original: Moreover, COCOA is beneficial even in a supervised setting.
Revised: Moreover, COCOA is beneficial even in a supervised setting.

Original: Existing feature attribution methods only explain the classes the model has been trained to predict, so they can only explain classes which are fully labeled in the training set.
Revised: Existing feature attribution methods only explain the classes the model has been trained to predict, so they can only explain classes which are fully labeled in the training set.

Original: Instead, COCOA only requires a few class labels after the training process is complete, so it can be used more flexibly.
Revised: Instead, COCOA only requires a few class labels after the training process is complete, so it can be used more flexibly.

Original: For instance, if we train a supervised model on CIFAR-10, existing methods can only explain the ten classes the model was trained on.
Revised: For instance, if we train a supervised model on CIFAR-10 (Krizhevsky et al., 2009), existing methods can only explain the ten classes the model was trained on.

Original: Instead, we can collect new samples from an unseen class, and apply COCOA to a representation layer of the trained model to understand this new class.
Revised: Instead, we can collect new samples from an unseen class, and apply COCOA to a representation layer of the trained model to understand this new class.

Original:  N OTATION
Revised:  N OTATION

Original: We consider an arbitrary input space X and output space Z .
Revised: We consider an arbitrary input space X and output space Z .

Original: Given a model f : X → Z and an explicand x e ∈ X to be explained, local feature attribution methods assign a score to each input feature based on the feature’s importance to a scalar explanation target .
Revised: Given a model f : X → Z and an explicand x e ∈ X to be explained, local feature attribution methods assign a score to each input feature based on the feature’s importance to a scalar explanation target .

Original: A model’s intermediate or final output is usually not a scalar.
Revised: A model’s intermediate or final output is usually not a scalar.

Original: Hence, feature attribution methods require an explanation target function γ f, ∗ : X → R that transforms a model’s behavior into an explanation target.
Revised: Hence, feature attribution methods require an explanation target function γ f, ∗ : X → R that transforms a model’s behavior into an explanation target.

Original: The subscript f indicates that a model is considered a fixed parameter of an explanation target function, and ∗ denotes an arbitrary number of additional parameters.
Revised: The subscript f indicates that a model is considered a fixed parameter of an explanation target function, and ∗ denotes an arbitrary number of additional parameters.

Original: To make this concrete, consider the following example with a classifier f class : X → [0 , 1] C , where C is the number of classes.
Revised: To make this concrete, consider the following example with a classifier f class : X → [0 , 1] C , where C is the number of classes.

Original: Example 2.1.
Revised: Example 2.1.

Original: Given an explicand x e ∈ X , let the explanation target be the predicted probability of the explicand’s predicted class.
Revised: Given an explicand x e ∈ X , let the explanation target be the predicted probability of the explicand’s predicted class.

Original: Then the explanation target function is
Revised: Then the explanation target function is

Original: [Equation]
Revised: [Equation]

Original: e ) ( x ) , for all x ∈ X , where f classi ( · ) denotes the i th element of f class ( · ) .
Revised: for all x ∈ X , where f classi ( · ) denotes the i th element of f class ( · ) .

Original: Here, the explanation target function has the additional subscript x e to indicate that the explicand is a fixed parameter.
Revised: Here, the explanation target function has the additional subscript x e to indicate that the explicand is a fixed parameter.

Original: Let R X denote the set of functions that map from X to R .
Revised: Let R X denote the set of functions that map from X to R .

Original: Formally, a local feature attribution method ϕ : R X ×X → R |X| takes an explanation target function γ f, ∗ and an explicand x e as inputs.
Revised: Formally, a local feature attribution method ϕ : R X ×X → R |X| takes an explanation target function γ f, ∗ and an explicand x e as inputs.

Original: The attribution method returns ϕ ( γ f, ∗ , x e ) ∈ R |X| as feature importance scores.
Revised: The attribution method returns ϕ ( γ f, ∗ , x e ) ∈ R |X| as feature importance scores.

Original: For k = 1 , ..., |X| , let x ek denote the k th feature of x e and ϕ k ( γ f, ∗ , x e ) the corresponding feature importance of x ek .
Revised: For k = 1 , ..., |X| , let x ek denote the k th feature of x e and ϕ k ( γ f, ∗ , x e ) the corresponding feature importance of x ek .

Original: We demonstrate this definition of local feature attribution methods with the following example.
Revised: We demonstrate this definition of local feature attribution methods with the following example.

Original: Example 2.2.
Revised: Example 2.2.

Original: Given an explicand x e , the above classifier f class , and the explanation target function in Example 2.1 , consider Vanilla Gradients (Simonyan et al., 2013) as the local feature attribution method.
Revised: Given an explicand x e , the above classifier f class , and the explanation target function in Example 2.1 , consider Vanilla Gradients (Simonyan et al., 2013) as the local feature attribution method.

Original: The feature importance scores of x e are computed as
Revised: The feature importance scores of x e are computed as

Original: [Equation]

Original: [Figure]
Revised: [Figure]

Original: g., abnormal cases).
Revised: feature importance) for an unsupervised model identifies important features that, when removed, make the representation dissimilar to the explicand’s original representation. (c) Contrastive corpus attribution for an unsupervised model identifies important features that, when removed, make the representation similar to a foil set (e.g., normal cases) and dissimilar to a corpus set (e.g., abnormal cases).

Original:  A PPROACH
Revised:  A PPROACH

Original: In this section, we motivate and propose a novel explanation target function for local feature attributions in the setting of explaining representations learned by an encoder model.
Revised: In this section, we motivate and propose a novel explanation target function for local feature attributions in the setting of explaining representations learned by an encoder model.

Original: We also describe how to combine the proposed explanation target function with existing attribution methods and how to interpret the obtained attribution scores.
Revised: We also describe how to combine the proposed explanation target function with existing attribution methods and how to interpret the obtained attribution scores.

Original: All proofs are in Appendix H.
Revised: All proofs are in Appendix H.

Original: 3. D EFINING CONTRASTIVE CORPUS SIMILARITY AS AN EXPLANATION TARGET FUNCTION
Revised: 3. D EFINING CONTRASTIVE CORPUS SIMILARITY AS AN EXPLANATION TARGET FUNCTION

Original: We first define the notion of similarity between representations.
Revised: We first define the notion of similarity between representations.

Original: Without loss of generality, suppose the representation space is R d for some integer d > 0 .
Revised: Without loss of generality, suppose the representation space is R d for some integer d > 0 .

Original: Definition 3.1 (Representation similarity) .
Revised: Definition 3.1 (Representation similarity) .

Original: Let f : X → R d be a representation encoder.
Revised: Let f : X → R d be a representation encoder.

Original: For all x , x ′ ∈ X , the representation similarity between x and x ′ based on f is
Revised: For all x , x ′ ∈ X , the representation similarity between x and x ′ based on f is

Original: [Equation]
Revised: [Equation]

Original: where ∥·∥ denotes the Euclidean norm of a vector.
Revised: where ∥·∥ denotes the Euclidean norm of a vector.

Original: We note that the above representation similarity is the cosine similarity of representations.
Revised: We note that the above representation similarity is the cosine similarity of representations.

Original: Motivations for this choice are in Appendix B.
Revised: Motivations for this choice are in Appendix B.

Original: With Definition 3.1 , the explanation target functions for RELAX and label-free feature importance are γ f, x e ( · ) =
Revised: With Definition 3.1 , the explanation target functions for RELAX and label-free feature importance are γ f, x e ( · ) =

Original: s f ( · , x e ) and γ f, x e ( · ) =
Revised: s f ( · , x e ) and γ f, x e ( · ) =

Original: s f ( · , x e ) ∥ f
Revised: s f ( · , x e ) ∥ f

Original: ( · ) ∥∥ f ( x e ) ∥ , respectively (Wickstrøm et al., 2021; Crabb´e & van der Schaar, 2022).
Revised: ( · ) ∥∥ f ( x e ) ∥ , respectively (Wickstrøm et al., 2021; Crabb´e & van der Schaar, 2022).

Original: Intuitively, these explanation target functions identify explicand features that, when removed or perturbed, point the altered representation away from the explicand’s original representation.
Revised: Intuitively, these explanation target functions identify explicand features that, when removed or perturbed, point the altered representation away from the explicand’s original representation.

Original: As mentioned in Section 1, we address the limitations of prior works by (i) relaxing the representation reference for feature attribution to any corpus set of samples; and (ii) enabling explicit specification of a foil to answer contrastive questions.
Revised: As mentioned in Section 1, we address the limitations of prior works by (i) relaxing the representation reference for feature attribution to any corpus set of samples; and (ii) enabling explicit specification of a foil to answer contrastive questions.

Original: For generality, we define the foil to be a distribution of inputs, which can also remove the extra burden of identifying particular foil samples.
Revised: For generality, we define the foil to be a distribution of inputs, which can also remove the extra burden of identifying particular foil samples.

Original: Nevertheless, we acknowledge that a fixed set of foil samples can indeed be selected for tailored use cases.
Revised: Nevertheless, we acknowledge that a fixed set of foil samples can indeed be selected for tailored use cases.

Original: Overall, our proposed improvements lead to contrastive corpus similarity as defined below.
Revised: Overall, our proposed improvements lead to contrastive corpus similarity as defined below.

Original: Definition 3.2 (Contrastive corpus similarity) .
Revised: Definition 3.2 (Contrastive corpus similarity) .

Original: Let C ⊂
Revised: Let C ⊂

Original: X be a finite set of corpus samples and
Revised: X be a finite set of corpus samples and

Original: D foil be a foil distribution over X .
Revised: D foil be a foil distribution over X .

Original: The contrastive corpus similarity of any x ∈ X to C in contrast to D foil is
Revised: The contrastive corpus similarity of any x ∈ X to C in contrast to D foil is

Original: [Equation]
Revised: [Equation]

Original: Given a set of foil samples F = { x (1) , ..., x ( m ) } , where x (1) , ..., x ( m ) i.i.d.
Revised: Given a set of foil samples F = { x (1) , ..., x ( m ) } , where x (1) , ..., x ( m ) i.i.d.

Original: ∼ D foil , the empirical estimator for the contrastive corpus similarity is
Revised: ∼ D foil , the empirical estimator for the contrastive corpus similarity is

Original: [Equation]
Revised: [Equation]

Original: The unbiasedness of and a concentration bound for the empirical estimator are stated in Appendix C.
Revised: The unbiasedness of and a concentration bound for the empirical estimator are stated in Appendix C.

Original: Practically, these analyses allow us to choose an empirical foil set size with theoretical justification.
Revised: Practically, these analyses allow us to choose an empirical foil set size with theoretical justification.

Original: 3. G ENERATING AND INTERPRETING FEATURE ATTRIBUTIONS
Revised: 3. G ENERATING AND INTERPRETING FEATURE ATTRIBUTIONS

Original: As in Example 2.2 , given a feature attribution method ϕ : R X × X → R |X| which takes an explanation target function γ f, ∗ and an explicand x e as inputs, COCOA computes attributions by explaining the empirical contrastive corpus similarity:
Revised: As in Example 2.2 , given a feature attribution method ϕ : R X × X → R |X| which takes an explanation target function γ f, ∗ and an explicand x e as inputs, COCOA computes attributions by explaining the empirical contrastive corpus similarity:

Original: [Equation]
Revised: [Equation]

Original: In practice, this amounts to wrapping the representation encoder f with the empirical estimator for contrastive corpus similarity and applying a feature attribution method.
Revised: In practice, this amounts to wrapping the representation encoder f with the empirical estimator for contrastive corpus similarity and applying a feature attribution method.

Original: In particular, there are two classes of feature attribution methods which can be used to explain contrastive corpus similarity and other similarity-based explanation target functions.
Revised: In particular, there are two classes of feature attribution methods which can be used to explain contrastive corpus similarity and other similarity-based explanation target functions.

Original: The first are removal-based methods (e.g.,
Revised: The first are removal-based methods (e.g.,

Original: RISE, KernelSHAP) and the second are gradient-based methods (e.g., Vanilla Gradients, Integrated
Revised: RISE, KernelSHAP) and the second are gradient-based methods (e.g., Vanilla Gradients, Integrated

Original: Gradients).
Revised: Gradients).

Original: Since the explanation target functions we consider in this paper can always be evaluated and are differentiable with respect to the inputs, we can directly apply these methods without modification.
Revised: Since the explanation target functions we consider in this paper can always be evaluated and are differentiable with respect to the inputs, we can directly apply these methods without modification.

Original: Finally, it is worth noting that methods which satisfy desirable properties (Sundararajan et al., 2017; Sundararajan & Najmi, 2020) in the supervised feature attribution setting also satisfy analogous properties for similarity-based explanation targets (Appendix D).
Revised: Finally, it is worth noting that methods which satisfy desirable properties (Sundararajan et al., 2017; Sundararajan & Najmi, 2020) in the supervised feature attribution setting also satisfy analogous properties for similarity-based explanation targets (Appendix D).

Original: To interpret feature attribution results from COCOA, consider the contrastive direction toward the corpus representation mean and away from the foil representation mean.
Revised: To interpret feature attribution results from COCOA, consider the contrastive direction toward the corpus representation mean and away from the foil representation mean.

Original: Features with high COCOA scores have a large impact on the explicand’s representation direction pointing along this contrastive direction.
Revised: Features with high COCOA scores have a large impact on the explicand’s representation direction pointing along this contrastive direction.

Original: This interpretation of COCOA is formalized in the following proposition.
Revised: This interpretation of COCOA is formalized in the following proposition.

Original: Proposition 3.3.
Revised: Proposition 3.3.

Original: The empirical estimator of contrastive corpus similarity is the dot product of the input’s representation direction and the empirical contrastive direction.
Revised: The empirical estimator of contrastive corpus similarity is the dot product of the input’s representation direction and the empirical contrastive direction.

Original: That is,
Revised: That is,

Original: [Equation]
Revised: [Equation]

Original: Proposition E.1 is an analogous statement for contrastive corpus similarity.
Revised: Proposition E.1 is an analogous statement for contrastive corpus similarity.

Original: An extension of Proposition 3.3 and Proposition E.1 to general kernel functions is in Appendix F.
Revised: An extension of Proposition 3.3 and Proposition E.1 to general kernel functions is in Appendix F.

Original: This extension implies that the directional interpretation of COCOA generally holds for kernel-based similarity functions.
Revised: This extension implies that the directional interpretation of COCOA generally holds for kernel-based similarity functions.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we quantitatively evaluate whether important features identified by COCOA are indeed related to the corpus and foil, using multiple datasets, encoders, and feature attribution methods (Section 4.1).
Revised: In this section, we quantitatively evaluate whether important features identified by COCOA are indeed related to the corpus and foil, using multiple datasets, encoders, and feature attribution methods (Section 4.1).

Original: Furthermore, we demonstrate the utility of COCOA by its application to understanding image data augmentations (Section 4.2) and to mixed modality object localization (Section 4.3).
Revised: Furthermore, we demonstrate the utility of COCOA by its application to understanding image data augmentations (Section 4.2) and to mixed modality object localization (Section 4.3).

Original: 4. Q UANTITATIVE EVALUATION
Revised: 4. Q UANTITATIVE EVALUATION

Original: To evaluate COCOA across different representation learning models and datasets, we apply them to (i) SimCLR, a contrastive self-supervised model (Chen et al., 2020), trained on ImageNet (Russakovsky et al., 2015); (ii) SimSiam, a non-contrastive self-supervised model (Chen & He, 2021), trained on CIFAR-10 (Krizhevsky et al., 2009); and (iii) representations extracted from the penultimate layer of a ResNet18 (He et al., 2016), trained on the abnormal-vs.-normal musculoskeletal
Revised: To evaluate COCOA across different representation learning models and datasets, we apply them to (i) SimCLR, a contrastive self-supervised model (Chen et al., 2020), trained on ImageNet (Russakovsky et al., 2015); (ii) SimSiam, a non-contrastive self-supervised model (Chen & He, 2021), trained on CIFAR-10 (Krizhevsky et al., 2009); and (iii) representations extracted from the penultimate layer of a ResNet18 (He et al., 2016), trained on the abnormal-vs.-normal musculoskeletal

Original: X-ray dataset MURA (Rajpurkar et al., 2017).
Revised: X-ray dataset MURA (Rajpurkar et al., 2017).

Original: Setup.
Revised: Setup.

Original: We consider four explanation targets: representation similiarity, contrastive similarity, corpus similarity, and contrastive corpus similarity; in our tables, these correspond to the methods: labelfree, contrastive label-free, corpus, and COCOA, respectively.
Revised: We consider four explanation targets: representation similiarity, contrastive similarity, corpus similarity, and contrastive corpus similarity; in our tables, these correspond to the methods: labelfree, contrastive label-free, corpus, and COCOA, respectively.

Original: To conduct ablation studies for the components of COCOA, we introduce contrastive similarity , which is contrastive corpus similarity when the corpus similarity term is replaced with similarity to an explicand; and corpus similarity , which is contrastive corpus similarity without a foil similarity term.
Revised: To conduct ablation studies for the components of COCOA, we introduce contrastive similarity , which is contrastive corpus similarity when the corpus similarity term is replaced with similarity to an explicand; and corpus similarity , which is contrastive corpus similarity without a foil similarity term.

Original: We use cosine similarity in all methods for the main text results (analogous results with dot product similarity are in Appendix M).
Revised: We use cosine similarity in all methods for the main text results (analogous results with dot product similarity are in Appendix M).

Original: To evaluate the feature attributions for a given explicand image x e ∈ R h,w,c and explanation target, we compute feature attributions using either Integrated Gradients (Sundararajan et al., 2017),
Revised: To evaluate the feature attributions for a given explicand image x e ∈ R h,w,c and explanation target, we compute feature attributions using either Integrated Gradients (Sundararajan et al., 2017),

Original: GradientSHAP (Smilkov et al., 2017; Erion et al., 2021), or RISE (Petsiuk et al., 2018).
Revised: GradientSHAP (Smilkov et al., 2017; Erion et al., 2021), or RISE (Petsiuk et al., 2018).

Original: Then, we average the attributions across the channel dimension in order to use insertion and deletion metrics (Petsiuk et al., 2018).
Revised: Then, we average the attributions across the channel dimension in order to use insertion and deletion metrics (Petsiuk et al., 2018).

Original: For deletion, we start with the original explicand and “remove” the M most important pixels by masking 1 them with a mask m ∈ { 0 , 1 } h,w,c and a blurred version of the explicand x m ∈ R h,w,c .
Revised: For deletion, we start with the original explicand and “remove” the M most important pixels by masking 1 them with a mask m ∈ { 0 , 1 } h,w,c and a blurred version of the explicand x m ∈ R h,w,c .

Original: We then evaluate the impact of each masked image with an evaluation measure η ( m ⊙ x e
Revised: We then evaluate the impact of each masked image with an evaluation measure η ( m ⊙ x e

Original: + (1 − m ) ⊙ x m ) .
Revised: + (1 − m ) ⊙ x m ) .

Original: In the supervised setting, this evaluation measure is typically the predicted probability of a class of interest.
Revised: In the supervised setting, this evaluation measure is typically the predicted probability of a class of interest.

Original: If we plot the number of removed features on the x-axis and the evaluation measure on the y-axis, we would expect the curve to drop sharply initially, leading to a low area under the curve.
Revised: If we plot the number of removed features on the x-axis and the evaluation measure on the y-axis, we would expect the curve to drop sharply initially, leading to a low area under the curve.

Original: For insertion, which “adds” important features, we would expect a high area under the curve.
Revised: For insertion, which “adds” important features, we would expect a high area under the curve.

Original: In the setting of explaining representations, we measure (i) the contrastive corpus similarity, which is calculated with the same corpus and foil sets used for explanations; and (ii) the corpus majority prediction, which is the predicted probability of the majority class based on each corpus sample’s prediction, where the prediction is from a downstream linear classifier trained to predict classes based on representations 2 .
Revised: In the setting of explaining representations, we measure (i) the contrastive corpus similarity, which is calculated with the same corpus and foil sets used for explanations; and (ii) the corpus majority prediction, which is the predicted probability of the majority class based on each corpus sample’s prediction, where the prediction is from a downstream linear classifier trained to predict classes based on representations 2 .

Original: We evaluate and average our insertion and deletion metrics for 250 explicands drawn for ImageNet and CIFAR-10 and 50 explicands for MURA, all from held-out sets.
Revised: We evaluate and average our insertion and deletion metrics for 250 explicands drawn for ImageNet and CIFAR-10 and 50 explicands for MURA, all from held-out sets.

Original: We consider both scenarios in which explicands and the corpus are in the same class and from different classes.
Revised: We consider both scenarios in which explicands and the corpus are in the same class and from different classes.

Original: For our experiments, 100 training samples are randomly selected to be the corpus set for each class in CIFARand MURA.
Revised: For our experiments, 100 training samples are randomly selected to be the corpus set for each class in CIFARand MURA.

Original: For ImageNet, 100 corpus samples are drawn from each of the 10 classes from ImageNette (Howard, 2019) instead of all 1000 classes for computational feasibility.
Revised: For ImageNet, 100 corpus samples are drawn from each of the 10 classes from ImageNette (Howard, 2019) instead of all 1,000 classes for computational feasibility.

Original: Foil samples are randomly drawn from each training set.
Revised: Foil samples are randomly drawn from each training set.

Original: Because ResNets output non-negative representations from
Revised: Because ResNets output non-negative representations from

Original: ReLU activations, we can apply Equation (C.3) and set the foil size to 1500 based on a δ = 0 .and ε = 0 .
Revised: ReLU activations, we can apply Equation (C.3) and set the foil size to 1,500 based on a δ = 0 .and ε = 0 .

Original: 05 in Proposition C.2 .
Revised: 05 in Proposition C.2 .

Original: We find that, even with a corpus size = 1, COCOA can have good performance, and results with corpus size ≥ 20 are similar to each other (Appendix J).
Revised: We find that, even with a corpus size = 1, COCOA can have good performance, and results with corpus size ≥ 20 are similar to each other (Appendix J).

Original: Varying the foil size does not seem to impact COCOA performance (Appendix K).
Revised: Varying the foil size does not seem to impact COCOA performance (Appendix K).

Original: Additional experiment details are in Appendix I.
Revised: Additional experiment details are in Appendix I.

Original: Results.
Revised: Results.

Original: First, our evaluations show that COCOA consistently has the strongest performance in contrastive corpus similarity insertion and deletion metrics across all feature attribution methods, models, and datasets (Appendix L).
Revised: First, our evaluations show that COCOA consistently has the strongest performance in contrastive corpus similarity insertion and deletion metrics across all feature attribution methods, models, and datasets (Appendix L).

Original: These metrics imply that COCOA generates feature attributions that successfully describe whether pixels in the explicand make it more similar to the corpus and
Revised: These metrics imply that COCOA generates feature attributions that successfully describe whether pixels in the explicand make it more similar to the corpus and

Original: [Table]
Revised: [Table]

Original: confidence intervals) across 5 experiment runs are reported.
Revised: intervals) across 5 experiment runs are reported.

Original: Higher insertion and lower deletion values indicate better performance, respectively.
Revised: Higher insertion and lower deletion values indicate better performance, respectively.

Original: Each method is a combination of a feature attribution method and an explanation target function (e.g., COCOA under
Revised: Each method is a combination of a feature attribution method and an explanation target function (e.g., COCOA under

Original: RISE corresponds to feature attributions computed by RISE for the contrastive corpus similarity).
Revised: RISE corresponds to feature attributions computed by RISE for the contrastive corpus similarity).

Original: Performance results of random attributions (last row) are included as benchmarks.
Revised: Performance results of random attributions (last row) are included as benchmarks.

Original: [Table]
Revised: [Table]

Original: dissimilar to the foil based on cosine similarity.
Revised: dissimilar to the foil based on cosine similarity.

Original: Although this is an important sanity check, it is perhaps unsurprising that COCOA performs strongly in this evaluation metric, because COCOA’s explanation target is exactly the same as this evaluation metric.
Revised: Although this is an important sanity check, it is perhaps unsurprising that COCOA performs strongly in this evaluation metric, because COCOA’s explanation target is exactly the same as this evaluation metric.

Original: Next, in Tables 1 and 2, COCOA has the strongest performance in corpus majority probability insertion and deletion metrics across nearly all feature attribution methods, models, and datasets.
Revised: Next, in Tables 1 and 2, COCOA has the strongest performance in corpus majority probability insertion and deletion metrics across nearly all feature attribution methods, models, and datasets.

Original: This is arguably a more interesting evaluation compared to contrastive corpus similarity, given that COCOA is not explicitly designed to perform this task.
Revised: This is arguably a more interesting evaluation compared to contrastive corpus similarity, given that COCOA is not explicitly designed to perform this task.

Original: Evaluation using the corpus majority predicted probability is semantically meaningful, because it tests whether features with high attributions move an explicand representation closer to the corpus as opposed to other classes across a decision boundary.
Revised: Evaluation using the corpus majority predicted probability is semantically meaningful, because it tests whether features with high attributions move an explicand representation closer to the corpus as opposed to other classes across a decision boundary.

Original: Furthermore, evaluation with a linear classifier has direct implications for downstream classification tasks and follows the widely used linear evaluation protocol for representation learning (Bachman et al., 2019; Kolesnikov et al., 2019; Chen et al., 2020).
Revised: Furthermore, evaluation with a linear classifier has direct implications for downstream classification tasks and follows the widely used linear evaluation protocol for representation learning (Bachman et al., 2019; Kolesnikov et al., 2019; Chen et al., 2020).

Original: In Table 1, we evaluate each method in the setting where explicands are from the same class as the corpus.
Revised: In Table 1, we evaluate each method in the setting where explicands are from the same class as the corpus.

Original: This enables us to compare to label-free attribution methods, which can still perform strongly on these metrics, because large changes in the explicand representation likely correlate with changes in the predicted probability of the explicand’s class.
Revised: This enables us to compare to label-free attribution methods, which can still perform strongly on these metrics, because large changes in the explicand representation likely correlate with changes in the predicted probability of the explicand’s class.

Original: Here, we can see that COCOA consistently outperforms label-free attribution methods and also benefits from the combination of a corpus and foil set.
Revised: Here, we can see that COCOA consistently outperforms label-free attribution methods and also benefits from the combination of a corpus and foil set.

Original: Importantly, in Table 2, corpus-based attribution methods perform strongly especially when the corpus is from a different class compared to the explicand.
Revised: Importantly, in Table 2, corpus-based attribution methods perform strongly especially when the corpus is from a different class compared to the explicand.

Original: This is significant, because it implies that we can use COCOA to generate explanations that answer questions such as, ”What pixels in my dog image make my representation similar to cat images?”, which was previously impossible using label-free approaches.
Revised: This is significant, because it implies that we can use COCOA to generate explanations that answer questions such as, ”What pixels in my dog image make my representation similar to cat images?”, which was previously impossible using label-free approaches.

Original: 4. U NDERSTANDING DATA AUGMENTATIONS IN S IM CLR
Revised: 4. U NDERSTANDING DATA AUGMENTATIONS IN S IM CLR

Original: In this section, we aim to visualize augmentation-invariant features that preserve class labels in
Revised: In this section, we aim to visualize augmentation-invariant features that preserve class labels in

Original: SimCLR.
Revised: SimCLR.

Original: Because class labels are associated with original images, we use COCOA to find features that make the representation of an augmented image similar to that of the original image.
Revised: Because class labels are associated with original images, we use COCOA to find features that make the representation of an augmented image similar to that of the original image.

Original: SimCLR is a self-supervised contrastive learning method that pushes images from the same class close together and images from different classes far apart.
Revised: SimCLR is a self-supervised contrastive learning method that pushes images from the same class close together and images from different classes far apart.

Original: This is achieved by its contrastive objective, which maximizes the similarity between representations from two augmentations of the same image and minimizes the similarity between representations from different images.
Revised: This is achieved by its contrastive objective, which maximizes the similarity between representations from two augmentations of the same image and minimizes the similarity between representations from different images.

Original: Empirically, linear classifiers trained on representations learned by SimCLR perform comparably to strong supervised baselines (Chen et al., 2020).
Revised: Empirically, linear classifiers trained on representations learned by SimCLR perform comparably to strong supervised baselines (Chen et al., 2020).

Original: Theoretical analyses of SimCLR performance rely on the idea that data augmentations preserve the class label of an image (Arora et al., 2019; HaoChen et al., 2021).
Revised: Theoretical analyses of SimCLR performance rely on the idea that data augmentations preserve the class label of an image (Arora et al., 2019; HaoChen et al., 2021).

Original: Here, we show which input features are associated with label preservation, which has not been previously shown.
Revised: Here, we show which input features are associated with label preservation, which has not been previously shown.

Original: Setup.
Revised: Setup.

Original: The same SimCLR model and its downstream classifier for ImageNet from Section 4.1 are used here.
Revised: The same SimCLR model and its downstream classifier for ImageNet from Section 4.1 are used here.

Original: For each image, we investigate augmentations used to train SimCLR including flipping, cropping, flipping with cropping, grayscaling, and color jittering.
Revised: For each image, we investigate augmentations used to train SimCLR including flipping, cropping, flipping with cropping, grayscaling, and color jittering.

Original: We also consider cutout and 90 ◦ rotation, which are not included in SimCLR training.
Revised: We also consider cutout and 90 ◦ rotation, which are not included in SimCLR training.

Original: Class prediction based on the linear classifier is computed for the original version and all augmentations of each image to check class preservation.
Revised: Class prediction based on the linear classifier is computed for the original version and all augmentations of each image to check class preservation.

Original: With each original and augmented image as the explicand, the original image as the corpus, and 1500 random images from the ImageNet training set as the foil, COCOA is paired with RISE to visually identify class-preserving features.
Revised: With each original and augmented image as the explicand, the original image as the corpus, and 1,500 random images from the ImageNet training set as the foil, COCOA is paired with RISE to visually identify class-preserving features.

Original: The RISE parameters are the same as in Section 4.1.
Revised: The RISE parameters are the same as in Section 4.1.

Original: Results.
Revised: Results.

Original: We first note that the original images and the augmented images after flipping, cropping, flipping with cropping, grayscaling, and color jittering yield correct class predictions (Figure 2).
Revised: We first note that the original images and the augmented images after flipping, cropping, flipping with cropping, grayscaling, and color jittering yield correct class predictions (Figure 2).

Original: However, cutout and rotated images, which are not part of SimCLR training, are less robust and can result in misclassifications.
Revised: However, cutout and rotated images, which are not part of SimCLR training, are less robust and can result in misclassifications.

Original: Because class labels are generated based on the original images, important features identified by COCOA with each original image as the corpus are class-preserving for correctly classified augmentations.
Revised: Because class labels are generated based on the original images, important features identified by COCOA with each original image as the corpus are class-preserving for correctly classified augmentations.

Original: Qualitatively, the class-preserving region of the English springer seems to be its face, as the face is highlighted in all augmentations with the correct classification and the original image (Figure 2a).
Revised: Qualitatively, the class-preserving region of the English springer seems to be its face, as the face is highlighted in all augmentations with the correct classification and the original image (Figure 2a).

Original: Contrarily, in the rotated image, which is misclassified as a llama,
Revised: Contrarily, in the rotated image, which is misclassified as a llama,

Original: COCOA highlights the body of the English springer instead of its face.
Revised: COCOA highlights the body of the English springer instead of its face.

Original: Similarly, the piping in the
Revised: Similarly, the piping in the

Original: French horn is the class-preserving region (Figure 2b).
Revised: French horn is the class-preserving region (Figure 2b).

Original: When the piping is cut out, the image is misclassified, and COCOA highlights only the face of the performer.
Revised: When the piping is cut out, the image is misclassified, and COCOA highlights only the face of the performer.

Original: Finally, the co-occurrence of the parachute and parachuter is the important feature for an augmented image to have the correct classification of parachute (Figure 2c).
Revised: Finally, the co-occurrence of the parachute and parachuter is the important feature for an augmented image to have the correct classification of parachute (Figure 2c).

Original: Additional results are in Appendix O.
Revised: Additional results are in Appendix O.

Original: 4. M IXED MODALITY OBJECT LOCALIZATION
Revised: 4. M IXED MODALITY OBJECT LOCALIZATION

Original: Here, we showcase the utility of COCOA by performing zero-shot object localization based on mixed modality representations as in (Gadre et al., 2022).
Revised: Here, we showcase the utility of COCOA by performing zero-shot object localization based on mixed modality representations as in Gadre et al.

Original: In this experiment, we work with CLIP which trains a text and image encoder on paired data: images with associated captions (Radford et al., 2021).
Revised: In this experiment, we work with CLIP which trains a text and image encoder on paired data: images with associated captions (Radford et al., 2021).

Original: To apply COCOA, a necessary prerequisite is a set of corpus examples.
Revised: To apply COCOA, a necessary prerequisite is a set of corpus examples.

Original: These examples are not necessarily hard to gather, since small sets of corpus examples can be sufficient (Appendix J).
Revised: These examples are not necessarily hard to gather, since small sets of corpus examples can be sufficient (Appendix J).

Original: [Figure]
Revised: [Figure]

Original: However, if our image representations are generated via the CLIP image encoder, we can circumvent the need to gather a set of corpus images by instead explaining similarity of our explicand image to a corpus caption of interest.
Revised: However, if our image representations are generated via the CLIP image encoder, we can circumvent the need to gather a set of corpus images by instead explaining similarity of our explicand image to a corpus caption of interest.

Original: This enables us to easily ask semantically meaningful questions such as, ”What in this image makes its representation similar to a woman?”.
Revised: This enables us to easily ask semantically meaningful questions such as, ”What in this image makes its representation similar to a woman?”.

Original: Setup.
Revised: Setup.

Original: In this experiment, we have two CLIP representation encoders f text : X text → R d and f image : X image → R d .
Revised: In this experiment, we have two CLIP representation encoders f text : X text → R d and f image : X image → R d .

Original: Both encoders map their respective input domains into a common representation space such that the representations of a caption and image pair have high cosine similarity when the caption is descriptive of the paired image.
Revised: Both encoders map their respective input domains into a common representation space such that the representations of a caption and image pair have high cosine similarity when the caption is descriptive of the paired image.

Original: We use the original implementation of CLIPand model ViT-B/32.
Revised: We use the original implementation of CLIPand model ViT-B/32.

Original: The feature attributions are computed using RISE with the same parameters as in Section 4.1, except with 20,000 masks.
Revised: The feature attributions are computed using RISE with the same parameters as in Section 4.1, except with 20,000 masks.

Original: Here, we aim to explain an explicand image x e ∈ X image .
Revised: Here, we aim to explain an explicand image x e ∈ X image .

Original: For label-free importance, the explanation target is the representation similarity based on the image encoder: γ f image , x e ( · ) .
Revised: For label-free importance, the explanation target is the representation similarity based on the image encoder: γ f image , x e ( · ) .

Original: Then, we use an adapted version of the COCOA explanation target, which uses a text corpus set
Revised: Then, we use an adapted version of the COCOA explanation target, which uses a text corpus set

Original: C text ⊂
Revised: C text ⊂

Original: X text and a text foil set F text ⊂
Revised: X text and a text foil set F text ⊂

Original: X text and maps from images to contrastive corpus similarity ˆ γ f image ,f text , C text , F text ( x ) : X image → R , defined as follows:|C [Equation] x c ∈C text
Revised: X text and maps from images to contrastive corpus similarity ˆ γ f image ,f text , C text , F text ( x ) : X image → R , defined as follows:

Original: [Equation]

Original: image

Original: [Equation]

Original: ∥ f image

Original: [Equation]
Revised: [Equation]

Original: In particular, the corpus set consists of a single caption C
Revised: In particular, the corpus set consists of a single caption C

Original: = { x c } of the following form, x c
Revised: = { x c } of the following form, x c

Original: = “This is a photo of a P ”, where P is manually chosen.
Revised: = “This is a photo of a P ”, where P is manually chosen.

Original: Then, the foil set is either (i) a single caption
Revised: Then, the foil set is either (i) a single caption

Original: F = { x f } of the form, x f = “This is a photo of a Q ”, where Q is manually chosen or (ii) many captions F = { x f i } of the forms, x f i =
Revised: F = { x f } of the form, x f = “This is a photo of a Q ”, where Q is manually chosen or (ii) many captions F = { x f i } of the forms, x f i =

Original: “This is a photo of a Q i ” where Q i ∈ C cifar 100 \ P is each of the 100 classes in CIFAR-100 excluding the corpus class P .
Revised: “This is a photo of a Q i ” where Q i ∈ C cifar 100 \ P is each of the 100 classes in CIFAR-100 excluding the corpus class P .

Original: Each explicand image is a public domain image from https://www.pexels.com/ that the CLIP model is unlikely to have been trained on 4 . 
Revised: Each explicand image is a public domain image from https://www.pexels.com/ that the CLIP model is unlikely to have been trained on 4 . 3 https://github.com/openai/CLIP

Original: With the exception of the astronaut image, which is available in the scikit-image package.
Revised: With the exception of the astronaut image, which is available in the scikit-image package.

Original: [Figure]
Revised: [Figure]

Original: Results.
Revised: Results.

Original: We visualize the label-free and COCOA attributions for several explicands in Figure 3.
Revised: We visualize the label-free and COCOA attributions for several explicands in Figure 3.

Original: First, we find that label-free attributions highlight a variety of regions in the explicand image.
Revised: First, we find that label-free attributions highlight a variety of regions in the explicand image.

Original: Although these pixels contribute to similarity to the explicand’s representation, with the exception of the astronaut image, the important pixels are largely not localized and therefore hard to interpret.
Revised: Although these pixels contribute to similarity to the explicand’s representation, with the exception of the astronaut image, the important pixels are largely not localized and therefore hard to interpret.

Original: For instance, in the mountain image we cannot say whether the mountain or cloud is the most important to the explicand’s representation, because the most positively and negatively important pixels are distributed across both objects.
Revised: For instance, in the mountain image we cannot say whether the mountain or cloud is the most important to the explicand’s representation, because the most positively and negatively important pixels are distributed across both objects.

Original: Instead, we find that using COCOA, we can ask specific questions.
Revised: Instead, we find that using COCOA, we can ask specific questions.

Original: For instance, we may ask, “compared to many other classes, what pixels in this image look like P ?”
Revised: For instance, we may ask, “Compared to many other classes, what pixels in this image look like P ?”

Original: (Figure 3a).
Revised: (Figure 3a).

Original: We evaluate a number of these questions in the first three images.
Revised: We evaluate a number of these questions in the first three images.

Original: In the astronaut image, we can see the woman’s face and the rocket are localized.
Revised: In the astronaut image, we can see the woman’s face and the rocket are localized.

Original: Likewise in the image with both a bicycle and a car, we can localize either the bicycle or the car.
Revised: Likewise in the image with both a bicycle and a car, we can localize either the bicycle or the car.

Original: In the zebra image, we are able to localize the zebras by using corpus texts that aim to identify a “zebra” or “stripes”.
Revised: In the zebra image, we are able to localize the zebras by using corpus texts that aim to identify a “zebra” or “stripes”.

Original: Finally, in the last three images we find that we can formulate specific contrastive questions that ask, “what pixels in this image look like P rather than
Revised: Finally, in the last three images we find that we can formulate specific contrastive questions that ask, “What pixels in this image look like P rather than

Original: Q ?”
Revised: Q ?”

Original: (Figure 3b).
Revised: (Figure 3b).

Original: Doing so we successfully identify a bee rather than a flower, a dog rather than a cat, and a mountain rather than clouds (and vice versa respectively).
Revised: Doing so we successfully identify a bee rather than a flower, a dog rather than a cat, and a mountain rather than clouds (and vice versa respectively).

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this work, we introduce COCOA, a method to explain unsupervised models based on similarity in a representation space.
Revised: In this work, we introduce COCOA, a method to explain unsupervised models based on similarity in a representation space.

Original: We theoretically analyze our method and demonstrate its compatibility with many feature attribution methods.
Revised: We theoretically analyze our method and demonstrate its compatibility with many feature attribution methods.

Original: We quantitatively validate COCOA in several datasets and on contrastive self-supervised, non-contrastive self-supervised, and supervised models.
Revised: We quantitatively validate COCOA in several datasets and on contrastive self-supervised, non-contrastive self-supervised, and supervised models.

Original: Then, we use COCOA to better understand data augmentations in a self-supervised setting and to perform zero-shot object localization.
Revised: Then, we use COCOA to better understand data augmentations in a self-supervised setting and to perform zero-shot object localization.

Original: Although COCOA is already a powerful technique to understand representations, there are many interesting future research directions.
Revised: Although COCOA is already a powerful technique to understand representations, there are many interesting future research directions.

Original: First, we can adaptexisting automatic concept discovery methods (Ghorbani et al., 2019; Yeh et al., 2020) to automatically identify  corpus sets.
Revised: First, we can adapt automatic concept discovery methods (Ghorbani et al., 2019; Yeh et al., 2020) to automatically identify homogeneous corpus sets.

Original: Second, we can use other explanation methods for our novel contrastive corpus similarity explanation target.
Revised: Second, we can use other explanation methods for our novel contrastive corpus similarity explanation target.

Original: This includes counterfactual explanation methods (Verma et al., 2020), which only require gradients to optimize a counterfactual objective.
Revised: This includes counterfactual explanation methods (Verma et al., 2020), which only require gradients to optimize a counterfactual objective.

Original: Finally, a fundamental limitation of COCOA is that it cannot explain corpus sets if the model has not learned anything about the given corpus.
Revised: Finally, a fundamental limitation of COCOA is that it cannot explain corpus sets if the model has not learned anything about the given corpus.

Original: Therefore, a valuable future direction is to investigate complementary methods to detect whether the model’s representation contains information about a corpus prior to application of COCOA.
Revised: Therefore, a valuable future direction is to investigate complementary methods to detect whether the model’s representation contains information about a corpus prior to application of COCOA.

Original: A R ELATED WORK
Revised: A A DDITIONAL RELATED WORK

Original: Representation learning.
Revised: Representation learning.

Original: COCOA can be used to explain any representation learning model.
Revised: COCOA can be used to explain any representation learning model.

Original: Representation learning aims to automatically discover latent dimensions that are valuable to downstream tasks (e.g., classification) (Bengio et al., 2013).
Revised: Representation learning aims to automatically discover latent dimensions that are valuable to downstream tasks (e.g., classification) (Bengio et al., 2013).

Original: It has been applied to great effect within natural language processing (Deng et al., 2013; Van Den Oord et al., 2017) and computer vision (Ciregan et al., 2012; Gidaris et al., 2018) and encompasses both supervised and unsupervised models ( f : X → Z ).
Revised: It has been applied to great effect within natural language processing (Deng et al., 2013; Van Den Oord et al., 2017) and computer vision (Ciregan et al., 2012; Gidaris et al., 2018) and encompasses both supervised and unsupervised models ( f : X → Z ).

Original: For supervised models the output space corresponds to known labels ( Z = Y ), whereas for unsupervised models the outputs are latent representations ( Z = H ).
Revised: For supervised models the output space corresponds to known labels ( Z = Y ), whereas for unsupervised models the outputs are latent representations ( Z = H ).

Original: Within unsupervised representation learning, self-supervised learning leverages unlabelled data to formulate learning tasks (Grill et al., 2020; Misra & Maaten, 2020).
Revised: Within unsupervised representation learning, self-supervised learning leverages unlabelled data to formulate learning tasks (Grill et al., 2020; Misra & Maaten, 2020).

Original: One particularly successful instantiation of self-supervised learning is contrastive learning, which optimizes the similarity between different views of the same sample and dissimilarity to random samples (Chen et al., 2020; Radford et al., 2021).
Revised: One particularly successful instantiation of self-supervised learning is contrastive learning, which optimizes the similarity between different views of the same sample and dissimilarity to random samples (Chen et al., 2020; Radford et al., 2021).

Original: Representation-space explanations.
Revised: Representation-space explanations.

Original: COCOA explains a representation (latent) space.
Revised: COCOA explains a representation (latent) space.

Original: An early post-hoc explanation method based on representation spaces is TCAV (Kim et al., 2018) which defines concepts based on the representation space and produces a global explanation of whether a given class depends on a concept of interest.
Revised: An early post-hoc explanation method based on representation spaces is TCAV (Kim et al., 2018) which defines concepts based on the representation space and produces a global explanation of whether a given class depends on a concept of interest.

Original: A number of approaches similarly define concepts in the representation space and build upon TCAV with automatic concept discovery (Ghorbani et al., 2019; Yeh et al., 2020).
Revised: A number of approaches similarly define concepts in the representation space and build upon TCAV with automatic concept discovery (Ghorbani et al., 2019; Yeh et al., 2020).

Original: Finally, Basaj et al.
Revised: Finally, Basaj et al.

Original: (2021) build upon TCAV to define visual probes to validate whether a representation encodes a property of interest.
Revised: (2021) build upon TCAV to define visual probes to validate whether a representation encodes a property of interest.

Original: These approaches detect whether a given concept is important to the model, whereas our approach aims to understand which features make a representation similar to a given concept captured by a corpus.
Revised: These approaches detect whether a given concept is important to the model, whereas our approach aims to understand which features make a representation similar to a given concept captured by a corpus.

Original: Supervised explanations.
Revised: Supervised explanations.

Original: COCOA builds upon existing supervised explanation methods to explain an unsupervised setting.
Revised: COCOA builds upon existing supervised explanation methods to explain an unsupervised setting.

Original: The majority of explanation methods operate on supervised models and aim to explain a scalar output, often the model’s predicted probability for a specific class.
Revised: The majority of explanation methods operate on supervised models and aim to explain a scalar output, often the model’s predicted probability for a specific class.

Original: These methods vary in the unit of explanation (i.e., the way in which they describe what was important) and the explanation target (i.e., the model behavior they aim to explain).
Revised: These methods vary in the unit of explanation (i.e., the way in which they describe what was important) and the explanation target (i.e., the model behavior they aim to explain).

Original: The unit of explanation for example attributions and counterfactual attributions are samples which are either important to or modify the explanation target Koh & Liang (2017); Wachter et al.
Revised: The unit of explanation for example attributions and counterfactual attributions are samples which are either important to or modify the explanation target (Koh & Liang, 2017; Wachter et al., 2017).

Original: In contrast, the unit of explanation of feature attributions or concept attributions are importances which quantify how features/concepts contribute to the explanation target Lundberg & Lee (2017); Kim et al.
Revised: In contrast, the unit of explanation of feature attributions or concept attributions are importances which quantify how features or concepts contribute to the explanation target (Lundberg & Lee, 2017; Kim et al., 2018).

Original: In this paper, we focus on feature attribution methods, which are the most widely-used explanation technique, but note that it is possible to apply many  explanation methods to understand our new explanation target.
Revised: In this paper, we focus on feature attribution methods, which are the most widely-used explanation technique, but note that it is possible to apply many other explanation techniques to understand our new explanation target.

Revised: It has been shown that different feature attribution methods can generate disagreeing results (Krishna et al., 2022), so we empirically evaluate contrastive corpus similarity and other explanation target functions with a number of varied attribution methods.

Original: In particular, we compute local feature attributions using three popular methods.
Revised: In particular, we compute local feature attributions using three popular methods.

Original: The first is Integrated Gradients (Sundararajan et al., 2017), apopular gradient-based feature attribution method, which accumulates gradients between a baseline and an explicand.
Revised: The first is Integrated Gradients (Sundararajan et al., 2017), a gradient-based feature attribution method, which accumulates gradients between a baseline and an explicand.

Original: The second is GradientSHAP 5 , which  combines SmoothGrad (Smilkov et al., 2017) and Expected Gradients (Erion et al., 2021), a generalization of Integrated Gradients.
Revised: The second is GradientSHAP 5 , which is a SHAPrelated method that combines SmoothGrad (Smilkov et al., 2017) and Expected Gradients (Erion et al., 2021), a generalization of Integrated Gradients.

Original: Third, we use RISE (Petsiuk et al., 2018), a removal-based explanation method (Covert et al., 2021), which computes feature attributions by randomly masking portions of an explicand.
Revised: Third, we use RISE (Petsiuk et al., 2018), a removal-based explanation method (Covert et al., 2021), which computes feature attributions by randomly masking portions of an explicand.

Original: Unsupervised explanations.
Revised: Unsupervised explanations.

Original: COCOA explicitly aims to provide explanations for unsupervised models.
Revised: COCOA explicitly aims to provide explanations for unsupervised models.

Original: There is largely a deficit of explanation methods suitable to this task and we are aware of only two recent methods that address this problem: RELAX (Wickstrøm et al., 2021) and label-free explainability (Crabb´e & van der Schaar, 2022).
Revised: There is largely a deficit of explanation methods suitable to this task and we are aware of only two recent methods that address this problem: RELAX (Wickstrøm et al., 2021) and label-free explainability (Crabb´e & van der Schaar, 2022).

Original: Both methods define an explanation target function based on similarity to the explicand, where RELAX uses the cosine kernel ( γ f, x e ( x ) =
Revised: Both methods define an explanation target function based on similarity to the explicand, where RELAX uses the cosine kernel ( γ f, x e ( x ) =

Original: ( f ( x ) T f
Revised: ( f ( x ) T f

Original: ( x e )) / ( ∥  x  ∥∥  x e  ∥ ) ) and label-free explainability uses the dot product ( γ f, x e ( x ) =
Revised: ( x e )) / ( ∥ f ( x ) ∥∥ f ( x e ) ∥ ) ) and label-free explainability uses the dot product ( γ f, x e ( x ) =

Original: f ( x ) T f
Revised: f ( x ) T f

Original: ( x e ) ).
Revised: ( x e ) ).

Original: Then, both approaches utilize pre-existing local feature attribution methods to explain their respective explanation targets, where RELAX specfically utilizes RISE, and label-free explainability is an agnostic wrapper function.
Revised: Then, both approaches utilize pre-existing local feature attribution methods to explain their respective explanation targets, where RELAX specfically utilizes RISE, and label-free explainability is an agnostic wrapper function.

Original: A final distinction between the two papers is that label-free explainability additionally provides example attributions which aim to identify important training examples.
Revised: A final distinction between the two papers is that label-free explainability additionally provides example attributions which aim to identify important training examples.

Original: While COCOA, RELAX, and label-free explainability are motivated to explain unsupervised models, they all can be applied to explain representations in general, whether obtained through unsupervised or supervised learning.
Revised: While COCOA, RELAX, and label-free explainability are motivated to explain unsupervised models, they all can be applied to explain representations in general, whether obtained through unsupervised or supervised learning.

Original: Contrastive explanations.
Revised: Contrastive explanations.

Original: COCOA makes unsupervised explanations more interpretable by making them contrastive.
Revised: COCOA makes unsupervised explanations more interpretable by making them contrastive.

Original: Contrastive explanations aim to understand an event P (the fact) by contrasting it to another event Q (the foil)
Revised: Contrastive explanations aim to understand an event P (the fact) by contrasting it to another event Q (the foil)

Original: (Jacovi et al., 2021).
Revised: (Jacovi et al., 2021).

Original: Social science studies recognize that contrastive explanations align with human intuition and reduce an explanation’s complexity by ignoring factors common to both the fact and the foil (Kahneman & Miller, 1986; Lipton, 1990; Miller, 2019).
Revised: Social science studies recognize that contrastive explanations align with human intuition and reduce an explanation’s complexity by ignoring factors common to both the fact and the foil (Kahneman & Miller, 1986; Lipton, 1990; Miller, 2019).

Original: One strategy to obtain contrastive explanations is to simply define an explanation target function which describes a contrastive model behavior (e.g., the difference between the model’s prediction for the fact and the foil)
Revised: One strategy to obtain contrastive explanations is to simply define an explanation target function which describes a contrastive model behavior (e.g., the difference between the model’s prediction for the fact and the foil)

Original: (Jacovi et al., 2021).
Revised: (Jacovi et al., 2021).

Original: B M OTIVATIONS FOR COSINE SIMILARITY AS THE REPRESENTATION SIMILARITY
Revised: B M OTIVATIONS FOR COSINE SIMILARITY AS THE REPRESENTATION SIMILARITY

Original: The representation similarity defined in Definition 3.1 is the cosine similarity of representations.
Revised: The representation similarity defined in Definition 3.1 is the cosine similarity of representations.

Original: There are multiple motivations for this choice.
Revised: There are multiple motivations for this choice.

Original: First, this representation similarity is used in RELAX as the explanation target function (Wickstrøm et al., 2021).
Revised: First, this representation similarity is used in RELAX as the explanation target function (Wickstrøm et al., 2021).

Original: Similarly, label-free feature importance uses the dot product without normalization (Crabb´e & van der Schaar, 2022).
Revised: Similarly, label-free feature importance uses the dot product without normalization (Crabb ´ e & van der Schaar, 2022).

Original: Hence, the choice of cosine similarity facilitates comparisons with prior works.
Revised: Hence, the choice of cosine similarity facilitates comparisons with prior works.

Original: Second, many self-supervised models are trained with the cosine similarity or the related cross-correlation in their loss functions (Chen et al., 2020;
Revised: Second, many self-supervised models are trained with the cosine similarity or the related cross-correlation in their loss functions (Chen et al., 2020;

Original: Grill et al., 2020; Chen & He, 2021;
Revised: Grill et al., 2020; Chen & He, 2021;

Original: Radford et al., 2021; Zbontar et al., 2021), so cosine similarity is a natural choice for measuring similarity between representations learned by these models.
Revised: Radford et al., 2021; Zbontar et al., 2021), so cosine similarity is a natural choice for measuring similarity between representations learned by these models.

Original: Finally, other similarity measures, such as the Gaussian kernel function, may contain parameters that impact explanation outcomes, but to our knowledge there are no principled ways of tuning these parameters for the purpose of generating explanations.
Revised: Finally, other similarity measures, such as the Gaussian kernel function, may contain parameters that impact explanation outcomes, but to our knowledge there are no principled ways of tuning these parameters for the purpose of generating explanations.

Original: C S OME PROPERTIES OF THE EMPIRICAL ESTIMATOR OF CONTRASTIVE CORPUS SIMILARITY
Revised: C S OME PROPERTIES OF THE EMPIRICAL ESTIMATOR OF CONTRASTIVE CORPUS SIMILARITY

Original: Proposition C.1.
Revised: Proposition C.1.

Original: The estimator ˆ γ f, C , F ( x ) is an unbiased estimator of γ f, C , D foil ( x ) .
Revised: The estimator ˆ γ f, C , F ( x ) is an unbiased estimator of γ f, C , D foil ( x ) .

Original: Proposition C.2.
Revised: Proposition C.2.

Original: For any δ ∈ (0 , 1) and ε > 0 , if the sample size m in ˆ γ f, C , F ( x ) satisfies
Revised: For any δ ∈ (0 , 1) and ε > 0 , if the sample size m in ˆ γ f, C , F ( x ) satisfies

Original: [Equation]
Revised: [Equation]

Original: then
Revised: then

Original: P

Original: [Equation]
Revised: [Equation]

Original: If it is further assumed that f maps from X to the set of non-negative vectors R d + (e.g., a neural network with a ReLU output layer), then the same concentration bound is achieved with
Revised: If it is further assumed that f maps from X to the set of non-negative vectors R d + (e.g., a neural network with a ReLU output layer), then the same concentration bound is achieved with

Original: [Equation]
Revised: [Equation]

Original: Proposition C.2 allows us to choose an empirical foil sample size with theoretical justification.
Revised: Proposition C.2 allows us to choose an empirical foil sample size with theoretical justification.

Original: D F EATURE ATTRIBUTION PROPERTIES
Revised: D F EATURE ATTRIBUTION PROPERTIES

Original: There are a multitude of feature attribution methods.
Revised: There are a multitude of feature attribution methods.

Original: In order to compare these methods, researchers have debated their respective merits in terms of the properties they satisfy (Sundararajan & Najmi, 2020).
Revised: In order to compare these methods, researchers have debated their respective merits in terms of the properties they satisfy (Sundararajan & Najmi, 2020).

Original: Most feature attribution methods satisfy certain properties with respect to the model output (explanation target) they explain.
Revised: Most feature attribution methods satisfy certain properties with respect to the model output (explanation target) they explain.

Original: Methods which satisfy a property when the explanation target is the predicted probability of a particular class will also satisfy a related property for other explanation targets such as representation similarity or contrastive corpus similarity.
Revised: Methods which satisfy a property when the explanation target is the predicted probability of a particular class will also satisfy a related property for other explanation targets such as representation similarity or contrastive corpus similarity.

Original: As an example, we may consider the property of completeness, which is satisfied by Integrated
Revised: As an example, we may consider the property of completeness, which is satisfied by Integrated

Original: Gradients, KernelSHAP, DeepLIFT, and a number of other explanation methods.
Revised: Gradients, KernelSHAP, DeepLIFT, and a number of other explanation methods.

Original: Methods which satisfy completeness generate feature attributions which sum up to the explanation target function applied to the explicand minus a baseline value b 0 :
Revised: Methods which satisfy completeness generate feature attributions which sum up to the explanation target function applied to the explicand minus a baseline value b 0 :

Original: [Equation]
Revised: [Equation]

Original: For attribution methods that use a single baseline (e.g., Integrated Gradients, DeepLIFT), the baseline value is equal to the explanation target function applied to a baseline of the user’s choosing (i.e., b 0 = γ f, ∗ ( x b ) ).
Revised: For attribution methods that use a single baseline (e.g., Integrated Gradients, DeepLIFT), the baseline value is equal to the explanation target function applied to a baseline of the user’s choosing (i.e., b 0 = γ f, ∗ ( x b ) ).

Original: Therefore, supervised feature attributions sum up to the difference between the predicted probability of a class of interest for the explicand x e and baseline x b :(cid:88)
Revised: Therefore, supervised feature attributions sum up to the difference between the predicted probability of a class of interest for the explicand x e and baseline x b :

Original: [Equation]
Revised: [Equation]

Original: Analogously, in the unsupervised explanation setting, COCOA feature attributions will sum up to the difference in the contrastive corpus similarity between the explicand and the baseline:(cid:88)
Revised: Analogously, in the unsupervised explanation setting, COCOA feature attributions will sum up to the difference in the contrastive corpus similarity between the explicand and the baseline:

Original: [Equation]
Revised: [Equation]

Original: Note that analogous completeness properties may be shown for representation similarity, contrastive similarity, and corpus similarity.
Revised: Note that analogous completeness properties may be shown for representation similarity, contrastive similarity, and corpus similarity.

Original: E P ROPOSITION FOR INTERPRETING COCOA WITH RESPECT TO THE CONTRASTIVE CORPUS SIMILARITY
Revised: E P ROPOSITION FOR INTERPRETING COCOA WITH RESPECT TO THE CONTRASTIVE CORPUS SIMILARITY

Original: Proposition E.1.
Revised: Proposition E.1.

Original: Contrastive corpus similarity is the dot product of the input’s representation direction and the contrastive direction.
Revised: Contrastive corpus similarity is the dot product of the input’s representation direction and the contrastive direction.

Original: That is,
Revised: That is,

Original: [Equation]
Revised: [Equation]

Original: (E.
Revised: (E.1)

Original: [Equation]

Original: F E XTENSION OF P ROPOSITION 3.3 AND P ROPOSITION E.1 TO GENERAL KERNEL FUNCTIONS
Revised: F E XTENSION OF P ROPOSITION 3.3 AND P ROPOSITION E.1 TO GENERAL KERNEL FUNCTIONS

Original: Let another kernel function s : R d × R d → R , such that s f ( · , · )
Revised: Let another kernel function s : R d × R d → R , such that s f ( · , · )

Original: = s ( f ( · ) , f ( · )) , be the representation similarity measure instead of the cosine similarity.
Revised: = s ( f ( · ) , f ( · )) , be the representation similarity measure instead of the cosine similarity.

Original: Then, in another latent feature space, the contrastive direction points toward the corpus’s mean representation and away from the foil’s mean representation.
Revised: Then, in another latent feature space, the contrastive direction points toward the corpus’s mean representation and away from the foil’s mean representation.

Original: Formally, we have the following proposition.
Revised: Formally, we have the following proposition.

Original: Proposition F.1.
Revised: Proposition F.1.

Original: Suppose the function s ( · , · ) is a continuous positive semi-definite kernel on a compact set
Revised: Suppose the function s ( · , · ) is a continuous positive semi-definite kernel on a compact set

Original: Z ⊂ R d , and f maps from X to Z .
Revised: Z ⊂ R d , and f maps from X to Z .

Original: Let L 2 ( Z ) denote the set of functions over
Revised: Let L 2 ( Z ) denote the set of functions over

Original: Z that are square integrable.
Revised: Z that are square integrable.

Original: Additionally, suppose that the integral operator T s , defined as ( T s g )( · ) =
Revised: Additionally, suppose that the integral operator T s , defined as ( T s g )( · ) =

Original: (cid:82) Z s ( · , ˜ z ) g
Revised: (cid:82) Z s ( · , ˜ z ) g

Original: (˜ z ) d ˜ z for all g ∈ L 2 ( Z ) , is positive semi-definite: (cid:90)
Revised: (˜ z ) d ˜ z for all g ∈ L 2 ( Z ) , is positive semi-definite: (cid:90)

Original: [Equation]
Revised: [Equation]

Original: (F.1)
Revised: (F.1)

Original: Let γ f, C , [Equation] foil ( [Equation]
Revised: Let γ f, C , D foil ( x ) = 1 |C| (cid:88) x

Original: = [Equation] (cid:88) x c 
Revised: c ∈C s ( f ( x ) , f ( x c ))

Original: [Equation] s ( f ( x ) , f ( x c )) − E x v ∼D foil [ s ( f ( x ) , f ( x v ))]
Revised: − E x v ∼D foil [ s ( f ( x ) , f ( x v ))]

Original: (F.2) and ˆ γ f, C , F ( x ) = 1 |C| (cid:88) x c ∈C s ( f ( x ) , f ( x c ))
Revised: (F.2) and ˆ γ f, C , F ( x ) = 1 |C| (cid:88) x c ∈C s ( f ( x ) , f ( x c ))

Original: − 1 m m (cid:88) i s ( f ( x ) , f ( x ( i ) )) .
Revised: − 1 m m (cid:88) i s ( f ( x ) , f ( x ( i ) )) .

Original: (F.3)
Revised: (F.3)

Original: Then γ f, C , D foil ( x ) = ψ ( f ( x )) T (cid:18)|C| (cid:88) x c ∈C ψ ( f ( x c ))
Revised: Then γ f, C , D foil ( x ) = ψ ( f ( x )) T (cid:18)|C| (cid:88) x c ∈C ψ ( f ( x c ))

Original: − E x v ∼D foil [ ψ ( f ( x v ))] (cid:19) (F.4) andˆ γ f, C , F ( x ) = ψ ( f ( x )) T (cid:18)|C| (cid:88) x c ∈C ψ ( f ( x c )) − 1 m m (cid:88) i = ψ ( f (
Revised: − E x v ∼D foil [ ψ ( f ( x v ))] (cid:19) (F.4) and

Original: [Equation]
Revised: [Equation]

Original: where ψ : Z → D is a feature map from the representation space to the feature space D induced by the kernel function.
Revised: where ψ : Z → D is a feature map from the representation space to the feature space D induced by the kernel function.

Original: G C ONTRASTIVE CORPUS SIMILARITY WHEN THE CORPUS CONSISTS OF MULTIPLE SUB - CORPORA
Revised: G C ONTRASTIVE CORPUS SIMILARITY WHEN THE CORPUS CONSISTS OF MULTIPLE SUB - CORPORA

Original: Here, we consider the scenario where a corpus consists of multiple groups (i.e., sub-corpora ) with different characteristics.
Revised: Here, we consider the scenario where a corpus consists of multiple groups (i.e., sub-corpora ) with different characteristics.

Original: In this setting, the contrastive corpus similarity and its empirical estimator are a weighted average of the contrastive sub-corpus similarities and a weighted average of the empirical estimators of contrastive sub-corpus similarities, respectively.
Revised: In this setting, the contrastive corpus similarity and its empirical estimator are a weighted average of the contrastive sub-corpus similarities and a weighted average of the empirical estimators of contrastive sub-corpus similarities, respectively.

Original: Importantly, the weight corresponding to each sub-corpus is proportional to its size.
Revised: Importantly, the weight corresponding to each sub-corpus is proportional to its size.

Original: This insight is formalized in the following proposition.
Revised: This insight is formalized in the following proposition.

Original: Proposition G.1.
Revised: Proposition G.1.

Original: Suppose a corpus C consists of disjoint sub-corpora C k for k = 1 , ..., K , for some integer K > 0 .
Revised: Suppose a corpus C consists of disjoint sub-corpora C k for k = 1 , ..., K , for some integer K > 0 .

Original: That is, C =
Revised: That is, C =

Original: (cid:83) Kk =1 C k , and C i ∩ C j = ∅ for all i ̸
Revised: (cid:83) Kk =1 C k , and C i ∩ C j = ∅ for all i ̸

Original: = j .
Revised: = j .

Original: Then the contrastive corpus similarity with respect to the corpus C is a weighted average of contrastive corpus similarities with respect to the sub-corpora:
Revised: Then the contrastive corpus similarity with respect to the corpus C is a weighted average of contrastive corpus similarities with respect to the sub-corpora:

Original: [Equation]
Revised: [Equation]

Original: Furthermore, the empirical estimator of contrastive corpus similarity with respect to the corpus C is a weighted average of empirical estimators of contrastive corpus similarity with respect to the sub-corpora:
Revised: Furthermore, the empirical estimator of contrastive corpus similarity with respect to the corpus C is a weighted average of empirical estimators of contrastive corpus similarity with respect to the sub-corpora:

Original: [Equation]
Revised: [Equation]

Original: H P ROOFS H. P ROOF OF P ROPOSITION C. 
Revised: P ROOFS H. P ROOF OF P ROPOSITION C.

Original: Proof.
Revised: Proof.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: where (H.1) follows from the linearity of expectation
Revised: H [Equation]

Original: [Equation]
Revised: [Equation]

Original:  and (H.2) from the fact that x (1) , ..., x ( m ) are identically distributed from D foil .
Revised: where (H.1) follows from the linearity of expectation, and (H.2) from the fact that x (1) , ..., x ( m ) are identically distributed from D foil .

Original: H. P ROOF OF P ROPOSITION C. 
Revised: H. P ROOF OF P ROPOSITION C.

Original: We first state Hoeffding’s inequality.
Revised: We first state Hoeffding’s inequality.

Original: Lemma H.1.
Revised: Lemma H.1.

Original: Let X 1 , ..., X n be independent random variables such that a i ≤ X i ≤
Revised: Let X 1 , ..., X n be independent random variables such that a i ≤ X i ≤

Original: b
Revised: b

Original: i almost surely.
Revised: i almost surely.

Original: Consider S n = X 1 + · · · + X n , then
Revised: Consider S n = X 1 + · · · + X n , then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See Hoeffding (1963).
Revised: See Hoeffding (1963).

Original: We now proceed to prove Proposition C.2 .
Revised: We now proceed to prove Proposition C.2 .

Original: Proof.
Revised: Proof.

Original: First, we have
Revised: First, we have

Original: [Equation]
Revised: [Equation]

Original: and note that
Revised: and note that

Original: [Equation]
Revised: [Equation]

Original: Because the cosine similarity is bounded: − 1 ≤ s f ( · , · ) ≤ 1 , it follows that − 1 /m ≤ s f ( · , · ) /m ≤ 1 /m .
Revised: Because the cosine similarity is bounded: − 1 ≤ s f ( · , · ) ≤ [Equation] it follows that − 1 /m ≤ s f ( · , · ) /m ≤ 1 /m .

Original: Applying the Hoeffding’s inequality, we obtain
Revised: Applying the Hoeffding’s inequality

Original: [Equation]
Revised: [Equation]

Original: H
Revised: we obtain

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: If it is further assumed that f maps from X to R d + , then 0 ≤ s f ( · , · ) ≤ 1 and hence 0 ≤ s f ( · , · ) /m ≤ 1 /m .
Revised: If it is further assumed that f maps from X to R d + , then 0 ≤ s f ( · , · ) ≤ 1 and hence 0 ≤ s f ( · , · ) /m ≤ 1 /m .

Original: Applying the Hoeffding’s inequality with m ≥ log(2 /δ ) 2 εyields
Revised: Applying the Hoeffding’s inequality with m ≥ log(2 /δ ) 2 εyields

Original: [Equation]
Revised: [Equation]

Original: H. P ROOF OF P ROPOSITION 3.3 AND P ROPOSITION E. 
Revised: H. P ROOF OF P ROPOSITION 3.3 AND P ROPOSITION E.

Original: Proof.
Revised: Proof.

Original: By the definition of ˆ γ f, C , F ( x ) and linear algebra, we have ˆ γ cf, x e , D foil ( x ) = 1 |C| (cid:88) x c ∈C (cid:18)f
Revised: By the definition of ˆ γ f, C , F ( x ) and linear algebra, we have ˆ γ f, C , F ( x ) = 1 |C| (cid:88) x c ∈C (cid:18)

Original:  ( x ) ∥ f
Revised: f ( x ) ∥ f

Original: ( x ) ∥ (cid:19) T f
Revised: ( x ) ∥ (cid:19) T f

Original: ( x c ) ∥
Revised: ( x c ) ∥

Original: f ( x c ) ∥ − m m (cid:88) i = (cid:18)
Revised: f ( x c ) ∥ − m m (cid:88) i = (cid:18)

Original: f
Revised: f

Original: ( x ) ∥ f
Revised: ( x ) ∥ f

Original: ( x ) ∥ (cid:19) T f ( x ( i ) )
Revised: ( x ) ∥ (cid:19) T f ( x ( i ) )

Original: ∥
Revised: ∥

Original: f ( x ( i ) ) ∥ (H.12) =
Revised: f ( x ( i ) ) ∥ (H.12) =

Original: (cid:18)
Revised: (cid:18)

Original: f ( x ) ∥ f ( x ) ∥ (cid:19) T (cid:18)|C| (cid:88) x x ∈C f ( x c ) ∥ f
Revised: f ( x ) ∥ f ( x ) ∥ (cid:19) T (cid:18)|C| (cid:88) x x ∈C f ( x c ) ∥ f

Original: ( x c ) ∥ (cid:19) − (cid:18)
Revised: ( x c ) ∥ (cid:19) − (cid:18)

Original: f ( x ) ∥ f ( x ) ∥ (cid:19) T (cid:18)m m (cid:88) i
Revised: f ( x ) ∥ f ( x ) ∥ (cid:19) T (cid:18)m m (cid:88) i

Original: = f
Revised: = f

Original: ( x ( i ) ) ∥
Revised: ( x ( i ) ) ∥

Original: f ( x ( i ) ) ∥ (cid:19) (H.13) =
Revised: f ( x ( i ) ) ∥ (cid:19) (H.13) =

Original: (cid:18)
Revised: (cid:18)

Original: f ( x ) ∥ f ( x ) ∥ (cid:19) T (cid:18)|C| (cid:88) x c ∈C f ( x c ) ∥
Revised: f ( x ) ∥ f ( x ) ∥ (cid:19) T (cid:18)|C| (cid:88) x c ∈C f ( x c ) ∥

Original: f ( x c ) ∥ − m m (cid:88) i =
Revised: f ( x c ) ∥ − m m (cid:88) i =

Original: f ( x ( i ) )
Revised: f ( x ( i ) )

Original: ∥
Revised: ∥

Original: f ( x ( i ) ) ∥ (cid:19) .
Revised: f ( x ( i ) ) ∥ (cid:19) .

Original: (H.14)
Revised: (H.14)

Original: By the definition of γ f, C , D foil ( x ) , linear algebra, and the linearity of expectation, we have γ f, C , D foil ( x ) = 1 |C| (cid:88) x c ∈C (cid:18)
Revised: By the definition of γ f, C , D foil ( x ) , linear algebra, and the linearity of expectation, we have γ f, C , D foil ( x ) = 1 |C| (cid:88) x c ∈C (cid:18)

Original: f ( x ) ∥ f ( x ) ∥ (cid:19) T f
Revised: f ( x ) ∥ f ( x ) ∥ (cid:19) T f

Original: ( x c ) ∥
Revised: ( x c ) ∥

Original: f ( x c ) ∥ − E x v ∼D foil (cid:34)(cid:18) f
Revised: f ( x c ) ∥ − E x v ∼D foil (cid:34)(cid:18) f

Original: ( x ) ∥ f
Revised: ( x ) ∥ f

Original: ( x ) ∥ (cid:19) T f ( x v ) ∥
Revised: ( x ) ∥ (cid:19) T f ( x v ) ∥

Original: f ( x v ) ∥ (cid:35) (H.15)
Revised: f ( x v ) ∥ (cid:35) (H.15)

Original: =
Revised: =

Original: (cid:18)
Revised: (cid:18)

Original: f ( x ) ∥ f ( x ) ∥ (cid:19) T (cid:18)|C| (cid:88) x c ∈C f ( x c ) ∥
Revised: f ( x ) ∥ f ( x ) ∥ (cid:19) T (cid:18)|C| (cid:88) x c ∈C f ( x c ) ∥

Original: f ( x c ) ∥ (cid:19) − (cid:18)
Revised: f ( x c ) ∥ (cid:19) − (cid:18)

Original: f ( x ) ∥ f ( x ) ∥ (cid:19) T E x v ∼D foil (cid:20) f
Revised: f ( x ) ∥ f ( x ) ∥ (cid:19) T E x v ∼D foil (cid:20) f

Original: ( x v )
Revised: ( x v )

Original: ∥ x v ∥ (cid:21) (H.16) =
Revised: ∥ x v ∥ (cid:21) (H.16) =

Original: (cid:18)
Revised: (cid:18)

Original: f ( x ) ∥ f ( x ) ∥ (cid:19) T (cid:18)|C| (cid:88) x c ∈C f ( x c ) ∥
Revised: f ( x ) ∥ f ( x ) ∥ (cid:19) T (cid:18)|C| (cid:88) x c ∈C f ( x c ) ∥

Original: f ( x c ) ∥ − E x v ∼D foil (cid:20) f
Revised: f ( x c ) ∥ − E x v ∼D foil (cid:20) f

Original: ( x v ) ∥
Revised: ( x v ) ∥

Original: f ( x v ) ∥ (cid:21)(cid:19) .
Revised: f ( x v ) ∥ (cid:21)(cid:19) .

Original: (H.17)
Revised: (H.17)

Original: H. P ROOF OF P ROPOSITION F. 
Revised: H. P ROOF OF P ROPOSITION F.

Original: Proof.
Revised: Proof.

Original: By Mercer’s Theorem (Mercer, 1909) (see Ghojogh et al.
Revised: By Mercer’s Theorem (Mercer, 1909) (see Ghojogh et al.

Original: (2021) for an accessible survey relevant to machine learning)
Revised: (2021) for an accessible survey relevant to machine learning)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Similarly
Revised: Similarly

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: H. P ROOF OF P ROPOSITION G. 
Revised: H. P ROOF OF P ROPOSITION G.

Original: Proof.
Revised: Proof.

Original: Because the sub-corpora are disjoint sets
Revised: Because the sub-corpora are disjoint sets

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Similarly,
Revised: Similarly,

Original: [Equation]
Revised: [Equation]

Original: I E XPERIMENT DETAILSI. D ATASETS
Revised: I E XPERIMENT DETAILS

Revised: Code is available at https://github.com/suinleelab/cl-explainability .

Revised: I. D ATASETS

Original: ImageNet.
Revised: ImageNet.

Original: The ImageNet ILSVRC dataset contains 1.2 million labeled training images and 50, labeled validation images over 1,000 object classes (Russakovsky et al., 2015).
Revised: The ImageNet ILSVRC dataset contains 1.2 million labeled training images and 50, labeled validation images over 1,000 object classes (Russakovsky et al., 2015).

Original: Since the SimCLR model and its downstream linear classifier are trained and tuned with only the training set, we use the validation set as a held-out set for quantitative evaluation (Section 4.1) and understanding data augmentations (Section 4.2).
Revised: Since the SimCLR model and its downstream linear classifier are trained and tuned with only the training set, we use the validation set as a held-out set for quantitative evaluation (Section 4.1) and understanding data augmentations (Section 4.2).

Original: For computational feasibility, a subset of 10 easily classified classes called ImageNette (including tench, English springer, cassette player, chainsaw, church, French horn, garbage truck, gas pump, golf ball, and parachute) (Howard, 2019) are used for quantitative evaluation.
Revised: For computational feasibility, a subset of 10 easily classified classes called ImageNette (including tench, English springer, cassette player, chainsaw, church, French horn, garbage truck, gas pump, golf ball, and parachute) (Howard, 2019) are used for quantitative evaluation.

Original: CIFAR-10.
Revised: CIFAR-10.

Original: The CIFAR-10 dataset consists of 50,000 labeled training images and 10,000 labeled test images of size 32 × 32 over 10 classes.
Revised: The CIFAR-10 dataset consists of 50,000 labeled training images and 10,000 labeled test images of size 32 × 32 over 10 classes.

Original: The test set is used for quantitative evaluation.
Revised: The test set is used for quantitative evaluation.

Original: MURA.
Revised: MURA.

Original: The MURA (MUsculoskeletal RAdiographs) dataset contains 36,808 training radiograph images from 13,457 musculoskeletal studies and 3,197 validation images from 1,199 studies.
Revised: The MURA (MUsculoskeletal RAdiographs) dataset contains 36,808 training radiograph images from 13,457 musculoskeletal studies and 3,197 validation images from 1,199 studies.

Original: Each study and its associated images are labeled by radiologists as either normal or abnormal.
Revised: Each study and its associated images are labeled by radiologists as either normal or abnormal.

Original: We further split the official training images into an 80% subset for our model training and a 20% subset for hyperparameter tuning.
Revised: We further split the official training images into an 80% subset for our model training and a 20% subset for hyperparameter tuning.

Original: The official validation set is held out and used for quantitative evaluation.
Revised: The official validation set is held out and used for quantitative evaluation.

Original: I. M ODELS
Revised: I. M ODELS

Original: SimCLR.
Revised: SimCLR.

Original: We obtained the weights of SimCLRv1 (with a ResNet50 backbone) and its downstream linear classifier, pre-trained with ImageNet, from https://github.com/ google-research/simclr by Chen et al.
Revised: We obtained the weights of SimCLRv1 (with a ResNet50 backbone) and its downstream linear classifier, pre-trained with ImageNet, from https://github.com/ google-research/simclr by Chen et al.

Original: The pre-trained SimCLR model and linear classifier were converted to the PyTorch format following the conversion in https://github. com/tonylins/simclr-converter , as recommended in the SimCLR authors’ official documentation.
Revised: The pre-trained SimCLR model and linear classifier were converted to the PyTorch format following the conversion in https://github. com/tonylins/simclr-converter , as recommended in the SimCLR authors’ official documentation.

Original: SimSiam.
Revised: SimSiam.

Original: We trained a SimSiam model (with a ResNet18 backbone) for CIFAR10 following the training procedure and hyperparameters outlined in Chen & He (2021), using the implementation in https://github.com/Reza-Safdari/SimSiam-91.
Revised: We trained a SimSiam model (with a ResNet18 backbone) for CIFAR10 following the training procedure and hyperparameters outlined in Chen & He (2021), using the implementation in https://github.com/Reza-Safdari/SimSiam-91.

Original: 9-top1-acc-on-CIFAR10 .
Revised: 9-top1-acc-on-CIFAR10 .

Original: The downstream linear classifier was trained with stochastic gradient descent with a learning rate of 30 . 0 and a batch size of 256 to achieve a top-1 accuracy of 92.06% on the CIFAR-10 test set.
Revised: The downstream linear classifier was trained with stochastic gradient descent with a learning rate of 30 . 0 and a batch size of 256 to achieve a top-1 accuracy of 92.06% on the CIFAR-10 test set.

Original: ResNet classifier.
Revised: ResNet classifier.

Original: We trained a ResNet18 classifier for normal vs. abnormal radiograph images with an 80% training subset and tuned hyperparameters with the other 20%.
Revised: We trained a ResNet18 classifier for normal vs. abnormal radiograph images with an 80% training subset and tuned hyperparameters with the other 20%.

Original: We randomly augmented the  flipping,  flipping, and rotation 
Revised: We randomly augmented the data during training by horizontal flipping, vertical flipping, and rotation between − 30 ◦ and 30 ◦ .

Original: The classifier was trained with an Adam optimizer (Kingma & Ba, 2014), with a batch size of 256 , a learning rate decay of 0 .
Revised: The classifier was trained with an Adam optimizer (Kingma & Ba, 2014), with a batch size of 256 , a learning rate decay of 0 .

Original: 1 every 10 steps, and a weight decay of 0 .
Revised: 1 every 10 steps, and a weight decay of 0 .

Original: The initial learning rate was tuned over { 0 . 1 , 0 . 01 , 0 . 001 , 0 . 0001 } to identify the optimal initial learning rate of 0 .
Revised: The initial learning rate was tuned over { 0 . 1 , 0 . 01 , 0 . 001 , 0 . 0001 } to identify the optimal initial learning rate of 0 .

Original: The trained ResNet18 classifier achieves an accuracy of 80.54% on the held-out official validation set.
Revised: The trained ResNet18 classifier achieves an accuracy of 80.54% on the held-out official validation set.

Original: I. F EATURE ATTRIBUTION METHODS
Revised: I. F EATURE ATTRIBUTION METHODS

Original: Integrated Gradients.
Revised: Integrated Gradients.

Original: Integrated Gradients (Sundararajan et al., 2017) with 50 steps for the Riemman approximation of integral were run in this work.
Revised: Integrated Gradients (Sundararajan et al., 2017) with 50 steps for the Riemman approximation of integral were run in this work.

Original: The implementation in the Captum package was used (Kokhlikyan et al., 2020).
Revised: The implementation in the Captum package was used (Kokhlikyan et al., 2020).

Original: The baseline values correspond to the explicand image with blurring.
Revised: The baseline values correspond to the explicand image with blurring.

Original: GradientSHAP.
Revised: GradientSHAP.

Original: GradientSHAP with 50 random Gaussian noise samples having a standard deviation of 0 . 2 for the gradient expectation estimation was run.
Revised: GradientSHAP with 50 random Gaussian noise samples having a standard deviation of 0 . 2 for the gradient expectation estimation was run.

Original: The implementation in the Captum package was used (Kokhlikyan et al., 2020).
Revised: The implementation in the Captum package was used (Kokhlikyan et al., 2020).

Original: The baseline values correspond to the explicand image with blurring.
Revised: The baseline values correspond to the explicand image with blurring.

Original: RISE.
Revised: RISE.

Original: RISE (Petsiuk et al., 2018) with 5000 random masks generated with a masking probability of 0 .
Revised: RISE (Petsiuk et al., 2018) with 5000 random masks generated with a masking probability of 0 .

Original: 5 was run.
Revised: 5 was run.

Original: For CIFAR-10, each binary mask before upsampling had size 4 × 4 .
Revised: For CIFAR-10, each binary mask before upsampling had size 4 × 4 .

Original: For ImageNet and MURA, each initial binary mask was 7 × 7 .
Revised: For ImageNet and MURA, each initial binary mask was 7 × 7 .

Original: The baseline values for replacing masked pixels correspond to the explicand image with blurring.
Revised: The baseline values for replacing masked pixels correspond to the explicand image with blurring.

Original: J C ORPUS SIZE SENSITIVITY ANALYSIS
Revised: J C ORPUS SIZE SENSITIVITY ANALYSIS

Original: K F OIL SIZE SENSITIVITY ANALYSIS
Revised: K F OIL SIZE SENSITIVITY ANALYSIS

Original: L S ANITY CHECK RESULTS
Revised: L S ANITY CHECK RESULTS

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: M A DDITIONAL QUANTITATIVE EVALUATION RESULTS WITH DOT PRODUCT SIMILARITY
Revised: M A DDITIONAL QUANTITATIVE EVALUATION RESULTS WITH DOT PRODUCT SIMILARITY

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: N Q UANTITATIVE EVALUATION RESULTS WITH MODEL PARAMETER RANDOMIZATION .
Revised: N Q UANTITATIVE EVALUATION RESULTS WITH MODEL PARAMETER RANDOMIZATION .

Revised: It has been shown that feature attribution methods, especially those based on gradients, can be undesirably invariant to model parameter randomization, suggesting that the attributions are not related to the trained models (Adebayo et al., 2018).

Revised: We performed additional experiments to show that our empirical evaluation results do not suffer from this issue.

Revised: We generated COCOA scores with randomized models, where the parameters are randomized by sampling from truncated normal distributions (as in Adebayo et al.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: and then evaluated the performance of these COCOA scores. As shown in Tables 9-12, the performance results of COCOA with randomized models are different from and worse than the performance of COCOA with trained models. These results suggest that COCOA performance is indeed sensitive to model parameter randomization and is related to model parameters.

Original: [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: O
Revised: O

Revised: A DDITIONAL RESULTS FOR UNDERSTANDING DATA AUGMENTATIONS IN S IM CLR

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]