Original: C ONDITIONAL C ONTRASTIVE L EARNING WITH K ERNEL
Revised: C ONDITIONAL C ONTRASTIVE L EARNING WITH K ERNEL

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Conditional contrastive learning frameworks consider the conditional sampling procedure that constructs positive or negative data pairs conditioned on speciﬁc variables.
Revised: Conditional contrastive learning frameworks consider the conditional sampling procedure that constructs positive or negative data pairs conditioned on speciﬁc variables.

Original: Fair contrastive learning constructs negative pairs, for example, from the same gender (conditioning on sensitive information), which in turn reduces undesirable information from the learned representations; weakly supervised contrastive learning constructs positive pairs with similar annotative attributes (conditioning on auxiliary information), which in turn are incorporated into the representations.
Revised: Fair contrastive learning constructs negative pairs, for example, from the same gender (conditioning on sensitive information), which in turn reduces undesirable information from the learned representations; weakly supervised contrastive learning constructs positive pairs with similar annotative attributes (conditioning on auxiliary information), which in turn are incorporated into the representations.

Original: Although conditional contrastive learning enables many applications, the conditional sampling procedure can be challenging if we cannot obtain sufﬁcient data pairs for some values of the conditioning variable.
Revised: Although conditional contrastive learning enables many applications, the conditional sampling procedure can be challenging if we cannot obtain sufﬁcient data pairs for some values of the conditioning variable.

Original: This paper presents Conditional
Revised: This paper presents Conditional

Original: Contrastive Learning with Kernel (CCL-K) that converts existing conditional contrastive objectives into alternative forms that mitigate the insufﬁcient data problem.
Revised: Contrastive Learning with Kernel (CCL-K) that converts existing conditional contrastive objectives into alternative forms that mitigate the insufﬁcient data problem.

Original: Instead of sampling data according to the value of the conditioning variable, CCLK uses the Kernel Conditional
Revised: Instead of sampling data according to the value of the conditioning variable, CCLK uses the Kernel Conditional

Original: Embedding Operator that samples data from all available data and assigns weights to each sampled data given the kernel similarity between the values of the conditioning variable.
Revised: Embedding Operator that samples data from all available data and assigns weights to each sampled data given the kernel similarity between the values of the conditioning variable.

Original: We conduct experiments using weakly supervised, fair, and hard negatives contrastive learning, showing CCL-K outperforms state-of-the-art baselines.
Revised: We conduct experiments using weakly supervised, fair, and hard negatives contrastive learning, showing CCL-K outperforms state-of-the-art baselines.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Contrastive learning algorithms (Oord et al., 2018; Chen et al., 2020;
Revised: Contrastive learning algorithms (Oord et al., 2018; Chen et al., 2020;

Original: He et al., 2020; Khosla et al., 2020) learn similar representations for positively-paired data and dissimilar representations for negatively-paired data.
Revised: He et al., 2020; Khosla et al., 2020) learn similar representations for positively-paired data and dissimilar representations for negatively-paired data.

Original: For instance, self-supervised visual contrastive learning (Hjelm et al., 2018) deﬁne two views of the same image (applying different image augmentations to each view) as a positive pair and different images as a negative pair.
Revised: For instance, self-supervised visual contrastive learning (Hjelm et al., 2018) deﬁne two views of the same image (applying different image augmentations to each view) as a positive pair and different images as a negative pair.

Original: Supervised contrastive learning (Khosla et al., 2020) deﬁnes data with the same labels as a positive pair and data with different labels as a negative pair.
Revised: Supervised contrastive learning (Khosla et al., 2020) deﬁnes data with the same labels as a positive pair and data with different labels as a negative pair.

Original: We see that distinct contrastive approaches consider different positive pairs and negative pairs constructions according to their learning goals.
Revised: We see that distinct contrastive approaches consider different positive pairs and negative pairs constructions according to their learning goals.

Original: In conditional contrastive learning, positive and negative pairs are constructed conditioned on speciﬁc variables.
Revised: In conditional contrastive learning, positive and negative pairs are constructed conditioned on speciﬁc variables.

Original: The conditioning variables can be downstream labels (Khosla et al., 2020), sensitive attributes (Tsai et al., 2021c), auxiliary information (Tsai et al., 2021a), or data embedding features (Robinson et al., 2020; Wu et al., 2020).
Revised: The conditioning variables can be downstream labels (Khosla et al., 2020), sensitive attributes (Tsai et al., 2021c), auxiliary information (Tsai et al., 2021a), or data embedding features (Robinson et al., 2020; Wu et al., 2020).

Original: For example, in fair contrastive learning (Tsai et al., 2021c), conditioning on variables such as gender or race, is performed to remove undesirable information from the learned representations.
Revised: For example, in fair contrastive learning (Tsai et al., 2021c), conditioning on variables such as gender or race, is performed to remove undesirable information from the learned representations.

Original: Conditioning is achieved by constructing negative pairs from the same gender.
Revised: Conditioning is achieved by constructing negative pairs from the same gender.

Original: As a second example, in weakly-supervised contrastive learning (Tsai et al., 2021a), the aim is to include extra information in the learned representations.
Revised: As a second example, in weakly-supervised contrastive learning (Tsai et al., 2021a), the aim is to include extra information in the learned representations.

Original: This extra information could be, for example, some freely available attributes for images collected from social media.
Revised: This extra information could be, for example, some freely available attributes for images collected from social media.

Original: The conditioning is performed by constructing positive pairs with similar annotative attributes.
Revised: The conditioning is performed by constructing positive pairs with similar annotative attributes.

Original: The cornerstone of conditional contrastive learning is the conditional sampling procedure : efﬁciently constructing positive or negative pairs while properly enforcing conditioning.
Revised: The cornerstone of conditional contrastive learning is the conditional sampling procedure : efﬁciently constructing positive or negative pairs while properly enforcing conditioning.

Original: The conditional sampling procedure requires access to sufﬁcient data for each state of the conditioning variables.
Revised: The conditional sampling procedure requires access to sufﬁcient data for each state of the conditioning variables.

Original: For example, if we are conditioning on the “age” attribute to reduce the age bias, then the conditional sampling procedure will work best if we can create a sufﬁcient number of data pairs for each age group.
Revised: For example, if we are conditioning on the “age” attribute to reduce the age bias, then the conditional sampling procedure will work best if we can create a sufﬁcient number of data pairs for each age group.

Original: However, in many real-world situations, some values of the conditioning variable may not have enough data points or even no data points at all.
Revised: However, in many real-world situations, some values of the conditioning variable may not have enough data points or even no data points at all.

Original: The sampling problem exacerbates when the conditioning variable is continuous.
Revised: The sampling problem exacerbates when the conditioning variable is continuous.

Original: In this paper, we introduce Conditional Contrastive Learning with Kernel (CCL-K), to help mitigate the problem of insufﬁcient data, by providing an alternative formulation using similarity kernels (see Figure 1).
Revised: In this paper, we introduce Conditional Contrastive Learning with Kernel (CCL-K), to help mitigate the problem of insufﬁcient data, by providing an alternative formulation using similarity kernels (see Figure 1).

Original: Given a speciﬁc value of the conditioning variable, instead of sampling data that are exactly associated with this speciﬁc value, we can also sample data that have similar values of the conditioning variable.
Revised: Given a speciﬁc value of the conditioning variable, instead of sampling data that are exactly associated with this speciﬁc value, we can also sample data that have similar values of the conditioning variable.

Original: We leverage the Kernel Conditional
Revised: We leverage the Kernel Conditional

Original: Embedding Operator (Song et al., 2013) for the sampling process, which considers kernels (Schölkopf et al., 2002) to measure the similarities between the values of the conditioning variable.
Revised: Embedding Operator (Song et al., 2013) for the sampling process, which considers kernels (Schölkopf et al., 2002) to measure the similarities between the values of the conditioning variable.

Original: This new formulation with a weighting scheme based on similarity kernel allows us to use all training data when conditionally creating positive and negative pairs.
Revised: This new formulation with a weighting scheme based on similarity kernel allows us to use all training data when conditionally creating positive and negative pairs.

Original: It also enables contrastive learning to use continuous conditioning variables.
Revised: It also enables contrastive learning to use continuous conditioning variables.

Original: To study the generalization of CCL-K, we conduct experiments on three tasks.
Revised: To study the generalization of CCL-K, we conduct experiments on three tasks.

Original: The ﬁrst task is weakly supervised contrastive learning , which incorporates auxiliary information by conditioning on the annotative attribute to improve the downstream task performance.
Revised: The ﬁrst task is weakly supervised contrastive learning , which incorporates auxiliary information by conditioning on the annotative attribute to improve the downstream task performance.

Original: For the second task, fair contrastive learning , we condition on the sensitive attribute to remove its information in the representations.
Revised: For the second task, fair contrastive learning , we condition on the sensitive attribute to remove its information in the representations.

Original: The last task is hard negative contrastive learning , which samples negative pairs to learn dissimilar representations where the negative pairs have similar outcomes from the conditioning variable.
Revised: The last task is hard negative contrastive learning , which samples negative pairs to learn dissimilar representations where the negative pairs have similar outcomes from the conditioning variable.

Original: We compare CCL-K with the baselines tailored for each of the three tasks, and CCL-K outperforms all baselines on downstream evaluations.
Revised: We compare CCL-K with the baselines tailored for each of the three tasks, and CCL-K outperforms all baselines on downstream evaluations.

Original:  C ONDITIONAL S AMPLING IN C ONTRASTIVE L EARNING
Revised:  C ONDITIONAL S AMPLING IN C ONTRASTIVE L EARNING

Original: In Section 2.1, we ﬁrst present the technical preliminaries of contrastive learning.
Revised: In Section 2.1, we ﬁrst present the technical preliminaries of contrastive learning.

Original: Next, we introduce the conditional sampling procedure in Section 2.2, showing that it instantiates recent conditional contrastive frameworks.
Revised: Next, we introduce the conditional sampling procedure in Section 2.2, showing that it instantiates recent conditional contrastive frameworks.

Original: Last, in Section 2.3, we discuss the limitation of insufﬁcient data in the current framework, presenting to convert existing objectives into alternative forms with kernels to alleviate the limitation.
Revised: Last, in Section 2.3, we discuss the limitation of insufﬁcient data in the current framework, presenting to convert existing objectives into alternative forms with kernels to alleviate the limitation.

Original: In the paper, we use uppercase letters (e.g., X ) for random variables, P · · for the distribution (e.g., P X denotes the distribution of X ), lowercase letters (e.g., x ) for the outcome from a variable, and the calligraphy letter (e.g., X ) for the sample space of a variable.
Revised: In the paper, we use uppercase letters (e.g., X ) for random variables, P · · for the distribution (e.g., P X denotes the distribution of X ), lowercase letters (e.g., x ) for the outcome from a variable, and the calligraphy letter (e.g., X ) for the sample space of a variable.

Original: 2. T ECHNICAL P RELIMINARIES - U NCONDITIONAL C ONTRASTIVE L EARNING
Revised: 2. T ECHNICAL P RELIMINARIES - U NCONDITIONAL C ONTRASTIVE L EARNING

Original: Contrastive methods learn similar representations for positive pairs and dissimilar representations for negative pairs (Chen et al., 2020;
Revised: Contrastive methods learn similar representations for positive pairs and dissimilar representations for negative pairs (Chen et al., 2020;

Original: He et al., 2020; Hjelm et al., 2018).
Revised: He et al., 2020; Hjelm et al., 2018).

Original: In prior literature (Oord et al., 2018;
Revised: In prior literature (Oord et al., 2018;

Original: Tsai et al., 2021b; Bachman et al., 2019; Hjelm et al., 2018), the construction of the positive and negative pairs can be understood as sampling from the joint distribution P XY and product of marginal P X P Y .
Revised: Tsai et al., 2021b; Bachman et al., 2019; Hjelm et al., 2018), the construction of the positive and negative pairs can be understood as sampling from the joint distribution P XY and product of marginal P X P Y .

Original: To see this, we begin by reviewing one popular contrastive approach, the InfoNCE objective (Oord et al., 2018):
Revised: To see this, we begin by reviewing one popular contrastive approach, the InfoNCE objective (Oord et al., 2018):

Original: [Equation]
Revised: [Equation]

Original: where X and Y represent the data and x is the anchor data.
Revised: where X and Y represent the data and x is the anchor data.

Original: ( x, y pos ) are positively-paired and are sampled from P XY ( x and y are different views to each other; e.g., augmented variants of the sameimage), and { ( x, y neg , i ) } ni =1 are negatively-paired and are sampled from P X P Y (e.g., x and y are two random images). f ( · , · ) deﬁnes a mapping X × Y → R , which is parameterized via neural nets (Chen et al., 2020) as: [Equation]
Revised: ( x, y pos ) are positively-paired and are sampled from P XY ( x and y are different views to each other; e.g., augmented variants of the same

Original: Framework
Revised: Framework

Original: Conditioning Variable Z
Revised: Conditioning Variable Z

Original: Positive Pairs from
Revised: Positive Pairs from

Original: Negative Pairs from
Revised: Negative Pairs from

Original: Weakly Supervised (Tsai et al., 2021a)
Revised: Weakly Supervised (Tsai et al., 2021a)

Original: Auxiliary information
Revised: Auxiliary information

Original: P X | Z = z P Y | Z
Revised: P X | Z = z P Y | Z

Original: = z
Revised: = z

Original: P X P Y
Revised: P X P Y

Original: Fair (Tsai et al., 2021c)
Revised: Fair (Tsai et al., 2021c)

Original: Sensitive information
Revised: Sensitive information

Original: P XY | Z =
Revised: P XY | Z =

Original: z
Revised: z

Original: P X | Z = z P Y
Revised: P X | Z = z P Y

Original: | Z
Revised: | Z

Original: =
Revised: =

Original: z
Revised: z

Original: Hard Negatives (Wu et al., 2020)
Revised: Hard Negatives (Wu et al., 2020)

Original: Feature embedding of X
Revised: Feature embedding of X

Original: P XY
Revised: P XY

Original: P X | Z = z P Y | Z 
Revised: P X | Z = z P Y | Z = z image), and { ( x, y neg , i ) } ni =1 are negatively-paired and are sampled from P X P Y (e.g., x and y are two random images). f ( · , · ) deﬁnes a mapping X × Y → R , which is parameterized via neural nets (Chen et al., 2020) as:

Original: = z where g θ X ( x ) and g θ Y ( y ) are embedded features, g θ X
Revised: [Equation] 2) where g θ X ( x ) and g θ Y ( y ) are embedded features, g θ X

Original: and g θ Y are neural networks ( g θ X can be the same as g θ Y ) parameterized by parameters θ X
Revised: and g θ Y are neural networks ( g θ X can be the same as g θ Y ) parameterized by parameters θ X

Original: and θ Y , and τ is a hyper-parameter that rescales the score from the cosine similarity.
Revised: and θ Y , and τ is a hyper-parameter that rescales the score from the cosine similarity.

Original: The InfoNCE objective aims to maximize the similarity score between a data pair sampled from the joint distribution (i.e., ( x, y pos ) ∼ P XY ) and minimize the similarity score between a data pair sampled from the product of marginal distribution (i.e., ( x, y neg ) ∼ P X P Y ) (Tschannen et al., 2019).
Revised: The InfoNCE objective aims to maximize the similarity score between a data pair sampled from the joint distribution (i.e., ( x, y pos ) ∼ P XY ) and minimize the similarity score between a data pair sampled from the product of marginal distribution (i.e., ( x, y neg ) ∼ P X P Y ) (Tschannen et al., 2019).

Original: 2. C ONDITIONAL C ONTRASTIVE L EARNING
Revised: 2. C ONDITIONAL C ONTRASTIVE L EARNING

Original: Recent literature (Robinson et al., 2020; Tsai et al., 2021a;c) has modiﬁed the InfoNCE objective to achieve different learning goals by sampling positive or negative pairs under conditioning variable Z (and its outcome z ).
Revised: Recent literature (Robinson et al., 2020; Tsai et al., 2021a;c) has modiﬁed the InfoNCE objective to achieve different learning goals by sampling positive or negative pairs under conditioning variable Z (and its outcome z ).

Original: These different conditional contrastive learning frameworks have one common technical challenge: the conditional sampling procedure.
Revised: These different conditional contrastive learning frameworks have one common technical challenge: the conditional sampling procedure.

Original: The conditional sampling procedure samples the positive or negative data pairs from the product of conditional distribution: ( x, y ) ∼
Revised: The conditional sampling procedure samples the positive or negative data pairs from the product of conditional distribution: ( x, y ) ∼

Original: P X | Z = z P Y | Z
Revised: P X | Z = z P Y | Z

Original: = z , where x and y are sampled given the same outcome from the conditioning variable (e.g., two random images with blue sky background, when selecting z as blue sky background).
Revised: = z , where x and y are sampled given the same outcome from the conditioning variable (e.g., two random images with blue sky background, when selecting z as blue sky background).

Original: We summarize how different frameworks use the conditional sampling procedure in Table 1.
Revised: We summarize how different frameworks use the conditional sampling procedure in Table 1.

Original: Weakly Supervised Contrastive Learning.
Revised: Weakly Supervised Contrastive Learning.

Original: Tsai et al.
Revised: Tsai et al.

Original: (2021a) consider the auxiliary information from data (e.g., annotation attributes of images) as a weak supervision signal and propose a contrastive objective to incorporate the weak supervision in the representations.
Revised: (2021a) consider the auxiliary information from data (e.g., annotation attributes of images) as a weak supervision signal and propose a contrastive objective to incorporate the weak supervision in the representations.

Original: This work is motivated by the argument that the auxiliary information implies semantic similarities.
Revised: This work is motivated by the argument that the auxiliary information implies semantic similarities.

Original: With this motivation, the weakly supervised contrastive learning framework learns similar representations for data with the same auxiliary information and dissimilar representations for data with different auxiliary information.
Revised: With this motivation, the weakly supervised contrastive learning framework learns similar representations for data with the same auxiliary information and dissimilar representations for data with different auxiliary information.

Original: Embracing this idea, the original InfoNCE objective can be modiﬁed into the weakly supervised
Revised: Embracing this idea, the original InfoNCE objective can be modiﬁed into the weakly supervised

Original: InfoNCE (abbreviated as WeaklySup-InfoNCE) objective:
Revised: InfoNCE (abbreviated as WeaklySup-InfoNCE) objective:

Original: [Equation]
Revised: [Equation]

Original: Here Z is the conditioning variable representing the auxiliary information from data, and z is the outcome of auxiliary information that we sample from P Z .
Revised: Here Z is the conditioning variable representing the auxiliary information from data, and z is the outcome of auxiliary information that we sample from P Z .

Original: ( x, y pos ) are positive pairs sampled from P X | Z
Revised: ( x, y pos ) are positive pairs sampled from P X | Z

Original: = z P Y | Z
Revised: = z P Y | Z

Original: = z .
Revised: = z .

Original: In this design, the positive pairs always have the same outcome from the conditioning variable.
Revised: In this design, the positive pairs always have the same outcome from the conditioning variable.

Original: { ( x, y neg , i ) } ni =1 are negative pairs that are sampled from P X P Y .
Revised: { ( x, y neg , i ) } ni =1 are negative pairs that are sampled from P X P Y .

Original: Fair Contrastive Learning.
Revised: Fair Contrastive Learning.

Original: Another recent work (Tsai et al., 2021c) presented to remove undesirable sensitive information (such as gender) in the representation, by sampling negative pairs conditioning on sensitive attributes.
Revised: Another recent work (Tsai et al., 2021c) presented to remove undesirable sensitive information (such as gender) in the representation, by sampling negative pairs conditioning on sensitive attributes.

Original: The paper argues that ﬁxing the outcome of the sensitive variable prevents the model from using the sensitive information to distinguish positive pairs from negative pairs (since all positive and negative samples share the same outcome), and the model will ignore the effect of the sensitive attribute during contrastive learning.
Revised: The paper argues that ﬁxing the outcome of the sensitive variable prevents the model from using the sensitive information to distinguish positive pairs from negative pairs (since all positive and negative samples share the same outcome), and the model will ignore the effect of the sensitive attribute during contrastive learning.

Original: Embracing this idea, the original InfoNCE objective can be modiﬁed into the Fair-InfoNCE objective:
Revised: Embracing this idea, the original InfoNCE objective can be modiﬁed into the Fair-InfoNCE objective:

Original: [Equation]
Revised: [Equation]

Original: Here Z is the conditioning variable representing the sensitive information (e.g., gender), z is the outcome of the sensitive information (e.g., female), and the anchor data x is associated with z (e.g., a data point that has the gender attribute being female).
Revised: Here Z is the conditioning variable representing the sensitive information (e.g., gender), z is the outcome of the sensitive information (e.g., female), and the anchor data x is associated with z (e.g., a data point that has the gender attribute being female).

Original: ( x, y pos ) are positively-paired that are sampled from P XY | Z
Revised: ( x, y pos ) are positively-paired that are sampled from P XY | Z

Original: = z , and x and y pos are constructed to have the same z .
Revised: = z , and x and y pos are constructed to have the same z .

Original: { ( x, y neg , i ) } ni =1 are negativelypaired that are sampled from P X | Z
Revised: { ( x, y neg , i ) } ni =1 are negativelypaired that are sampled from P X | Z

Original: = z P Y | Z
Revised: = z P Y | Z

Original: = z .
Revised: = z .

Original: In this design, the positively-paired samples and the negatively-paired samples are always having the same outcome from the conditioning variable.
Revised: In this design, the positively-paired samples and the negatively-paired samples are always having the same outcome from the conditioning variable.

Original: Hard-negative Contrastive Learning.
Revised: Hard-negative Contrastive Learning.

Original: Robinson et al.
Revised: Robinson et al.

Original: (2020) and Kalantidis et al.
Revised: (2020) and Kalantidis et al.

Original: (2020) argue that contrastive learning can beneﬁt from hard negative samples (i.e., samples y that are difﬁcult to distinguish from an anchor x ).
Revised: (2020) argue that contrastive learning can beneﬁt from hard negative samples (i.e., samples y that are difﬁcult to distinguish from an anchor x ).

Original: Rather than considering two arbitrary data as negatively-paired, these methods construct a negative data pair from two random data that are not too far from each other 1 .
Revised: Rather than considering two arbitrary data as negatively-paired, these methods construct a negative data pair from two random data that are not too far from each other 1 .

Original: Embracing this idea, the original InfoNCE objective is modiﬁed into the Hard Negative InfoNCE (abbreviated as HardNeg-InfoNCE) objective:
Revised: Embracing this idea, the original InfoNCE objective is modiﬁed into the Hard Negative InfoNCE (abbreviated as HardNeg-InfoNCE) objective:

Original: [Equation]
Revised: [Equation]

Original: Here Z is the conditioning variable representing the embedding feature of X , in particular z
Revised: Here Z is the conditioning variable representing the embedding feature of X , in particular z

Original: = g θ X ( x ) (see deﬁnition in equation 2, we refer g θ X ( x ) as the embedding feature of x and g θ Y ( y ) as the embedding feature of y ).
Revised: = g θ X ( x ) (see deﬁnition in equation 2, we refer g θ X ( x ) as the embedding feature of x and g θ Y ( y ) as the embedding feature of y ).

Original: ( x, y pos ) are positively-paired that are sampled from P XY .
Revised: ( x, y pos ) are positively-paired that are sampled from P XY .

Original: To construct negative pairs { ( x, y neg ,i ) } ni =1 , we sample { ( y neg , i ) } ni =1 from P Y | Z
Revised: To construct negative pairs { ( x, y neg ,i ) } ni =1 , we sample { ( y neg , i ) } ni =1 from P Y | Z

Original: = z = g θX ( x ) .
Revised: = z = g θX ( x ) .

Original: We realize the sampling from P Y | Z
Revised: We realize the sampling from P Y | Z

Original: = z = g θX ( x ) as sampling data points from Y whose embedding features are close to z=
Revised: = z = g θ X ( x ) as sampling data points from Y whose embedding features are close to z

Original:  g θ X ( x ) : sampling y such that g θ Y ( y ) is close to g θ X ( x ) .
Revised: = g θ X ( x ) : sampling y such that g θ Y ( y ) is close to g θ X ( x ) .

Original: 2. C ONDITIONAL C ONTRASTIVE L EARNING WITH K ERNEL
Revised: 2. C ONDITIONAL C ONTRASTIVE L EARNING WITH K ERNEL

Original: The conditional sampling procedure common to all these conditional contrastive frameworks has a limitation when we have insufﬁcient data points associated with some outcomes of the conditioning variable.
Revised: The conditional sampling procedure common to all these conditional contrastive frameworks has a limitation when we have insufﬁcient data points associated with some outcomes of the conditioning variable.

Original: In particular, given an anchor data x and its corresponding conditioning variable’s outcome z , if z is uncommon, then it will be challenging for us to sample y that is associated with z via y ∼ P Y | Z = z 2 .
Revised: In particular, given an anchor data x and its corresponding conditioning variable’s outcome z , if z is uncommon, then it will be challenging for us to sample y that is associated with z via y ∼ P Y | Z = z 2 .

Original: The insufﬁcient data problem can be more serious when the cardinality of the conditioning variable | Z | is large, which happens when Z contains many discrete values, or when
Revised: The insufﬁcient data problem can be more serious when the cardinality of the conditioning variable | Z | is large, which happens when Z contains many discrete values, or when

Original: Z is a continuous variable (cardinality | Z | = ∞ ).
Revised: Z is a continuous variable (cardinality | Z | = ∞ ).

Original: In light of this limitation, we present to convert these objectives into alternative forms that can avoid the need to sample data from P Y | Z and can retain the same functions as the original forms.
Revised: In light of this limitation, we present to convert these objectives into alternative forms that can avoid the need to sample data from P Y | Z and can retain the same functions as the original forms.

Original: We name this new family of formulations C onditional
Revised: We name this new family of formulations C onditional

Original: C ontrastive L earning with K ernel (CCL-K).
Revised: C ontrastive L earning with K ernel (CCL-K).

Original: High Level Intuition.
Revised: High Level Intuition.

Original: The high level idea of our method is that, instead of sampling y from
Revised: The high level idea of our method is that, instead of sampling y from

Original: P Y | Z
Revised: P Y | Z

Original: = z , we sample y from existing data of Y whose associated conditioning variable’s outcome is close to z .
Revised: = z , we sample y from existing data of Y whose associated conditioning variable’s outcome is close to z .

Original: For example, assuming the conditioning variable Z to be age and z to be 80 years old, instead of sampling the data points at the age of 80 directly, we sample from all data points, assigning highest weights to the data points at the age from 70 to 90 , given their proximity to 80 .
Revised: For example, assuming the conditioning variable Z to be age and z to be 80 years old, instead of sampling the data points at the age of 80 directly, we sample from all data points, assigning highest weights to the data points at the age from 70 to 90 , given their proximity to 80 .

Original: Our intuition is that data with similar outcomes from the conditioning variable should be used in support of the conditional sampling.
Revised: Our intuition is that data with similar outcomes from the conditioning variable should be used in support of the conditional sampling.

Original: Mathematically, instead of sampling from P Y
Revised: Mathematically, instead of sampling from P Y

Original: | Z =
Revised: | Z =

Original: z , we sample from a distribution proportional to the weighted sum (cid:80) Nj =1 w ( z j , z ) P Y | Z
Revised: z , we sample from a distribution proportional to the weighted sum (cid:80) Nj =1 w ( z j , z ) P Y | Z

Original: =
Revised: =

Original: z j , where w ( z j , z ) represents how similar in the space of of the conditioning variable Z .
Revised: z j , where w ( z j , z ) represents how similar in the space of of the conditioning variable Z .

Original: This similarity is computed for all data points j = 1 · · · N .
Revised: This similarity is computed for all data points j = 1 · · · N .

Original: In this paper, we use the Kernel Conditional
Revised: In this paper, we use the Kernel Conditional

Original: Embedding Operator (Song et al., 2013) for such approximation, where we represent the similarity using kernel (Schölkopf et al., 2002).
Revised: Embedding Operator (Song et al., 2013) for such approximation, where we represent the similarity using kernel (Schölkopf et al., 2002).

Original: Step I - Problem Setup.
Revised: Step I - Problem Setup.

Original: We want to avoid the conditional sampling procedure from existing conditional learning objectives (equation 3, 4, 5), and hence we are not supposed to have access to data pairs from the conditional distribution P X | Z P Y | Z .
Revised: We want to avoid the conditional sampling procedure from existing conditional learning objectives (equation 3, 4, 5), and hence we are not supposed to have access to data pairs from the conditional distribution P X | Z P Y | Z .

Original: Instead, the only given data will be a batch of triplets { ( x i , y i , z i ) } bi =1 , which are independently sampled from the joint distribution P ⊗ b XY Z with b being the batch size.
Revised: Instead, the only given data will be a batch of triplets { ( x i , y i , z i ) } bi =1 , which are independently sampled from the joint distribution P ⊗ b XY Z with b being the batch size.

Original: In particular, when ( x i , y i , z i ) ∼ P XY Z , ( x i , y i ) is a pair of data sampled from the joint distribution P XY (e.g., two augmented views of the same image) and z i is the associated conditioning variable’s outcome (e.g., the annotative attribute of the image).
Revised: In particular, when ( x i , y i , z i ) ∼ P XY Z , ( x i , y i ) is a pair of data sampled from the joint distribution P XY (e.g., two augmented views of the same image) and z i is the associated conditioning variable’s outcome (e.g., the annotative attribute of the image).

Original: To convert previous objectives into alternative forms that avoid the need of the conditional sampling procedure, we need to perform an estimation of the scoring function e f ( x,y ) for ( x, y ) ∼ P X | Z P Y | Z in equation 3, 4,given only { ( x i , y i , z i ) } bi =1 ∼ P ⊗ b XY Z .
Revised: To convert previous objectives into alternative forms that avoid the need of the conditional sampling procedure, we need to perform an estimation of the scoring function e f ( x,y ) for ( x, y ) ∼ P X | Z P Y | Z in equation 3, 4,given only { ( x i , y i , z i ) } bi =1 ∼ P ⊗ b XY Z .

Original: Step II - Kernel Formulation.
Revised: Step II - Kernel Formulation.

Original: We present to reformulate previous objectives into kernel expressions.
Revised: We present to reformulate previous objectives into kernel expressions.

Original: We denote
Revised: We denote

Original: K XY ∈ R
Revised: K XY ∈ R

Original: b × b as a kernel gram matrix between X and Y : let the i th row and j th column of K XY be the exponential scoring function e f ( x i ,y j )
Revised: b × b as a kernel gram matrix between X and Y : let the i th row and j th column of K XY be the exponential scoring function e f ( x i ,y j )

Original: (see equation 2) with [ K XY ] ij :=
Revised: (see equation 2) with [ K XY ] ij :=

Original: e f ( x i ,y j ) .
Revised: e f ( x i ,y j ) .

Original: K XY is a gram matrix because the design of e f ( x,y ) satisﬁes a kernel 3 between g θ X ( x ) and g θ Y ( y ) : 
Revised: K XY is a gram matrix because the design of e f ( x,y ) satisﬁes a kernel 3 between g θ X ( x ) and g θ Y ( y ) : e f ( x,y ) = exp (cid:16) cosine similarity( g [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where (cid:104)· , ·(cid:105) H is the inner product in a Reproducing Kernel Hilbert Space (RKHS)
Revised: (6) where (cid:104)· , ·(cid:105) H is the inner product in a Reproducing Kernel Hilbert Space (RKHS)

Original: H and φ is the corresponding feature map.
Revised: H and φ is the corresponding feature map.

Original: K XY can also be represented as K XY =
Revised: K XY can also be represented as K XY =

Original: Φ X Φ (cid:62) Y with Φ X = (cid:2) φ (cid:0) g θ X ( x 1 ) (cid:1) , · · · , φ (cid:0) g θ X ( x b ) (cid:1)(cid:3) (cid:62) and Φ Y = (cid:2) φ (cid:0) g θ Y ( y 1 ) (cid:1) , · · · , φ (cid:0) g θ Y ( y b ) (cid:1)(cid:3) (cid:62) .
Revised: Φ X Φ (cid:62) Y with Φ X = (cid:2) φ (cid:0) g θ X ( x 1 ) (cid:1) , · · · , φ (cid:0) g θ X ( x b ) (cid:1)(cid:3) (cid:62) and Φ Y = (cid:2) φ (cid:0) g θ Y ( y 1 ) (cid:1) , · · · , φ (cid:0) g θ Y ( y b ) (cid:1)(cid:3) (cid:62) .

Original: Similarly, we denote K Z ∈ R b × b as a kernel gram matrix for Z , where [ K Z ] ij represents the similarity between z i and z j : [ K Z ] ij :=
Revised: Similarly, we denote K Z ∈ R b × b as a kernel gram matrix for Z , where [ K Z ] ij represents the similarity between z i and z j : [ K Z ] ij :=

Original: (cid:10) γ ( z i ) , γ ( z j ) (cid:11) G , where γ ( · ) is an arbitrary kernel embedding for
Revised: (cid:10) γ ( z i ) , γ ( z j ) (cid:11) G , where γ ( · ) is an arbitrary kernel embedding for

Original: Z and G is its corresponding RKHS space.
Revised: Z and G is its corresponding RKHS space.

Original: K Z can also be represented as K Z = Γ Z Γ (cid:62) Z with Γ Z = (cid:2) γ ( z 1 ) , · · · , γ ( z b ) (cid:3) (cid:62) .
Revised: K Z can also be represented as K Z = Γ Z Γ (cid:62) Z with Γ Z = (cid:2) γ ( z 1 ) , · · · , γ ( z b ) (cid:3) (cid:62) .

Original: Step III - Kernel-based Scoring Function e f ( x,y ) Estimation.
Revised: Step III - Kernel-based Scoring Function e f ( x,y ) Estimation.

Original: We present the following:
Revised: We present the following:

Original: Deﬁnition 2.1 (Kernel Conditional
Revised: Deﬁnition 2.1 (Kernel Conditional

Original: Embedding Operator (Song et al., 2013)) .
Revised: Embedding Operator (Song et al., 2013)) .

Original: By Kernel
Revised: By Kernel

Original: Conditional Embedding Operator (Song et al., 2013), the ﬁnite-sample kernel estimation of
Revised: Conditional Embedding Operator (Song et al., 2013), the ﬁnite-sample kernel estimation of

Original: E y ∼ P Y | Z = z [Equation]
Revised: E y ∼ P Y | Z = z (cid:104) φ (cid:16) g θ Y ( y ) (cid:17)(cid:105) is Φ (cid:62) Y ( K Z

Original:  , where λ is a hyper-parameter.
Revised: + λ I ) − 1 Γ Z γ ( z ) , where λ is a hyper-parameter.

Original: Proposition 2.2 (Estimation of e f ( x i ,y ) when y ∼ P Y | Z
Revised: Proposition 2.2 (Estimation of e f ( x i ,y ) when y ∼ P Y | Z

Original: = z i ) .
Revised: = z i ) .

Original: Given { ( x i , y i , z i ) } bi =1 ∼ P ⊗ b XY Z , the ﬁnite-sample kernel estimation of e f ( x i ,y ) when y ∼ P Y | Z
Revised: Given { ( x i , y i , z i ) } bi =1 ∼ P ⊗ b XY Z , the ﬁnite-sample kernel estimation of e f ( x i ,y ) when y ∼ P Y | Z

Original: =
Revised: =

Original: z i is (cid:104) K XY
Revised: z i is (cid:104) K XY

Original: ( K Z + λ I ) − 1 K Z (cid:105) ii .
Revised: ( K Z + λ I ) − 1 K Z (cid:105) ii .

Original: (cid:104) K [Equation] K [Equation]
Revised: (cid:104) K XY ( K Z

Original: I

Original: [Equation] 1 K [Equation] (cid:105) 
Revised: + λ I ) − 1 K Z (cid:105) ii

Original: [Equation]

Original: = (cid:80) bj =1 w (
Revised: = (cid:80) bj =1 w (

Original: z j , z i )
Revised: z j , z i )

Original: e f ( x i ,y j ) with w ( z j , z i ) =
Revised: e f ( x i ,y j ) with w ( z j , z i ) =

Original: (cid:104)
Revised: (cid:104)

Original: ( K Z + λ I ) − 1 K Z (cid:105) ji .
Revised: ( K Z + λ I ) − 1 K Z (cid:105) ji .

Original: Proof.
Revised: Proof.

Original: For any Z = z , along with Deﬁnition 2.1, we estimate φ (cid:16) g θ Y ( y ) (cid:17) when y ∼ P Y | Z
Revised: For any Z = z , along with Deﬁnition 2.1, we estimate φ (cid:16) g θ Y ( y ) (cid:17) when y ∼ P Y | Z

Original: = z ≈
Revised: = z ≈

Original: E y ∼ P Y | Z = z (cid:104) φ (cid:16) g θ Y ( y ) (cid:17)(cid:105) ≈
Revised: E y ∼ P Y | Z = z (cid:104) φ (cid:16) g θ Y ( y ) (cid:17)(cid:105) ≈

Original: Φ (cid:62) Y ( K Z
Revised: Φ (cid:62) Y ( K Z

Original: + λ I ) − 1 Γ Z γ ( z ) .
Revised: + λ I ) − 1 Γ Z γ ( z ) .

Original: Then, we plug in the result for the data pair ( x i , z i ) to estimate e f ( x i ,y ) when y ∼ P Y | Z
Revised: Then, we plug in the result for the data pair ( x i , z i ) to estimate e f ( x i ,y ) when y ∼ P Y | Z

Original: = zi
Revised: = z

Revised: i : (cid:10) φ (cid:16) g

Original: [Equation]
Revised: [Equation]

Revised: X ( x i ) (cid:17) , Φ (cid:62)

Revised: [Equation]

Revised: [ K XY ] i ∗ ( K Z

Revised: + λ I )

Revised: [Equation]

Original:  [ K XY ( K Z
Revised: K Z ] ∗ i = [ K XY ] i [Equation]

Original:  + λ I ) − 1 K Z ] ii is the kernel estimation of e f ( x i ,y ) when ( x i , z i ) ∼ P XZ , y ∼ P Y | Z
Revised: (cid:2) [Equation] [ K XY ( K Z + λ I ) − 1 K Z ] ii is the kernel estimation of e f ( x i ,y ) when ( x i , z i ) ∼ P XZ , y ∼ P Y | Z

Original: = z i .
Revised: = z i .

Original: It deﬁnes the similarity between the data pair sampled from P X | Z
Revised: It deﬁnes the similarity between the data pair sampled from P X | Z

Original: =
Revised: =

Original: z i P Y | Z
Revised: z i P Y | Z

Original: = z i .
Revised: = z i .

Original: Hence, ( K Z + λ I ) − 1 K Z can be seen as a transformation applied on K XY ( K XY deﬁnes the similarity between X and
Revised: Hence, ( K Z + λ I ) − 1 K Z can be seen as a transformation applied on K XY ( K XY deﬁnes the similarity between X and

Original: Y ), converting unconditional to conditional similarities between X and Y (conditioning on Z ).
Revised: Y ), converting unconditional to conditional similarities between X and Y (conditioning on Z ).

Original: Proposition 2.2 also re-writes this estimation as a weighted sum over { e f ( x i ,y j ) } nj =1 with the weights w ( z j , z i ) = (cid:2) ( K Z + λ I ) − 1 K Z (cid:3) ji .
Revised: Proposition 2.2 also re-writes this estimation as a weighted sum over { e f ( x i ,y j ) } nj =1 with the weights w ( z j , z i ) = (cid:2) ( K Z + λ I ) − 1 K Z (cid:3) ji .

Original: We provide an illustration to compare ( K Z + λ I ) − 1 K Z and K Z in Figure 2, showing that ( K Z + λ I ) − 1 K Z can be seen as a smoothed version of K Z , suggesting the weight (cid:2) ( K Z + λ I ) − 1 K Z (cid:3) ji captures the similarity between ( z j , z i ) .
Revised: We provide an illustration to compare ( K Z + λ I ) − 1 K Z and K Z in Figure 2, showing that ( K Z + λ I ) − 1 K Z can be seen as a smoothed version of K Z , suggesting the weight (cid:2) ( K Z + λ I ) − 1 K Z (cid:3) ji captures the similarity between ( z j , z i ) .

Original: To conclude, we use the Kernel
Revised: To conclude, we use the Kernel

Original: Conditional Embedding Operator (Song et al., 2013) to avoid explicitly sampling y ∼ P Y | Z
Revised: Conditional Embedding Operator (Song et al., 2013) to avoid explicitly sampling y ∼ P Y | Z

Original: = z , which alleviates the limitation of having insufﬁcient data from Y that are associated with z .
Revised: = z , which alleviates the limitation of having insufﬁcient data from Y that are associated with z .

Original: It is worth noting that our method neither generates raw data directly nor includes additional training.
Revised: It is worth noting that our method neither generates raw data directly nor includes additional training.

Original: In terms of computational complexity, calculating the inverse ( K Z + λI ) − 1 costs O ( b 3 ) where b is the batch size, or O ( b 2 . 376 ) using more efﬁcient inverse algorithms like Coppersmith and Winograd (1987).
Revised: In terms of computational complexity, calculating the inverse ( K Z + λI ) − 1 costs O ( b 3 ) where b is the batch size, or O ( b 2 . 376 ) using more efﬁcient inverse algorithms like Coppersmith and Winograd (1987).

Original: We use the inverse approach with O ( b 3 ) computational cost, which will not be an issue for our method.
Revised: We use the inverse approach with O ( b 3 ) computational cost, which will not be an issue for our method.

Original: This is because we consider a mini-batch training to constrain the size of b , and the inverse ( K Z + λI ) − 1 does not contain gradients.
Revised: This is because we consider a mini-batch training to constrain the size of b , and the inverse ( K Z + λI ) − 1 does not contain gradients.

Original: The computational bottlenecks are gradients computation and their updates.
Revised: The computational bottlenecks are gradients computation and their updates.

Original: Step IV - Converting Existing Contrastive Learning Objectives.
Revised: Step IV - Converting Existing Contrastive Learning Objectives.

Original: We short hand K XY ( K Z
Revised: We short hand K XY ( K Z

Original: + λ I ) − 1 K Z as K X ⊥⊥ Y | Z , following notation from prior work (Fukumizu et al., 2007) that shorthands P X | Z P Y |
Revised: + λ I ) − 1 K Z as K X ⊥⊥ Y | Z , following notation from prior work (Fukumizu et al., 2007) that shorthands P X | Z P Y |

Original: Z as P X ⊥⊥ Y | Z .
Revised: Z as P X ⊥⊥ Y | Z .

Original: Now, we plug in the estimation of e f ( x,y ) using Proposition 2.2 into
Revised: Now, we plug in the estimation of e f ( x,y ) using Proposition 2.2 into

Original: WeaklySup-InfoNCE (equation 3), Fair-InfoNCE (equation 4), and HardNeg-InfoNCE (equation 5) objectives, coverting them into Conditional Contrastive learning with Kernel (CCL-K) objectives:
Revised: WeaklySup-InfoNCE (equation 3), Fair-InfoNCE (equation 4), and HardNeg-InfoNCE (equation 5) objectives, coverting them into Conditional Contrastive learning with Kernel (CCL-K) objectives:

Original: [Figure]
Revised: [Figure]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Figure] [Equation] [Figure] [Equation]
Revised: [ K X ⊥⊥ Y | Z ] ii (cid:105) .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The majority of the literature on contrastive learning focuses on self-supervised learning tasks (Oord et al., 2018), which leverages unlabeled samples for pretraining representations and then uses the learned representations for downstream tasks.
Revised: The majority of the literature on contrastive learning focuses on self-supervised learning tasks (Oord et al., 2018), which leverages unlabeled samples for pretraining representations and then uses the learned representations for downstream tasks.

Original: Its applications span various domains, including computer vision (Hjelm et al., 2018), natural language processing (Kong et al., 2019), speech processing (Baevski et al., 2020), and even interdisciplinary (vision and language) across domains (Radford et al., 2021).
Revised: Its applications span various domains, including computer vision (Hjelm et al., 2018), natural language processing (Kong et al., 2019), speech processing (Baevski et al., 2020), and even interdisciplinary (vision and language) across domains (Radford et al., 2021).

Original: Besides the empirical success, Arora et al.
Revised: Besides the empirical success, Arora et al.

Original: ; Lee et al.
Revised: ; Lee et al.

Original: ; Tsai et al.
Revised: ; Tsai et al.

Original: (2021d) provide theoretical guarantees, showing that contrastively learning can reduce the sample complexity on downstream tasks.
Revised: (2021d) provide theoretical guarantees, showing that contrastively learning can reduce the sample complexity on downstream tasks.

Original: The standard self-supervised contrastive frameworks consider the objective that requires only data’s pairing information: it learns similar representations between different views of a data (cid:0) augmented variants of the same image (Chen et al., 2020) or an image-caption pair (Radford et al., 2021) (cid:1) and dissimilar representations between two random data.
Revised: The standard self-supervised contrastive frameworks consider the objective that requires only data’s pairing information: it learns similar representations between different views of a data (cid:0) augmented variants of the same image (Chen et al., 2020) or an image-caption pair (Radford et al., 2021) (cid:1) and dissimilar representations between two random data.

Original: We refer to these frameworks as unconditional contrastive learning, in contrast to our paper’s focus - conditional contrastive learning, which considers contrastive objectives that take additional conditioning variables into account.
Revised: We refer to these frameworks as unconditional contrastive learning, in contrast to our paper’s focus - conditional contrastive learning, which considers contrastive objectives that take additional conditioning variables into account.

Original: Such conditioning variables can be sensitive information from data (Tsai et al., 2021c), auxiliary information from data (Tsai et al., 2021a), downstream labels (Khosla et al., 2020), or data’s embedded features (Robinson et al., 2020; Wu et al., 2020; Kalantidis et al., 2020).
Revised: Such conditioning variables can be sensitive information from data (Tsai et al., 2021c), auxiliary information from data (Tsai et al., 2021a), downstream labels (Khosla et al., 2020), or data’s embedded features (Robinson et al., 2020; Wu et al., 2020; Kalantidis et al., 2020).

Original: It is worth noting that, with additional conditioning variables, the conditional contrastive frameworks extend the self-supervised learning settings to the weakly supervised learning (Tsai et al., 2021a) or the supervised learning setting (Khosla et al., 2020).
Revised: It is worth noting that, with additional conditioning variables, the conditional contrastive frameworks extend the self-supervised learning settings to the weakly supervised learning (Tsai et al., 2021a) or the supervised learning setting (Khosla et al., 2020).

Original: Our paper also relates to the literature on few-shot conditional generation (Sinha et al., 2021), which aims to model the conditional generative probability (generating instances according to a conditioning variable) given only a limited amount of paired data (paired between an instance and its corresponding conditioning variable).
Revised: Our paper also relates to the literature on few-shot conditional generation (Sinha et al., 2021), which aims to model the conditional generative probability (generating instances according to a conditioning variable) given only a limited amount of paired data (paired between an instance and its corresponding conditioning variable).

Original: Its applications span conditional mutual information estimation (Mondal et al., 2020), noisy signals recovery (Candes et al., 2006), image manipulation (Park et al., 2020;
Revised: Its applications span conditional mutual information estimation (Mondal et al., 2020), noisy signals recovery (Candes et al., 2006), image manipulation (Park et al., 2020;

Original: Sinha et al., 2021), etc.
Revised: Sinha et al., 2021), etc.

Original: These applications require generating authentic data, which is notoriously challenging (Goodfellow et al., 2014; Arjovsky et al., 2017).
Revised: These applications require generating authentic data, which is notoriously challenging (Goodfellow et al., 2014; Arjovsky et al., 2017).

Original: On the contrary, our method models the conditional generative probability via
Revised: On the contrary, our method models the conditional generative probability via

Original: Kernel Conditional
Revised: Kernel Conditional

Original: Embedding Operator (Song et al., 2013), which generates kernel embeddings but not raw data.
Revised: Embedding Operator (Song et al., 2013), which generates kernel embeddings but not raw data.

Original: Ton et al.
Revised: Ton et al.

Original: (2021) relates to our work and also uses conditional mean embedding to perform estimation regarding the conditional distribution.
Revised: (2021) relates to our work and also uses conditional mean embedding to perform estimation regarding the conditional distribution.

Original: The differences is that Ton et al.
Revised: The differences is that Ton et al.

Original: (2021) tries to improve conditional density estimation while this paper aims to resolve the challenge of insufﬁcient samples of the conditional variable.
Revised: (2021) tries to improve conditional density estimation while this paper aims to resolve the challenge of insufﬁcient samples of the conditional variable.

Original: Also, both Ton et al.
Revised: Also, both Ton et al.

Original: (2021) and
Revised: (2021) and

Original: this work consider noise contrastive method, more speciﬁcally, Ton et al.
Revised: this work consider noise contrastive method, more speciﬁcally, Ton et al.

Original: (2021) discusses noise contrastive estimation (NCE) (Gutmann and Hyvärinen, 2010), while this work discusses
Revised: (2021) discusses noise contrastive estimation (NCE) (Gutmann and Hyvärinen, 2010), while this work discusses

Original: InfoNCE (Oord et al., 2018) objective which is inspired from NCE.
Revised: InfoNCE (Oord et al., 2018) objective which is inspired from NCE.

Original: Our proposed method can also connect to domain generalization (Blanchard et al., 2017), if we treat each z i as a domain or a task indicator (Tsai et al., 2021c).
Revised: Our proposed method can also connect to domain generalization (Blanchard et al., 2017), if we treat each z i as a domain or a task indicator (Tsai et al., 2021c).

Original: In speciﬁc, Tsai et al.
Revised: In speciﬁc, Tsai et al.

Original: (2021c) considers a conditional contrastive learning setup, and one task of it performs contrastive learning from data across multiple domains, by conditioning on domain indicators to reduce domain-speciﬁc information for better generalization.
Revised: (2021c) considers a conditional contrastive learning setup, and one task of it performs contrastive learning from data across multiple domains, by conditioning on domain indicators to reduce domain-speciﬁc information for better generalization.

Original: This paper further extends this idea from Tsai et al.
Revised: This paper further extends this idea from Tsai et al.

Original: (2021c), by using conditional mean embedding to address the challenge of insufﬁcient data in certain conditioning variables (in this case, domains).
Revised: (2021c), by using conditional mean embedding to address the challenge of insufﬁcient data in certain conditioning variables (in this case, domains).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We conduct experiments on various conditional contrastive learning frameworks that are discussed in Section 2.2: Section 4.1 for the weakly supervised contrastive learning, Section 4.2 for the fair contrastive learning; and Section 4.3 for the hard-negatives contrastive learning.
Revised: We conduct experiments on various conditional contrastive learning frameworks that are discussed in Section 2.2: Section 4.1 for the weakly supervised contrastive learning, Section 4.2 for the fair contrastive learning; and Section 4.3 for the hard-negatives contrastive learning.

Original: Experimental Protocal.
Revised: Experimental Protocal.

Original: We consider the setup from the recent contrastive learning literature (Chen et al., 2020;
Revised: We consider the setup from the recent contrastive learning literature (Chen et al., 2020;

Original: Robinson et al., 2020; Wu et al., 2020), which contains stages of pre-training, ﬁne-tuning, and evaluation.
Revised: Robinson et al., 2020; Wu et al., 2020), which contains stages of pre-training, ﬁne-tuning, and evaluation.

Original: In the pre-training stage, on data’s training split, we update the parameters in the feature encoder (i.e., g θ · ( · ) s in equation 2) using the contrastive learning objectives (cid:0) e.g., InfoNCE (equation 1), WeaklySup InfoNCE (equation 3), or WeaklySup CCLK (equation 7) (cid:1) .
Revised: In the pre-training stage, on data’s training split, we update the parameters in the feature encoder (i.e., g θ · ( · ) s in equation 2) using the contrastive learning objectives (cid:0) e.g., InfoNCE (equation 1), WeaklySup InfoNCE (equation 3), or WeaklySup CCLK (equation 7) (cid:1) .

Original: In the ﬁne-tuning stage, we ﬁx the parameters of the feature encoder and add a small ﬁne-tuning network on top of it.
Revised: In the ﬁne-tuning stage, we ﬁx the parameters of the feature encoder and add a small ﬁne-tuning network on top of it.

Original: On the data’s training split, we ﬁne-tune this small network with the downstream labels.
Revised: On the data’s training split, we ﬁne-tune this small network with the downstream labels.

Original: In the evaluation stage, we evaluate the ﬁne-tuned representations on the data’s test split.
Revised: In the evaluation stage, we evaluate the ﬁne-tuned representations on the data’s test split.

Original: We adopt
Revised: We adopt

Original: ResNet-50 He et al.
Revised: ResNet-50 He et al.

Original: (2016) or LeNet-5 LeCun et al.
Revised: (2016) or LeNet-5 LeCun et al.

Original: (1998) as the feature encoder and a linear layer as the ﬁne-tuning network.
Revised: (1998) as the feature encoder and a linear layer as the ﬁne-tuning network.

Original: All experiments are performed using LARS optimizer You et al.
Revised: All experiments are performed using LARS optimizer You et al.

Original: More details can be found in Appendix and our released code.
Revised: More details can be found in Appendix and our released code.

Original: 4. W EAKLY S UPERVISED C ONTRASTIVE L EARNING
Revised: 4. W EAKLY S UPERVISED C ONTRASTIVE L EARNING

Original: In this subsection, we perform experiments within the weakly supervised contrastive learning framework (Tsai et al., 2021a), which considers auxiliary information as the conditioning variable Z .
Revised: In this subsection, we perform experiments within the weakly supervised contrastive learning framework (Tsai et al., 2021a), which considers auxiliary information as the conditioning variable Z .

Original: It aims to learn similar representations for a pair of data with similar outcomes from the conditioning variable (i.e., similar auxiliary information), and vice versa.
Revised: It aims to learn similar representations for a pair of data with similar outcomes from the conditioning variable (i.e., similar auxiliary information), and vice versa.

Original: Datasets and Metrics.
Revised: Datasets and Metrics.

Original: We consider three visual datasets in this set of experiments.
Revised: We consider three visual datasets in this set of experiments.

Original: Data X and Y represent images after applying arbitrary image augmentations.
Revised: Data X and Y represent images after applying arbitrary image augmentations.

Original: 1) UT-Zappos (Yu and
Revised: 1) UT-Zappos (Yu and

Original: Grauman, 2014):
Revised: Grauman, 2014):

Original: It contains 50 , 025 shoe images over 21 shoe categories.
Revised: It contains 50 , 025 shoe images over 21 shoe categories.

Original: Each image is annotated with 7 binomially-distributed attributes as auxiliary information, and we convert them intobinary attributes (Bernoulli-distributed).
Revised: Each image is annotated with 7 binomially-distributed attributes as auxiliary information, and we convert them intobinary attributes (Bernoulli-distributed).

Original: 2) CUB (Wah et al., 2011):
Revised: 2) CUB (Wah et al., 2011):

Original: It contains 11 , 788 bird images spanning 200 ﬁne-grain bird species, meanwhile 312 binary attributes are attached to each image.
Revised: It contains 11 , 788 bird images spanning 200 ﬁne-grain bird species, meanwhile 312 binary attributes are attached to each image.

Original: ImageNet-100 (Russakovsky et al., 2015):
Revised: ImageNet-100 (Russakovsky et al., 2015):

Original: It is a subset of ImageNet-1k Russakovsky et al.
Revised: It is a subset of ImageNet-1k Russakovsky et al.

Original: (2015) dataset, containing 0 .
Revised: (2015) dataset, containing 0 .

Original: 12 million images spanning 100 categories.
Revised: 12 million images spanning 100 categories.

Original: This dataset does not come with auxiliary information, and hence we extract the 512 -dim.
Revised: This dataset does not come with auxiliary information, and hence we extract the 512 -dim.

Original: visual features from the CLIP (Radford et al., 2021) model (a large pre-trained visual model with natural language supervision) to be its auxiliary information.
Revised: visual features from the CLIP (Radford et al., 2021) model (a large pre-trained visual model with natural language supervision) to be its auxiliary information.

Original: Note that we consider different types of auxiliary information.
Revised: Note that we consider different types of auxiliary information.

Original: For UT-Zappos and CUB , we consider discrete and human-annotated attributes.
Revised: For UT-Zappos and CUB , we consider discrete and human-annotated attributes.

Original: For ImageNet-100 , we consider continuous and pre-trained language-enriched features.
Revised: For ImageNet-100 , we consider continuous and pre-trained language-enriched features.

Original: We report the top-1 accuracy as the metric for the downstream classiﬁcation task.
Revised: We report the top-1 accuracy as the metric for the downstream classiﬁcation task.

Original: Methodology.
Revised: Methodology.

Original: We consider the WeaklySup CCLK objective (equation 7) as the main method.
Revised: We consider the WeaklySup CCLK objective (equation 7) as the main method.

Original: For
Revised: For

Original: WeaklySup CCLK , we perform the sampling process { ( x i , y i , z i ) } bi =1 ∼ P ⊗ b XY Z by ﬁrst sampling an image
Revised: WeaklySup CCLK , we perform the sampling process { ( x i , y i , z i ) } bi =1 ∼ P ⊗ b XY Z by ﬁrst sampling an image

Original: im i along with its auxiliary information z
Revised: im i along with its auxiliary information z

Original: i and then applying different image augmentations on im i to obtain ( x i , y i ) .
Revised: i and then applying different image augmentations on im i to obtain ( x i , y i ) .

Original: We also study different types of kernels for K Z .
Revised: We also study different types of kernels for K Z .

Original: On the other hand, we select two baseline methods.
Revised: On the other hand, we select two baseline methods.

Original: The ﬁrst one is the unconditional contrastive learning method: the InfoNCE objective (Oord et al., 2018; Chen et al., 2020) (equation 1).
Revised: The ﬁrst one is the unconditional contrastive learning method: the InfoNCE objective (Oord et al., 2018; Chen et al., 2020) (equation 1).

Original: The second one is the conditional contrastive learning baseline: the WeaklySup InfoNCE objective (equation 3).
Revised: The second one is the conditional contrastive learning baseline: the WeaklySup InfoNCE objective (equation 3).

Original: The difference between
Revised: The difference between

Original: WeaklySup CCLK and WeaklySup InfoNCE is that the latter requires sampling a pair of data with the same outcome from the conditioning variable (cid:0) i.e., ( x, y ) ∼ P X | Z =
Revised: WeaklySup CCLK and WeaklySup InfoNCE is that the latter requires sampling a pair of data with the same outcome from the conditioning variable (cid:0) i.e., ( x, y ) ∼ P X | Z =

Original: z P Y | Z
Revised: z P Y | Z

Original: = z (cid:1) .
Revised: = z (cid:1) .

Original: However, as suggested in Section 2.3, directly performing conditional sampling is challenging if there is not enough data to support the conditioning sampling procedure.
Revised: However, as suggested in Section 2.3, directly performing conditional sampling is challenging if there is not enough data to support the conditioning sampling procedure.

Original: Such limitation exists in our datasets:
Revised: Such limitation exists in our datasets:

Original: CUB has on average 1 .
Revised: CUB has on average 1 .

Original: 001 data points per Z ’s conﬁguration, and ImageNet-100 has only 1 data point per Z ’s conﬁguration since its conditioning variable is continuous and each instance from the dataset has a unique Z .
Revised: 001 data points per Z ’s conﬁguration, and ImageNet-100 has only 1 data point per Z ’s conﬁguration since its conditioning variable is continuous and each instance from the dataset has a unique Z .

Original: To avoid this limitation, in WeaklySup InfoNCE clusters the data to ensure that data within the same cluster are abundant and have similar auxiliary information, and then treating  the clustering information as the new conditioning variable.
Revised: To avoid this limitation, in WeaklySup InfoNCE clusters the data to ensure that data within the same cluster are abundant and have similar auxiliary information, and then treating [Table] the clustering information as the new conditioning variable.

Original: The result of WeaklySup InfoNCE is reported by selecting the optimal number of clusters via cross-validation.
Revised: The result of WeaklySup InfoNCE is reported by selecting the optimal number of clusters via cross-validation.

Original: Results.
Revised: Results.

Original: We show the results in Table 2.
Revised: We show the results in Table 2.

Original: First, WeaklySup CCLK shows consistent improvements over unconditional baseline InfoNCE, with absolute improvements of 8 . 8% , 15 . 8% , and 6 . 2% on
Revised: First, WeaklySup CCLK shows consistent improvements over unconditional baseline InfoNCE, with absolute improvements of 8 . 8% , 15 . 8% , and 6 . 2% on

Original: UT-Zappos, CUB and, ImageNet-100 respectively.
Revised: UT-Zappos, CUB and, ImageNet-100 respectively.

Original: This is because the conditional method utilizes the[Table] additional auxiliary information (Tsai et al., 2021a).
Revised: This is because the conditional method utilizes the additional auxiliary information (Tsai et al., 2021a).

Original: Second, WeaklySup CCLK performs better than
Revised: Second, WeaklySup CCLK performs better than

Original: WeaklySup InfoNCE , with absolute improvements of 2% , 9 .
Revised: WeaklySup InfoNCE , with absolute improvements of 2% , 9 .

Original: 3% , and 1% .
Revised: 3% , and 1% .

Original: We attribute better performance of WeaklySup CCLK over WeaklySup InfoNCE to the following fact: WeaklySup InfoNCE ﬁrst performs clustering on the auxiliary information and considers the new clusters as the conditioning variable, while WeaklySup CCLK directly considers the auxiliary information as the conditioning variable.
Revised: We attribute better performance of WeaklySup CCLK over WeaklySup InfoNCE to the following fact: WeaklySup InfoNCE ﬁrst performs clustering on the auxiliary information and considers the new clusters as the conditioning variable, while WeaklySup CCLK directly considers the auxiliary information as the conditioning variable.

Original: The clustering in WeaklySup InfoNCE may lose precision of the auxiliary information and may negatively affect the quality of the auxiliary information incorporated in the representation.
Revised: The clustering in WeaklySup InfoNCE may lose precision of the auxiliary information and may negatively affect the quality of the auxiliary information incorporated in the representation.

Original: Ablation study on the choice of kernels has shown the consistent performance of WeaklySup CCLK across different kernels on the UT-Zappos, CUB and ImageNet-100 dataset, where we consider the following kernel functions: RBF, Laplacian, Linear and Cosine.
Revised: Ablation study on the choice of kernels has shown the consistent performance of WeaklySup CCLK across different kernels on the UT-Zappos, CUB and ImageNet-100 dataset, where we consider the following kernel functions: RBF, Laplacian, Linear and Cosine.

Original: Most kernels have similar performances, except that linear kernel is worse than others on ImageNet-100 (by at least 2 . 7% ).
Revised: Most kernels have similar performances, except that linear kernel is worse than others on ImageNet-100 (by at least 2 . 7% ).

Original: 4. F AIR C ONTRASTIVE L EARNING
Revised: 4. F AIR C ONTRASTIVE L EARNING

Original: In this subsection, we perform experiments within the fair contrastive learning framework (Tsai et al., 2021c), which considers sensitive information as the conditioning variable Z .
Revised: In this subsection, we perform experiments within the fair contrastive learning framework (Tsai et al., 2021c), which considers sensitive information as the conditioning variable Z .

Original: It ﬁxes the outcome of the sensitive variable for both the positively-paired and negatively-paired samples in the contrastive learning process, which leads the representations to ignore the effect from the sensitive variable.
Revised: It ﬁxes the outcome of the sensitive variable for both the positively-paired and negatively-paired samples in the contrastive learning process, which leads the representations to ignore the effect from the sensitive variable.

Original: Datasets and Metrics.
Revised: Datasets and Metrics.

Original: Our experiments focus on continuous sensitive information, to echo with the limitation of the conditional sampling procedure in Section 2.3.
Revised: Our experiments focus on continuous sensitive information, to echo with the limitation of the conditional sampling procedure in Section 2.3.

Original: Nonetheless, existing datasets mostly consider discrete sensitive variables, such as gender or race.
Revised: Nonetheless, existing datasets mostly consider discrete sensitive variables, such as gender or race.

Original: Therefore, we synthetically create
Revised: Therefore, we synthetically create

Original: ColorMNIST dataset, which randomly assigns a continuous RBG color value for the background in each handwritten digit image in the MNIST dataset (LeCun et al., 1998).
Revised: ColorMNIST dataset, which randomly assigns a continuous RBG color value for the background in each handwritten digit image in the MNIST dataset (LeCun et al., 1998).

Original: We consider the background color as sensitive information.
Revised: We consider the background color as sensitive information.

Original: For statistics, ColorMNIST has 60 , 000 colored digit images across 10 digit labels.
Revised: For statistics, ColorMNIST has 60 , 000 colored digit images across 10 digit labels.

Original: Similar to Section 4.1, data X and Y represent images after applying arbitrary image augmentations.
Revised: Similar to Section 4.1, data X and Y represent images after applying arbitrary image augmentations.

Original: Our goal is two-fold: we want to see how well the learned representations 1) perform on the downstream classiﬁcation task and 2) ignore the effect from the sensitive variable.
Revised: Our goal is two-fold: we want to see how well the learned representations 1) perform on the downstream classiﬁcation task and 2) ignore the effect from the sensitive variable.

Original: For the former one, we report the top-1 accuracy as the metric; for the latter one, we report the Mean Square
Revised: For the former one, we report the top-1 accuracy as the metric; for the latter one, we report the Mean Square

Original: Error (MSE) when trying to predict the color information.
Revised: Error (MSE) when trying to predict the color information.

Original: Note that the MSE score is higher the better since we would like the learned representations to contain less color information.
Revised: Note that the MSE score is higher the better since we would like the learned representations to contain less color information.

Original: [Table]
Revised: [Table]

Original: under the fair contrastive learning setup, and the MSE (higher the

Original: Methodology.
Revised: Methodology.

Original: We consider the Fair CCLK objective (equation 8) as the main method.
Revised: We consider the Fair CCLK objective (equation 8) as the main method.

Original: For Fair CCLK , we perform the sampling process { ( x i , y i , z i ) } bi =1 ∼ P ⊗ b XY Z by ﬁrst sampling a digit image im i along with its sensitive information
Revised: For Fair CCLK , we perform the sampling process { ( x i , y i , z i ) } bi =1 ∼ P ⊗ b XY Z by ﬁrst sampling a digit image im i along with its sensitive information

Original: z i and then applying different image augmentations on im i to obtain ( x i , y i ) .
Revised: z i and then applying different image augmentations on im i to obtain ( x i , y i ) .

Original: We select the unconditional contrastive learning method - the InfoNCE objective (equation 1) as our baseline.
Revised: We select the unconditional contrastive learning method - the InfoNCE objective (equation 1) as our baseline.

Original: We also consider the
Revised: We also consider the

Original: Fair InfoNCE objective (equation 4) as a baseline, by clustering the continuous conditioning variable Z into one of the following: 3, 5, 10, 15, or 20 clusters using K-means.
Revised: Fair InfoNCE objective (equation 4) as a baseline, by clustering the continuous conditioning variable Z into one of the following: 3, 5, 10, 15, or 20 clusters using K-means.

Original: Results.
Revised: Results.

Original: We show the results in Table 3.
Revised: We show the results in Table 3.

Original: Fair CCLK is consistently better than the InfoNCE, where the absolute accuracy improvement is 2 .
Revised: Fair CCLK is consistently better than the InfoNCE, where the absolute accuracy improvement is 2 .

Original: 3% and the relative improvement of MSE (higher the better) is 32 .
Revised: 3% and the relative improvement of MSE (higher the better) is 32 .

Original: 6% over InfoNCE.
Revised: 6% over InfoNCE.

Original: We report the result of using the Cosine kernel and provide the ablation study of different kernel choices in the Appendix.
Revised: We report the result of using the Cosine kernel and provide the ablation study of different kernel choices in the Appendix.

Original: This result suggests that the proposed Fair CCLK can achieve better downstream classiﬁcation accuracy while ignoring more sensitive information (color information) compared to the unconditional baseline, suggesting that our method can achieve a better level of fairness (by excluding color bias) without sacriﬁcing performance.
Revised: This result suggests that the proposed Fair CCLK can achieve better downstream classiﬁcation accuracy while ignoring more sensitive information (color information) compared to the unconditional baseline, suggesting that our method can achieve a better level of fairness (by excluding color bias) without sacriﬁcing performance.

Original: Next we compare to Fair InfoNCE baseline, and we report the result using the 10-cluster partition of Z as it achieves the best top-1 accuracy.
Revised: Next we compare to Fair InfoNCE baseline, and we report the result using the 10-cluster partition of Z as it achieves the best top-1 accuracy.

Original: Compared to Fair InfoNCE , Fair CCLK is better in downstream accuracy, while slightly worse in MSE (a difference of 0.2).
Revised: Compared to Fair InfoNCE , Fair CCLK is better in downstream accuracy, while slightly worse in MSE (a difference of 0.2).

Original: For Fair InfoNCE , if the number of discretized value of Z increases, the MSE in general grows, but the accuracy peaks atclusters and then declines.
Revised: For Fair InfoNCE , if the number of discretized value of Z increases, the MSE in general grows, but the accuracy peaks at 10 clusters and then declines.

Original: This suggests that Fair InfoNCE can remove more sensitive information as the granularity of Z increases, but may hurt the downstream task performance.
Revised: This suggests that Fair InfoNCE can remove more sensitive information as the granularity of Z increases, but may hurt the downstream task performance.

Original: Overall, the Fair CCLK performs slightly better than Fair InfoNCE , and do not need clustering to discretize Z .
Revised: Overall, the Fair CCLK performs slightly better than Fair InfoNCE , and do not need clustering to discretize Z .

Original: 4. H ARD -N EGATIVES C ONTRASTIVE L EARNING
Revised: 4. H ARD -N EGATIVES C ONTRASTIVE L EARNING

Original: In this subsection, we perform experiments within the hard negative contrastive learning framework (Robinson et al., 2020), which considers the embedded features as the conditioning variable Z .
Revised: In this subsection, we perform experiments within the hard negative contrastive learning framework (Robinson et al., 2020), which considers the embedded features as the conditioning variable Z .

Original: Different from conventional contrastive learning methods (Chen et al., 2020; He et al., 2020) that considers learning dissimilar representations for a pair of random data, the hard negative contrastive learning methods learn dissimilar representations for a pair of random data when they have similar embedded features (i.e., similar outcomes from the conditioning variable).
Revised: Different from conventional contrastive learning methods (Chen et al., 2020; He et al., 2020) that considers learning dissimilar representations for a pair of random data, the hard negative contrastive learning methods learn dissimilar representations for a pair of random data when they have similar embedded features (i.e., similar outcomes from the conditioning variable).

Original: Datasets and Metrics.
Revised: Datasets and Metrics.

Original: We consider two visual datasets in this set of experiments.
Revised: We consider two visual datasets in this set of experiments.

Original: Data X
Revised: Data X

Original: and
Revised: and

Original: Y represent images after applying arbitrary image augmentations.
Revised: Y represent images after applying arbitrary image augmentations.

Original: 1) CIFAR10 (Krizhevsky et al., 2009):
Revised: 1) CIFAR10 (Krizhevsky et al., 2009):

Original: It contains 60 , 000 images spanning 10 classes, e.g. automobile, plane, or dog. 2) ImageNet100 (Russakovsky et al., 2015):
Revised: It contains 60 , 000 images spanning 10 classes, e.g. automobile, plane, or dog. 2) ImageNet100 (Russakovsky et al., 2015):

Original: It is the same dataset that we used in Section 2.
Revised: It is the same dataset that we used in Section 2.

Original: We report the topaccuracy as the metric for the downstream classiﬁcation task.
Revised: We report the topaccuracy as the metric for the downstream classiﬁcation task.

Original: Methodology.
Revised: Methodology.

Original: We consider the HardNeg CCLK objective (equation 9) as the main method.
Revised: We consider the HardNeg CCLK objective (equation 9) as the main method.

Original: For
Revised: For

Original: HardNeg CCLK , we perform the sampling process { ( x i , y i , z i ) } bi =1 ∼ P ⊗ b XY Z by ﬁrst sampling an image
Revised: HardNeg CCLK , we perform the sampling process { ( x i , y i , z i ) } bi =1 ∼ P ⊗ b XY Z by ﬁrst sampling an image

Original: im i , then applying different image augmentations on im i to obtain ( x i , y i ) , and last deﬁning z
Revised: im i , then applying different image augmentations on im i to obtain ( x i , y i ) , and last deﬁning z

Original: i
Revised: i

Original: =
Revised: =

Original: g θ X ( x i ) .
Revised: g θ X ( x i ) .

Original: We select two baseline methods.
Revised: We select two baseline methods.

Original: The ﬁrst one is the unconditional contrastive learning method: the InfoNCE objective (Oord et al., 2018; Chen et al., 2020) (equation 1).
Revised: The ﬁrst one is the unconditional contrastive learning method: the InfoNCE objective (Oord et al., 2018; Chen et al., 2020) (equation 1).

Original: The second one is the conditional contrastive learning baseline: the HardNeg InfoNCE , objective (Robinson et al., 2020), and we report its result directly using the released code from the author (Robinson et al., 2020).
Revised: The second one is the conditional contrastive learning baseline: the HardNeg InfoNCE , objective (Robinson et al., 2020), and we report its result directly using the released code from the author (Robinson et al., 2020).

Original: [Table]
Revised: [Table]

Original: Results.
Revised: Results.

Original:  Table 4 presents the results. First, HardNeg CCLK consistently shows better performances over the InfoNCE baseline, with absolute improvements of 1 . 8% and 3 .
Revised: From Table 4, ﬁrst, HardNeg CCLK consistently shows better performances over the InfoNCE baseline, with absolute improvements of 1 . 8% and 3 .

Original: 1% on CIFAR-10 and ImageNet-100 respectively.
Revised: 1% on CIFAR-10 and ImageNet-100 respectively.

Original: This suggests that the hard negative sampling effectively improves the downstream performance, which is in accordance with the observation by Robinson et al.
Revised: This suggests that the hard negative sampling effectively improves the downstream performance, which is in accordance with the observation by Robinson et al.

Original: Next, HardNeg CCLK also performs better than the HardNeg InfoNCE baseline, with absolute improvements of 0 .
Revised: Next, HardNeg CCLK also performs better than the HardNeg InfoNCE baseline, with absolute improvements of 0 .

Original: 3% and 2 . 0% on CIFARand ImageNet-100 respectively.
Revised: 3% and 2 . 0% on CIFAR-10 and ImageNet-100 respectively.

Original: Both methods construct hard negatives by assigning a higher weight to a random paired data that are close in the embedding space and a lower weight to a random paired data that are far in the embedding space.
Revised: Both methods construct hard negatives by assigning a higher weight to a random paired data that are close in the embedding space and a lower weight to a random paired data that are far in the embedding space.

Original: The implementation by Robinson et al.
Revised: The implementation by Robinson et al.

Original: (2020) uses Euclidean distances to measure the similarity and HardNeg CCLK uses the smoothed kernel similarity (i.e., ( K Z + λ I ) − 1 K Z in Proposition 2.2) to measure the similarity.
Revised: (2020) uses Euclidean distances to measure the similarity and HardNeg CCLK uses the smoothed kernel similarity (i.e., ( K Z + λ I ) − 1 K Z in Proposition 2.2) to measure the similarity.

Original: We empirically observe that our approach performs better.
Revised: Empirically our approach performs better.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we present CCL-K, the C onditional C ontrastive L earning objectives with K ernel expressions.
Revised: In this paper, we present CCL-K, the C onditional C ontrastive L earning objectives with K ernel expressions.

Original: CCL-K avoids the need to perform explicit conditional sampling in conditional contrastive learning frameworks, which can help when we do not have access to sufﬁcient data for each variable state of the conditioning variables.
Revised: CCL-K avoids the need to perform explicit conditional sampling in conditional contrastive learning frameworks, alleviating the insufﬁcient data problem of the conditioning variable.

Original: In speciﬁc, to sample data given a speciﬁc value of the conditioning variable, CCL-K considers the sampling process using the Kernel Conditional Embedding Operator, which ﬁrst deﬁnes the kernel similarities between the conditioning variable’s values and then samples data that have similar values of the conditioning variable.
Revised: CCL-K uses the Kernel Conditional Embedding Operator, which ﬁrst deﬁnes the kernel similarities between the conditioning variable’s values and then samples data that have similar values of the conditioning variable.

Original: An advantage of CCL-K isthat it can directly work with continuous conditioning variables, which prior work often needs to bin or cluster continuous variables to ensure that data within the same bin or cluster are abundant. Not only does
Revised: CCL-K is can directly work with continuous conditioning variables, while prior work requires binning or clustering to ensure sufﬁcient data for each bin or cluster.

Original:  CCL-Kbroaden the applicability of conditional contrastive learning, we also empirically observe CCL-K outperforming conditional contrastive baselines tailored for weakly-supervised contrastive learning, fair contrastive learning, and hard negatives contrastive learning.
Revised: Empirically, CCL-K also outperforms conditional contrastive baselines tailored for weakly-supervised contrastive learning, fair contrastive learning, and hard negatives contrastive learning.

Original: An interesting future direction is to add more ﬂexibility to CCL-K, by relaxing the kernel similarity to arbitrary similarity measurements.
Revised: An interesting future direction is to add more ﬂexibility to CCL-K, by relaxing the kernel similarity to arbitrary similarity measurements.

Original:  E THICS S TATEMENT
Revised:  E THICS S TATEMENT

Original: Because our method can improve removing sensitive information in contrastive learning representations, our contribution can have a positive impact on fairness and privacy, where biases or user-speciﬁc information should be excluded from the representation.
Revised: Because our method can improve removing sensitive information in contrastive learning representations, our contribution can have a positive impact on fairness and privacy, where biases or user-speciﬁc information should be excluded from the representation.

Original: However, the conditioning variable must be predeﬁned, so our method cannot directly remove any biases that are implicit and are not captured by a variable in the dataset.
Revised: However, the conditioning variable must be predeﬁned, so our method cannot directly remove any biases that are implicit and are not captured by a variable in the dataset.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: We provide an anonymous source code link for reproducing our result in the supplementary material and include complete ﬁles which can reproduce the data processing steps for each dataset we use in the supplementary material.
Revised: We provide an anonymous source code link for reproducing our result in the supplementary material and include complete ﬁles which can reproduce the data processing steps for each dataset we use in the supplementary material.

Original: A C ODE
Revised: A CKNOWLEDGEMENTS

Revised: The authors would like to thank the anonymous reviewers for helpful comments and suggestions. This work is partially supported by the National Science Foundation IIS1763562, IARPA D17PC00340, ONR Grant N000141812861, Facebook PhD Fellowship, BMW, National Science Foundation awards 1722822 and 1750439, and National Institutes of Health awards R01MH125740, R01MH096951 and

Revised: U01MH116925.

Revised: Any opinions, ﬁndings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reﬂect the views of the sponsors, and no ofﬁcial endorsement should be inferred.

Original: We include our code in an anonymous Github: https://anonymous.4open.science/r/
Revised: We include our code in a Github link: https://github.com/Crazy-Jack/

Original: CCLK2-7BEF/ B A BLATION OF D IFFERENT C HOICES OF K ERNELS
Revised: CCLK-release B A BLATION OF D IFFERENT C HOICES OF K ERNELS

Original: We report the kernel choice ablation study for Fair CCLK and HardNeg CCLK (Table 5).
Revised: We report the kernel choice ablation study for Fair CCLK and HardNeg CCLK (Table 5).

Original: For the
Revised: For the

Original: Fair CCLK , we consider the synthetically created ColorMNIST dataset, where the background color for each image digit image is randomly assigned.
Revised: Fair CCLK , we consider the synthetically created ColorMNIST dataset, where the background color for each image digit image is randomly assigned.

Original: We report the accuracy of the downstream classiﬁcation task, as well as the Mean Square Error (MSE) between the assigned background color and a color predicted by the learned representation.
Revised: We report the accuracy of the downstream classiﬁcation task, as well as the Mean Square Error (MSE) between the assigned background color and a color predicted by the learned representation.

Original: A higher MSE score is better because we would like the learned representations to contain less color information.
Revised: A higher MSE score is better because we would like the learned representations to contain less color information.

Original: In Table 5, we consider the following kernel functions: RBF, Polynomial (degree of 3), Laplacian and Cosine.
Revised: In Table 5, we consider the following kernel functions: RBF, Polynomial (degree of 3), Laplacian and Cosine.

Original: The performances using different kernels are consistent.
Revised: The performances using different kernels are consistent.

Original: For the HardNeg CCLK , we consider the CIFAR-10 dataset for the ablation study and use the topaccuracy for object classiﬁcation as the metric.
Revised: For the HardNeg CCLK , we consider the CIFAR-10 dataset for the ablation study and use the topaccuracy for object classiﬁcation as the metric.

Original: In Table 5, we consider the following kernel functions:
Revised: In Table 5, we consider the following kernel functions:

Original: Linear, RBF, and Polynomial (degree of 3), Laplacian and Cosine.
Revised: Linear, RBF, and Polynomial (degree of 3), Laplacian and Cosine.

Original: The performances using different kernels are consistent.
Revised: The performances using different kernels are consistent.

Original: Lastly, we also provide the ablation of different σ 2 when using the RBF kernel.
Revised: Lastly, we also provide the ablation of different σ 2 when using the RBF kernel.

Original: First, the performance trend does change too much and is sensitive to different bandwidths for the RBF kernel.
Revised: First, the performance trend does change too much and is sensitive to different bandwidths for the RBF kernel.

Original: In Tablewe show the result of using different σ 2 in the RBF kernel.
Revised: In Tablewe show the result of using different σ 2 in the RBF kernel.

Original: We found that using σ 2 = 1 signiﬁcantly hurts performance (only 3.0%), using σ 2 = 1000 is also sub-optimal.
Revised: We found that using σ 2 = 1 signiﬁcantly hurts performance (only 3.0%), using σ 2 = 1000 is also sub-optimal.

Original: The performances of σ 2 = 10 , 100 , 500 are close.
Revised: The performances of σ 2 = 10 , 100 , 500 are close.

Original: C A DDITIONAL R ESULTS ON CIFAR 
Revised: C A DDITIONAL R ESULTS ON CIFAR

Original: In Section 4.3 in the main text, we report the results of HardNeg CCLK as well as baseline methods on CIFAR10 dataset by training with 400 epochs using 256 batch size.
Revised: In Section 4.3 in the main text, we report the results of HardNeg CCLK as well as baseline methods on CIFAR10 dataset by training with 400 epochs using 256 batch size.

Original: Here we also include the results with larger batch size (512 batch size) and longer training time (1000 epochs).
Revised: Here we also include the results with larger batch size (512 batch size) and longer training time (1000 epochs).

Original: The results are summarized in Table 6.
Revised: The results are summarized in Table 6.

Original: For reference, we name the training procedure with 256 batch size andepochs setting 1 and the one with 512 batch size and 1000 epochs as setting 2 .
Revised: For reference, we name the training procedure with 256 batch size andepochs setting 1 and the one with 512 batch size and 1000 epochs as setting 2 .

Original: From Table 6 we observe that HardNeg CCLK still has better performance comparing to HardNeg InfoNCE , suggesting that our method is solid.
Revised: From Table 6 we observe that HardNeg CCLK still has better performance comparing to HardNeg InfoNCE , suggesting that our method is solid.

Original: However, the performance differences shrink between the vallina InfoNCE method and the Hard Negative mining approaches in general because the beneﬁt of hard negative mining mainly lies in the training efﬁciency, i.e., spend less training iteration on discriminating the negative samples that have been already pushed away.
Revised: However, the performance differences shrink between the vallina InfoNCE method and the Hard Negative mining approaches in general because the beneﬁt of hard negative mining mainly lies in the training efﬁciency, i.e., spend less training iteration on discriminating the negative samples that have been already pushed away.

Original: D F AIR InfoNCE ON C OLOR MNIST
Revised: D F AIR InfoNCE ON C OLOR MNIST

Original: Here we include the results of performing Fair InfoNCE on the ColorMNIST dataset as a baseline.
Revised: Here we include the results of performing Fair InfoNCE on the ColorMNIST dataset as a baseline.

Original: We implemented Fair InfoNCE based on Tsai et al.
Revised: We implemented Fair InfoNCE based on Tsai et al.

Original: (2021c), where the idea is to remove the information of the conditional variable by sampling positive and negative pairs from the same outcome of the conditional variable at each iteration.
Revised: (2021c), where the idea is to remove the information of the conditional variable by sampling positive and negative pairs from the same outcome of the conditional variable at each iteration.

Original: Same as our previous setup in Section 4.2, we evaluate the results by the accuracy of the downstream classiﬁcation task as well as the MSE value, both of which are deemed higher as the better (as we want the learned representation to contain less color information, so a larger MSE is desirable because it means the representation contains less color information for reconstruction.)
Revised: Same as our previous setup in Section 4.2, we evaluate the results by the accuracy of the downstream classiﬁcation task as well as the MSE value, both of which are deemed higher as the better (as we want the learned representation to contain less color information, so a larger MSE is desirable because it means the representation contains less color information for reconstruction.)

Original: To perform Fair InfoNCE and condition on the continuous color information, we need to discretize the color variable using clustering method such as K-means.
Revised: To perform Fair InfoNCE and condition on the continuous color information, we need to discretize the color variable using clustering method such as K-means.

Original: We use K-means to cluster samples based on their color variable, and we use the following numbers of clusters: { 3 , 5 , 10 , 15 , 20 } .
Revised: We use K-means to cluster samples based on their color variable, and we use the following numbers of clusters: { 3 , 5 , 10 , 15 , 20 } .

Original: As shown in Table 8, with the number of clusters increasing, the downstream accuracy ﬁrst increases then drops, peaking at number of clusters being 10.
Revised: As shown in Table 8, with the number of clusters increasing, the downstream accuracy ﬁrst increases then drops, peaking at number of clusters being 10.

Original: The MSE values continue increase as number of clusters increase.
Revised: The MSE values continue increase as number of clusters increase.

Original: This suggests that Fair InfoNCE can remove more sensitive information as the granularity of Z increases, but the downstream task performance may decrease.
Revised: This suggests that Fair InfoNCE can remove more sensitive information as the granularity of Z increases, but the downstream task performance may decrease.

Original: E P ERFORMANCE UNDER INSUFFICIENT NUMBER OF SAMPLES
Revised: E P ERFORMANCE UNDER INSUFFICIENT NUMBER OF SAMPLES

Original: We would illustrate why the insufﬁcient sample would be a problem for conditional contrastive learning.
Revised: We would illustrate why the insufﬁcient sample would be a problem for conditional contrastive learning.

Original: We provide comparison of performances under different number of conditioning data.
Revised: We provide comparison of performances under different number of conditioning data.

Original: To be speciﬁc, we provide Figure 3, where the x-axis is the averaged number of data samples per discretized conditioning variable (cluster) for conditional contrastive learning, if we use framework like the WeaklySup InfoNCE .
Revised: To be speciﬁc, we provide Figure 3, where the x-axis is the averaged number of data samples per discretized conditioning variable (cluster) for conditional contrastive learning, if we use framework like the WeaklySup InfoNCE .

Original: The dataset is UT-Zappos and the conditioning variable is the annotative attributes.
Revised: The dataset is UT-Zappos and the conditioning variable is the annotative attributes.

Original: The discretization is done by grouping instances that share the same annotative attributes to the same cluster.
Revised: The discretization is done by grouping instances that share the same annotative attributes to the same cluster.

Original: The blue line is WeaklySup InfoNCE , which requires discretized conditioning variables.
Revised: The blue line is WeaklySup InfoNCE , which requires discretized conditioning variables.

Original: The black line represents the proposed WeaklySup CCLK which does not require discretization.
Revised: The black line represents the proposed WeaklySup CCLK which does not require discretization.

Original: As we can see from the ﬁgure, the performance of WeaklySup InfoNCE suffers when the number of data per cluster (conditioning variable) is small, and WeaklySup CCLK outperforms
Revised: As we can see from the ﬁgure, the performance of WeaklySup InfoNCE suffers when the number of data per cluster (conditioning variable) is small, and WeaklySup CCLK outperforms

Original: WeaklySup InfoNCE in all cases.
Revised: WeaklySup InfoNCE in all cases.

Original: From this example, we can see that when the data is very insufﬁcient (towards the origin in this ﬁgure), the proposed WeaklySup CCLK outperforms WeaklySup InfoNCE signiﬁcantly.
Revised: From this example, we can see that when the data is very insufﬁcient (towards the origin in this ﬁgure), the proposed WeaklySup CCLK outperforms WeaklySup InfoNCE signiﬁcantly.

Original: F D ATASET D ETAILS
Revised: F D ATASET D ETAILS

Original: We provide the training details of experiments conducted on the following datasets: UT-Zappos50 (Yu and Grauman, 2014), CUB-200-2011 (Wah et al., 2011), CIFAR-10 (Krizhevsky et al., 2009),
Revised: We provide the training details of experiments conducted on the following datasets: UT-Zappos50 (Yu and Grauman, 2014), CUB-200-2011 (Wah et al., 2011), CIFAR-10 (Krizhevsky et al., 2009),

Original: ColorMNIST (our creation), and ImageNet-100 (Russakovsky et al., 2015).
Revised: ColorMNIST (our creation), and ImageNet-100 (Russakovsky et al., 2015).

Original: F. UT-Z APPOS 50K
Revised: F. UT-Z APPOS 50K

Original: The following section describes the experiments we performed on UT-Zappos50K dataset.
Revised: The following section describes the experiments we performed on UT-Zappos50K dataset.

Original: Accessiblity
Revised: Accessiblity

Original: The dataset is attributed to (Yu and Grauman, 2014) and available at the link: http://vision.cs.utexas.edu/projects/finegrained/utzap50k .
Revised: The dataset is attributed to (Yu and Grauman, 2014) and available at the link: http://vision.cs.utexas.edu/projects/finegrained/utzap50k .

Original: The datasetis for non-commercial use only.
Revised: The dataset

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Data Processing
Revised: Data Processing

Original: The dataset contains images of shoe from Zappos.com.
Revised: The dataset contains images of shoe from Zappos.com.

Original: We downsamples the images to 32 × 32 .
Revised: We downsamples the images to 32 × 32 .

Original: The ofﬁcial dataset has 4 large categories following 21 sub-categories.
Revised: The ofﬁcial dataset has 4 large categories following 21 sub-categories.

Original: We utilize 21 subcategories for all our classiﬁcation tasks.
Revised: We utilize 21 subcategories for all our classiﬁcation tasks.

Original: The dataset comes with 7 attributes as auxiliary information.
Revised: The dataset comes with 7 attributes as auxiliary information.

Original: We binarize the 7 discrete attributes into 126 binary attributes.
Revised: We binarize the 7 discrete attributes into 126 binary attributes.

Original: We consider our conditional variable Z is this 128 dimensional variable.
Revised: We consider our conditional variable Z is this 128 dimensional variable.

Original: Training and Test Split : We randomly split train-validation images by 7 : 3 ratio, resulting in 35 ,train data and 15 , 008 validation dataset.
Revised: Training and Test Split : We randomly split train-validation images by 7 : 3 ratio, resulting in 35 ,train data and 15 , 008 validation dataset.

Original: Network Design and Optimization We use ResNet-50 architecture to serve as a backbone for the encoder.
Revised: Network Design and Optimization We use ResNet-50 architecture to serve as a backbone for the encoder.

Original: To compensate the 32x32 image size, we change the ﬁrst 7x7 2D convolution to 3x 2D convolution and remove the ﬁrst max pooling layer in the normal ResNet-50 (See code for details).
Revised: To compensate the 32x32 image size, we change the ﬁrst 7x7 2D convolution to 3x 2D convolution and remove the ﬁrst max pooling layer in the normal ResNet-50 (See code for details).

Original: This allows a ﬁner grain of information processing.
Revised: This allows a ﬁner grain of information processing.

Original: After using the modiﬁed ResNetas the encoder, we include a 2048-2048-128 Multi-Layer Perceptron (MLP) as the projection head.
Revised: After using the modiﬁed ResNetas the encoder, we include a 2048-2048-128 Multi-Layer Perceptron (MLP) as the projection head.

Original: Batch normalization is used after each 2048 activation layers.
Revised: Batch normalization is used after each 2048 activation layers.

Original: During the evaluation, we discard the 
Revised: During the evaluation, we discard the projection head and train a linear layer on top of the encoder’s output. We train 1000 epochs for all experiments with LARS optimizer (base learning rate 1.5 and scale learning rate based on our batch size divided by 256) with batch size 152 on 4 NVIDIA 1080ti GPUs. It takes about 16 hours to ﬁnish 1000 epochs training.

Original: F. CUB-200- 
Revised: F. CUB-200-

Original: The following section describes the experiments we performed on CUB-200-2011 dataset.
Revised: The following section describes the experiments we performed on CUB-200-2011 dataset.

Original: Accessiblity CUB-200-2011 is created by Wah et al.
Revised: Accessiblity CUB-200-2011 is created by Wah et al.

Original: and is a ﬁne-grained dataset for bird species.
Revised: and is a ﬁne-grained dataset for bird species.

Original: It can be downloaded from the link: http://www.vision.caltech.edu/ visipedia/CUB-200-2011.html .
Revised: It can be downloaded from the link: http://www.vision.caltech.edu/ visipedia/CUB-200-2011.html .

Original: The usage is restricted to noncommercial research and educational purposes.
Revised: The usage is restricted to noncommercial research and educational purposes.

Original: Data Processing The original dataset contains 200 birds categories over 11 , 788 images withbinary attributes attached to each image.
Revised: Data Processing The original dataset contains 200 birds categories over 11 , 788 images withbinary attributes attached to each image.

Original: The image is rescaled to 224 × 224 .
Revised: The image is rescaled to 224 × 224 .

Original: Train Test Split : We follow the original train-validation split, resulting in 5 , 994 train images and 5 , 794 validation images.
Revised: Train Test Split : We follow the original train-validation split, resulting in 5 , 994 train images and 5 , 794 validation images.

Original: We combine the original training and validation set as our training set and use the original test set as our validation set.
Revised: We combine the original training and validation set as our training set and use the original test set as our validation set.

Original: The resulting training set contains 6 , 871 images and the validation set contains 6 , 918 images.
Revised: The resulting training set contains 6 , 871 images and the validation set contains 6 , 918 images.

Original: Network Design and Optimization We use ResNet-50 architecture as an encoder.
Revised: Network Design and Optimization We use ResNet-50 architecture as an encoder.

Original: We choose 2048-2048-128 MLP as the projection head.
Revised: We choose 2048-2048-128 MLP as the projection head.

Original: Batch normalization is used after each 2048 activation layers.
Revised: Batch normalization is used after each 2048 activation layers.

Original: Different than UT-Zappos dataset, we directly employ the original design of ResNet-50 since we are training on 224x224 images.
Revised: Different than UT-Zappos dataset, we directly employ the original design of ResNet-50 since we are training on 224x224 images.

Original: Similarly, LARS is used for optimization during the contrastive learning pretraining and we ﬁne tune a linear layer and use Limited-memory BFGS (L-BFGS (Liu and Nocedal, 1989)) optimizer after pretraining.
Revised: Similarly, LARS is used for optimization during the contrastive learning pretraining and we ﬁne tune a linear layer and use Limited-memory BFGS (L-BFGS (Liu and Nocedal, 1989)) optimizer after pretraining.

Original: All experiments are run with 1000 pretraining iterations and 500 L-BFGS ﬁne tuning steps.
Revised: All experiments are run with 1000 pretraining iterations and 500 L-BFGS ﬁne tuning steps.

Original: We use 128 batch size and train it on 4 1080ti NVIDIA
Revised: We use 128 batch size and train it on 4 1080ti NVIDIA

Original: GPUs.
Revised: GPUs.

Original: It takes about 13 hours to ﬁnish 1000 epochs training.
Revised: It takes about 13 hours to ﬁnish 1000 epochs training.

Original: F. CIFAR- 
Revised: F. CIFAR-

Original: The following section describes the experiments we performed on CIFAR-10.
Revised: The following section describes the experiments we performed on CIFAR-10.

Original: Accessibility CIFAR-10 (Krizhevsky et al., 2009) is an object detection dataset with 60 ,32 × 32 images in 10 classes.
Revised: Accessibility CIFAR-10 (Krizhevsky et al., 2009) is an object detection dataset with 60 ,32 × 32 images in 10 classes.

Original: The test set includes 10 , 000 images.
Revised: The test set includes 10 , 000 images.

Original: The dataset can be downloaded at https://www.cs.toronto.edu/~kriz/cifar.html .
Revised: The dataset can be downloaded at https://www.cs.toronto.edu/~kriz/cifar.html .

Original: Data Processing and Train and Test split We use the training and test split from the original dataset.
Revised: Data Processing and Train and Test split We use the training and test split from the original dataset.

Original: Network Design and Optimization We employ ResNet-50 backbone architecture, but we change the ﬁrst 7x7 2D convolution to 3x3 2D convolution and remove the ﬁrst max pooling layer in the normal ResNet-50 (See code for details).
Revised: Network Design and Optimization We employ ResNet-50 backbone architecture, but we change the ﬁrst 7x7 2D convolution to 3x3 2D convolution and remove the ﬁrst max pooling layer in the normal ResNet-50 (See code for details).

Original: This allows better results on CIFAR10 as this dataset consists of 32x32 resolution images.
Revised: This allows better results on CIFAR10 as this dataset consists of 32x32 resolution images.

Original: 2048-2048-128 projection head is employed during contrastive learning.
Revised: 2048-2048-128 projection head is employed during contrastive learning.

Original: Batch normalization is used after each 2048 activation layers.
Revised: Batch normalization is used after each 2048 activation layers.

Original: There are two CIFAR training settings we consider.
Revised: There are two CIFAR training settings we consider.

Original: The ﬁrst setting, which is reported in the main text, trains contrastive learning with 256 batch size for 400 epochs.
Revised: The ﬁrst setting, which is reported in the main text, trains contrastive learning with 256 batch size for 400 epochs.

Original: It takes a 4 GPU 1080ti Machine 8 hours to ﬁnish the pretraining.
Revised: It takes a 4 GPU 1080ti Machine 8 hours to ﬁnish the pretraining.

Original: For the second setting where we train with 512 batch size for 1000 epochs, it takes an DGX-1 machine 48 hours to ﬁnish training.
Revised: For the second setting where we train with 512 batch size for 1000 epochs, it takes an DGX-1 machine 48 hours to ﬁnish training.

Original: We use LARS optimizer for all CCL-K related experiments with base lr=1.5 and base batch size equals 256.
Revised: We use LARS optimizer for all CCL-K related experiments with base lr=1.5 and base batch size equals 256.

Original: F. C REATION OF C OLOR MNIST
Revised: F. C REATION OF C OLOR MNIST

Original: Accessiblity We create ColorMNIST dataset for experiments in Section 4.2 in the main text.
Revised: Accessiblity We create ColorMNIST dataset for experiments in Section 4.2 in the main text.

Original: The train and test split images can be accessed from our anonymous Github link (Section A).
Revised: The train and test split images can be accessed from our anonymous Github link (Section A).

Original: We allow any non-commerical usage of our dataset.
Revised: We allow any non-commerical usage of our dataset.

Original: Data Processing As discussed in Section 4.2 in the main text, we create the ColorMNIST dataset by assigning a random sampled color as MNIST’s background.
Revised: Data Processing As discussed in Section 4.2 in the main text, we create the ColorMNIST dataset by assigning a random sampled color as MNIST’s background.

Original: Images are converted into 32x resolution and only the background is augmented with the sampled color while the digit stroke pixel remains black.
Revised: Images are converted into 32x resolution and only the background is augmented with the sampled color while the digit stroke pixel remains black.

Original: Examples of the ColorMNIST images are shown in Figure 4.
Revised: Examples of the ColorMNIST images are shown in Figure 4.

Original: Training and Test Split
Revised: Training and Test Split

Original: We follow the original MNIST train/test split, resulting in 60,000 training images and 10,000 testing images spanning 10 digit categories.
Revised: We follow the original MNIST train/test split, resulting in 60,000 training images and 10,000 testing images spanning 10 digit categories.

Original: Network Design and Optimization To train our model, we use LeNet-5 (LeCun et al., 1998) as backbone architecture and use 2 layer linear projection head to project it to 128 dimension.
Revised: Network Design and Optimization To train our model, we use LeNet-5 (LeCun et al., 1998) as backbone architecture and use 2 layer linear projection head to project it to 128 dimension.

Original: We use LARS (You et al., 2017) as our optimizer.
Revised: We use LARS (You et al., 2017) as our optimizer.

Original: After our network is pretrained using contrastive learning, we discard the head annd ﬁne tune a linear layer use Limited-memory BFGS (L-BFGS (Liu and Nocedal, 1989)) as optimizer.
Revised: After our network is pretrained using contrastive learning, we discard the head annd ﬁne tune a linear layer use Limited-memory BFGS (L-BFGS (Liu and Nocedal, 1989)) as optimizer.

Original: All experiments are run with 1175 pretraining iterations and
Revised: All experiments are run with 1175 pretraining iterations and

Original: L-BFGS ﬁne tuning steps.
Revised: L-BFGS ﬁne tuning steps.

Original: F. I MAGE N ET - 
Revised: F. I MAGE N ET -

Original: The following section describes the experiments we performed on ImageNet-100 dataset in Sectionin the main text.
Revised: The following section describes the experiments we performed on ImageNet-100 dataset in Sectionin the main text.

Original: Accessibility This dataset is a subset of ImageNet-1K dataset, which comes from the ImageNet
Revised: Accessibility This dataset is a subset of ImageNet-1K dataset, which comes from the ImageNet

Original: Large Scale Visual Recognition Challenge (ILSVRC) 2012-2017 (Russakovsky et al., 2015).
Revised: Large Scale Visual Recognition Challenge (ILSVRC) 2012-2017 (Russakovsky et al., 2015).

Original: ILSVRC is for non-commercial research and educational purposes and we refer to the ImageNet ofﬁcial site for more information: https://www.image-net.org/download.php .
Revised: ILSVRC is for non-commercial research and educational purposes and we refer to the ImageNet ofﬁcial site for more information: https://www.image-net.org/download.php .

Original: Data Processing We select 100 classes from ImageNet-1K to conduct experiments.
Revised: Data Processing We select 100 classes from ImageNet-1K to conduct experiments.

Original: Selected class names can be accessed from our Github link.
Revised: Selected class names can be accessed from our Github link.

Original: Training and Test Split : The training split contains 128 , 783 images and the test split contains 5 ,images.
Revised: Training and Test Split : The training split contains 128 , 783 images and the test split contains 5 ,images.

Original: The images are rescaled to size 224 × 224 .
Revised: The images are rescaled to size 224 × 224 .

Original: Network Design and Optimization Hyper-parameters We use conventional ResNet-50 as the backbone for the encoder.
Revised: Network Design and Optimization Hyper-parameters We use conventional ResNet-50 as the backbone for the encoder.

Original:  MLP layer and a l 2 normalization layer is used after the encoder during training and discarded in the linear evaluation protocol
Revised: 2048-2048-128 MLP layer and a l 2 normalization layer is used after the encoder during training and discarded in the linear evaluation protocol.

Original: Batch normalization is used after each 2048 activation layers
Revised: Batch normalization is used after each 2048 activation layers.

Original: For optimization , we choose 128 batch size for WeaklySup CCLK setting and 512 batch size for HardNeg CCLK .
Revised: For optimization, we choose 128 batch size for WeaklySup CCLK setting and 512 batch size for HardNeg CCLK .

Original: Open AI CLIP model (Radford et al., ) is used to extract continuous feature from the raw image.
Revised: Open AI CLIP model (Radford et al., 2021) is used to extract continuous feature from the raw image.

Original: WeaklySup InfoNCE , we discretize the Z space using
Revised: WeaklySup InfoNCE , we discretize the Z space using

Original: Kmeans clustering with k=100, 200, 500, 2500.
Revised: Kmeans clustering with k=100, 200, 500, 2500.

Original: The best result of WeaklySup InfoNCE is produced by k=200.
Revised: The best result of WeaklySup InfoNCE is produced by k=200.

Original: All experiments are trained with 200 epochs and require 53 hours of training on DGX machine with 8 Tesla P100 GPUs
Revised: All experiments are trained with 200 epochs and require 53 hours of training on DGX machine with 8 Tesla P100 GPUs.