C ONTRASTIVE A TTRACTION AND C ONTRASTIVE R EPULSION FOR R EPRESENTATION L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Contrastive learning (CL) methods effectively learn data representations without label supervision, where the encoder contrasts each positive sample over multiple negative samples via a one-vs-many softmax cross-entropy loss.
By leveraging large amounts of unlabeled image data, recent CL methods have achieved promising results when pretrained on ImageNet, a well-curated dataset with balanced image classes.
However, they tend to yield worse performance when pretrained on images in the wild.
In this paper, to further improve the performance of CL and enhance its robustness on uncurated datasets, we propose a doubly CL strategy that contrasts positive samples and negative ones within themselves separately.
We realize this strategy with contrastive attraction and contrastive repulsion (CACR), which makes the query not only exert a greater force to attract more distant positive samples but also do so to repel closer negative samples.
Theoretical analysis reveals that CACR generalizes CL’s behavior by positive attraction and negative repulsion.
It further considers the intra-contrastive relation within the positive and negative pairs to narrow the gap between the sampled and true distribution, which is important when datasets are less curated.
Extensive large-scale experiments on standard vision tasks show that CACR not only consistently outperforms existing CL methods on benchmark datasets in representation learning, but also shows better robustness when generalized to pretrain on imbalanced image datasets.
I NTRODUCTION
The conventional Contrastive Learning (CL) loss (Oord et al., 2018; Poole et al., 2018) has achieved remarkable success in representation learning, beneﬁting downstream tasks in a variety of areas (Misra & Maaten, 2020;
He et al., 2020; Chen et al., 2020a; Fang & Xie, 2020; Giorgi et al., 2020).
This loss typically appears in a one-vs-many softmax form to make the encoder distinguish the positive sample within multiple negative samples.
In image representation learning, this scheme is widely used to encourage the encoder to learn representations that are invariant to unnecessary details in the representation space, for which the unit hypersphere is the most common assumption (Wang et al., 2017; Davidson et al., 2018;
Hjelm et al., 2018; Tian et al., 2019; Bachman et al., 2019).
Meanwhile, the contrast with negative samples is demystiﬁed as avoiding the collapse issue, where the encoder outputs a trivial constant, and uniformly distributing samples on the hypersphere (Wang & Isola, 2020).
To improve the quality of the contrast, various methods, such as large negative memory bank (Chen et al., 2020c), hard negative mining (Chuang et al., 2020; Kalantidis et al., 2020), and using strong or multi-view augmentations (Chen et al., 2020a; Tian et al., 2019; Caron et al., 2020), are proposed and succeed in learning powerful representations.
Since the conventional
CL loss achieves the one-vs-many contrast with a softmax cross-entropy loss, a notable concern is that the contrast could be sensitive to the sampled positive and negative pairs (Saunshi et al., 2019;
Chuang et al., 2020).
Given a sampled query, conventional CL methods usually randomly take one positive sample and multiple negative samples, and equally treat them in a softmax cross-entropy form, regardless of how informative they are to the query.
The sampled positive pair could make the contrast either easy or difﬁcult, while trivially selecting hard negative pairs could make the pretraining inefﬁcient.
Thus in practice, a large negative sample size is desired to ensure diverse samples are incorporated to make the contrast more effective (Wu et al., 2021), and the pretraining becomes less effective in large-scale real-world data, where the label could be noisy and rarely distributed in a balanced manner (Li et al., 2020b; 2021).
[Table]
(cid:8)(cid:30)(cid:25)(cid:24)(cid:28)(cid:24)(cid:23)(cid:22)(cid:21)(cid:12)(cid:21) (cid:8)(cid:30)(cid:25)(cid:24)(cid:28)(cid:24)(cid:23)(cid:22)(cid:21)(cid:21)(cid:11) (cid:8)(cid:30)(cid:25)(cid:24)(cid:28)(cid:24)(cid:23)(cid:22)(cid:21)(cid:10)
Considering the CL loss aims to train the encoder to distinguish the positive sample from multiple negative samples, an alternative intuition is that the positive samples need to be pulled close, while negative samples need to be pushed far away from the given query in the representation space.
This motivates the construction of Contrastive Attraction and Contrastive Repulsion (CACR), a doubly
CL framework where the positive and negative samples are ﬁrst contrasted within themselves before getting pulled and pushed from the query, respectively.
As shown in Figure 1, unlike conventional
CL, which equally treats samples and pulls/pushes them in the softmax cross-entropy contrast, CACR not only considers moving positive/negative samples close/away, but also models two conditional distributions to guide the movement of different samples.
The conditional distributions apply a doubly-contrastive strategy to compare the positive samples and the negative ones within themselves separately.
As an interpretation, if a selected positive sample is far from the query, it indicates the encoder does not sufﬁciently capture some information.
CACR will then assign a higher probability for the query to pull this positive sample.
Conversely, if a selected negative sample is too close to the query, it indicates the encoder has difﬁculty distinguishing them, and CACR will assign a higher probability for the query to push away this negative sample.
This double-contrast method determines if a positive/negative sample is easy or hard to move, making the optimization more effective.
Our main contributions include:
i)
We propose CACR, which achieves contrastive learning and produces useful representations by attracting the positive samples towards the query and repelling the negative samples away from the query, guided by two conditional distributions.
ii)
Our theoretical analysis shows that CACR generalizes the conventional CL loss.
The conditional distributions help treat the samples differently by modeling the intra-relation of positive/negative samples, which is proved to be important when the datasets are less curated.
iii)
Our experiments show that CACR consistently outperforms existing CL methods in a variety of settings, achieving state-of-the-art results on various benchmark datasets.
Moreover, in the case where the dataset has an imbalanced label distribution, CACR has better robustness and provides consistent better pretraining results than conventional CL methods.
R ELATED WORK
Plenty of unsupervised representation learning (Bengio et al., 2013) methods have been developed to learn good data representations, e.g., PCA (Tipping & Bishop, 1999), RBM (Hinton & Salakhutdinov, 2006), VAE (Kingma & Welling, 2014).
Among them, CL (Oord et al., 2018) is investigated as a lower bound of mutual information in early-stage (Gutmann & Hyv¨arinen, 2010; Hjelm et al., 2018).
Recently, many studies reveal that the effectiveness of CL is not just attributed to the maximization of mutual information (Tschannen et al., 2019; Tian et al., 2020a), motivating various works to demystify the contrastive learning scheme.
Contrastive representation learning.
In vision tasks, SimCLR (Chen et al., 2020a;b) studies extensive augmentations for positive and negative samples and intra-batch-based negative sampling.
A memory bank that caches representations (Wu et al., 2018) and a momentum update strategy are introduced to enable the use of an enormous number of negative samples (He et al., 2020; Chen et al.,
2020c).
Tian et al.
(2019; 2020b) consider the image views in different modalities and minimize the irrelevant mutual information between them.
Empirical researches observe the merits of using “hard” negative samples in CL, motivating additional techniques, such as Mixup and adversarial noise (Bose et al., 2018; Cherian & Aeron, 2020; Li et al., 2020a).
In a view that not all negative pairs are “true” negatives (Saunshi et al., 2019), Chuang et al.
(2020) propose a decomposition of the data distribution to approximate the true negative distribution.
RingCL (Wu et al., 2021) proposes to use “neither too hard nor too easy” negative samples by predeﬁned percentiles, and HN-CL (Robinson et al., 2021) applies Monte-Carlo sampling for selecting hard negative samples.
CL has also been developed in learning representations for text (Logeswaran & Lee, 2018), sequential data (Oord et al., 2018; H´enaff et al., 2019), structural data like graphs (Sun et al., 2020a;
Li et al., 2019; Hassani & Khasahmadi, 2020; Velickovic et al., 2019), reinforcement learning (Srinivas et al., 2020), and few-shot scenarios (Khosla et al., 2020; Sylvain et al., 2020).
Along with the success of these designs, works like Wang & Isola (2020) reveal the contrastive scheme is optimizing the alignment of positive samples and the uniformity of negative pairs in the limit of an inﬁnite number of negative samples.
Besides vision tasks, CL methods are widely applied to beneﬁt in a variety of areas such as NLP, graph learning, and cross-modality learning (Misra & Maaten, 2020;
He et al., 2020; Chen et al., 2020d; Fang & Xie, 2020; Giorgi et al., 2020; Gao et al., 2021; Korbar et al., 2018; Jiao et al., 2020;
Monfort et al., 2021).
Deep representation learning.
Besides contrastive methods for representation learning, a variety of relevant methods share the same purpose to learn powerful representation without label supervision.
Triplet loss and its extensions consider the pair-wise similarity between the query and all other samples in the given mini-batch, where the positive/negative similarities need to be maximized/minimized (Hoffer & Ailon, 2015; Sohn, 2016;
Wang et al., 2019; Sun et al., 2020b).
Instead of using both positive and negative pairs, some recent self-supervised methods require only positive pairs, deploying the spirit of Siamese networks and avoiding the collapse issue with stop gradient or a momentum updating strategy (Chen & He, 2021;
Grill et al., 2020; Caron et al., 2021).
In addition, Zbontar et al.
(2021) propose to train the encoder to make positive feature pairs have higher correlation and decrease the cross-correlation in different feature dimensions to avoid the collapse.
Another category is based on clustering, Caron et al.
(2020) and Li et al.
(2020c) introduce the prototypes as a proxy and train the encoder by learning to predict the cluster assignment.
CACR is closely related to the previous methods, where the conditional probability can be regarded as a re-weighting mechanism in triplet loss.
The CA part could also be naturally combined with the recent self-supervised methods without negative pairs and replace the CR part with stop-gradient or momentum updating to avoid collapse issues.
T HE PROPOSED APPROACH
In CL, for observations x 0: M ∼ p data ( x ) , we commonly assume that each x i can be transformed in certain ways, with the samples transformed from the same and different data regarded as positive and negative samples, respectively.
Speciﬁcally, we denote T ( x i , (cid:15) i ) as a random transformation of x i , where (cid:15) i ∼ p ( (cid:15) ) represents the randomness injected into the transformation.
In computer vision, (cid:15) i often represents a composition of random cropping, color jitter, Gaussian blurring, etc.
For each x 0 , with query x = T ( x 0 , (cid:15) 0 ) , we sample a positive pair ( x , x + ) , where x + = T ( x 0 , (cid:15) + ) , and M negative pairs { ( x , x − i ) } 1: M , where x − i = T ( x i , (cid:15) − i ) .
Denote τ ∈ R + , where R + := { x : x > 0 } , as a temperature parameter.
With encoder f θ : R n → S d − 1 , where we follow the convention to restrict the learned d -dimensional features with a unit norm, we desire to have similar and distinct representations for positive and negative pairs, respectively, via the contrastive loss as
[Equation]
Note by construction, the positive sample x + is independent of x given x 0 and the negative samples x − i are independent of x .
Intuitively, this 1-vs- M softmax cross-entropy encourages the encoder to not only pull the representation of a randomly selected positive sample closer to that of the query, but also push the representations of M randomly selected negative samples away from that of the query.
𝑥 !
[Figure]
[Figure]
conditionaldistribution sampleconstruction embeddinghypersphere 𝑥 # 𝑧 $# parametersharing parametersharing distancemetric
X
X 𝑥 :positiveconditionalweights :negativeconditionalweights :positivepaircost :negativepaircost :query/positive/negativeembedding distancemetric costmetric conditionaldistribution 𝑓 " 𝑓 " Π !" Π !
# 𝑧 $ 𝑧 $!
contrastiveattraction contrastiverepulsion
3. C ONTRASTIVE ATTRACTION AND CONTRASTIVE REPULSION
In the same spirit of letting the query attract positive samples and repel negative samples,
Contrastive Attraction and Contrastive Repulsion (CACR) directly models the cost of moving from the query to positive/negative samples with a doubly contrastive strategy:
[Equation]
where we denote π + and π − as the conditional distributions of intra-positive contrasts and intranegative contrasts, respectively, and c ( z 1 , z 2 ) as the point-to-point cost of moving between two vectors z 1 and z
, e.g. , the squared Euclidean distance (cid:107) z 1 − z 2 (cid:107) 2 or the negative inner product − z T 1 z 2 .
In the following we explain the doubly contrastive components with more details.
Contrastive attraction : The intra-positive contrasts is deﬁned in a form of the conditional probability, where the positive samples compete to gain a larger probability to be moved from the query.
Here we adapt to CACR a Bayesian strategy in Zheng & Zhou (2020), which exploits the combination of an energy-based likelihood term and a prior distribution, to quantify the difference between two implicit probability distributions given their empirical samples.
Speciﬁcally, denoting d t + ( · , · ) as a distance metric with temperature t + ∈ R + , e.g. , d t + ( z 1 , z 2 )
=
t + (cid:107) z 1 − z 2 (cid:107) 2 , given a query x = T ( x 0 , (cid:15) 0 ) , we deﬁne the conditional probability for moving it to positive sample x + = T ( x 0 , (cid:15) + ) as
[Equation]
where f θ ( · ) is an encoder parameterized by θ and Q + ( x ) is a normalization term.
This construction makes it more likely to pull x towards a positive sample that is more distant in their latent representation space.
With (3), the contrastive attraction loss L CA measures the expected cost of moving a query to its positive samples, as deﬁned in (2), which more heavily weighs c ( f θ ( x ) , f θ ( x + )) if f θ ( x ) and f θ ( x + ) are further away from each other.
Contrastive repulsion : On the contrary of the contrastive attraction shown in (3), we deﬁne the conditional probability for moving query x to a negative sample as
[Equation]
where t − ∈ R
+ is the temperature.
This construction makes it more likely to move query x to a negative sample that is closer from it in their representation space.
With (4), the contrastive repulsion loss L CR measures the expected cost to repel negative samples from the query shown in (2), which more heavily weighs c ( f θ ( x ) , f θ ( x − ))
if f θ ( x ) and f θ ( x − ) are closer to each other.
Choice of c ( · , · ) , d t + ( · , · ) and d t − ( · , · ) .
There could be various choices for the point-to-point cost function c ( · , · ) , distance metric d t + ( · , · ) in (3), and d t − ( · , · ) in (4).
Considering the encoder f θ outputs normalized vectors on the surface of a hypersphere, maximizing the inner product is equivalent to minimizing squared Euclidean distance.
Without loss of generality, we deﬁne them as
[Equation]
, where t + , t − ∈ R + .
There are other choices for c ( · , · ) and we show the ablation study in Appendix B.5.
3. M INI - BATCH BASED STOCHASTIC OPTIMIZATION
Under the CACR loss as in (2), to make the learning of f θ ( · ) amenable to mini-batch stochastic gradient descent (SGD) based optimization, we draw ( x data i , (cid:15) i ) ∼ p data
[Equation]
x ) p (
[Equation]
) for i = 1 , . . .
, M and then approximate the distribution of the query using an empirical distribution of M samples as
[Equation]
where the δ ( · ) denotes the Dirac delta function.
With query x
i and (cid:15) 1: K iid ∼ p ( (cid:15) ) , we approximate p ( x − i ) for Eqn.
(4) and p ( x + i | x data i ) for Eqn.
(3) as
[Equation]
Note we may improve the accuracy of ˆ p ( x − i ) in (5) by adding previous queries into the support of this empirical distribution.
Other more sophisticated ways to construct negative samples (Oord et al., 2018;
He et al., 2020; Khosla et al., 2020) could also be adopted to deﬁne ˆ p ( x − i ) .
We will elaborate these points when describing experiments.
Plugging (5) into (3) and (4), we approximate the conditional distributions with discrete distributions and obtain a mini-batch based CACR loss as ˆ L CACR =
ˆ L CA
+ ˆ L CR , where
[Equation]
+ ik ))
(cid:80) Kk (cid:48)
=1 e dt
+( f θ ( x i ) ,f θ ( x + ik (cid:48) ))
× c ( f θ ( x i ) , f θ ( x + ik )) , ˆ L CR := − 1 M (cid:80) Mi
=1 (cid:80) j (cid:54) =
i e − dt − ( f θ ( x i ) ,f θ ( x j )) (cid:80) j (cid:48)(cid:54) =
i e − dt − ( f θ ( x i ) ,f θ ( x j (cid:48) ))
× c ( f θ ( x i ) , f θ ( x j )) .
We optimize θ via SGD using ∇ θ ˆ L CACR , with the framework instantiated as in Figure 2.
[Table]
Relation with typical CL loss :
As shown in (2), with both the contrastive attraction component and contrastive repulsion component, CACR loss shares the same intuition of conventional
CL in pulling positive samples closer to and pushing negative samples away from the query in their representation space.
However, CACR realizes this intuition by introducing the double-contrast strategy on the point-to-point moving cost, where the contrasts appear in the intra-comparison within positive and negative samples, respectively.
The use of the double-contrast strategy clearly differs the CACR loss in (2) from the conventional CL loss in (1), which typically relies on a softmax-based contrast formed with a single positive sample and multiple equally-weighted independent negative samples.
A summary of the comparison between some representative CL losses and CACR is shown in Table 1.
P ROPERTY ANALYSIS OF CACR 4. O N CONTRASTIVE ATTRACTION
We ﬁrst analyze the effects w.r.t.
the positive samples.
With contrastive attraction, the property below suggests that the optimal encoder produces representations invariant to the noisy details.
Property 1.
The contrastive attraction loss L CA is optimized if and only if all positive samples of a query share the same representation as that query.
More speciﬁcally, for query x that is transformed from x 0 ∼ p data ( x ) , its positive samples share the same representation with it, which means
[Equation]
This property coincides with the characteristic (learning invariant representation) of the CL loss in
Wang & Isola (2020) when achieving the optima.
However, the optimization dynamic in contrastive attraction evolves in the context of x + ∼ π θ ( x + | x , x 0 ) , which is different from that in the CL.
Lemma 1.
Let us instantiate c ( f θ ( x ) , f θ ( x + ))
=
− f θ ( x ) (cid:62) f θ ( x + ) .
Then, the contrastive attraction loss L CA in (2) can be re-written as
[Equation]
which could further reduce to the alignment loss E x
[Equation]
( x
[Equation]
E x
[Equation]
x
[Equation]
∼ p (
[Equation]
x
[Equation]
(cid:2) − f θ ( x ) (cid:62)
f θ ( x + )) (cid:3) in
Wang & Isola (2020), iff π + θ ( x + | x , x 0 ) =
p ( x + | x 0 ) .
Property 1 and Lemma 1 jointly show contrastive attraction in CACR and the alignment loss in CL reach the same optima, while working in different sampling mechanism.
In practice x + and x are usually independently sampled augmentations in a mini-batch, as shown in Section 3.2, which raises a gap between the empirical distribution and the true distribution.
Our method makes the alignment more efﬁcient by considering the intra-relation of these positive samples to the query.
4. O N CONTRASTIVE REPULSION
Next we analyze the effects w.r.t.
the contribution of negative samples.
Wang & Isola (2020) reveal that a perfect encoder will uniformly distribute samples on a hypersphere under an uniform isometric assumption, i.e. , for any uniformly sampled x , x − iid ∼ p ( x ) , their latent representations z
= f θ ( x ) and z − = f θ ( x − ) also satisfy p ( z ) =
p ( z − ) .
We follow their assumption to analyze contrastive repulsion via the following lemma.
Lemma 2.
Without loss of generality, we deﬁne the moving cost and metric in the conditional distribution as c ( z 1 , z 2 ) =
d ( z 1 , z 2 ) =
(cid:107) z
1 − z 2 (cid:107) 22 .
When we are with an uniform prior, namely p ( x ) = p ( x − ) for any x , x − iid ∼ p ( x ) and p ( z ) = p ( z − ) given their latent representations z = f θ ( x ) and z − = f θ ( x − ) , then optimizing θ with L CR in (2) is the same as optimizing θ to minimize the mutual information between x and x − :
[Equation]
and is also the same as optimizing θ to maximize the conditional differential entropy of x − given x :
[Equation]
Here the minimizer θ (cid:63) of L CR is also that of I ( X ; X − ) , whose global minimum zero is attained iff
X and X − are independent, and the equivalent maximum of H ( X − | X ) indicates the optimization of L CR is essentially aimed towards the uniformity of representation about negative samples.
We notice that one way to reach the optimum suggested in the above lemma is optimizing θ by contrastive repulsion until that for any x ∼ p ( x ) , d ( f θ ( x ) , f θ ( x − )) is equal for all x − ∼ π − θ ( · | x ) .
This means for any sampled negative samples, their representations are also uniformly distributed after contrastive repulsion.
Interestingly, this is consistent with the uniformity property achieved by CL (Wang & Isola, 2020), which connects contrastive repulsion with CL in the perspective of negative sample effects.
Note that, although the above analysis builds upon the uniform isometric assumption, our method actually does not rely on it.
Here, we formalize a more general relation between the contrastive repulsion and the contribution of negative samples in CL without this assumption as follows.
Lemma 3.
As the number of negative samples M goes to inﬁnity, the contribution of the negative samples to the CL loss become the Uniformity Loss in AU-CL (Wang & Isola, 2020), termed as
L uniform for simplicity.
It can be expressed as an upper bound of L CR by adding the mutual information
I ( X ; X − ) in (7) :
[Equation]
As shown in Lemma 3, the mutual information I ( X ; X − ) helps quantify the difference between
L uniform and L CR .
The difference between drawing x − ∼ π − θ ( x − | x ) (in CR) and drawing x − independently in a mini-batch (in CL) is non-trivial as long as I ( X ; X − ) is non-zero.
In practice, this is true almost everywhere since we have to handle the skewed data distribution in real-world applications, e.g. , the label-shift scenarios (Garg et al., 2020).
In this view, CR does not require the representation space to be uniform like CL does, and is more robust to the complex cases through considering the intra-contrastive relation within negative samples.
E XPERIMENTS AND EMPIRICAL ANALYSIS
In this section, we ﬁrst study the CACR behaviors with small-scale experiments, where we use
CIFAR-10, CIFAR-100 (Hinton, 2007) and create two class-imbalanced CIFAR datasets as empirical veriﬁcation of our theoretical analysis.
We mainly compare with representative CL methods, divided into two different categories according to their positive sampling size: K = 1 and K = 4 .
For methods with a single positive sample ( K = 1 ), the baseline methods include the conventional CL loss (Oord et al., 2018), AlignUniform CL loss (AU-CL)
(Wang & Isola, 2020), and the CL loss with hard negative sampling (HN-CL)
(Robinson et al., 2021).
In the case of K = 4 , we take contrastive multi-view coding (CMC) loss (Tian et al., 2019)
(align with our augmentation settings and use augmentation views instead of channels) as the comparison baseline.
For a fair comparison, we keep for all methods with the same experiment setting including learning-rate, training epochs, etc. , but use their best temperature parameters; the mini-batch size for K = 4 is divided by 4 from those when
K = 1 to make sure the encoder leverages same samples in each iteration.
For large-scale datasets, we use ImageNet-1K
(Deng et al., 2009)
and compare with the state-of-theart frameworks (He et al., 2020; Zbontar et al., 2021;
Chen et al., 2020a; Caron et al., 2020; Grill et al., 2020; Huynh et al., 2020) on linear probing, where we report the Top-1 validation accuracy on ImageNet-1K data.
We also report the results of object detection/segmentaion following the transfer learning protocol.
To further justify our analysis, we also leverage two large-scale but label-imbalanced datasets (Webvision v1 and ImageNet-22K) for linear probing pretraining.
The reported numbers for baselines are from the original papers if available, otherwise we report the best ones reproduced with the settings according to their corresponding papers.
Please refer to Appendix C for detailed experiment setups.
5. S TUDIES AND ANALYSIS ON SMALL - SCALE DATASETS
Classiﬁcation accuracy: To facilitate the analysis, we apply all methods with an AlexNet-based encoder following the setting in Wang & Isola (2020), trained in 200 epochs.
We pretrained the encoder on regular CIFAR-10/100 data and create class-imbalanced cases by randomly sampling a certain number of samples from each class with a “linear” or “exponentional” rule by following the setting in Kim et al.
Speciﬁcally, given a dataset with C classes, for class l ∈ { 1 , 2 , ..., C } , we randomly take samples with proportion (cid:98) lC (cid:99) for “linear” rule and proportion exp( (cid:98) lC (cid:99) ) for “exponential” rule.
For evaluation we keep the standard validation/testing datasets.
Thus there is a label-shift between the training and testing data distributions.
Summarized in Table 2 are the results on both regular and class-imbalanced datasets.
The ﬁrst two columns show the results pretrained with curated data, where we can observe that in the case of
K = 1 , where the intra-positive contrast of CACR degenerates, CACR slightly outperforms all
CL methods.
When K = 4 , it is interesting to observe an obvious boost in performance, where
CMC improves CL by around 2-3% while CACR improves CL by around 3-4%, which supports our analysis that CA is helpful when the intra-positive contrast is not degenerated.
The right four columns present the linear probing results pretrained with class-imbalanced data, which show all the methods have a performance drop.
It is clear that CACR has the least performance decline in most cases.
Especially, when K = 4 , CACR shows better performance robustness due to the characteristic of doubly contrastive within positive and negative samples.
For example, in the “exponentional” setting of CIFAR-100, CL and HN-CL drop 12.57% and 10.73%, respectively, while CACR ( K = 4 ) drops 9.24%.
It is also interesting to observe HN-CL is relatively better among the baseline methods.
According to Robinson et al.
(2021), in HN-CL the negative samples are sampled according to the “hardness” w.r.t.
the query samples with an intra-negative contrast.
Its loss could converge to CACR ( K = 1 ) with inﬁnite negative samples.
This performance gap indicates that directly optimizing the
CACR loss could be superior when we have a limited number of samples.
With this class-imbalanced datasets, we provide the empirical support to our analysis: When the condition in Lemma 2 is violated,
CACR shows a clearer difference than CL and a better robustness with its unique doubly contrastive strategy within positive and negative samples.
On the effect of CA and CR: To further study the contrasts within positive and negative samples, in each epoch, we calculate the conditional entropy with (8) on every mini-batch of the validation data and take the average across mini-batches.
Then, we illustrate in Figure 3 the evolution of conditional entropy H ( X − | X ) w.r.t.
the training epoch on regular CIFAR-10 and class-imbalanced CIFAR-10.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
As shown, H ( X − | X ) is getting maximized as the encoder is getting optimized, indicating the encoder learns to distinguish the negative samples from given query.
It is also interesting to observe that in the case with multiple positive samples, this process is much more efﬁcient, where the conditional entropy reaches the possible biggest value rapidly.
This implies the CA module can further boost the repulsion of negative samples.
From the gap between CACR and CMC, we can learn although
CMC uses multiple positive in CL loss, the lack of intra-positive contrast shows the gap of repulsion efﬁciency.
In the right panel of Figure 3, the difference between CACR and baseline methods are more obvious, where we can ﬁnd the conditional entropy of baselines is slightly lower than pretrained with regular CIFAR-10 data.
Especially for vanilla CL loss, we can observe the conditional entropy has a slight decreasing tendency, indicating the encoder hardly learns to distinguish negative samples in this case.
Conversely, CACR still shows to remain the conditional entropy at a higher level, which explains the robustness shown in Table 2, and indicating a superior learning efﬁciency of CACR.
See
Appendix B.2 for similar observations on CIFAR-100 and exponential label-imbalanced cases.
In that part, we provide more quantitative and qualitative studies on the effects of conditional distributions.
5. C OMPARISON WITH SOTA S ON LARGE - SCALE DATASETS
For large-scale experiments, following the convention, we adapt CACR loss into the MoCo-v framework and pretrain a ResNet50 encoder with mini-batch size 256 on ImageNet-1K.
Linear probing: Table 3 summarizes the results of linear classiﬁcation, where a linear classiﬁer is trained on ImageNet-1K on top of ﬁxed representations of the pretrained ResNet50 encoder.
Similar to the case on small-scale datasets, CACR consistently shows better performance than the baselines using contrastive loss, improving SimCLR and MoCov2 by 2.7% and 2.2% respectively.
Compared with other self-supervised SOTAs, CACR also shows on par performance.
Label-imbalanced case: To strengthen our analysis on small-scale label-imbalanced data, we specially deploy two real-world, but less curated datasets Webvision v1 and ImageNet-22K that have long-tail label distributions for encoder pretraining and evaluate the linear classiﬁcation accuracy on ImageNet-1K. We pretrain encoder with 100/20 epochs on Webvision v1/ImageNet-22K and compare with the encoder pretrained with 200 epochs on ImageNet-1K to make sure similar samples
[Table]
[Table] [Table]
have been seen in the pretraining.
The results are shown in Table
[Table]
where we can see CACR still outperforms the MoCov2 baseline and shows better robustness when generalized to wild image data.
Object detection and segmentation:
Besides the linear classiﬁcation evaluation, following the protocols in previous works (Tian et al., 2019;
He et al., 2020; Chen et al., 2020c; Wang & Isola, 2020), we use the pretrained ResNet50 on ImageNet-1K for object detection and segmentation task on Pascal VOC (Everingham et al., 2010) and COCO (Lin et al., 2014) by using detectron2 (Wu et al., 2019).
The experimental setting details are shown in Appendix C.2 and kept the same as He et al.
(2020) and Chen et al.
(2020c).
The test AP, AP 50 , and AP 75 of bounding boxes in object detection and test AP, AP 50 , and AP 75 of masks in segmentation are reported in Table 5.
We can observe that the performances of CACR is consistently better than baselines using contrastive objectives, and even better than recent self-supervised learning SOTAs.
C ONCLUSION
In this paper, we rethink the limitation of conventional contrastive learning (CL) methods that use the contrastive loss but merely consider the intra-relation between samples.
We introduce a novel
Contrastive Attraction and Contrastive Repulsion (CACR) loss with a doubly contrastive strategy, which constructs for two conditional distributions to respectively model the importance of a positive sample and that of a negative sample to the query according to their distances to the query.
Our theoretical analysis and empirical results show that optimizing the CACR loss can effectively attract positive samples and repel negative ones from the query as CL intends to do, but is more robust in more general cases.
Extensive experiments on small, large-scale, and imbalanced datasets consistently demonstrate the superiority and robustness of CACR over the state-of-the-art methods in contrastive representation learning and related downstream tasks.
E THICS S TATEMENT
Contrastive learning (CL) is effective in learning data representations without label supervision and has led to notable recent progresses in a variety of research areas, such as computer vision.
Recently proposed advanced CL methods often require a huge amount of data and thus cost large computational energy.
Especially in the case where one needs to use multiple positive pairs in the contrast.
Instead of contrasting each positive pair over multiple negative pairs with the classic softmax cross-entropy, our work discovers that the contrastive attraction within positives and contrastive repulsion within negatives bring new insight in self-supervised representation learning.
CACR, which naturally takes multiple positive samples in the contrast without making the contrast complexity become combinatorial in the number of positive pairs, has demonstrated clear improvements over existing CL methods.
However, the same as existing CL methods, our method is not designed to resist the potential biases existing in the dataset, e.g. the false negatives in data.
At the current stage, CACR relies on the positive contrast to implicitly alleviate this issue: if a false negative sample is repelled too far away from the query, in the positive attraction, it will be assigned with larger probability to be pulled back.
This raises the risk of the quality of learned representations.
In the future work, we aim and also encourage other researchers to consider the resistance of these potential risks to make the learned representations more robust and powerful.
R EPRODUCIBILITY S TATEMENT
Our paper provides both theoretical analysis and empirical studies on the proposed algorithm.
For theoretical results, to make sure the properties hold and the analysis are correct, we provide stepby-step proof in Appendix A. For empirical results on both small-scale and large-scale datasets, we provide detailed experiment settings and Pytorch-style pseudo-code in Appendix C. We also include an example code with instruction as supplementary material to ensure the reproducibility.
A P ROOFS AND DETAILED DERIVATION
Proof of Property 1 .
By deﬁnition, the point-to-point cost c ( z 1 , z 2 ) is always non-negative.
Without loss of generality, we deﬁne it with the Euclidean distance.
When (6) is true, the expected cost of moving between a pair of positive samples, as deﬁned as L CA in (2), will reach its minimum at 0.
When (6) is not true, by deﬁnition we will have L CA > 0 , i.e. , L CA = 0 is possible only if (6) is true.
Proof of Lemma 1 .
By changing the reference distribution of the expectation from π + θ ( · | x , x 0 ) to p ( · | x 0 ) , we can directly re-write the CA loss as:
[Equation]
which complete the proof.
Proof of Lemma 2 .
Denoting
[Equation]
we have
[Equation]
Thus we have
[Equation]
9) where C 1 = E x ∼ p ( x ) E x − ∼ π − θ ( · | x ) [ln p ( x − )] and C 2 = − E x ∼ p ( x ) ln Z ( x ) .
Under the assumption of a uniform prior on p ( x ) , C 1 becomes a term that is not related to θ .
Under the assumption of a uniform prior on p ( z ) , where z = f θ ( x ) , we have
[Equation]
which is also not related to θ .
Therefore, under the uniform prior assumption on both p ( x ) and p ( z ) , minimizing L CR is the same as maximizing H ( X − | X ) , as well as the same as minimizing
I ( X, X − ) .
Proof of Lemma 3 .
The CL loss can be decomposed as an expected dissimilarity term and a logsum-exp term:
L CL :=
E
[Equation]
=
E ( x , x + ) (cid:20) −τ f
[Equation]
( x ) (cid:62) f
[Equation]
( x + ) (cid:21)
+
E (
[Equation]
−
[Equation] [Equation]
ln (cid:32) e f
[Equation]
[Equation] [Equation]
M
[Equation]
i =
e f θ ( x − i ) (cid:62)
[Equation]
where the positive sample x + is independent of x given x 0 and the negative samples x − i are independent of x .
As the number of negative samples goes to inﬁnity, following Wang & Isola (2020), the normalized CL loss is decomposed into the sum of the align loss, which describes the contribution of the positive samples, and the uniform loss, which describes the contribution of the negative samples:
[Equation]
[Figure]
[Equation]
[Table]
[Equation]
With importance sampling, the second term in the RHS of the above equation can be further derived into:
[Equation]
Apply the Jensen inequality, the second term is decomposed into the negative cost plus a log density ratio:
[Equation]
Deﬁning the point-to-point cost function between two unit-norm vectors as c ( z 1 , z 2 ) = − z (cid:62) 1 z(same as the Euclidean cost since (cid:107) z 1 − z 2 (cid:107) 22 / 2 = 1 − z (cid:62) 1 z 2 ), we have
[Equation]
This concludes the relation between the contribution of the negative samples in CL and that in CACR.
B A DDITIONAL EXPERIMENTAL RESULTS
In this section
[Table]
we provide additional results in our experiments, including ablation studies, and corresponding qualitative results.
B. A DDITIONAL RESULTS WITH A LEX N ET AND R ES N ET 50 ENCODER ON SMALL - SCALE DATASETS
Following benchmark works in contrative learning, we add STL-10 dataset to evaluate CACR in small-scale experiments.
As an additional results on small-scale datasets, we test the performance of CACR two different encoder backbones.
Here we strictly follow the same setting of Wang &
Isola (2020) and Robinson et al. (2021), and the results are shown in Table 6 and 7.
We can observe with ResNet50 encoder backbone, CACR with single positive or multiple positive pairs consistently outperform the baselines.
Compared with the results in Table 6, the CACR shows a more clear improvement over the CL baselines
[Table] [Table]
[Table]
B. O N THE EFFECTS OF CONDITIONAL DISTRIBUTION
Supplementary studies of CA and CR: As a continuous ablation study shown in Figure 3, we also conduct similar experiments on CIFAR-100, where we study the evolution of conditional entropy
H ( X − | X ) w.r.t.
the training epoch.
The results are shown in Figure 4, and the results of exponential label-imbalanced data are shown in Figure 5.
Similar to the observation on CIFAR-10, shown in
Figure 3, we can observe H ( X − | X ) is getting maximized as the encoder is getting optimized with these methods, as suggested in Lemma 2.
In the right panel, We can observe baseline methods have lower conditional entropy, which indicates the encoder is less effective in distinguish the nagative samples from query, while CACR consistently provides better performance than the other methods indicating the better robustness of CACR.
As a qualitative veriﬁcation, we randomly take a query from a mini-batch, and illustrate its positive and negative samples and their conditional probabilities in Figure 6.
As shown, given this query of a dog image, the positive sample with the largest weight contains partial dog information, indicating the encoder to focus on texture information; the negatives with larger weights are more related to the dog category, which encourages the encoder to focus on distinguishing these “hard” negative samples.
In total, the weights learned by CACR enjoy the interpretability compared to the conventional CL.
We study different deﬁnition of the conditional distribution.
From Table 8, we can observe that the results are not sensitive to the distance space.
In addition, as we change π + to assign larger probability to closer samples, the results are similar to those using single positive pair (K=1).
Moreover, the performance drops if we change π − to assign larger probability to more distant negative samples.
Uniform Attraction and Uniform Repulsion: A degenerated version of CACR
[Figure]
[Figure]
To reinforce the necessity of the contrasts within positives and negatives before the attraction and repulsion, we introduce a degenerated version of CACR here, where the conditional distributions are forced to be uniform.
Remind c
( z 1 , z 2 ) as the point-to-point cost of moving between two vectors z 1 and z 2 , e.g. , the squared Euclidean distance (cid:107) z 1 − z 2 (cid:107) 2 or the negative inner product − z T 1 z 2 .
In the same spirit of (1), we have considered a uniform attraction and uniform repulsion (UAUR) without doubly contrasts within positive and negative samples, whose objective is
[Equation]
[Equation]
The intuition of UAUR is to minimize/maximize the expected cost of moving the representations of positive/negative samples to that of the query, with the costs of all sample pairs being uniformly weighted.
While (1) has been proven to be effective for representation learning, our experimental results do not ﬁnd (11) to perform well, suggesting that the success of representation learning is not guaranteed by uniformly pulling positive samples towards and pushing negative samples away from the query.
Distinction between CACR and UAUR :
Compared to UAUR in (11) that uniformly weighs different pairs, CACR is distinct in considering the dependency between samples: as the latent-space distance between the query and its positive sample becomes larger, the conditional probability becomes higher, encouraging the encoder to focus more on the alignment of this pair.
In the opposite, as the distance between the query and its negative sample becomes smaller, the conditional probability becomes higher, encouraging the encoder to push them away from each other.
In order to further explore the effects of the conditional distribution, we conduct an ablation study to compare the performance of different variants of CACR with/without conditional distributions.
Here, we compare 4 conﬁgurations of CACR ( K = 4 ): ( i ) CACR with both positive and negative conditional distribution; ( ii ) CACR without the positive conditional distribution; ( iii ) CACR without the negative conditional distribution; ( iv ) CACR without both positive and negative conditional distributions, which refers to UAUR model (see Equation 11).
As shown in Table 9, when discarding the positive conditional distribution, the linear classiﬁcation accuracy slightly drops.
As the negative
[Table]
(cid:10)(cid:22)(cid:24)(cid:24)(cid:22)(cid:17)(cid:18)(cid:13)(cid:18)(cid:12)(cid:29)(cid:11)(cid:25)(cid:24)(cid:20)(cid:19)(cid:29)(cid:18)(cid:21)(cid:25)(cid:17)(cid:16)(cid:15)(cid:29)(cid:21)(cid:18) conditional distribution is discarded, there is a large performance drop compared to the full CACR objective.
With the modeling of neither positive nor negative conditional distribution, the UAUR shows a continuous performance drop, suggesting that the success of representation learning is not guaranteed by uniformly pulling positive samples closer and pushing negative samples away.
The comparison between these CACR variants shows the necessity of the conditional distribution.
experiment is done
[Table]
CIFAR-10 CIFAR-
[Table]
CACR
[Table] [Table]
[Table] [Table]
As qualitative illustrations, we randomly ﬁx one mini-batch, and randomly select one sample as the query.
Then we extract the features with the encoder trained with CL loss and CACR ( K = 1 ) loss at epochs 1, 20, and 200, and visualize the (four) positives and negatives in the embedding space with t -SNE (van der Maaten & Hinton, 2008).
For more clear illustration, we center the query in the middle of the plot and only show samples appearing in the range of [ − 10 , 10] on both x and y axis.
The results are shown in Figure 7(c), from which we can ﬁnd that as the the encoder is getting trained, the positive samples are aligned closer and the negative samples are pushed away for both methods.
Compared to the encoder trained with CL, we can observe CACR shows better performance in achieving this goal.
Moreover, we can observe the distance between any two data points in the plot is more uniform, which conﬁrms that CACR shows better results in the maximization of the conditional entropy H ( X − | X ) .
B. A BLATION STUDY
Does CACR( K (cid:62) 2 ) outperform by seeing more samples?
To address this concern, in our main paper, we intentionally decrease the mini-batch size as M = 128 .
Thus the total number of samples
QueryPositiveNegative 10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 10.
QueryPositiveNegative
[Figure]
[Figure] [Figure]
positive samples are aligned closer to the query (Property 1), and the conditional differential entropy H ( X − | X ) is progressively maximized, driving the distances d ( f θ ( x ) , f θ ( x − )) towards uniform (Lemma 2).
[Table]
used per iteration is not greater than those used when K = 1 .
To further justify if the performance boost comes from seeing more samples when using multiple positive pairs, we also let the methods allowing single positive pair train with double epochs.
As shown in Table 10, we can observe even trained with 400 epochs, the performance of methods using single positive pair still have a gap from those using multiple positive pairs.
On the effects of negative sampling size: We investigate the model performance and robustness with different sampling size by varying the mini-batch size used in the training.
On all the small-scale datasets, the mini-batches are applied with size 64, 128, 256, 512, 768 and the corresponding linear classiﬁcation results are shown in Figure 8.
From this ﬁgure, we can see that CACR ( K = 4 ) consistently achieves better performance than other objectives.
For example, when mini-batch size is 256, CACR ( K = 4 ) outperforms CMC by about 0.4%-1.2%.
CACR ( K = 1 ) shows better performance in most of the cases, while slightly underperforms than the baselines with mini-batch size 64.
A possible explanation could be the estimation of the conditional distribution needs more samples to provide good guidance for the encoder.
[Figure]
On the effects of positive sampling size: We conduct experiments to investigate the model performance with different positive sampling size by using different K values in the pretraining:
K ∈ { 1 , 2 , 4 , 6 , 8 , 10 } on CIFAR-10/100 and K ∈ { 1 , 2 , 3 , 4 } on ImageNet-1K. Similar to our experiment setting, in 200 epochs, we apply AlexNet encoder on CIFAR-10 and CIFAR-100 and apply ResNet50 encoder on ImageNet-1K. Shown in Figure 9, we can observe as K increases, the linear classiﬁcation accuracy increases accordingly.
Correspondingly, we also evaluate the GPU time of CACR loss with different choices of K, as shown in Table 11 and Table 16.
1   10 K 84.0 84.
A cc u r a c
y
( % ) (a) CIFAR1   10 K 56.
A cc u r a c y ( % ) (
[Figure]
CIFAR4 K 69.
A cc u r a c y ( % )
[Figure]
[Table]
On the effects of hyper-parameter t + , t − : Remind in the deﬁnition of positive and negative conditional distribution, two hyper-parameters t + and t − are involved as following:
[Equation]
:= e − t −(cid:107)
f θ ( x ) − f θ ( x − ) (cid:107) 2 p ( x − ) (cid:82) e − t −(cid:107)
f θ ( x ) − f θ ( x − ) (cid:107) 2 p ( x − ) d x − .
In this part, we investigate the effects of t + and t − on representation learning performance on small-scale datasets, with mini-batch size 768 ( K = 1 ) and 128 ( K = 4 ) as an ablation study.
We search in a range { 0 . 5 , 0 .
The results are shown in Table 12 and Table 13.
As shown in these two tables, from Table 12, we observe the CACR shows better performance with smaller values for t + .
Especially when t + increases to 3 . 0 , the performance drops up to about
[Table]
1.9% on CIFAR-100.
For analysis, since we have K = 4 positive samples for the computation of positive conditional distribution, using a large value for t + could result in an over-sparse conditional distribution, where the conditional probability is dominant by one or two positive samples.
This also explains why the performance when t + = 3 . 0 is close to the classiﬁcation accuracy of CACR ( K = 1 ).
Similarly, from Table 13, we can see that a small value for t − will lead to the degenerated performance.
Here, since we are using mini-batches of size 768 ( K = 1 ) and 128 ( K = 4 ), a small value for t − will ﬂatten the weights of the negative pairs and make the conditional distribution closer to a uniform distribution, which explains why the performance when t − = 0 .
5 is close to those without modeling π − θ .
Based on these results, the values of t + ∈ [0 . 5 , 1 . 0] and t − ∈ [0 . 9 , 2 . 0] could be good empirical choices according to our experiment settings on these datasets.
B. A DDITIONAL COMPARISONS
In this part we provide more comparisons with baseline methods.
For small-scale experiments, we still compare with contrastive learning methods, conventional CL loss, align-uniform loss, and hard negative sampling CL loss.
For large-scale experiments, we continue to compare with contrastive learning loss on ImageNet-100 and ImagNet-1K with MoCov2 framework, and provide comparisons with SOTAs pretrained with different epochs.
Training efﬁciency on small-scale datasets: On CIFAR-10, CIFAR-100 and STL-10, we pretrained
AlexNet encoder in 200 epochs and save linear classiﬁcation results with learned representations every 10 epochs.
Shown in Figure 10, CACR consistently outperforms the other methods in linear classiﬁcation with the learned representations at the same epoch, indicating a superior learning efﬁciency of CACR.
Comparison with contrastive learning methods on ImageNet: For large-scale experiments, following the convention, we adapt all methods into the MoCo-v2 framework and pre-train a ResNet encoder in 200 epochs with mini-batch size 128/256 on ImageNet-100/ImageNet-1k.
Tablesummarizes the results of linear classiﬁcation on these two large-scale datasets.
Similar to the case on small-scale datasets, CACR consistently shows better performance, improving the baselines at least by 1.74% on ImageNet-100 and 0.71% on ImageNet-1K. In MoCo-v2, with multiple positive samples,
CACR improves the baseline methods by 2.92% on ImageNet-100 and 2.75% on ImageNet-1K.
It is worth highlighting that the improvement of CACR is more signiﬁcant on these large-scale datasets, where the data distribution could be much more diverse compared to these small-scale ones.
This is not surprising, as according to our theoretical analysis, CACR’s double-contrast within samples enhances the effectiveness of the encoder’s optimization.
Moreover, we can see CACR ( K = 1 )
[Table]
CIFAR51.
STL80.
CACR ( K = 4 )
CIFAR85.
CIFAR58.
STL83.
[Figure]
shows a clear improvement over HN-CL.
A possible explanation is that although both increasing the negative sample size and selecting hard negatives are proposed to improve the CL loss, the effectiveness of hard negatives is limited when the sampling size is increased over a certain limit.
As
CACR targets to repel the negative samples away, the conditional distribution still efﬁciently guides the repulsion when the sampling size becomes large.
[Table] [Table]
Comparison with other SOTAs:
Besides the methods using contrastive loss
[Table]
we continue to compare with the self-supervised learning methods like BYOL, SWaV, SimSiam, etc.
that do not involve the contrasts with negative samples.
Table 15 provides more detailed comparison with all state-of-the-arts in different epochs and could better support the effectiveness of CACR:
We can observe CACR achieves competitive results and generally outperforms most of SOTAs at the same epoch in linear classiﬁcation tasks.
We also compare the computation complexity.
Table 16 reports computation complexity to provide quantitative results in terms of positive number K, where we can observe the computation cost of CACR slightly increases as K increase
[Table]
but does not increase as that when using multi-positives in CL loss.
[Table]
B. C ONNECTION TO OTHER REPRESENTATION LEARNING METHODS Results of different cost metrics
Recall that the deﬁnition of the point-to-point cost metric is usually set as the quadratic Euclidean distance:
[Equation]
In practice, the cost metric deﬁned in our method is ﬂexible to be any valid metrics.
Here, we also investigate the performance when using the Radial Basis Function (RBF) cost metrics:
[Equation]
where t ∈ R + is the precision of the Gaussian kernel.
With this deﬁnition of the cost metric, our method is closely related to the baseline method AU-CL (Wang & Isola, 2020), where the authors calculate pair-wise RBF cost for the loss w.r.t.
negative samples.
Following Wang & Isola (2020)
[Equation]
we replace the cost metric when calculate the negative repulsion cost with the RBF cost and modify ˆ L CR as:
[Equation]
[Table]
Here the negative cost is in log scale for numerical stability.
When using the RBF cost metric, we use the same setting in the previous experiments and evaluate the linear classiﬁcation on all small-scale datasets.
The results of using Euclidean and RBF cost metrics are shown in Table 17.
From this table, we see that both metrics achieve comparable performance, suggesting the RBF cost is also valid in our framework.
In CACR, the cost metric measures the cost of different sample pairs and is not limited on speciﬁc formulations.
More favorable cost metrics can be explored in the future.
[Table]
Discussion: Relation to triplet loss CACR is also related to the widely used triplet loss (Schroff et al., 2015; Sun et al., 2020b).
A degenerated version of CACR where the conditional distributions are all uniform can be viewed as triplet loss, while underperform the proposed CACR, as discussed in
Section B.2.
In the view of triplet loss, CACR is dealing with the margin between expected positive pair similarity and negative similarity:
[Equation]
which degenerates to the generic triplet loss if the conditional distribution degenerates to a uniform distribution:
[Equation]
This degeneration also highlights the importance of the Bayesian derivation of the conditional distribution.
The experimental results of the comparison between CACR and the degenerated uniform version (equivalent to generic triplet loss) are presented in Table 9.
Moreover, CACR loss can degenerate to a triplet loss with hard example mining if π t + ( x + | x ) and π t + ( x + | x ) are sufﬁciently concentrated, where the density shows a very sharp peak:
[Equation]
which corresponds to the loss shown in Schroff et al.
As shown in Table 12 and 13, when varying t + and t − to sharpen/ﬂatten the conditional distributions.
Based on our observations, when t + = 3 and t − = 3 , the conditional distributions are dominated by 1-2 samples, where CACR can be regarded as the above-mentioned triplet loss, and this triplet loss with hard mining slightly underperforms CACR.
From these views, CACR provides a more general form to connect the triplet loss.
Meanwhile, it is interesting to notice CACR explains how triplet loss is deployed in the self-supervised learning scenario.
Relation to CT .
The CT framework of Zheng & Zhou (2020) is primarily focused on measuring the difference between two different distributions, which are referred to as the source and target distributions, respectively.
It deﬁnes the expected CT cost from the source to target distributions as the forward CT, and that from the target to source as the backward CT.
Minimizing the combined backward and forward CT cost, the primary goal is to optimize the target distribution to approximate the source distribution.
By contrast, in CACR, since the marginal distributions of x and x + are the same and these of x and x − are also the same, there is no need to differentiate the transporting directions.
In addition, the primary goal of CACR is not to regenerate the data but to learn f θ ( · ) that can provide good latent representations for downstream tasks.
C E XPERIMENT DETAILS
On small-scale datasets, all experiments are conducted on a single GPU, including NVIDIA
Ti and RTX 3090; on large-scale datasets, all experiments are done on 8 Tesla-V100-32G GPUs.
C.
S MALL - SCALE DATASETS : CIFAR-10, CIFAR-100, AND STLFor experiments on CIFAR-10, CIFAR-100, and STL-10, we use the following conﬁgurations:
• Data Augmentation : We strictly
follow the standard
data augmentations
to
construct positive and negative samples introduced in prior works in contrastive learning (Wu et al., 2018;
Tian et al., 2019; Hjelm et al., 2018; Bachman et al., 2019; Chuang et al., 2020; He et al., 2020; Wang
Isola, 2020)
The augmentations include image resizing, random cropping, ﬂipping, color jittering, and gray-scale conversion.
We
provide a Pytorch-style augmentation code in Algorithm 1, which is exactly the same as the one used in Wang & Isola (2020).
import torchvision.transforms as transforms # CIFAR-10 Transformation def transform_cifar10(): returntransforms.
] transforms.
[transforms.
RandomResizedCrop(32, scale=(0.2, 1)), transforms.
RandomHorizontalFlip(),
default
5 transforms.
ColorJitter(0.4, 0.4, 0.4, 0.4), transforms.
RandomGrayscale(p=0.2),transforms.
ToTensor(),#normalize to value in [0,1] transforms.Normalize((0.5071,0.4867,0.4408), (0.2675, 0.2565, 0.2761), ) ])
.ToTensor(),#normalize to value in [0,1] transforms.Normalize((0.4409,0.4279,0.3868), (0.2683, 0.2610, 0.2687), ) ]) • Feature Encoder :
Following the experiments in Wang & Isola (2020), we use an AlexNetbased encoder as the feature encoder for these three datasets, where encoder architectures are the same as those used in the corresponding experiments in Tian et al.
(2019) and Wang
[Table]
& Isola
Moreover, we also follow the setups in Robinson et al.
and test the performance of CACR with a ResNet50 encoder (results are shown in Table 7).
• Model Optimization : We apply the mini-batch SGD with 0.9 momentum and 1e-4 weight decay.
The learning rate is linearly scaled as 0.12 per 256 batch size (Goyal et al., 2017).
The optimization is done over 200 epochs, and the learning rate is decayed by a factor of 0.
at epoch 155, 170, and 185.
• Parameter Setup :
On CIFAR-10, CIFAR-100, and STL-10, we follow Wang & Isola (2020) to set the training batch size as M = 768 for baselines.
The hyper-parameters of CL, AU-CL 1 , and HN-CL 2 are set according to the original paper or online codes.
Speciﬁcally, the temperature parameter of CL is τ = 0 .
19 , the hyper-parameters of AU-CL are t = 2 . 0 , τ = 0 .
19 , and the hyper-parameter of HN-CL are τ = 0 .
5 , β = 1 . 0 3 , which shows the best performance according to our tuning.
For CMC and CACR with multiple positives, the positive sampling size is K = 4 .
To make sure the performance is not improved by using more samples, the training batch size is set as M = 128 .
For CACR, in both single and multi-positive sample settings, we set t + = 1 . 0 for all small-scale datasets.
As for t − , for CACR ( K = 1 ), t − is 2.0, 3.0, and 3.0 on CIFAR-10,CIFAR100, and STL-10, respectively.
For CACR ( K = 4 ), t − is 0.9, 2.0, and 2.0 on CIFAR-10, CIFAR100, and STL-10, respectively.
For further ablation studies, we test t + and t − with the search in the range of [0 . 5 , 0 . 7 , 0 . 9 , 1 . 0 , 2 . 0 , 3 . 0] , and we test all the methods with several mini-batch sizes M ∈ { 64 , 128 , 256 , 512 , 768 } .
• Evaluation : The feature encoder is trained with the default built-in training set of the datasets.
In the evaluation, the feature encoder is frozen, and a linear classiﬁer is trained and tested on the default training set and validation set of each dataset, respectively.
Following Wang & Isola (2020), we train the linear classiﬁer with Adam optimizer over 100 epochs, with β 1 = 0 .
5 , β 2 = 0 .
999 , (cid:15) =
10 − 8 , and 128 as the batch size.
The initial learning rate is 0.001 and decayed by a factor of 0.2 at epoch 60 and epoch 80.
Extracted features from “fc7” are employed for the evaluation.
For the ResNet50 setting in Robinson et al.
(2021), the extracted features are from the encoder backbone with dimension 2048.
C. L ARGE - SCALE DATASETS : I MAGE N ET -100 AND I MAGE N ET -1K
For large-scale datasets, the Imagenet-1K is the standard ImageNet dataset that has about 1.28 million images of 1000 classes.
The ImageNet-100 contains randomly selected 100 classes from the standard
ImageNet-1K dataset and the classes used here are the same with Tian et al.
and Wang &
Isola (2020), listed in Table 18.
For pretraining with less curated data 4 , considering Webvision v and ImageNet-22k respectively have 2.4 and 14.2 million images, we decrease the training epoch to 1/2 and 1/10 when pretraining on these two datasets as done in Li et al.
We follow the standard settings in these works and describe the experiment conﬁgurations as follows:
• Data Augmentation :
Following Tian et al.
(2019; 2020b)
; Wang & Isola (2020); Chuang et al.
; He et al.
(2020), and Chen et al.
(2020c)
, the data augmentations are the
https
align uniform
Algorithm
# ImageNet-100 and ImageNet-1K Transformation
# MoCo v2’s aug: similar to SimCLR https://arxiv.org/abs /2002.
def transform_imagenet():returntransforms.
Compose([ transforms.
RandomResizedCrop(224, scale=(0.2, 1.)), transforms.
RandomApply([transforms.
ColorJitter(0.4, 0.4, 0.4, 0.1)],p=0.8), transforms.
RandomGrayscale(p=0.2),transforms.
RandomApply([moco.loader.
GaussianBlur([.1, 2.])], p=0.5), transforms.
RandomHorizontalFlip(), transforms.
ToTensor(),transforms.
Normalize(mean=[0.485,0.456, 0.406], std=[0.229, 0.224, 0.225]) ])
same as the standard protocol, including resizing, 1x image cropping, horizontal ﬂipping, color jittering, and gray-scale conversion with speciﬁc probability.
The full augmentation combination is shown in Algorithm 2.
• Feature Encoder : On these two datasets, we apply the MoCo-v2 framework (Chen et al., 2020c), where the ResNet50 (He et al., 2016) is a commonly chosen feature encoder architecture.
The output dimension of the encoder is set as 128. • Model Optimization : Following the standard setting in He et al.
; Chen et al.
(2020c)
; Wang & Isola (2020), the training mini-batch size is set as 128 on ImageNet-100 and 256 on ImageNet-1K. We use a mini-batch stochastic gradient descent (SGD) optimizer with 0.
momentum and 1e-4 weight decay.
The total number of training epochs is set as 200.
The learning rate is initialized as 0.03, decayed by a cosine scheduler for MoCo-V2 at epochand epoch 160.
In all experiments, the momentum of updating the ofﬂine encoder is 0.999.
• Parameter Setup :
On ImageNet-100 and ImageNet-1K, for all methods, the queue size for negative sampling is 65,536.
The training batch size is 128 on ImageNet-100 and 256 on ImageNet.
For CACR, we train with two positive sampling sizes K =
and K = 4 and the parameters in the conditional weight metric are respectively set as t + = 1 . 0 , t − = 2 . 0 .
For baselines, according to their papers and Github pages (Tian et al., 2019; Wang & Isola, 2020; Robinson et al., 2021), the temperature parameter of CL is τ = 0 .
2 , the hyper-parameters of AU-CL are t = 3 . 0 , τ = 0 . 2 , and the hyper-parameters of HN-CL are τ = 0 .
5 , β = 1 . 0 .
Note that CMC ( K = 1 ) reported in the main paper is trained with 240 epochs and with its own augmentation methods (Tian et al., 2019).
For CMC ( K = 4 ), the temperature is set τ = 0 .
according to the setting in Tian et al.
and the loss is calculated with Equation (8) in the paper, which requires more GPU resources than 8 Tesla-V100-32G GPUs with the setting on ImageNet-1K. • Linear Classiﬁcation Evaluation :
Following the standard linear classiﬁcation evaluation (He et al., 2020; Chen et al., 2020c; Wang & Isola, 2020), the pre-trained feature encoders are ﬁxed, and a linear classiﬁer added on top is trained on the train split and test on the validation split.
The linear classiﬁer is trained with SGD over 100 epochs, with the momentum as 0.9, the mini-batch size as 256, and the learning rate as 30.0, decayed by a factor of 0.1 at epoch 60 and epoch 80.
• Feature Transferring Evaluation (
Detection and Segmentation) :
The pre-trained models are transferred to various tasks including PASCAL VOC 5 and COCO 6 datsets.
Strictly following the same setting in He et al.
(2020), for the detection on Pascal VOC,
a Faster
R-CNN (Ren et al., 2016) with an R50-C4 backbone is ﬁrst ﬁne-tuned end-to-end on the
VOC 07+12 trainval set and then evaluated on the VOC 07 test set with the COCO suite of
metrics (Lin et al., 2014).
The image scale is [480, 800] pixels during training and 800 at inference.
For the detection and segmentation on COCO dataset, a Mask R-CNN (He et al., 2017) with C4 backbone (1x schedule) is applied for the end-to-end ﬁne-tuning.
The model is tuned on the train2017 set and evaluate on val2017 set, where the image scale is in [640, 800] pixels during training and is 800 in the inference.
• Estimation of ˆ π − θ with MoCo-v2: Following the strategy in Wang & Isola (2020), we estimate ˆ π − θ with not only the cost between queries and keys, but also with the cost between queries.
Speciﬁcally, at each iteration, let M be the mini-batch size, N be the queue size, { f q i } Mi =1 be the query features, and { f k j } Nj =1 be the key features.
The conditional distribution is calculated as:
[Equation]
To be clear the Pytorch-like pseudo-code is provided in Algorithm 3.
In MoCo-v2 framework, as the keys are produced from the momentum encoder this estimation could help
the main encoder get involved
with the gradient from
the conditional distribution
which
is consistent
with
the
formulation in Section
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
if with_intra_batch: # compute the distance of negative samples in the mini-batch: Mx(M -1)
intra_batch_sq_dists = torch.norm(q[:,None] - q, dim=-1).pow(2).masked_select(˜torch.
eye(q.shape[0], dtype=
bool).cuda()).view(q.shape[0], q.shape[0] - 1) # combine the distance of negative samples from the queue and intra-batch: Mx(K+M-1) sq_dists_for_cost =
torch.cat([sq_dists_for_cost, intra_batch_sq_dists], dim=1) sq_dists_for_weights =
torch.cat([sq_dists_for_weights, intra_batch_sq_dists], dim=1)
# calculate the negative conditional distribution: if with_intra_batch==True Mx(K
[Table]