E Q C O : E QUIVALENT R ULES FOR S ELF - SUPERVISED C ONTRASTIVE L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we propose a method, named EqCo ( Eq uivalent Rules for
Co ntrastive Learning), to make self-supervised learning irrelevant to the number of negative samples in the contrastive learning framework.
Inspired by the
InfoMax principle, we point that the margin term in contrastive loss needs to be adaptively scaled according to the number of negative pairs in order to keep steady mutual information bound and gradient magnitude.
EqCo bridges the performance gap among a wide range of negative sample sizes, so that for the ﬁrst time, we can use only a few negative pairs (e.g. smaller than 256 per query) in the InfoNCE loss of self-supervised contrastive training on large-scale vision datasets like ImageNet with almost no accuracy drop.
This is quite a contrast to the widely used large batch training or memory bank mechanism in current practices.
Equipped with EqCo, our simpliﬁed MoCo (SiMo) achieves comparable accuracy with MoCo v on ImageNet (linear evaluation protocol) while only involves 16 negative pairs per query instead of 65536, suggesting that large quantities of negative samples might not be a critical factor in contrastive learning frameworks.
I NTRODUCTION AND B ACKGROUND
Self-supervised learning has recently received much attention in the ﬁeld of visual representation learning (Hadsell et al.
; Dosovitskiy et al.
Oord et al.
; Bachman et al.
H ´ enaff et al.
; Wu et al.
; Tian et al.
; He et al.
; Misra & Maaten (2020);
Grill et al.
(2020); Cao et al.
(2020)), as its potential to learn universal representations from unlabeled data.
Among various self-supervised methods, one of the most promising research paths is contrastive learning (Oord et al.
(2018)), which has been demonstrated to achieve comparable or even better performances than supervised training for many downstream tasks such as image classiﬁcation, object detection, and semantic segmentation (Chen et al., 2020c;
He et al., 2020; Chen et al., 2020a;b).
The core idea of contrastive learning is brieﬂy summarized as follows:
ﬁrst, extracting a pair of embedding vectors ( q ( I ) , k
( I ))
(named query and key respectively) from the two augmented views of each instance I ; then, learning to maximize the similarity of each positive pair ( q ( I ) , k ( I )) while pushing the negative pairs ( q ( I ) , k
( I (cid:48) ))
(i.e., query and key extracted from different instances accordingly) away from each other.
To learn the representation, an InfoNCE loss (Oord et al.
Wu et al.
(2018)) is conventionally employed in the following formulation (slightly modiﬁed with an additional margin term):
[Equation]
where q and k i ( i = 0 , . . .
, K ) stand for the query and keys sampled from the two (augmented) data distributions D and D (cid:48) respectively.
Speciﬁcally, k 0 is associated to the same instance as q ’s while other k i s not; hence we name k 0 and k i ( i > 0 ) positive sample and negative samples respectively in the remaining text, in which K is the number of negative samples (or pairs) for each query.
The temperature τ and the margin m are hyper-parameters.
In most previous works, m is trivially set to zero (e.g. Oord et al.
; He et al.
; Chen et al.
(2020a))
or some handcraft values (e.g.
Xie et al.
In contrastive learning research, it has been widely believed that enlarging the number of negative samples K boosts the performance (H´enaff et al.
; Tian et al.
; Bachman et al.
For example, in MoCo (He et al.
the ImageNet accuracy rises from 54.7% to 60.6% under linear classiﬁcation protocol when K grows from 256 to 65536.
Such observation further drives a line of studies how to effectively optimize under a number of negative pairs, such as memory bank methods (Wu et al.
; He et al.
and large batch training (Chen et al.
(2020a)), either of which empirically reports superior performances when K becomes large.
Analogously, in the ﬁeld of supervised metric learning (Deng et al.
; Wang et al.
; Sun et al.
Wang et al.
(2020)), loss in the similar form as Eq.
1 is often applied on a lot of negative pairs for hard negative mining.
Besides, there are also a few theoretical studies supporting the viewpoint.
For instance,
Oord et al.
(2018) points out that the mutual information between the positive pair tends to increase with the number of negative pairs K ; Wang & Isola (2020)
ﬁnd that the negative pairs encourage features’ uniformity on the hypersphere; Chuang et al.
(2020) suggests that large K leads to more precise estimation of the debiased contrastive loss; etc.
Despite the above empirical or theoretical evidence, however, we point out that the reason for using many negative pairs is still less convincing.
First , unlike the metric learning mentioned above, in self-supervised learning, the negative terms k i in Eq.
1 include both “true negative” (whose underlying class label is different from the query’s, similarly hereinafter) and “false negative” samples, since the actual ground truth label is not available.
So, intuitively large K should not always be beneﬁcial because the risk of false negative samples also increases (known as class collision problem).
Arora et al.
(2019) thus theoretically concludes that a large number of negative samples could not necessarily help.
Second , some recent works have proven that by introducing new architectures ( e.g. , a predictor network in BYOL (Grill et al., 2020)), or designing new loss functions ( e.g. , Caron et al. ; Ermolov et al.
(2020)), state-of-the-art performance can still be obtained even without any explicit negative pairs.
In conclusion, it is still an open question whether large quantities of negative samples are essential to contrastive learning.
After referring to the above two aspects, we rise a question: is a large K really essential in the contrastive learning framework ?
We propose to rethink the question from a different view: note that in Eq. 1, there are three hyper-parameters: the number of negative samples K , temperature τ , and margin m .
In most of previous empirical studies (He et al.
; Chen et al.
(2020a))
, only K is changed while τ and m are usually kept constant.
Do the optimal hyper-parameters of τ and m varies with K ?
If so, the performance gains observed from larger K s may be a wrong interpretation – merely brought by suboptimal hyper-parameters’ choices for small K s, rather than much of an essential.
In the the paper, we investigate the relationship among three hyper-parameters and suggest an equivalent rule :
[Equation]
where α is a constant.
We ﬁnd that if the margin m is adaptively adjusted based on the above rule, the performance of contrastive learning is irrelevant to the size of K , in a very large range (e.g. K ≥ 16 ).
For example, in MoCo framework, by introducing EqCo the performance gap between K = 256 and K = 65536 (the best conﬁguration reported in He et al.
(2020)) almost disappears (from 6.1% decrease to 0.2%).
We call this method “ Eq uivalent Rules for Co ntrastive learning” ( EqCo ).
For completeness, as the other part of EqCo we point that adjusting the learning rate according to the conventional linear scaling rule satisﬁes the equivalence for different number of queries per batch.
Theoretically, following the InfoMax principle (Linsker (1988)) and the derivation in CPC (Oord et al.
, we prove that in EqCo , the lower bound of the mutual information keeps steady under various numbers of negative samples K .
Moreover, from the back-propagation perspective, we further prove that in such conﬁguration the upper bound of the gradient norm is also free of
K ’s scale.
The proposed equivalent rule implies that, by assigning α = K 0 , it can “mimic” the optimization behavior under K 0 negative samples even if the physical number of negatives K (cid:54) =
K 0 .
The “equivalant” methodology of EqCo follows the well-known linear scaling rule (Krizhevsky (2014);
Goyal et al.
(2017)), which suggests scaling the learning rate proportional to the batch size if the loss satisﬁes with the linear averaged form: L = 1 N (cid:80)
Ni =1 f ( x i ; θ ) .
However, linear scaling rule cannot be directly applied on InfoNCE loss (Eq. 1), which is partially because InfoNCE loss includes two batch sizes (number of queries and keys respectively) while linear scaling rule only involves one, in addition to the nonlinearity of the keys in InfoNCE loss.
In the experiments of
SimCLR (Chen et al.
(2020a))
, learning rates under different batch sizes are adjusted with linear scaling rule (or square root rule for LARS (You et al.
(2017)) optimizer), but the accuracy gap is still very large (57.5%@batch=256 vs. 64+%@batch=8192, 100 epochs training).
EqCo challenges the belief that self-supervised contrastive learning requires large quantities of negative pairs to obtain competitive performance, making it possible to design simpler algorithms.
We thus present SiMo , a simpliﬁed contrastive learning framework based on MoCo v2 (Chen et al.
(2020c)).
SiMo is elegant, efﬁcient, free of large batch training and memory bank; moreover, it can achieve superior performances over state-of-the-art even if the number of negative pairs is extremely small (e.g. 16), without bells and whistles.
The contributions of our paper are summarized as follows:
• We challenge the widely accepted belief that on large-scale vision datasets like ImageNet, large size of negative samples is critical for contrastive learning. We interpret it from a different view: it may be a wrong interpretation as the hyper-parameters are not set to the optimum.
• We propose EqCo, an equivalent rule to adaptively set hyper-parameters between small and large numbers of negative samples, which proves to bridge the performance gap.
• We present SiMo, a simpler but stronger baseline for contrastive learning.
E Q C O : E QUIVALENT R ULES FOR C ONTRASTIVE L EARNING
In this section we introduce EqCo .
We mainly consider the circumstance of optimizing the InfoNCE loss (Eq. 1) with SGD .
For each batch of training, there are two meanings of the concept “batch size”, i.e., the size of negative samples/pairs K per query, and the number of queries (or positive pairs ) N per batch.
Hence our equivalent rules accordingly consist of two parts, which will be introduced in the next subsections.
2. T HE C ASE OF N EGATIVE P AIRS
Our derivation is mainly inspired by the model of Contrastive Predictive Coding (CPC)
(Oord et al.
(2018)), in which InfoNCE loss is interpreted as a mutual information estimator.
We further extend the method so that it is applicable to InfoNCE loss with a margin term (Eq. 1), which is not considered in Oord et al.
Following the concept in Oord et al.
(2018), given an query embedding q (namely the context in Oord et al.
and suppose K + 1 random key embeddings x = { x i } i =0 ,...,K , where there exists exactly one entry (e.g., x i ) sampled from the conditional distribution P ( x i | q ) while others (e.g., x j ) sampled from the “proposal” distribution P ( x j ) independently.
According to which entry corresponds to the conditional distribution, we therefore deﬁnes K + 1 candidate distributions for x (denoted by { H i } i =0 ,...,K ), where the probability density of x under H i is
P H i ( x ) =
P ( x i | q ) (cid:81) j
(cid:54) =
i P ( x j ) .
So, given the observed data X = { k 0 , . . .
, k K } of x , the probability where x is sampled from H 0 rather than other candidates is thus derived with Bayes theorem:
[Equation]
where we denote P + and P − as the prior probabilities of H 0 and H i ( i > 0) respectively.
We point that Eq.
2 introduces a generalized form to that in Oord et al.
(2018) by taking the priors into account.
Referring to the notations in Eq.
1, we suppose that H 0 is the ground truth distribution of x (since kis the only positive sample).
By modeling the density ratio P ( k i | q ) / P ( k i ) ∝ e q (cid:62) k
i /τ ( i = 0 , . . .
, K ) and letting P + / P − =
e − m/τ , the negative log-likelihood L opt (cid:44) E q ,X − log
Pr [ x ∼ H 0 | q , X ] can be regarded as the optimal value of L NCE .
Similar to the methodology of Oord et al.
(2018), we explore the lower bound of L opt :
[Equation]
where I ( · , · ) means mutual information.
The approximation in the second row is guaranteed by
Law of Large Numbers as well as the fact P ( k i | q ) ≈ P ( k i ) since k i ( i > 0) and q are “almost” independent.
The inequality in the last row is resulted from P ( k 0 | q ) ≥ P ( k 0 ) as k 0 and q are extracted from the same instance.
Therefore the lower bound of the mutual information (noted as f bound ( m, K ) ) between the positive pair ( k 0 , q ) is:
[Equation]
So, minimizing L NCE (Eq. 1) towards L opt implies maximizing the lower bound of the mutual information, which is also satisﬁed when m (cid:54) = 0 .
In the case of m = 0 , the result is consistent with that in Oord et al.
Oord et al.
(2018) further points out the bound increases with K , which indicates larger K encourages to learn more mutual information thus could help to improve the performance.
Nevertheless, different from Oord et al.
our model does not require m to be zero, so the lower bound in Eq.
4 is also a function of e m/τ .
Thus we have the following theorem:
Theorem 1.
(Main, EqCo for negative pairs)
The mutual information lower bound of InfoNCE loss in Eq.
1 is irrelevant to the number of negative pairs K , if
[Equation]
where α is a constant coefﬁcient.
And in the circumstances the bound is given by:
[Equation]
which can be immediately obtained by substituting Eq. 5 into Eq.
We name Eq.
5 as “ equivalent condition ”.
Theorem 1 suggests a property of equivalency : under the condition of Eq. 5, no matter what the number of physical negative pairs K is, the optimal solution of L NCE (Eq. 1) is “equivalent” in the sense of the same mutual information lower bound.
The bound is controlled by a hyper-parameter α rather than K .
Eq.
6 further implies that the lower bound also correlates to the conﬁguration of
K = α without margin , which suggests we can “mimic” the InfoNCE loss’s behavior of K
= Kunder a different physical negative sample size K 1 , just by applying Eq.
5 with α = K 0 .
It inspires us to simplify the existing state-of-the-art frameworks (e.g. MoCo (He et al.
with fewer negative samples but as accurate as the original conﬁgurations, which will be introduced next.
We empirically validate Theorem 1 as follows.
Notice that f bound is difﬁcult to calculate directly because L opt is not known.
Instead, we plot the empirical mutual information lower bound ˆ f bound ( m, K ) (cid:44) log (1 + Ke m/τ ) − L NCE .
So, we have ˆ f bound ≤ f bound ; when L NCE converges to the optimum L opt ,
ˆ f bound is an approximation of f bound .
In Fig.
1, we plot the evolution of ˆ f bound during the training of MoCo v2 under different conﬁgurations.
Obviously, when it converges, without EqCo ˆ f bound keeps increasing with the number of negative pairs K ; in contrast, after applying the equivalent condition (Eq. 5) ˆ f bound converges to almost the same value under different K s. The empirical results are thus consistent with Theorem 1.
[Figure]
Remarks 1.
The equivalent condition in Eq.
5 suggests the margin m is inversely correlated with
K .
It is intuitive, because the larger K is, the more risks of class collision (Arora et al.
it suffers from, so we need to avoid over-penalty for negative samples near the query, thus smaller m is used; in contrast, if K is very small, we use larger m to exploit more “hard” negative samples.
Besides, recall that the margin term e m/τ is deﬁned as the ratio of the prior probabilities P − / P + in
Eq.
If the the equivalent condition Eq.
5 satisﬁes, i.e., P − / P + = α/K , we have P + = 1 / (1 + α ) (notice that K P − + P + ≡ 1 ), suggesting that the prior probability of the ground truth distribution His supposed to be a constant ignoring the number of negative samples K .
While in previous works (usually without the margin term, or m = 0 ) we have P + = 1 / ( K + 1) .
It is hard to distinguish which prior is more reasonable.
However at least, we intuitively suppose keeping a constant prior for the ground truth distribution may help to keep the optimal choices of hyper-parameters steady under different K s, which is also consistent with our empirical observations.
Remarks 2.
In Theorem 1, it is worth noting that K refers to the number of negative samples per query .
In the conventional batched training scheme, negative samples for different queries could be either (fully or partially) shared or isolated, i.e., the total number of distinguishing negatives samples per batch could be different, which is not ruled by Theorem 1.
However, we empirically ﬁnd the differences in implementation do not result in much of the performance variation.
The following theorem further supports the equivalent rule (Theorem 1) from back-propagation view:
Theorem 2.
Given the equivalent condition (Eq. 5) and a query embedding q as well as the corresponding positive sample k 0 , for L NCE in Eq.
1 the expectation of the gradient norm w.r.t.
q is bounded by 1 :
[Equation]
Please refer to the Appendix A.1 for the detailed proof.
Note that we assume the embedding vectors are normalized , i.e., (cid:107) k
i (cid:107) = 1( i = 0 , · · · , K ) , which is also a convention in recent contrastive learning works.
Theorem 2 indicates that, equipped with the equivalent rule (Eq. 5), the upper bound of the gradient norm is irrelevant to the number of negative samples K .
Fig.
4 (see the Appendix A.2) further validates our theory: the gradient norm becomes much more steady after using EqCo under different
K s. Since the size of K affects little on the gradient magnitude, gradient scaling techniques, e.g. linear scaling rule , are not required speciﬁcally for different K s. Eq.
7 also implies that the temperature τ signiﬁcantly affects the gradient norm even EqCo is applied – it is why we only recommend to modify m for equivalence (Eq. 5), though the mutual information lower bound is determined by e m/τ as a whole.
2. T HE C ASE OF P OSITIVE P AIRS
In practice the InfoNCE loss (Eq. 1) is usually optimized with batched SGD , which can be represented as empirical risk minimization :
[Equation]
where N is the number of queries (or positive pairs ) per batch ; ( q j , k j, 0 ) ∼ ( D , D (cid:48) ( q j )) is the j -th positive pair, and L ( j ) NCE ( q j , k j, 0 ) is the corresponding loss.
For different j , L ( j ) NCE is (almost) independent of each other, because q j is sampled independently.
Hence, Eq. 8 satisﬁes the form of linear scaling rule (Krizhevsky (2014); Goyal et al.
, suggesting that the learning rate should be adjusted proportional to the number of queries N per batch .
Remarks 3.
Previous work like SimCLR (Chen et al.
(2020a)) also proposes to apply linear scaling rule.
The difference is, in SimCLR it does not clarify the concept of “batch size” refers to the number of queries or the number of keys.
However in our paper, we explicitly point that the linear scaling rule needs to be applied corresponding to the number of queries per batch ( N ) rather than K .
2. E MPIRICAL E VALUATION
In this subsection we conduct experiments on the three state-of-the-art self-supervised contrastive learning frameworks – MoCo (He et al.
MoCo v2 (Chen et al. (2020c)) and SimCLR (Chen et al. (2020a)) to verify our theory in Sec. 2.1 and Sec. 2.2.
We propose to alter K and N separately to examine the correctness of our equivalent rules .
Implementation details.
We follow most of the training and evaluation settings recommended in the original papers respectively.
The only difference is, for SimCLR, we adopt SGD with momentum rather than LARS as the optimizer.
We use ResNet-50 (He et al.
as the default network architecture.
128-d features are employed for query and key embeddings.
Unless specially mentioned, all models are trained on ImageNet (Deng et al.
for 200 epochs without using the ground truth labels.
We report the top-1 accuracy under the conventional linear evaluation protocol according to the original paper respectively.
The number of queries per batch ( N ) is set to 256 by default.
All models are trained with 8 GPUs.
It is worth noting the way we alter the number of negative samples K independent of N during training.
For MoCo and MoCo v2, we simply need to set the size of the memory bank to K .
Specially, if K < N , in the current batch the memory bank is actually composed of K random keys sampled from the previous batch.
While for SimCLR, if K < N we random sample K negative keys for each query independently.
We do not study the case that K > N for SimCLR.
We mainly consider the ease of implementation in designing the strategies; as mentioned in Remarks 2 (Sec. 2.1), it does not affect the empirical conclusion.
Quantitative results.
Fig.
2 illustrates the effect of our equivalent rule under different K s. Our experiments start with the best conﬁgurations (i.e. K = 65536 for MoCo and MoCo v2, and
K = 256 for SimCLR 2 ), then we gradually reduce K and benchmark the performance.
Results in
Fig.
2 indicates that, without EqCo the accuracy signiﬁcantly drops if K becomes very small (e.g.
K < 64 ).
While with EqCo, by setting α to “mimic” the optimal K , the performance surprisingly keeps steady under a wide range of K s. The comparison fully demonstrates EqCo is essential especially when the number of negative pairs is small.
4096 16384 65536 K
T o pA cc u r a c y ( % ) 37.
[Figure]
256 K
T o pA cc u r a c y ( % ) 45. SimCLR SimCLR w/ EqCo SimCLR w/
[Figure] [Figure]
= 4096) Besides, Table 1 compares the results of MoCo v2 under different number of queries N , while
K = 65536 is ﬁxed.
It is clear that, with linear scaling rule (Krizhevsky (2014);
Goyal et al.
(2017)), the ﬁnal performance is almost unchanged under different N , suggesting the effectiveness of our equivalent rule for N . [Table]
[Table]
S I M O : A S IMPLER BUT S TRONGER B ASELINE
EqCo inspires us to rethink the design of contrastive learning frameworks.
The previous state-ofthe-arts like MoCo and SimCLR heavily rely on large quantities of negative pairs to obtain high performances, hence implementation tricks such as memory bank and large batch training are introduced, which makes the system complex and tends to be costly.
Thanks to EqCo, we are able to design a simpler contrastive learning framework with fewer negative pairs.
We propose SiMo , a simpliﬁed variant of MoCo v2 (Chen et al.
(2020c)) equipped with EqCo.
We follow most of the design in Chen et al.
(2020c), where the key differences are as follows:
Memory bank.
Both MoCo and MoCo v2 employ memory bank to maintain large number of negative embeddings k i , in which there is a side effect: every positive embedding k 0 is always extracted from a “newer” network than the negatives’ in the same batch, which could harm the performance.
In SiMo, we thus cancel the memory bank as we only rely on a few negative samples per batch.
Instead, we use the momentum encoder to extract both positive and negative key embeddings from the current batch.
Shufﬂing BN vs. Sync BN.
In MoCo v1/v2, shufﬂing BN (He et al.
(2020)) is proposed to remove the obvious dissimilarities of the BN (Ioffe & Szegedy (2015))
statistics between the positive (from current mini-batch) and the negatives (from memory bank), so that the model can make predictions based on the semantic information of images rather than the BN statistics.
In contrast, since the positive and negatives are from the same batch in SiMo, therefore, we use sync BN
(Peng et al.
(2018)) for simplicity and more stable statistics.
There are a few other differences, including 1) we use a BN attached to each of the fully-connected layers; 2) we introduce a warm-up stage at the beginning of the training, which follows the method- ology in SimCLR (Chen et al.
(2020a)).
Apart from all the differences mentioned above, the architecture and the training (including data augmentations) details in SiMo are exactly the same as
MoCo v2’s.
In the following text, the number of queries per batch (N) is set to 256, and the backbone network is ResNet-50 by default.
Quantitative results.
First, we empirically demonstrate the necessity of EqCo in SiMo framework.
We choose the number of negative samples K = 256 as the baseline, then reduce K to evaluate the performance.
Fig.
3 shows the result on ImageNet using linear evaluation protocol .
Without EqCo, the accuracy signiﬁcantly drops when K is very small.
In contrast, using EqCo to “mimic” the case of large K (by setting α to 256), the accuracy almost keeps steady even under very small K s.
[Figure] [Figure]
[Table]
Table 2 further compares our SiMo with state-of-the-art self-supervised contrastive learning methods on ImageNet.
Using only 16 negative samples per query, SiMo outperforms MoCo v2 (68.1% vs.
If we increase α to 65536 to “simulate” the case under huge number of negative pairs, the accuracy further increases to 68.5%.
Moreover, when we extend the training epochs to 800, we get the accuracy of 72.1%, surpassing previous state-of-the-art MoCo v2 by 1.0%.
The experiments indicate that SiMo is a simpler but more powerful baseline for self-supervised contrastive learning.
Readers can refer to the Appendix B for more experimental results of SiMo.
L IMITATIONS AND F UTURE W ORK
Theorem 1 suggests that given the equivalent condition (Eq. 5), InfoNCE losses under various K s are “equivalent” in the sense of the same mutual information lower bound, which is also backed up with the experiments in Fig.
However, Fig. 2 (a) shows that if K is smaller than a certain value (e.g. K ≤ 16 ), some frameworks like MoCo v2 start to degrade signiﬁcantly even with EqCo; while for other frameworks like SiMo (Fig. 3), the accuracy almost keeps steady for very small K s.
Tschannen et al.
(2019) also point that the principle of InfoMax cannot explain all the phenomena in contrastive learning.
We will investigate the problem in the future, e.g. from other viewpoints such as gradient noise brought by small K s (Fig. 4 in Appendix A.2 gives some insights).
Though the formulation of Eq. 1 is very common in the ﬁeld of supervised metric learning , which is usually named margin softmax cross-entropy loss (Deng et al., 2019;
Wang et al., 2018; Sun et al., 2020).
Nevertheless, unfortunately, our equivalent rule seems invalid to be generalized to those problems (e.g. face recognition).
The major issue lies in the approximation in Eq.
3, we need the negative samples
k i to be independent of the query q , which is not satisﬁed in supervised tasks.
According to Fig.
2 and Fig.
3, the beneﬁts of EqCo becomes signiﬁcant if K is sufﬁciently small (e.g. K < 64 ).
But in practice, for modern computing devices (e.g. GPUs) it is not that difﬁcult to use ∼ 256 negative pairs per query.
Applying EqCo to “simulate” more negative pairs via adjusting α can further boost the performance, however, whose accuracy gains becomes relatively marginal.
For example, in Table 2 under 200 epochs training, SiMo with α = 65536 outperforms that of α = 256 by only 0.5%.
It could be a fundamental limitation of InfoNCE loss.
We will investigate the problem in the future.
A D ETAILS ABOUT T HEOREMA. P ROOF OF E Q .
Given the equivalent condition (Eq. 5) and a query embedding q as well as the corresponding positive sample k 0 , for L NCE in Eq.
1 the expectation of the gradient norm w.r.t.
q is bounded by:
[Equation]
Proof.
For simplicity, we denote the term exp( q (cid:62) k
i /τ )
as s i ( i = 0 , . . .
, K ) .
Then L NCE can be rewritten as:
[Equation]
The gradient of L NCE with respect to q is easily to derived:
[Equation]
Owing to the Triangle Inequality and the fact that k i ( i = 0 , . .
. , K ) is normalized, the norm of gradient is bounded by:
[Equation]
Since the cosine similarity between q and k i ( i = 1 , . . . , K ) is bounded in [ − 1 , 1] , we know the expectation of E k i ∼D (cid:48)
[ s i ] exists.
According to Inequality (12) and Jensen’s Inequality , we have:
[Equation]
Replacing s i by exp( q (cid:62) k
i /τ ) , the proof of Theorem 2 is completed.
A. E MPIRICAL E VALUATION ON THE M AGNITUDE OF G RADIENTS
[Figure]
B M ORE E XPERIMENTS ON S I M O
For the following experiments of this section, we report the top-1 accuracy of SiMo on ImageNet (Deng et al., 2009) under the linear evaluation protocol.
The backbone of SiMo is ResNet-50 (He et al., 2016) and we train SiMo for 200 epochs unless noted otherwise.
B. A BLATION ON M OMENTUM U PDATE
In MoCo (He et al., 2020) and MoCo v2 (Chen et al., 2020c), the key encoder is updated by the following rule:
[Equation]
where θ q and θ k stand for the weights of query encoder and key encoder respectively, and β is the momentum coefﬁcient.
For SiMo, we also adopt the momentum update and use the key encoder to compute the features of positive sample and negative samples.
In Table 3, we report the results of SiMo with different momentum coefﬁcients.
The number of training epochs is set to be 100, so the top-1 accuracy of baseline ( β = 0 . 999 ) drops to 64.4%.
Compared to the baseline, SiMo without momentum update ( β = 0 ) is inferior, showing the advantage of momentum update.
[Table]
Table 4 shows the performance of SiMo equipped with shufﬂing BN or Sync BN.
Likewise, we train
SiMo for 100 epochs.
It is easy to check out that SiMo with shufﬂing BN struggles to perform well.
Besides, compared to MoCo v2, SiMo with shufﬂing BN degrades signiﬁcantly, and we conjecture that it is because the MLP structure of SiMo is more suitable for Sync BN, rather than shufﬂing BN.
[Table] [Table]
WITH D IFFERENT α
As shown in Sec.2.1, α is related to the lower bound of mutual information.
Table 5 reveals how accuracy of SiMo varies with the choice of α .
As we increase α to 65536, the accuracy tends to improve, in accordance with the Eq.6.
However, when α is too large (e.g., 262144), the performance slightly drops by 0.2%.
[Table]
Similar results can be found in MoCo v2.
We increase K to 262144 in MoCo v2, the accuracy also descends (in Table 6).
[Table]
B. S I M O WITH W IDER M ODELS
Results using wider models are presented in Table 7.
For SiMo, the performance is further boosted with wider models (more channels).
For instance, SiMo with ResNet-50 (2x) and ResNet-50 (4x) outperforms the baseline (68.5%) by 2% and 3.8% respectively.
[Table]
B. T RANSFER TO OBJECT DETECTION
Setup We utilize FPN (Lin et al., 2017) with a stack of 4 3 × 3 convolution layers in R-CNN head to validate the effectiveness of SiMo.
Following the MoCo training protocol, we ﬁne-tune with synchronized batch-normalization (Peng et al., 2018) across GPUs.
The additional initialized layers are also equipped with BN for stable training.
To effectively validate the transferability of the features, the training schedule is set to be 12 epochs (known as 1 × )
[Table]
in which learning rate is initialized as 0.2 and decreased at 7 and 11 epochs with a factor of 0.1.
The image scales are random sampled of [640 , 800] pixels during training and ﬁxed with 800 at inference.
Results Table 8 summarizes the ﬁne-tuning results on COCO val2017 of different pre-training methods.
Random initialization indicates training COCO from scratch
[Table]
and supervised represents conventional pre-training with ImageNet labels.
Compared with MoCo, SiMo achieves competitive performance without large quantities of negative pairs.
It is also on a par with the supervised counterpart and signiﬁcantly outperforms random initialized one.
[Table]
init
[Table]
supervised
[Table]
C T OY EXAMPLE OF ESTIMATING MUTUAL INFORMATION
To evaluate the effectiveness of EqCo as mutual information (MI) estimator, we apply it to the simple toy problem and compare it with InfoNCE (Oord et al., 2018).
Speciﬁcally, given that ( X, Y ) are drawn from the known correlated Gaussian distribution, we calculate the lower bound of MI between X and Y based on their embedding.
X is a 20-dimentional random variables drawn from a standard Gaussian distribution.
And we sampled Y with the following rule:
[Equation]
where ρ is a the given correlation coefﬁcient and (cid:15) is a random variable sampled from a standard
Gaussian distribution and independent from X .
With a known ρ , the ground truth MI between X and Y is easy to compute:
[Equation]
Here, d is the dimension of X and Y , and as mentioned above we set d = 20 .
To embed X and Y , we adopt two MLPs respectively, and each MLP has 1 hidden layer ofunits, followed by ReLU activation function.
We use Adam optimizer with learning rate of 0.
to optimize InfoNCE or EqCo for 5000 steps.
For each training iteration, K pairs of ( X, Y ) are independently sampled, which means there are K − 1 negative samples for each query.
After training, the weights of MLPs are frozen and we repeat estimating the lower bound of MI for 1000 times to reduce the estimating variance.
For experiments with EqCo, we set the α = 512 .
As shown in Table 9, I NCE varies with K , while I EqCo remains steady.
When the grouth MI is relately high (e.g., 8, 10), obvious difference between EqCo and InfoNCE can be observed.
K
Mutual Information = 2 .
I NCE 1.
I EqCo 1.
Mutual Information = 4 .
I NCE 2.
I EqCo 3.
Mutual Information = 6 .
I NCE 3.
I EqCo 5.
Mutual Information = 8 .
I NCE 3.
I EqCo 5.
[Table]