Less can be more in contrastive learning
Abstract
Unsupervised representation learning provides an attractive alternative to its supervised counterpart because of the abundance of unlabelled data.
Contrastive learning has recently emerged as one of the most successful approaches to unsupervised representation learning.
Given a datapoint, contrastive learning involves discriminating between a matching, or positive, datapoint and a number of non-matching, or negative, ones.
Usually the other datapoints in the batch serve as the negatives for the given datapoint.
It has been shown empirically that large batch sizes are needed to achieve good performance, which led the the belief that a large number of negatives is preferable.
In order to understand this phenomenon better, in this work investigate the role of negatives in contrastive learning by decoupling the number of negatives from the batch size.
Surprisingly, we discover that for a ﬁxed batch size performance actually degrades as the number of negatives is increased.
We also show that using fewer negatives can lead to a better signal-to-noise ratio for the model gradients, which could explain the improved performance.
Introduction
Large amounts of supervised data are usually needed to train deep neural networks successfully and acquiring the necessary amount of supervised signal can be costly or impractical.
On the other hand, large amounts of unsupervised data are often readily available.
We can leverage this data by pretraining representations for the unknown downstream tasks of interest.
Self-supervised methods utilize proxy tasks deﬁned on unsupervised data in order to pretrain representations.
Recently, a particular kind of self-supervised methods, contrastive learning, has achieved state-of-the-art performance in unsupervised representation learning [10, 6, 7, 2, 3].
Contrastive representation learning methods work by learning to discriminate between datapoints that are similar to the current one and randomly sampled other datapoints, known as the negative examples, which are presumed to be dissimilar.
To perform this classiﬁcation, similarity scores of pairs of similar examples and negative examples are computed.
Usually, similar examples are different augmentations of the original datapoint.
The most performant contrastive methods such as CPC [10, 6], AMDIM [2] and SimCLR [3] all follow this general approach.
The main differences between these approaches are in the data augmentations and encoder architecture used.
While CPC and AMDIM use large custom networks, SimCLR uses a standard ResNet50 [5] in combination with strong data augmentations.
Notwithstanding the differences in encoder architectures employed in contrastive learning coupled with some differences in the computation of the similarity scores between representations, [10, 6, 7, 2, 3] all argue for using large sets of negative examples.
They argue that this leads to more hard negatives being used in learning, which improves the quality of the resulting representation.
Thus, they use very large batches in order to have larger number of negatives available and have reported signiﬁcant performance gains with increasing batch sizes.
Recently, [4] proposed to include a queue of data samples in order to increase the number of negative examples beyond the batch size.
In this work we examine the role of negatives in contrastive learning and test the widely held belief that using large numbers of negatives is important for learning good representations.
We decouple the batch size from the number of negatives by randomly sampling negatives for each datapoint.
We perform experiments on CIFAR100 [9] and downsampled ImageNet [12] for two highly performant contrastive methods, SimCLR [3] and R E LIC [1].
Keeping the batch size ﬁxed, we vary the number of negative examples and discover that using a large number of negatives actually hurts performance.
This shows that the widely used heuristic of using as many negatives as feasible can lead to suboptimal performance.
For CIFAR100, we ﬁnd that the best performance is attained with just 1 andnegative examples for R E LIC and SimCLR, respectively.
For downsampled ImageNet, we ﬁnd that the best performance is achieved with 100 negative examples which is much lower than the batch size of 4096 .
Background
Denote by X the unlabelled observed data and by Y the targets of the unknown downstream task.
We want to pretrain a representation f ( X ) using unsupervised data only such that it is useful for solving the downstream task Y .
Contrastive methods learn representations by comparing a data point to a series of similar and dissimilar data points.
Speciﬁcally, for an observation x from X , the representation f ( x ) is optimized such that the similarity between f ( x ) and f ( x + ) is maximized, while the similarity between f ( x ) and f ( x − ) is minimized; x + and x − are data points that are semantically similar and dissimilar to x , respectively, and are called positive and negative examples.
Given a mini-batch { x i } Bi =1 , a general form of the contrastive loss is:
[Equation]
with z i
= g ( f ( x i )) , τ a temperature parameter, P i the set of positive examples and N i the set of negative examples for data point x i , and g is a simple function (e.g. identity or a small fully connected network) called the projection head.
Both SimCLR [3] and R E LIC [1] apply augmentations to datapoints before computing the similarities between them and use one positive example per datapoint.
The objective in SimCLR is
[Equation]
with z i,a = g ( f ( x i,a )) and x i,a denotes the datapoint x i under augmentation a .
For R E LIC, we use the following version of the objective
[Equation]
with p vwi ( m ) =
exp (cid:0)
z
Ti,v z m,w /τ (cid:1) /Z
vwi , where Z vwi is the normalizing constant.
KL denotes the
Kullback-Leibler divergence, M negative examples are sampled uniformly at random from the batch for each datapoint and α is the invariance penalty weight.
Experiments
We study the impact of the number of negative examples on the quality of the learned representation.
We measure representation quality under the linear evaluation protocol as proposed in [8] by training a linear classiﬁer on top the frozen representation and reporting the top 1 accuracy on the test set.
While in previous research the batch size and number of negatives were tightly coupled, we believe this work is the ﬁrst one to consider the effect of the number of negative examples separately from the batch size.
We achieve this by decoupling the batch size and the number of negatives by restricting the sum in the denominator in (2) and (3) to be over a random subset of all negative examples available in the current batch.
We assess the performance SimCLR and R E LIC, both of which use the data-augmentation scheme proposed in [3] but differ in their objectives as described in Section 2.
For each method and each number of negatives we sweep over the temperature, learning rate, and, for R E LIC the invariance penalty weight.
For SimCLR we optimized over the following ranges: learning rate in [0 . 2 , 1 . 5] , temperature in [0 . 2 , 10] .
For R E LIC we considered: learning rate in the range [0 . 5 , 1 . 5] , temperature in the range [0 . 2 , 50] and invariance pentalty weight in the interval [0 . 1 , 500] .
In all experiments we use Resnet50 [5] as the encoder.
For the non-linear projection head we use fully-connected networks with dimensions [2048 , 128] for SimCLR and [4092 , 2048 , 1024 , 512 , 128] for R E LIC.
All models are trained for 2000 epochs using the LARS optizimer [13] with the same parameters as in [3] and a batch size of 1120.
In Figure 1a and 1b we report the top 1 accuracy on CIFAR100 obtained for different numbers of negative examples under the linear evaluation protocol [9].
We report the average performance over 5 random seeds, with the errors bars denoting the standard deviation.
In Figure 1a we observe that performance decreases with as number of negatives increases apart from when 1 negative is used.
In
Figure 1b we see that performance decreases with as number of negatives increases.
Unlike SimCLR,
R E LIC has the highest performance with 1 negative.
Moreover, the gain when using fewer negatives is more pronounced with R E LIC than with SimCLR.
These results directly contradict the commonly held belief that a large number of negatives is needed for good performance.
[Figure]
We also examined this phenomenon for R E LIC and SimCLR on the ImageNet dataset [12] after downsampling the images to 64 × 64 pixels.
We followed the same protocol and experimental setup as for CIFAR100 apart from the batch size which we set to be 4096 .
In Figure 1c we see that the best performance for R E LIC is achieved with 100 negatives which is signiﬁcantly lower than what the full batch size of 4096; top 1 test accuracy with 100 negatives examples is 0 .
7% higher than under the standard approach to using negatives that utilizes all the remaining points in the batch as negatives.
For SimCLR, we found that we could achieve the same performance as the standard approach utilizing all the remaining points in the batch as negatives with only 75 negatives.
While not leading to improved performance, using fewer negatives in SimCLR can help reduce the computational cost signiﬁcantly.
Hypothesis: Better intra-class concentration
We ﬁrst hypothesized that the improved performance with fewer negatives was due to better intra-class concentration of the learned representations.
Better intra-class concentration would make the learned latent space of representations more easily linearly separable and thus directly lead to improved classiﬁcation performance under the linear evaluation protocol.
To evaluate intra-class concentration we used two measures – the intra-class average pairwise distance between representations and the
Silhouette score.
The Silhouette score is calculated using the mean intra-cluster distance (a) and the mean nearest-cluster distance (that the datapoint is not part of) (b) and computed as ( b − a ) /max
( a, b ) .
Note that higher Silhouette score indicates better intra-class concentration.
For R E LIC both the intra-class average pairwise distance and the Silhouette score indicate that for lower numbers of negatives there is a better intra-class concentration; see Figures 2a and 3a.
For SimCLR, neither the intra-class average pairwise distances nor the Silhouette scores do strongly correlate with downstream performance; see Figure 2b and Figure 3b.
[Figure] [Figure]
Hypothesis: Better gradient signal-to-noise ratio
Could the surprising performance obtained with fewer negative examples be partially explained by better gradient dynamics?
Gradient variance alone is not always a good indicator since gradients with different magnitudes will be affected differently by the same amount of noise.
We therefore focus on the gradient signal-to-noise ratio (SNR) which has been established as a useful measure to assess training dynamics [11].
SNR is deﬁned for each model parameter θ i as the absolute value of the gradient mean divided by its standard deviation:
[Equation]
where ∇ θ i L denotes the gradient of the loss L w.r.t.
θ i .
As we are dealing with a large number of model parameters and are primarily interested in the evolution of gradient SNR during training, we report both the SNR average over parameters 1# θ (cid:80) i SNR θ i and its standard deviation.
We computed
SNR by using an exponentially moving average to estimate the ﬁrst two moments of the gradient during training.
Figure 4 shows how these quantities evolve during training on CIFAR100.
We observe similar dynamics for SimCLR and R E LIC.
With a small number of negative examples, the
SNR variance is often higher (see bottom panels) but the average SNR is clearly improved (see top panels).
This is particularly noticeable for very small numbers of negatives such as 5 and 10 .
Using only 1 negative example seems to be a special case and the gradient behaviour looks qualitatively different.
1 From these ﬁndings, we conclude that the improvement in performance observed when
[Figure]
using fewer negatives is at least in part due to an improvement in the signal-to-noise ratio in the gradients, i.e. using fewer negatives yields improved gradient dynamics.
Conclusion
In this work we examined the role of negative examples in contrastive learning.
Speciﬁcally, we examined the widely held belief that using as many negatives as possible improves the quality of the learned representation.
We decoupled the batch size from the number of negatives in order to perform experiments with ﬁxed batch size and varying number of negatives for two very performant contrastive methods SimCRL and ReLIC.
For both methods, we observe that performance increases as the number of negatives decreases, which strongly suggests that the heuristic of using as many negatives as possible does not lead to the best performance.
We also notice increasing gradient SNR with decreasing number of negatives and hypothesize that this is one of the causes of improved performance.
Finally, together with previous results, our ﬁndings point towards large batch sizes and not the number of negatives having a decisive inﬂuence on performance.