C ONTRASTIVE L EARNING WITH H ARD N EGATIVE
S AMPLES
Anonymous authors
Paper under double-blind review
A BSTRACT
We consider the question: how can you sample good negative examples for contrastive learning?
We argue that, as with metric learning, learning contrastive representations beneﬁts from hard negative samples (i.e., points that are difﬁcult to distinguish from an anchor point).
The key challenge toward using hard negatives is that contrastive methods must remain unsupervised, making it infeasible to adopt existing negative sampling strategies that use true similarity information.
In response, we develop a new class of unsupervised methods for selecting hard negative samples where the user can control the amount of hardness.
A limiting case of this sampling results in a representation that tightly clusters each class, and pushes different classes as far apart as possible.
The proposed method improves downstream performance across multiple modalities, requires only few additional lines of code to implement, and introduces no computational overhead.
I NTRODUCTION
Owing to their empirical success, contrastive learning methods (Chopra et al., 2005; Hadsell et al., 2006) have become one of the most popular self-supervised approaches for learning representations (Oord et al., 2018; Tian et al., 2019; Chen et al., 2020a).
For instance, on certain computer vision tasks unsupervised contrastive learning methods have even outperformed supervised pre-training (Misra & Maaten, 2020;
He et al., 2020).
Contrastive learning relies on two key ingredients: notions of similar (positive) ( x, x + ) and dissimilar (negative) ( x, x − ) pairs of data points.
The training objective, typically noise-contrastive estimation (Gutmann & Hyvärinen, 2010), guides the learned representation f to map positive pairs to nearby locations, and negative pairs farther apart; other objectives have also been considered (Chen et al., 2020a).
The success of the associated methods depends on the design of informative of the positive and negative pairs, which cannot exploit true similarity information since there is no supervision.
Much research effort has addressed sampling strategies for positive pairs (Blum & Mitchell, 1998; Xu et al., 2013; Bachman et al., 2019; Tian et al., 2020).
For image data, positive sampling strategies often apply transformations that preserve semantic content, e.g., jittering, random cropping, separating color channels, etc.
(Chen et al., 2020a;c; Tian et al., 2019).
Such transformations have also been effective in learning control policies from raw pixel data (Srinivas et al., 2020).
Positive sampling techniques have also been proposed for sentence, audio, and video data (Logeswaran & Lee, 2018;
Oord et al., 2018; Purushwalkam & Gupta, 2020; Sermanet et al., 2018).
Surprisingly, the choice of negative pairs has drawn much lesser attention in contrastive learning.
Often, given an “anchor” point x , a “negative” x − is simply sampled uniformly from the training data, independent of how informative it may be for the learned representation.
In supervised and metric learning settings, “hard” (negative) examples can help guide a learning method to correct its mistakes more quickly (Schroff et al., 2015; Song et al., 2016).
For representation learning, informative negative examples are intuitively those pairs that are mapped nearby but should be far apart.
This idea is successfully applied in metric learning, where true pairs of dissimilar points are available, as opposed to unsupervised contrastive learning.
With this motivation, we address the challenge of selecting informative negatives for contrastive representation learning.
In particular, we propose a solution that builds a tunable sampling distribution that prefers negative pairs whose representations are currently very similar.
This solution faces two challenges: (1) we do not have access to any true similarity or dissimilarity information; (2) we need an efﬁcient sampling strategy for this tunable distribution.
We overcome (1) by building on ideas from positive-unlabeled learning (Elkan & Noto, 2008; Du Plessis et al., 2014), and (2) by designing an efﬁcient, easy to implement importance sampling technique that incurs no computational overhead.
Our theoretical analysis shows that, as a function of the tuning parameter, the optimal representations for our new method place similar inputs in tight clusters, whilst spacing the clusters as far apart as possible.
Empirically, our hard negative sampling strategy improves the downstream task performance for image, graph and text data, supporting that indeed, our negative examples are more informative.
Contributions.
In summary, we make the following contributions:
We formulate to our knowledge the ﬁrst sampling distribution for hard negative pairs for contrastive representation learning, and an efﬁcient sampling strategy that takes into account the lack of true dissimilarity information; 2.
We theoretically analyze the properties of the objective and optimal representations, showing that they capture desired goals for representations; 3.
We empirically observe that the proposed sampling method improves the downstream task performance on image, graph and text data.
Before moving onto the problem formulation and our results, we summarize related work below.
1. R ELATED W ORK
Contrastive Representation Learning.
Various frameworks for contrastive learning of visual representations have been proposed, including SimCLR (Chen et al., 2020a;b), which uses augmented views of other items in a minibatch as negative samples, and MoCo (He et al., 2020; Chen et al., 2020c), which uses a momentum updated memory bank of old negative representations to enable the use of very large batches of negative samples. Most contrastive methods are unsupervised, however there exist some that use label information (Sylvain et al., 2020; Khosla et al., 2020).
Many works study the role of positive pairs, and, e.g., propose to apply large perturbations for images Chen et al.
(2020a;c), or argue to minimize the mutual information within positive pairs, apart from relevant information for the ultimate prediction task (Tian et al., 2020).
Beyond visual data, contrastive methods have been developed for sentence embeddings (Logeswaran & Lee, 2018), sequential data (Oord et al., 2018;
Hénaff et al., 2020), graph (Sun et al., 2020; Hassani & Khasahmadi, 2020; Li et al., 2019) and node representation learning (Velickovic et al., 2019), and learning representations from raw images for off-policy control (Srinivas et al., 2020).
The role of negative pairs hase been much less studied.
Chuang et al.
(2020) propose a method for “debiasing”, i.e., correcting for the fact that not all negative pairs may be true negatives.
It does so by taking the viewpoint of Positive-Unlabeled learning, and exploits a decomposition of the true negative distribution.
Kalantidis et al.
(2020) consider applying Mixup to generate hard negatives, and Jin et al.
(2018) exploit the speciﬁc temporal structure of video to generate negatives for object detection.
Negative Mining in Deep Metric Learning.
As opposed to the contrastive representation learning literature, selection strategies for negative samples have been thoroughly studied in (deep) metric learning (Schroff et al., 2015; Song et al., 2016;
Harwood et al., 2017; Wu et al., 2017; Ge, 2018;
Suh et al., 2019).
Most of these works observe that it is helpful to use negative samples that are difﬁcult for the current embedding to discriminate.
Schroff et al.
(2015) qualify this, observing that some examples are simply too hard, and propose selecting “semi-hard” negative samples.
The well known importance of negative samples in metric learning, where (partial) true dissimilarity information is available, raises the question of negative samples in contrastive learning, the subject of this paper.
C ONTRASTIVE L EARNING S ETUP
We begin with the setup and the idea of contrastive representation learning.
We wish to learn an embedding f : X → S d − 1 /t that maps an observation x to a point on a hypersphere S d − 1 /t in R d of radius 1 /t , where t is the “temperature” scaling hyperparameter.
Following the setup of Arora et al.
(2019), we assume an underlying set of discrete latent classes C that represent semantic content, so that similar pairs ( x, x + ) have the same latent class.
Denoting the distribution over latent classes by ρ ( c ) for c ∈ C , we deﬁne the joint distribution p x,c ( x, c ) =
p ( x | c ) ρ ( c ) whose marginal p ( x ) we refer to simply as p , and assume supp ( p ) =
X .
For simplicity, [Figure] Anchor: Negative Batch:
Willow Sycamore
Maple
Sequoia
Oak Violets Mountain
Embedding Space:
[Figure]
we assume ρ ( c ) = τ + is uniform, and let τ − = 1 − τ + be the probability of another class.
Since the class-prior τ + is unknown in practice, it must either be treated as a hyperparameter, or estimated (Christoffel et al., 2016; Jain et al., 2016).
Let h : X → C be the true underlying hypothesis that assigns class labels to inputs.
We write x ∼ x (cid:48) to denote the label equivalence relation h ( x )
= h ( x (cid:48) ) .
We denote by p + x x (cid:48) ) =
p ( x (cid:48) | h ( x (cid:48) )
=
h ( x )) , the distribution over points with same label as x , and by p − x ( x (cid:48) )
= p ( x [-0 | h ( x 0 ) 
-](cid:48)
| h ( x (cid:48) )
(cid:54) =
h ( x )) , the distribution over points with labels different from x .
We drop the subscript x when the context is clear.
Following the usual convention, we overload ‘ ∼ ’ and also write x ∼ p to denote a point sampled from p .
For each data point x ∼ p , the noise-contrastive estimation (NCE) objective (Gutmann & Hyvärinen, 2010) for learning the representation f uses a positive example x + with the same label as x , and negative examples { x − i } Ni =1 with (supposedly) different labels, h ( x − i ) (cid:54)
= h ( x ) , sampled from q :
[Equation]
The weighting parameter Q is introduced for the purpose of analysis.
When N is ﬁnite we take Q = N , yielding the usual form of the contrastive objective.
The negative sample distribution q is frequently chosen to be the marginal distribution p , or, in practice, an empirical approximation of it (Tian et al., 2019;
Chen et al., 2020a;c; He et al., 2020;
Chen et al., 2020c; Oord et al., 2018; Hénaff et al., 2020).
In this paper we ask: is there a better way to choose q ?
H ARD N EGATIVE S AMPLING
In this section we describe our approach for hard negative sampling.
We begin by asking what makes a good negative sample?
To answer this question we adopt the following two guiding principles:
Principle 1.
q should only sample “negatives” x − i whose labels differ from that of the anchor x .
Principle 2.
The most useful negative samples are ones that the embedding currently believes to be similar to the anchor.
In short, negative samples that have different label from the anchor, but that are embedded nearby are likely to be most useful and provide signiﬁcant gradient information during training.
In metric learning there is access to true negative pairs, automatically fulﬁlling the ﬁrst principle.
In contrastive learning there is no supervision, so upholding Principle 1 is impossible to do exactly.
In this paper we propose a method that upholds Principle 1 approximately, and simultaneously combines this idea with the key additional conceptual ingredient of “hardness” (encapsulated in Principle 2).
The level of “hardness” in our method can be smoothly adjusted, allowing the user to select the hardness that best trades-off between an improved learning signal from hard negatives, and the harm due to the correction of false negatives being only approximate. This important since the hardest points are those closest to the anchor, and are expected to have a high propensity to have the same label.
Therefore the damage from the approximation not removing all false negatives becomes larger for harder samples, creating the trade-off.
As a special case our our method, when the hardness level is tuned fully down, we obtain the method proposed in (Chuang et al., 2020) that only upholds
Principle 1 (approximately) but not Principle 2. Finally, beyond Principles 1 and 2, we wish to design an efﬁcient sampling method that does not add additional computational overhead during training.
3. P ROPOSED H ARD S AMPLING M ETHOD
Our ﬁrst goal is to design a distribution q on X that is allowed to depend on the embedding f and the anchor x .
From q we sample a batch of negatives { x − i } Ni =
according to the principles noted above.
We propose sampling negatives from the distribution q − β deﬁned as
[Equation]
where q β ( x − )
∝
e βf ( x ) (cid:62) f ( x − ) · p ( x − ) , for β ≥ 0 .
Note that q − β and q β both depend on x , but we suppress the dependance from the notation.
The exponential term in q β is an unnormalized von Mises–Fisher distribution with mean direction f ( x ) and “concentration parameter” β (Mardia & Jupp, 2000).
There are two key components to q − β , corresponding to each principle: 1) conditioning on the event { h ( x ) (cid:54) = h ( x − ) } which guarantees that ( x, x − ) correspond to different latent classes (Principle 1 ); 2) the concentration parameter β term controls the degree by which q β up-weights points x − that have large inner product (similarity) to the anchor x (Principle 2 ).
Since f lies on the surface of a hypersphere of radius 1 /t , we have [Equation]
so preferring points with large inner product is equivalent to preferring points with small squared Euclidean distance.
Although we have designed q − β to have all of the desired components, it is not clear how to sample efﬁciently from it.
To work towards a practical method, note that we can rewrite this distribution by adopting a PU-learning viewpoint (Elkan & Noto, 2008; Du Plessis et al., 2014; Chuang et al., 2020).
That is, by conditioning on the event { h ( x ) = h ( x − ) } we can split q β ( x − ) as
[Equation]
where q + β ( x − ) =
q β ( x − | h ( x ) = h ( x − ))
∝ e βf ( x ) (cid:62) f ( x − ) · p + ( x − ) .
Rearranging equation 2 yields a formula q − β ( x − )
= (cid:0) q β ( x − ) − τ + q + β ( x − ) (cid:1)
/τ − for the negative sampling distribution q − β in terms of two distributions that are tractable since we have samples from p and can approximate samples from p + using a set of semantics-preserving transformations, as is typical in contrastive learning methods.
It is possible to generate samples from q β and (approximately from) q + β using rejection sampling.
However, rejection sampling involves an algorithmic complication since the procedure for sampling batches must be modiﬁed.
To avoid this, we instead take an importance sampling approach.
To obtain this, ﬁrst note that ﬁxing the number Q and taking the limit N → ∞ in the objective (1) yields,
[Equation]
The original objective (1) can be viewed as a ﬁnite negative sample approximation to L ( f, q ) (note implicitly L ( f, q ) depends on Q ) .
Inserting q = q − β and using the rearrangement of equation (2) we obtain the following hardness-biased objective:
[Equation]
This objective suggests that we need only to approximate expectations E x − ∼ q β [ e f ( x ) T f ( x − ) ] and
E v ∼ q + β [ e f ( x ) T f ( v ) ] over q β and q + β (rather than explicily sampling).
This can be achieved using classical Monte-Carlo importance sampling techniques using samples from p and p + as follows:
[Equation]
where Z β , Z + β are the partition functions of q β and q + β respectively.
The right hand terms readily admit empirical approximations by replacing p and p + with ˆ p ( x )
= 1 N {+(cid:80) Ni =
+}
δ x − i ( x ) and ˆ p + ( x ) = 1 M (cid:80) Mi
{+=
+}
δ x + i ( x ) respectively ( δ w denotes the Dirac delta function centered at w ).
The only unknowns left are the partition functions, Z β = E x − ∼ p
[ e βf ( x ) T f ( x − ) ] and Z
+ β = E x + ∼ p + [ e βf ( x ) T f ( x + ) ] which themselves are expectations over p and p + and therefore admit empirical estimates,
[Equation]
It is important to emphasize the simplicity of the implementation of our proposed approach.
Since we propose to reweight the objective instead of modifying the sampling procedure, only two extra lines of code are needed to implement our approach, with no additional computational overhead.
PyTorch-style pseudocode for the objective is given in Figure 13 in Appendix D.
A NALYSIS OF H ARD N EGATIVE S AMPLING 4. H ARD S AMPLING I NTERPOLATES B ETWEEN M ARGINAL AND W ORST -C ASE N EGATIVES
Intuitively, the concentration parameter β in our proposed negative sample distribution q − β controls the level of “hardness” of the negative samples.
As discussed earlier, the debiasing method of Chuang et al. (2020) can be recovered as a special case: taking β = [-
 (the “least hard” negatives) recovers-]0 to obtain the distribution q − 0 . This case amounts to correcting for the fact that some samples in a negative batch sampled from p will have the same label as the anchor.
But what interpretation does large β admit?
Speciﬁcally, what does the distribution q − β converge to in the limit β → ∞ , if anything?
We show that in the limit q − β approximates an inner solution to the following zero-sum two player game.
[Equation]
where Π = { q = q ( · ; x, f ) : supp (cid:0) q ( · ; x, f ) (cid:1) ⊆ { x (cid:48) ∈ X : x (cid:48) (cid:28) x } , ∀ x ∈ X} is the set of distributions with support that is disjoint from points with the same class as x (without loss of generality we assume { x (cid:48) ∈
X : x (cid:48) (cid:28) x } is non-empty).
Since q = q ( · ; x, f ) depends on x and f it can be thought of as a family of distributions.
The formal statement is as follows.
Proposition 3.
Let L ∗ ( f ) =
sup q ∈ Π L ( f, q ) .
Then for any t > 0 and f : X → S d − 1 /t
we observe the convergence L ( f, q − β ) −→ L ∗ ( f ) as β → ∞ .
Proof.
See Appendix A.1.
To develop a better intuitive understanding of the worst case negative distribution objective L ∗ ( f )
= sup q ∈ Π L ( f, q ) , we note that the supremum can be characterized analytically.
Indeed,
[Equation]
The supremum over q can be pushed inside the expectation since q is a family of distribution indexed by x , reducing the problem to maximizing E x − ∼ q
[ e f ( x ) T f ( x − ) ] , which is solved by any q ∗ whose support is a subset of arg sup x − : x − (cid:28) x e f ( x ) T f
( x − ) if the supremum is attained.
However, computing such points involves maximizing a neural network.
Instead of taking this challenging route, using q − β deﬁnes a lower bound by placing higher probability on x − for which f ( x ) T f ( x − ) is large. This lower bound becomes tight as β → ∞ (Proposition 3).
4. O PTIMAL E MBEDDINGS ON THE H YPERSPHERE FOR W ORST -C ASE N EGATIVE S AMPLES
What desirable properties does an optimal contrastive embedding (global minimizer of L ) possess that make the representation generalizable?
To study this question, we ﬁrst analyze the distribution of an optimal embedding f ∗ on the hypersphere when negatives are sampled from the adversarial worst-case distribution.
We consider a different limiting viewpoint of objective (1) as the number of negative samples N → ∞ .
Following the formulation of Wang & Isola (2020) we take Q = N in (1), and subtract log N .
This changes neither the set of minimizers, nor the geometry of the loss surface.
Taking the number of negative samples N → ∞ yields the limiting objective,
[Equation]
Theorem 4.
Suppose the downstream task is classiﬁcation (i.e. C is ﬁnite), and let L ∗∞ ( f )
= sup q ∈ Π L ∞ ( f, q ) .
The inﬁmum inf f : measurable L ∗∞ ( f ) is attained, and any f ∗ achieving the global minimum is such that f ∗ ( x ) =
f ∗ ( x + ) almost surely.
Furthermore, letting v c = f ∗ ( x ) for any x such that h ( x ) =
c (so v c is well deﬁned up to a set of x of measure zero), f ∗ is characterized as being any solution to the following ball-packing problem,
[Equation]
Proof.
See Appendix A.2.
Interpretation: The ﬁrst component of the result is that f ∗ ( x ) =
f ∗ ( x + ) almost surely for an optimal f ∗ .
That is, an optimal embedding f ∗ must be invariant across pairs of similar inputs x, x + .
The second component is characterizing solutions via the classical Tammes Ball-Packing Problem from geometry Tammes (1930)
(Eq. 7) that has only been solved exactly for uniform ρ , for speciﬁc of |C| and typically for S 2 Schütte & Van der Waerden (1951); Musin & Tarasov (2015); Tammes (1930).
When the distribution ρ over classes is uniform this problem is solved by a set of |C| points on the hypersphere such that the average squared- (cid:96) 2 distance from a point to the nearest other point is as large as possible.
In other words, suppose we wish to place |C| number of balls 1 on S d − 1 so that they do not intersect.
Then solutions to Tammes’ Problem (7) expresses (twice) the largest possible average squared radius that the balls can have.
So, we have a ball-packing problem where instead of trying to pack as many balls as possible of a ﬁxed size, we aim to pack a ﬁxed number of balls (one for each class) to have as big radii as possible.
Non-uniform ρ adds importance weights to each ﬁxed ball.
In summary, solutions of the problem min f L ∗∞ ( f ) are a maximum margin clustering. This understanding of global minimizers of L ∗∞ ( f ) = sup q ∈ Π L ∞ ( f, q ) can further developed into a better understanding of generalization on downstream tasks. The next result shows that representations that achieve small excess risk on the objective L ∗∞ still separate clusters well in the sense that a simple 1-nearest neighbor classiﬁer achieves low classiﬁcation error. Theorem 5.
Suppose ρ is uniform on C and f is such that L ∗∞ ( f ) − inf ¯ f measurable L ∗∞ ( ¯ f ) ≤ ε with ε ≤ 1 . Let { v ∗ c ∈ S d − 1 /t } c ∈C be a solution to Problem 7, and deﬁne ξ = min c,c − : c (cid:54) = c − (cid:13)(cid:13) v ∗ c − v ∗ c − (cid:13)(cid:13) > 0 .
Then there exists a set of vectors { v c ∈ S
d − 1 /t } c ∈C such that the following 1-nearest neighbor classiﬁer,
[Equation]
where
[Equation]
ties broken arbitrarily
[Equation]
achieves misclassiﬁcation risk,
[Equation]
Proof.
See Appendix A.3.
In particular, P (ˆ h ( x ) (cid:54) =
c )
=
O ( ε ) as ε → 0 , and in the limit ε → 0 we recover Theorem 4 as a special case.
The result can easily be generalized to arbitrary ρ by replacing |C| in the bound by 1 / min c ρ ( c ) .
The result also implies that it is possible to build simple classiﬁers for tasks that involve only a subset of classes from C , or classes that are a union of classes from C .
The constant ξ = min c,c − : c (cid:54) =
c − (cid:13)(cid:13) v ∗ c − v ∗ c − (cid:13)(cid:13) > 0 is a purely geometrical property of spheres, and describes the minimum separation distance between a set of points that solves the Tammes’ ball-packing problem.
Next, we evaluate our hard negative sampling method empirically, and apply it as a modiﬁcation to state-of-the-art contrastive methods on image, graph, and text data.
For all experiments β is treated as a hyper-parameter (see ablations in Figure 2 for more understanding of how to pick β ). Values for M and τ + must also be determined. We ﬁx M = 1 for all experiments, since taking M >would increase the number of inputs for the forward-backward pass. Lemma 11 in the appendix gives a theoretical justiﬁcation for the choice of M = 1 .
Choosing the class-prior τ + can be done in two ways: estimating it from data (Christoffel et al., 2016; Jain et al., 2016), or treating it as a hyper-parameter. The ﬁrst option requires the possession of labeled data before contrastive training.
5. I MAGE R EPRESENTATIONS
We begin by testing the hard sampling method on vision tasks using the STL10, CIFAR100 and
CIFAR10 data.
We use SimCLR (Chen et al., 2020a) as the baseline method.
The results in Figure 2 show consistent improvement over SimCLR ( q = p ) and the particular case of our method with β = 0 proposed in (Chuang et al., 2020) (called debiasing) on STL10 and CIFAR100.
For N =negative examples per data point we observe absolute improvements of 3% and 7 . 3% over SimCLR on CIFAR100 and STL10 respectively, and absolute improvements over the best debiased baseline of 1 . 9% and 3 .
On CIFAR10 there is a slight improvement for smaller N , which disappears at larger N .
See Appendix D.1 for full setup details [Figure]
5. G RAPH R EPRESENTATIONS
Second, we consider hard negative sampling in the context of learning graph representations.
We use the state-of-the-art InfoGraph method introduced by Sun et al.
(2020) as the baseline, which is suitable for downstream graph-level classiﬁcation.
The objective is of a slightly different form from the NCE loss.
Because of this we use a generalization of the formulation presented in Section 3 (See Appendix B for details).
In doing so, we illustrate that it is easy to adapt our hard sampling method to other contrastive objectives beyond the NCE loss.
Table 3 shows the results of ﬁne-tuning an SVM (Boser et al., 1992; Cortes & Vapnik, 1995) on the ﬁxed, learned embedding for a range of different values of β .
Hard sampling does as well as
InfoGraph in all cases, and better in 6 out of 8 cases.
For ENZYMES and REDDIT, hard negative samples improve the accuracy by 3 . 2% and 2 .
4% , respectively, for DD and PTC by 1 − 2% , and for IMDB-B and MUTAG by at least 0 .
Usually, multiple different choices of β > 0 were competitive with the InfoGraph baseline: 17 out of the 24 values of β > 0 tried (across all 8 datasets) achieve accuracy as high or better than InfoGraph ( β = 0 ).
5. S ENTENCE R EPRESENTATIONS
Third, we test hard negative sampling on learning representations of sentences using the quickthoughts (QT) vectors framework introduced by Logeswaran & Lee (2018), which uses adjacent sentences (before/after) as positive samples.
Embeddings are trained using the unlabeled BookCorpus [Figure] dataset (Kiros et al., 2015), and evaluated following the protocol of Logeswaran & Lee (2018) on six downstream tasks.
The results are reported in Table 1.
Hard sampling outperforms or equals the QT baseline in 5 out of 6 cases, the debiased baseline (Chuang et al., 2020) in 4 out of 6, and both inout of 6 cases.
Setting τ + > 0 led to numerical issues in optimization for hard sampling.
Objective
[Table]
A C LOSER L OOK AT H ARD S AMPLING 6. A RE H ARDER S AMPLES N ECESSARILY B ETTER ?
By setting β to large values, one can focus on only the hardest samples in a training batch.
But is this desirable?
Figure 4 (left, middle) shows that for vision problems, taking larger β does not necessarily lead to better representations, corroborating related observations by Wu et al.
In contrast, when one uses true positive pairs during training (green curve, uses label information for positive but not negative pairs), the downstream performance monotonically increases with β until convergence (Figure 4 , middle).
Interestingly, this is achieved without using label information for the negative pairs.
This observation suggests an explanation for why bigger β hurts performance in practice.
Debiasing (conditioning on the event { h ( x ) (cid:54) = h ( x − ) } ) using the true p + corrects for sampling x − with the same label as x .
However, since in practice we approximate p + using a set of data transformations, we can only partially correct.
This is harmful for large β since this regime strongly prefers x − for which f ( x − ) is close to f ( x ) , many of whom will have the same label as x if not corrected for.
We note also that by annealing β (gradually decreasing β to 0 throughout training; see Appendix D.1 for details) it is possible to be more robust to the choice of initial β , with only minor negative impact on downstream accuracy compared to the best ﬁxed value of β .
6. D OES A VOIDING F ALSE N EGATIVES I MPROVE H ARD S AMPLING ?
Our proposed hard negative sampling method conditions on the event { h ( x ) (cid:54) = h ( x − ) } in order to avoid false negatives (termed “debiasing” (Chuang et al., 2020)).
But does this help?
To test this, we train four embeddings: hard sampling with and without debiasing, and uniform sampling ( β = 0 ) with and without debiasing. The results in Figure 4 (right) show that hard sampling with debiasing [Figure] [Figure]
[Figure]
[Figure]
[Figure] T o pA cc u r a c y SimCLR ( =0, + =0) + =0. + =0.1( annealed ) 0.0 0.5 1.
6.0 (CIFAR100)
T o pA cc u r a c y
SimCLR ( =0, + =0) + =0.
+ =0.05( annealed )
True positives
Debiased Not Debiased
H a r d S a m p li n g
N o t H a r d
S a m p li n g 87.44 % 84.41 % 84.26 % 80.15 % 81838587 L i n e a r e v a l u a t
i
o
n a cc u r a c y
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
Cosine Similarity
P o s i t i v e
P a i r s
N e g a t
i v e
P a i r s
H & D (Ours)
H & not D not H & D not H & not D (SimCLR) obtains the highest linear readout accuracy on STL10, only using hard sampling or only debiasing yields (in this case) similar accuracy.
All improve over the SimCLR baseline.
Figure 5 compares the histograms of cosine similarities of positive and negative pairs for the four learned representations.
The representation trained with hard negatives and debiasing assigns much lower similarity score to a pair of negative samples than other methods.
On the other hand, the
SimCLR baseline assigns higher cosine similarity scores to pairs of positive samples.
However, to discriminate positive and negative pairs, a key property is the amount of overlap of positive and negative histograms.
Our hard sampling method achieves less overlap than SimCLR, by better trading off higher dissimilarity of negative pairs with less similarity of positive pairs.
Similar tradeoffs are observed for the debiased objective, and hard sampling without debiasing.
6. H OW DO H ARD N EGATIVES A FFECT O PTIMIZATION ?
Figure 11 (in Appendix C due to space constraints) shows the performance on STL10 and CIFAR of SimCLR versus using hard negatives throughout training.
We use weighted k -nearest neighbors with k = 200 as the classiﬁer and evaluate each model once every ﬁve epochs.
Hard sampling with β = 1 leads to much faster training: on STL10 hard sampling takes only 60 epochs to reach the same performance as SimCLR does in 400 epochs. On CIFAR100 hard sampling takes only 125 epochs to reach the same performance as SimCLR does in 400 epochs. We speculate that the speedup is, in part, due to hard negatives providing non-negligible gradient information during training.  C ONCLUSION
We argue for the value of hard negatives in unsupervised contrastive representation learning, and introduce a simple hard negative sampling method.
Our work connects two major lines of work: contrastive learning, and negative mining in metric learning.
Doing so requires overcoming an apparent roadblock: negative mining in metric learning uses pairwise similarity information as a core component, while contrastive learning is unsupervised.
Our method enjoys several nice aspects: having desirable theoretical properties, a very simple implementation that requires modifying only a couple of lines of code, not changing anything about the data sampling pipeline, introducing absolutely no extra computational overhead, and handling false negatives in a principled way.