S HARP L EARNING B OUNDS FOR C ONTRASTIVE U NSUPERVISED R EPRESENTATION L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Contrastive unsupervised representation learning (CURL) encourages data representation to make semantically similar pairs closer than randomly drawn negative samples, which has been successful in various domains such as vision, language, and graphs.
Although recent theoretical studies have attempted to explain its success by upper bounds of a downstream classiﬁcation loss by the contrastive loss, they are still not sharp enough to explain an experimental fact: larger negative samples improve the classiﬁcation performance.
This study establishes a downstream classiﬁcation loss bound with a tight intercept in the negative sample size.
By regarding the contrastive loss as a downstream loss estimator, our theory not only improves the existing learning bounds substantially but also explains why downstream classiﬁcation empirically improves with larger negative samples—because the estimation variance of the downstream loss decays with larger negative samples.
We verify that our theory is consistent with experiments on synthetic, vision, and language datasets.
I NTRODUCTION
The contrastive loss (Chopra et al., 2005) is one of the popular loss functions in metric learning (Kulis, 2012) and representation learning (Bengio et al., 2013).
The contrastive loss forces data representation of semantically similar pairs closer in some metric space than multiple random samples, called negative samples.
Many state-of-the-art representation learning algorithms use a type of contrastive losses in natural language processing (Mikolov et al., 2013; Logeswaran & Lee, 2018), vision (Chopra et al., 2005;
He et al., 2019; Chen et al., 2020), and graph (Lirong et al., 2021) domains.
A simple model built on top of the learned representation can achieve almost the same accuracy as supervised learning does.
See surveys (Le-Khac et al., 2020; Schmarje et al., 2021) and references therein for recent empirical progress.
Compared to its empirical success, we still do not know much about the theoretical perspective of the contrastive loss function.
Since the ﬁrst theoretical analysis of contrastive unsupervised representation learning (CURL) (Arora et al., 2019), a number of follow-up researches (Nozawa et al., 2020; Chuang et al., 2020; Nozawa & Sato, 2021; Ash et al., 2021) improved theoretical understanding of CURL.
They provided upper bounds of a downstream classiﬁcation loss by the contrastive loss.
While they partially explain why CURL provides good representation in terms of downstream classiﬁcation, a controversy has been held on the impact of the negative sample size, denoted by K in this work.
From the empirical viewpoint, it has been widely shown that downstream classiﬁcation accuracy tends to be better with the larger K (He et al., 2020; Chen et al., 2020).
1 From the theoretical viewpoint, although
Ash et al.
(2021) and Nozawa & Sato (2021) addressed the issue of the earlier analysis (Arora et al., 2019)—the bound grows exponentially in K —their improved bounds are still so huge that they may not capture the actual behavior of CURL related to K , as we will see in Figure 3.
In addition,
Ash et al.
(2021) claims that the optimal K exists in smaller K , which does not agree well with the empirical observations.
Henceforth, the relationship between the performance of downstream classiﬁcation and K needs to be investigated carefully.
This study aims to clarify the relationship between the performance of downstream classiﬁcation and the negative sample size K by deriving an upper bound of the downstream classiﬁcation loss
(Theorem 1).
By deriving the lower bound (Theorem 2), we found that the intercept of our upper bound is tight in K .
Notably, such a lower bound has yet to be known in CURL so far.
Our bounds support that the larger K reduces estimation variance of the downstream classiﬁcation loss (Section 3.2), while the smaller K could perform well.
This ﬁnding is evidence for the empirical success of the larger K .
In addition, our bounds are compared with the existing bounds to demonstrate that there is no optimal point in K and how accurately the derived bounds capture the downstream loss (Section 4).
Finally, we empirically verify our theory by experiments (Section 6) on a synthetic dataset, CIFAR-10/100 (Krizhevsky, 2009) datasets, and Wiki-3029 dataset (Arora et al., 2019).
F ORMULATION OF C ONTRASTIVE L EARNING
First, this section brieﬂy summarizes the problem setup and formulation of CURL.
Notation.
The C -dimensional vector whose elements are all one is denoted by 1 C :=
1] (cid:62) .
When it is clear from context, the subscript is abbreviated.
For a vector a ∈ R p , a ( i ) denotes the i -th largest element of a , namely, a (1) ≥ a (2) ≥ · · · ≥ a ( p ) .
Likewise, a ( − i ) denotes the i -th
smallest element of the vector a .
The indicator function is denoted by 1 { A } for a predicate A .
Let (cid:52) C :
= { p ∈ [0 , 1] C | p (cid:62) 1 = 1 } be the C -dimensional probability simplex.
For p ∈ (cid:52) C , the
Shannon entropy is denoted by H ( p ) :
= − (cid:80) c
∈ [ C ] p
c
ln p c .
Let H n be the n -th harmonic number.
Supervised classiﬁcation.
One of the goals in machine learning is supervised classiﬁcation, while we consider the setup where supervision is unavailable.
Here, we ﬁrst formulate the multi-class classiﬁcation problem.
Assume that there exists an unknown number of latent classes denoted by
C ∈ N .
Let X be d -dimensional feature space and Y :=
[ C ] be the supervised label set.
In the supervised setup, we are interested in the following risk quantity, the supervised loss , for a multi-class classiﬁer g : X → R C :
[Equation]
which is specialized for the softmax cross-entropy loss.
The expectation is taken over the unknown underlying joint distribution P .
The classiﬁer is eventually used for prediction by argmax y ∈Y g y ( · ) .
Contrastive unsupervised representation learning (CURL).
In the CURL framework (Arora et al., 2019), we target to learn meaningful data representation in an unsupervised manner.
By doing so, a similarity model is trained to make the representation of positive pairs more similar than randomly drawn K negative samples using a contrastive loss.
The class-conditional distribution is denoted by
D c := P ( X | Y = c ) for each c ∈ Y and the class-prior distribution by π :=
[ P ( Y = c )] c ∈Y ∈ (cid:52) C .
The data generating process is described as follows: (i) draw latent positive/negative classes:
c + , { c − k } Kk =1 ∼ π K +1 (ii) draw an anchor sample x ∼ D c + (iii) draw a positive sample x + ∼ D c
+(iv) draw K negative samples x
− k ∼ D c − k
(for each k ∈ [ K ] ).
Now we specify our model.
A multi-class classiﬁer g : X → R C consists of representation f : X → R h and linear parameters W ∈ R C × h as g ( · ) :=
Wf ( · ) , where h ∈ N denotes the dimensionality of the representation given in advance.
In CURL, the representation is learned through minimization of the following contrastive loss
[Equation]
E c
[Equation] E x
[Equation]
x [Equation]
∼D
[Equation]
c
[Equation] x
[Equation]
In the downstream task, the representation f shall be frozen, and the parameters W are to be learned via minimizing the supervised loss R supv .
Evaluation of representations.
For the sake of evaluation, a speciﬁc linear classiﬁer called mean classiﬁer is introduced.
Given representation f , the mean classiﬁer W µ is deﬁned as W µ :=
[ µ 1 · · · , µ C ] (cid:62) , where µ c :=
E x ∼D c
[ f ( x )] .
This will later be used for evaluating the representation f combined with the supervised loss, which is denoted by R µ -supv ( f ) :
= R supv ( W µ f ) .
We call it the mean supervised loss .
If the mean supervised loss is successfully bounded from above, we end up a bound on the supervised loss through inf W ∈ R C × h R supv ( Wf ) ≤ R µ -supv ( f ) .
For this reason, an upper bound on R µ -supv is an intermediate milestone that we seek throughout this paper.
L EARNING B OUNDS FOR C ONTRASTIVE L EARNING
In this section, our main theoretical results are provided.
We aim at showing that the contrastive loss
R cont ( f ) —the computable functional in CURL—serves as a good estimator of the mean supervised loss R µ -supv ( f ) —the inaccessible target in CURL—for any f .
We show this by establishing upper and lower bounds of R µ -supv ( f ) by R cont ( f ) .
Eventually, the minimization of R cont ( f ) may lead to a good minimizer of R µ -supv ( f ) .
All proofs are provided in Appendix B.
M AIN R ESULTS
First, we show a sharp upper bound of the mean supervised loss.
Unlike any of the existing learning bounds of CURL, the upper bound obtained here has a constant coefﬁcient in the contrastive loss and is applicable for all C and K (see discussions in Section 4).
Theorem 1.
For all f such that (cid:107) f
( x ) (cid:107) 2 ≤ L ( ∀ x ∈ X ), the following inequality holds.
[Equation]
(3) where ∆ U := 2 ln(( CKπ ( − 1) + 1) cosh( L 2 ))
− 2 ln
[Equation]
1 +
[Equation]
− ln Kπ
[Equation]
Next, the ﬁrst lower bound of the mean supervised loss is provided.
While the existing theoretical analyses only provided upper bounds, they often entail a huge coefﬁcient in the contrastive loss.
The lower bound provided below has the same constant coefﬁcient and intercept ( ∆ U and ∆ L ) rate as our upper bound, ensuring the tightness of our analysis.
Theorem 2.
For all f such that (cid:107) f
( x ) (cid:107) 2 ≤ L ( ∀ x ∈ X ), the following inequality holds.
[Equation]
where ∆ L := H ( π ) + ln
[Equation]
In Theorems 1 and 2, the size of the representation (cid:107) f
( x ) (cid:107) 2 is assumed to be bounded.
This assumption is reasonable from the experimental perspective since it is common to normalize representation to employ the cosine similarity as the similarity metric.
Several works reported that the normalized embeddings improve the performance (Chen et al., 2020; Wang & Isola, 2020).
Unlike the existing analyses (reviewed in Section 4), we take advantage of this assumption to derive the sharp bounds.
As we see in Section 3.2, ∆ U and ∆ L are the same order in K under the uniform class prior assumption.
By applying either the high-probability bound (Arora et al., 2019) or PAC-Bayesian analysis (Nozawa et al., 2020), Theorem 1 (Theorem 2 as well) can be naturally extended to the form
R µ -supv ( (cid:98) f ) ≤ R cont ( f ) + ∆ U + χ with a complexity term χ , where (cid:98) f is the empirical minimizer of the contrastive loss.
Since this is a routine, we omit the high-probability bounds.
3. D ISCUSSION
Subsequently, we discuss implications of our main results on the relationship between the mean supervised loss and the negative sample size K .
For the sake of simplicity, we assume π c = 1 / C for all c ∈
[ C ] (the uniform class prior) in this section.
R cont ( f )
R µ -supv ( f )
R ∗ µ -supv
R ∗ cont
R µ -supv ( f ) ≥ R cont ( f ) + ∆ L R µ -supv ( f ) ≤ R cont ( f ) + ∆ U
R ∗ cont + ∆ U
R ∗ µ -supv − ∆ L
Gap between learning upper and lower bounds.
Both of our upper (Theorem 1) and lower (Theorem 2) bounds draw the linear relationship between the mean supervised loss R µ -supv and the contrastive loss R cont , with the additional intercept terms ∆ U and ∆ L .
Under the uniform class prior assumption, the intercepts are in the same order:
[Equation]
and the gap between two bounds is ∆ U − ∆ L = 4 ln cosh( L 2 )
+ 2 ln (cid:0)
+ 1 K (cid:1) , meaning that the gap shrinks to 4 ln cosh( L 2 ) as K increases.
Hence, our bounds have the tight intercepts, and the larger
K is beneﬁcial for CURL from the viewpoint of the estimation variance of the mean supervised loss.
Learning bounds and feasible region.
Next, we consider the ( R cont , R µ -supv ) -plot, in which a point indicates ( R cont ( f ) , R µ -supv ( f )) for some f (see Figure 1).
Here, let us focus on the feasible region in the ( R cont , R µ -supv ) -plot by assuming (cid:107) f (cid:107) 2 ≤ L for any f (same as Theorems 1 and 2).
Then, the mean supervised loss and contrastive loss are essentially lower-bounded by the constants
[Equation]
respectively.
Hence, the feasible region is
[Equation]
as illustrated in Figure 1.
The ﬁrst two bounds (8a) restrict the mean supervised loss by the contrastive loss.
We speciﬁcally refer to these bounds as learning bounds .
The remaining two bounds (8b) represent the achievable limits for each loss separately.
One of the important questions is how the smallest possible value of R µ -supv in the feasible region (8) changes as K and C change.
In other words, we are interested in whether the optimal point ( R ∗ cont , R ∗ µ -supv ) ( (cid:70) in Figure 1) is always achievable regardless of the values of K and C .
To investigate it, we check the following two points.
• The feasible region at R cont ( f ) =
R ∗ cont (Figure 2a):
We plot the value R ∗ cont + ∆ U (solid line; the R µ -supv -value of the point • in Figure 1) and the minimum possible R µ -supv ( R ∗ µ -supv ; dotted line) numerically.
These two curves do not cross for all K , which means R µ -supv ( f )
=
R ∗ µ -supv is attainable no matter the values K and C .
In addition, the bound becomes sharper as K increases, but the gap between the upper bound and R ∗ µ -supv does remain even at the limit K (cid:37) ∞ .
• The feasible region at R µ -supv ( f )
=
R ∗ µ -supv
(Figure 2b): When R µ -supv ( f )
=
R ∗ µ -supv , the contrastive loss R cont ( f ) is upper-bounded by R ∗ µ -supv − ∆ L (the R cont -value of the point (cid:4) in Figure 1).
The curve of this value does not cross R ∗ cont , which tells us that the lower bound does not exclude the optimal point ( R ∗ cont , R ∗ µ -supv ) from the feasible region (8) at any K .
Note that the gap between R ∗ cont and R ∗ µ -supv − ∆ L gradually increases, meaning that it becomes much easier to attain R ∗ µ -supv as K increases.
[Figure]
[Table]
[Equation]
[Table]
[Equation]
[Table]
Arora et al.
[Table]
[Equation]
Nozawa & Sato (2021)
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Ash et al.
Hence, the optimal point (cid:70) stays in the feasible region (8) no matter the value K .
From this viewpoint, smaller K is not necessarily disadvantageous because the optimal point (cid:70) remains in the feasible region.
Note again that the estimation of R µ -supv may become harder with the smaller K because of the bound gap ∆ U − ∆ L = O
(ln 1 / K ) , even if the optimal solution is unaffected by K .
Summary.
We draw a connection between the mean supervised loss and the negative sample size
K by the following claim: if we regard CURL as an estimation process of the mean supervised loss by the contrastive loss, the small K does not necessarily have the estimation bias, but the larger K helps reduce the estimation variance.
C OMPARISON WITH E XISTING L EARNING B OUNDS
This section discusses the difference between our main results and the existing theoretical results on
CURL.
The learning bound analysis of CURL has been ﬁrst established by Arora et al.
(2019) and
later extended by Nozawa & Sato (2021) and Ash et al.
Here, we brieﬂy review these results.
For ease of the comparison, we assume the uniform class prior, namely, π c = 1 / C for all c ∈
[ C ] .
We introduce a notation Col := (cid:80) k ∈ [ K ] 1 { c + = c − k } .
Let v
K
be the probability that sampled K negative classes contains all classes
c
∈
[ C ] .
[Equation]
The value v K is often referred to as the coupon collector’s probability .
Let τ K be the probability that at least one of the negative classes c − k is the same as the positive class c
+ .
Under the uniform class prior, τ K = 1 − (1 − 1 / C ) K .
The learning bounds are summarized in Table 1.
the slightly et al
[Figure] [Figure]
Comparison.
First, the coefﬁcient of R cont ( f ) in the upper bounds are numerically compared in Figure 3a.
While Theorem 1 has the coefﬁcient 1 for all K , the existing bounds have different nature—Arora et al.’s
and
Ash et al.
’s coefﬁcients have unique minima, and Nozawa & Sato’s coefﬁcient has monotonically decreasing nature.
Since the larger K was shown advantageous in the several experimental work on contrastive learning (Chen et al., 2020;
He et al., 2020), the behaviors of Arora et al.’s
and Ash et al.
’s
bounds do not explain it well.
In Section 6, we also experimentally conﬁrm the beneﬁts of the large K .
Figure 3b compares the upper bound values at R cont ( f ) =
R ∗ cont , namely, the best possible mean supervised loss in terms of the upper bounds.
The tendencies are slightly different from Figure 3a—
Ash et al.
’s bound is monotonically increasing, Arora et al.’s
and Nozawa & Sato’s bounds have a unique minimum, and ours is monotonically decreasing.
Among the compared bounds, only ours agrees well with the experimental fact that the larger K is better.
Note that the coefﬁcients of the existing theories are so large—the vertical axes are log-scaled—that the upper bounds are vacuous in both situations.
In contrast, our theory can bound the downstream classiﬁcation loss most sharply in all ranges of K .
This sharpness can be empirically observed as well in Figure 4; the details are stated in Section 6.2.
R ELATED W ORK
While our work and the existing works reviewed in Section 4 attempt to understand CURL by connecting the contrastive loss to the mean supervised loss, there are other approaches; Wang &
Isola (2020) showed that the contrastive loss asymptotically favors data representation uniformly distributed over the unit sphere yet aligning across semantically similar samples.
Li et al.
(2021) proposed an alternative loss function to the contrastive loss based on a kernel metric, following the similar idea to Wang & Isola (2020).
Tosh et al.
(2021) showed that a (linear) mean classiﬁer learned in CURL can approximate the (potentially nonlinear) Bayes classiﬁer well.
While our work does not handle data augmentation (DA), several works analyzed how data augmentation affects the performance of downstream classiﬁcation.
Wen & Li (2021) showed that
DA is necessary to recover sparse signals under a speciﬁc assumption on the model architecture.
HaoChen et al.
(2021) introduces a notion of the augmentation graph, representing how likely the nearby samples are generated via data augmentation and showed that a type of contrastive loss could be viewed as a low-rank approximation of the adjacency matrix of the augmentation graph.
von
Kügelgen et al.
(2021) proposed a loss function that enables the model to identify invariant factors across DA.
We mention a few works analyzing the other types of self-supervised learning; Garg & Liang (2020) analyzed masked self-supervised learning, Wei et al.
(2021) analyzed the input consistency loss for unsupervised learning, and Saunshi et al.
(2021) analyzed auto-regressive language models.
As a ﬁnal remark, multi-sample estimators (Oord et al., 2018; Poole et al., 2019; Song & Ermon, 2020) popularly used in mutual information (MI) estimation are substantially related to the contrastive loss.
Although the multi-sample estimators have high bias and low variance compared to variational estimators in general (Poole et al., 2019; Song & Ermon, 2020; Guo et al., 2021), the quantitative analysis of the bias-variance trade-off of the multi-sample MI estimators has yet to be clearly known.
Note that Tian et al.
(2020) and Tschannen et al.
(2020) experimentally showed that maximizing MI does not necessarily lead to good representation.
E XPERIMENTS
We veriﬁed our theoretical ﬁndings with experiments on synthetic datasets (Section 6.1), vision, and language datasets (Section 6.2).
The details of the experimental setup are provided in Appendix F.
6. S MALL - SCALE E XPERIMENTS ON S YNTHETIC D ATASET
Dataset and learning setups.
We create a synthetic dataset circle , which is a 2D dataset created as follows: for each class c ∈ [ C ] , 1 000 samples are drawn from Uniform ([ − 0 .
5 − 0 .
5] , [0 . 5 , 0 . 5]) , normalized, and multiplied by c +1 / 2 .
The generated samples are nonlinear and require disentanglement to be linearly separable.
We treated 60% of the generated samples as a training dataset and the rest of the samples as a test dataset.
We did not use data augmentation.
As a feature extractor f , we used a multi-layer perceptron (the number of units 2 - 256 - 256 - 256 ) with the ReLU activation functions following after each hidden layer.
During the training, the extracted feature representations are normalized.
For negative samples, we sampled K ∈ { 1 , 4 , 16 , 64 , 256 } samples without replacement from 2 B − 2 points included in the same mini-batch to avoid the inﬂuence of mini-batch size B , inspired by Ash et al.
Results.
Figure 5 shows a single trajectory in the ( R cont , R µ -supv )
-plot and the feasible region (confer Figure 1) for each K .
We plotted the trajectories by tracking ( R cont ( f ( t ) ) , R µ -supv ( f ( t ) )) at each epoch t computed with the test dataset.
All trajectories were located in between the upper ( R cont + ∆ U ) and lower ( R cont + ∆ L ) bounds as a matter of course.
Given that the existing learning bounds provide the much larger upper bounds (Figure 3), our learning bounds provide the ﬁnest estimate of the mean supervised loss.
In addition, it is remarkable that all trajectories have nearly the same slopes as our learning bounds, which constitutes solid evidence that our learning bounds capture the learning dynamics well.
In Figure 6, the mean supervised loss and accuracy are compared with the different K .
For each K , we conducted the same experiments with eight different random seeds, and the standard deviations are shown in the ﬁgures.
From these ﬁgures, it can be concluded that the contrastive loss performance becomes better with the larger K in the sense that the supervised loss improved and the variance shrank.
The variance improvement can be theoretically observed from Figure 5 as well; the larger K is, the smaller the gap between upper and lower bounds becomes. [Figure] [Figure]
6. L ARGE - SCALE E XPERIMENTS ON V ISION AND L ANGUAGE D ATASETS
We used the same datasets as Arora et al.
(2019): CIFAR-100 (Krizhevsky, 2009) and {+Wiki
+}
(Arora et al., 2019) datasets.
In addition, we used CIFAR-10 (Krizhevsky, 2009) dataset.
Learning setups.
To create positive pairs for contrastive learning, we treated the supervised classes as latent classes as in Arora et al.
and Ash et al.
We treated the original supervised classes of CIFAR-10/100 as [ C ] ; C = 10 and C = 100 , respectively.
We used K in { 4 , 16 , 32 , 64 , 128 , 512 } and in { 4 , 64 , 128 , 512 } for CIFAR-10/100, respectively.
For Wiki-3029, we used C ∈ { 500 , 1 000 , 2 000 , 3 029 } and K ∈ { 8 , 64 , 256 , 1 024 } .
For each different ( C, K ) pair, we trained the feature extractor f on the training dataset.
We then evaluated its supervised performance on the test dataset with mean and linear classiﬁers.
We used ResNet-18 (He et al., 2016)-based feature extractor f for CIFAR-10/100 and the fasttext (Joulin et al., 2017)-based feature extractor for Wiki-3029.
Results.
Figure 4 shows the comparison between the estimated upper bounds using Theorem 1 and actual supervised loss on CIFAR-10 for different K on the test dataset.
We estimated the bounds by substituting the actual R cont ( f ) to the equations shown in Theorem 1 and Table 1.
Our learning bound gave the closest upper bound to the experimental value of the supervised loss.
The existing learning bounds of Arora et al.
and Ash et al.
(2021) were not sharp enough to explain the classiﬁcation performance.
Although Nozawa & Sato’s bound was comparable with our bound, it was valid only in K + 1 ≥ C and tended to increase near K + 1 = C , as we explained in Section 4.
We investigated how K affects the test accuracy for different C in Figure 7.
The test accuracy improved or was saturated with the larger K for all C on Wiki-3029.
In contrast, The test accuracy was degraded as K increased in mean and linear classiﬁers on CIFAR-10/100.
This behavior could be partly because of the gap between the cross-entropy loss and the supervised accuracy—the theory of CURL, including the existing studies, usually focuses on the cross-entropy loss only.
Figurein Appendix F.6 revealed that the supervised loss was not signiﬁcantly worse with the larger K on
CIFAR-10/100.
With the smaller K and large C , we found that long epochs were more effective to improve classiﬁcation accuracy than increasing the negative sample size K (Figure 8b).
While similar experimental results were reported by Chen et al.
(2020, Figure 9), it is important to remark that we randomly drew K negative samples from the 2 B − 2 samples in the given mini-batch at each iteration as in
Ash et al.
(2021)—a different approach was used by Chen et al.
(2020) to regard the all samples in the mini-batch except an anchor sample as negative samples. Under our experimental setup, a [Figure]
[Figure] [Figure] [Figure] learner may encounter less diverse samples with the smaller K even if the mini-batch size B is the same, which could make the performance of downstream classiﬁcation worse—the longer epochs are necessary to mitigate the issue.
Since the CIFAR-10 dataset has a smaller C and is simpler than the
CIFAR-100, all accuracies were saturated with similar epochs for all K (Figure 8a).
C ONCLUSION
In this study, we established novel theoretical bounds for contrastive unsupervised representation learning.
We derived sharp upper and lower bounds on a downstream classiﬁcation loss that are tight in the negative sample size.
They suggest that the contrastive loss can be used as a downstream loss estimator and its estimation variance decays with larger negative samples.
In the experiments, we veriﬁed that our bounds well explained learning dynamics on the synthetic dataset, and the estimation variance was large with smaller negative samples.
In addition, the empirical downstream classiﬁcation loss was best explained by our learning bound compared to the existing ones.
It is left open to ﬁll the gap between the supervised loss and accuracy.
In addition, we may understand contrastive unsupervised representation learning better by incorporating the optimization perspective into the theoretical analysis.
E THICS S TATEMENT
Since this study is theoretical in nature, we do not have speciﬁc ethical concerns.
R EPRODUCIBILITY S TATEMENT
All experimental codes are available in the supplementary material.
In addition, the details of experimental datasets and hyper-parameters of learning algorithms are described in Appendix F.
All theorems’ proofs are described in Appendix B. Appendix A provides Lemmas used in Appendix B.
Appendices C and D provide how to derive equations used in Section 3.2 and Section 4, respectively.
«A PPENDIX » S HARP L EARNING B OUNDS FOR C ONTRASTIVE U NSUPERVISED R EPRESENTATION L EARNING
As an additional notation, the d -dimensional ball of radius r associated with the L p -norm is denoted by B dp ( r ) :
=
{ x ∈ R d | (cid:107) x (cid:107) p ≤ r } .
For z ∈ R C , the log-sum-exp function is denoted by
LSE ( z ) :
= ln( (cid:80)
c ∈ [ C ] exp( z c )) .
A U SEFUL L EMMAS
In this section, a few lemmas are introduced in order to prove the main results.
Lemma 3.
For z ∈ R N ,
[Equation]
where ∆ 0 := 2 ln(1 + N ) .
Proof.
Deﬁne H ( z ) :
=
LSE ( − [0 z 1 . . .
z N ])
+ LSE ([0 z 1 . . .
z N ]) .
Then, ∆ 0 is a lower bound of H .
Since
[Equation]
for all i ∈ [ N ] , z = 0 N satisﬁes the ﬁrst-order optimality condition of H .
By noting that H is convex due to the convexity of the log-sum-exp functions, H is minimized at z = 0 N .
Hence, we can choose ∆
= H ( 0 N ) = 2 ln(1 + N ) .
Lemma 4.
For z 0 ∈ [ − L 2 , L 2 ] , z ∈ [ − L 2 , L 2 ] N , and ρ > 0 ,
[Equation]
[Equation]
where ∆ 1 := 2 ln (cid:16) ρ (cid:16) N + 1 + 1 ρ (cid:17) cosh( L 2 ) (cid:17) .
Proof.
The goal is to ﬁnd a tight upper bound ∆ 1 of −
[Equation]
where ˜ z :
=
[ z 0 z (cid:62) ] (cid:62) .
Equivalently, we aim at an upper bound of H (cid:48)
(˜ z ) for ˜ z ∈ B N +1 ∞ ( L 2 ) .
Observe that H (cid:48)
(˜ z ) =
LSE
[Equation]
is the sum of the two log-sum-exp functions after adding the constant vector hence it is convex in ˜ z .
In addition, the domain B N +1 ∞ ( L 2 ) is a compact convex polytope.
Henceforth, every vertex of the polytope, ˜ z ∈ {− L 2 , L 2 } N +1 , is a local maximizer because maximizing H (cid:48)
(˜ z ) is concave minimization over a convex polytope (Pardalos & Rosen, 1986).
Since H (cid:48)
(˜ z ) is symmetric in every z n except z 0 , we need to divide the cases depending on either z
=
L 2 or z
= − L 2 and compare the maximum values of
H (cid:48)
(˜ z ) .
If z 0 =
L 2 : In order to maximize H (cid:48) (˜ z ) , it is sufﬁcient to test the vertices ˜
z ∈ { z ∈ B N +1 ∞ ( L 2 )
| z
=
L 2 } and see the difference between ˜ H (cid:48) ( m ) and ˜ H (cid:48) ( m + 1) , where
[Equation]
for m ∈ { 0 , . . .
, N } to seek the maximum.
Here, the deﬁnition of ˜ H (cid:48) can be naturally extended over the real domain m ∈ R .
A simple algebra shows
[Equation]
Since ˜ H (cid:48) is concave over m ∈ R , ˜ H (cid:48) is maximized at m = (cid:104) N − 1 − 1 ρ [Equation] + ( < N ) .
If ρ < 1 N − 1 , then the maximum value is
[Equation]
[Equation]
If ρ ≥ 1 N − 1 , ˜ H (cid:48) is maximized at m =
and its value is
[Equation]
where the inequality follows from the AM-GM inequality exp(2 L 2 )+exp( − 2 L [Equation] ≥ 1 .
If z 0 = − L 2 : Similarly, we deﬁne
[Equation]
[Equation]
and extends its deﬁnition over the real line m ∈ [0 , N ] .
Since
[Equation]
˜ H
[Equation]
is maximized at m
[Equation]
min
[Equation]
and its value is
[Equation]
If ρ < 1 N − 1 , ˜ H (cid:48) is maximized at m = N and its value is
[Equation]
[Equation] After all
[Equation]
˜ H (cid:48) is bounded from above by either
[Equation]
These two values can be compared as follows.
[Equation]
where the ﬁrst inequality is due to the AM-GM inequality exp(2 L 2 )+exp( − 2 L 2 )≥ 1 .
Hence, regardless of ρ , 2 ln (cid:16)(cid:16) N + 1 + 1 ρ (cid:17) cosh
( L 2 ) (cid:17) is a tight upper bound of ˜ H (cid:48) .
Lemma 5.
For all z 0 ∈ R and z ∈ R K such that z 0 , z k ∈ [ − L 2 , L 2 ] ( ∀ k ∈ [ K ] ),
[Equation]
Proof.
We write ˜ z :=
[ z 0 z (cid:62) ] (cid:62) .
Let H (˜ z ) be a function such that
[Equation]
Our goal is to ﬁnd a tight upper bound of H (˜ z ) for ˜ z ∈ B K +1 ∞ ( L 2 ) .
Observe that H (˜ z ) is the sum of the two log-sum-exp functions hence it is convex in ˜ z .
In addition, the domain B K +1 ∞ ( L 2 ) is a compact convex polytope.
Henceforth, every vertex of the polytope, ˜ z ∈ {− L 2 , L 2 } K +1 , is a local maximizer because maximizing H (˜ z ) is concave minimization over a convex polytope (Pardalos & Rosen, 1986).
Since H (˜ z ) is symmetric in every element ˜
z k , it is sufﬁcient to test the vertices and see the difference between
[Equation]
and H
[Equation]
for j ∈ { 0 , . . .
, K } to seek out the global maximum.
For 0 ≤ j ≤ K , a simple algebra shows
[Equation]
from which we can tell that ˜ H ( j ) is maximized at j = K/ 2 when K is even and j =
( K + 1) /when
K is odd.
In addition, it is conﬁrmed that
[Equation]
where the AM-GM inequality is invoked at the last line.
Eventually, ˜ H
(( K + 1) / 2) turns out to be a tight upper bound of H (˜ z ) for ˜ z ∈ B K +1 ∞
( L 2 ) .
It is elementary to conﬁrm ˜ H
(( K + 1) / 2)
= 2 ln (cid:8) ( K + 1) cosh( L 2 ) (cid:9) .
B P ROOFS OF M AIN R ESULTS
In this section, we provide proofs for the main results, Theorems 1 and 2.
Theorem 1.
For all f such that (cid:107) f
( x ) (cid:107) 2 ≤ L ( ∀ x ∈ X ), the following inequality holds.
[Equation]
where ∆ U := 2 ln(( CKπ ( − 1) + 1) cosh( L 2 ))
− 2 ln(1 + K ) − ln
Kπ ( − 1) .
Proof of Theorem 1.
Deﬁne the two constants
[Equation]
[Equation]
The proof begins with the Jensen’s inequality applied on the contrastive loss:
[Equation]
where the second inequality follows from Lemma 3.
The ﬁrst term, the origin-symmetric transform of the contrastive loss, is further bounded as follows.
[Equation]
where the ﬁrst inequality is the Jensen’s inequality and the second inequality holds by noting that E
[Equation]
) A .
In order to proceed further, the origin-symmetric transform is needed to be applied on the lower-bounding term, resulting in a term aligning to the mean supervised loss.
[Equation]
) where the ﬁrst inequality is a consequence of Lemma 4 and the second inequality is simply observed by ln( 1 Kπ ( [Equation] 1) + x ) ≥ ln x for any x > 0 since 1 Kπ ( − 1) ≥ 0 .
Combining all the above, we get the ﬁnal bound R µ -supv ( f ) ≤ R cont ( f ) − ∆
+ ∆ 1 − ln
Kπ ( − 1) .
Even if Theorem 1 assumes Y =
[ C ] , it can be extended to the cases Y ⊆
[ C ] (subset) and Y is a coarse-grained set of [ C ] .
When Y (cid:54) =
[ C ] , R µ -supv is deﬁned over the class set Y , while R cont is deﬁned over the class set
[ C ] .
In the subset case, we can replace Eq.
(52) with the subset class bound:
[Equation]
In the coarse-grained case, Eq.
(52) holds without modiﬁcation.
Theorem 2.
For all f such that (cid:107) f
( x ) (cid:107) 2 ≤ L ( ∀ x ∈ X ), the following inequality holds.
[Equation]
where ∆ L := H ( π ) + ln K
[Equation]
cosh
[Equation]
Proof of Theorem 2.
The proof is essentially a consequence of the Fenchel’s inequality and the
Jensen’s inequality.
First, by noting that the convex conjugate of the log-sum-exp function is the
[Equation]
If we choose an arbitrary p ∈ (cid:52) C , R µ -supv ( f ) is lower bounded (Fenchel’s inequality).
Our choice is p = π .
Recall that K is the number of negative samples.
Then,
[Equation]
Here, we can proceed with the Jensen’s inequality to lower bound the ﬁrst term: for a non-negative vector z ∈ R N ≥ 0 , the inequality − N − 1 (cid:80) i ∈
[ N ] ln z i ≥ − ln( N − 1 (cid:80) i ∈
[ N ] z i ) holds.
If we set z k = exp( f ( x ) (cid:62) ( f ( x + ) − f ( x − k ))) for k ∈ [ K ] ,
[Equation]
Finally
[Equation]
by using Lemma
[Equation]
which concludes the proof
[Equation]
Unlike Theorem 1, Theorem 2 can only be extended to the coarse-grained Y from the case Y =
[ C ] .
Indeed, the ﬁrst expectation term on the right-hand side of Eq.
(56) can be transformed as
E
[Equation]
because of the linearity of the expectation.
On the other hand, it is not straightforward to lower-bound
Eq.
(56) by the expectation over the label set [ C ] when Y is a strict subset of [ C ] .
C E SSENTIAL B OUNDS OF M EAN S UPERVISED AND C ONTRASTIVE L OSSES
This section provides a supplementary explanation of the essential lower bounds of the mean supervised and contrastive losses.
The common approaches of CURL applies the normalization on representation, in order to employ the cosine similarity
[Equation]
as the similarity metric.
Then, it is reasonable to assume (cid:107) f
( x ) (cid:107) 2 ≤ L for all x with our data representation f .
The normalized representation corresponds to the case L = 1 .
When we introduce the constraint (cid:107) f ( x ) (cid:107) 2 ≤ L , the mean supervised loss and contrastive loss are restricted as well.
As for the mean supervised loss,
[Equation]
[Equation]
As for the contrastive loss,
[Equation]
where the Jensen’s inequality is applied in the second inequality.
D D ISCUSSION OF E XISTING L EARNING B OUNDS
In this section, we describe the existing learning bounds in details to make them comparable with our main results.
Before the discussion, we need to introduce the sub-class loss (of the mean classiﬁer), which is the supervised classiﬁcation loss over a subset of classes:
[Equation]
where T ⊆ [ C ] is a subset of classes and y is drawn from the subset of π with respect to T .
Arora et al.
’s bound.
We introduce additional notation that Arora et al.
(2019) use.
For a subset of classes T ,
• Q ⊆
[ C ] is the set of distinct classes in c + , c − 1 , . .
. , c − K • I
+ := { k ∈ [ K ] | c − k
= c + } • Col := (cid:80) k ∈
[ K ] 1 { c + = c − k }
= | I + | • ρ max ( T ) :
= max c ∈ T π c • ρ +min ( T ) :
= min c ∈ T P c + , { c − k } k ∼ π K + [Equation] +
[Equation] ) • τ K :=
P ( I + (cid:54) = ∅ )
Arora et al.
(2019) prove a ﬁnite-sample learning bound in Theorem B.1.
In its proof, Eq.
(26) is a learning bound established for a ﬁxed f .
For the comparison, we focus on their Eq.
[Equation]
We split the expectation term in the left-hand side as follows.
[Equation]
Under the uniform class prior assumption ( π = 1 / C · 1 ), ρ max ([ C ]) = 1 / C , and we can pick any class c 0 ∈ [ C ] by the symmetry and ρ +min ([ C ]) =
P ( c + = c 0 | Q =
[ C ] , I + = ∅ )
= 1 / C .
In addition, τ K
[Equation]
As a result, we obtain the following simpliﬁed expression in Table 1:
[Equation]
Nozawa & Sato’s bound.
The learning bound provided by Nozawa & Sato (2021, Theorem 8) involves a factor resulting from data augmentation and self-supervised learning setting.
By dropping this (negative) factor, the learning bound is
[Equation]
resulting in the bound in Table 1.
The sub-class loss may be safely dropped because it has the coefﬁcient 1 − v K +1 , which is expected to be exponentially small in K .
Ash et al.
’s bound.
Ash et al.
(2021, Theorem 5) provides the following learning bound
[Equation]
By substituting π = 1 / C · 1 and τ K E [ln(Col + 1) | I + (cid:54) = ∅ ] =
E [ln(Col + 1)] , the bound in Tableis obtained.
E R ELATIONSHIP TO M UTUAL I NFORMATION (MI) E STIMATION
The contrastive loss R cont we studied in this paper is also known as the InfoNCE loss (Oord et al., 2018), which is known to be deeply related to the multi-sample estimation of mutual information (MI) ( Oord et al., 2018; Poole et al., 2019; Song & Ermon, 2020). Recently, the theoretical limitations of sample-based MI estimation have been analyzed (Gao et al., 2015; McAllester & Stratos, 2020).
These studies revealed that a particular type of sample-based estimator of MI (Gao et al., 2015) or its lower bound (McAllester & Stratos, 2020) can be upper bounded by O (ln N ) for the number of samples N .
In this section, we discuss the implications of these limitations in the CURL setting.
Given two random variables X and Y , suppose that we have K + 1 randomly drawn pairs { ( x i , y i ) } K +1 i
=1 from these random variables such that for all ( i, j ) , ( x i , y j ) can be regarded as a positive pair when i = j , and otherwise can be regarded as a negative pair.
Poole et al.
Equation 10) derived the following lower bound for MI:
[Equation]
where I ( X ; Y ) is the MI between X and Y
[Equation]
and s ( x, y ) is a critic function.
This lower bound estimator can be rewritten using R cont as follows:
[Equation]
The ﬁrst equality is obtained by putting the constant in the denominator outside.
The third equality comes by replacing the notation ( x i , y i ) with ( x, x + ) under the assumption that all ( x i , y i ) come from the same iid distribution.
By setting s ( x, y ) :
= f ( x ) (cid:62) f ( y ) , we obtain the last equation.
Here, McAllester & Stratos (2020) give the following theorem for the sample-based estimator of the lower bound on MI.
Theorem 6 (McAllester & Stratos (2020) Theorem 1.1, informal) .
Let (cid:98)
I N be any mapping from N samples of ( X, Y ) to R that satisﬁes
[Equation]
in high probability, then the following relationship holds in high probability:
[Equation]
[Equation]
Since I K +1 NCE satisﬁes the condition for (cid:98)
I K +1 , we now have the following:
[Equation]
However, the right-hand statement always holds by the construction of R cont for all f ( ∀ f , R cont ( f ) ≥ 0 ).
In other words, in the case of the CURL setting, McAllester & Stratos (2020)’s theorem does not restrict R cont , which means that the large K effect investigated in our paper comes from a completely different mechanism from the above theorem.
The existing studies on sample-based MI estimation are worthwhile in the sense that these works revealed the O (ln N ) effect on the non-trivial estimators such as k -NN based estimator (Gao et al., 2015) or any kind of lower bound estimator (McAllester &
Stratos, 2020).
F E XPERIMENTAL DETAILS F. S YNTHETIC D ATASET
We used Adam (Kingma & Ba, 2015) optimizer with the weight decay of coefﬁcient 0 .
01 to all parameters.
The mini-batch size was set to B = 1 024 and the number of epochs was 300 .
The learning rate was set to 0 .
01 with ReduceLROnPlateau scheduler (patience: 10 epochs) provided by PyTorch (Paszke et al., 2019).
F. CIFAR-10/
We treated 10% training samples as a validation dataset by sampling class uniformly.
We used the original test dataset for testing.
We used the same data-augmentation as in the CIFAR-10 experiment by Chen et al.
(2020) during contrastive learning and linear supervised training of the linear classiﬁer.
As a feature extractor f , we modiﬁed the ResNet-18 (He et al., 2016) by following the convention of self-supervised representation learning (Chen et al., 2020, B.9); replacement of the ﬁrst convolutional layer with a smaller one, removal of the ﬁrst max-pooling layer, and replacement of the ﬁnal fullyconnected layer with a nonlinear projection head whose dimensional is 32 .
Since we need to enlarge the negative samples size K that depends on the size of mini-batches, we followed a large mini-batch training setting used in recent self-supervised learning (Chen et al., 2020;
Caron et al., 2020).
We used LARC (You et al., 2017) optimizer wrapping the momentum SGD, whose momentum term was 0 .
We applied weights decay of coefﬁcient 10 − 4 to all parameters except for all bias terms and batch norm’s parameters.
The base learning rate was initialized at lr × √ B , where lr ∈ { 2 , 4 , 6 } × 1 / 64 and mini-batch size B = 1 024 inspired by SimCLR’s squared learning rate scaling.
As a learning rate scheduler for each iteration, we used linear warmup during the ﬁrst 10 epochs and cosine annealing without restart (Loshchilov & Hutter, 2017) during the rest epochs.
The number of epochs was 2 000 .
We implemented our experimental code by using PyTorch (Paszke et al., 2019)’s distributed dataparallel training (Li et al., 2020) on 8 NVIDIA A100 GPUs provided by the internal cluster.
Therefore we replaced the all batch normalization layer with SyncBatchNorm module provided by PyTorch.
To accelerate contrastive learning, we used automatic mixed-precision training provided by PyTorch.
F. W IKI -
Wiki-3029 contains 3 029 English Wikipedia article pages.
Each page consists of 200 sentences.
Since the dataset does not have the explicit train/validation/test splits, we split the dataset into 70% / 10% / 20% train/validation/test datasets, respectively.
As a pre-processing, we tokenized the dataset using torchtext’s basic_english tokenizer.
After tokenization, we removed the tokens whose frequency is less than 5 in the training dataset.
We did not use data augmentation.
We used fasttext (Joulin et al., 2017)’s based feature extractor.
In our preliminary experiments, only using a word embedding layer and average pooling among words perform better than either additional linear or nonlinear projection heads.
A similar model to ours is also used in Ash et al.
The dimensionality of the word embedding layer was 256 .
We mainly followed the same optimization setting as our CIFAR-10/100 experiments.
We note that the mini-batch size B = 2 048 ; the initial learning rate lr was selected in { 1 , 2 , 3 , 4 } × 1 / 40 ; no weights decay; the number of epochs was 90 ; and perform linear warmup during the ﬁrst 3 epochs.
When we decrease C , the number of epochs is multiplied by 3000 / C for simplicity.
F. C ONTRASTIVE L EARNING
By following the data generation process in contrastive representation learning and existing work (Arora et al., 2019; Ash et al., 2021), we treated the supervised classes Y as latent classes [ C ] .
After obtaining training/validation/test datasets as described above, we carefully constructed positive pairs for contrastive learning before training 14 as follows; We treated each sample in the training data as an anchor sample.
We drew a different sample from the same latent class of each anchor sample as a positive sample in the training dataset.
For negative samples, we drew K negative samples from other samples in the same mini-batch by following the convention of self-supervised representation learning such as SimCLR (Chen et al., 2020).
Since Chen et al.
(2020) used all other samples as negative samples, the negative samples size and the size of mini-batches depend on each other: K = 2 B − 2 .
To relax the effect of the difference of the mini-batch size when we change K , we drew K samples without replacement from 2 B − 2 inspired by Ash et al.
In this sampling, we guaranteed to draw at most one sample from each positive pair because we are concerned about the relation between the number of latent classes and K .
We did not use validation and test datasets during contrastive representation learning.
F.  M EAN AND L INEAR C LASSIFIERS ’ E VALUATION
For evaluation, we reported the test accuracy values of mean and linear classiﬁers.
For a linear classiﬁer, we used Nesterov’s momentum SGD, whose momentum coefﬁcient was 0 . 9 without weight decay.
We set the mini-batch size B = 256 and B = 512 for CIFAR-10/100 and Wiki-3029, respectively.
We used cosine annealing without restart as a learning rate scheduler for each iteration.
We set 100 and 30 epochs for CIFAR-10/100 and Wiki-3029 datasets, respectively.
For CIFAR-10/100, we set learning rate as 0 .
For Wiki-3029, we searched the learning rate in { 0 . 5 , 1 , 5 , 10 , 50 }× 1 / 10 3 .
The learning rate was scaled by using squared learning rate scaling.
For linear evaluation of CIFAR10/100, we used PyTorch’s distributed data-parallel training.
We calculated the test accuracy by using the best combination of the contrastive model and the hyper-parameter of a linear classiﬁer that maximizes the validation accuracy.
We repeated contrastive learning and downstream task’s evaluation three times with different random seeds and reported the averaged values.
F. D ETAILS OF F IGURE
Before computing the upper bounds and supervised loss, we normalized feature representations f ( x ) learned in Appendix F.4 to ensure L = 1 , which is the upper bound of (cid:107) f
( x ) (cid:107) 2 , ∀ x .
For each random seed and the number of negative samples K , we selected learned feature encoder f
that got the highest validation mean supervised accuracy in different learning rates of the optimizer of the contrastive learning.
Then we calculated the test supervised loss value by using the selected contrastive models.
Using the same feature encoder with L 2 normalization, we calculated the contrastive loss on the test dataset.
To do so, we created positive pairs by the same procedure on the test dataset as described in Appendix F.4.
Negative samples were also drawn from the other samples in the mini-batches as the contrastive learning step described in Appendix F.4.
To calculate the contrastive loss, we used [Figure] the same batch size as the contrastive learning step and only one epoch.
Since this contrastive loss calculation was stochastic due to the sampling of positive and negative samples, we repeated the contrastive loss calculation 25 times and averaged them to create plot Figure 4.
Note that we used the theoretical values of τ K , v K +1 , E ln(Col + 1) that are shown in the existing upper bounds on Tablerather than the simulated values.
Figure 9 shows the enlarged version of Figure 4 and the same plot using CIFAR-100.
This ﬁgure focuses on the detailed comparison between the test datasets’ empirical supervised loss values and theoretical bounds.
For both CIFAR-10/100 datasets, there were almost no changes in the supervised loss as K varied, and the losses were slightly larger in the region where K was small.
These results are consistent with the theoretical estimation of the upper bounds (solid lines).
F. D ETAILS OF F IGURE
During minimization of the contrastive loss to learn f in Appendix F.4, we saved the model’s weight at every 200 epochs.
We reported the test mean supervised accuracy using f that maximized validation accuracy among different learning rate values.
F. A DDITIONALLY U SED L IBRARIES
In our experiments, we also used scikit-learn (Pedregosa et al., 2011) for train/val/test data splits.
We created all plots by using matplotlib (Hunter, 2007) and seaborn (Waskom, 2021) via pandas (Reback et al., 2020) except for Figure 1.
We managed our experiments’ conﬁguration using hydra (Yadan, 2019) and experimental results using Weights & Biases (Biewald, 2020).
For effective parallelized execution of our experimental codes, we use GNU Parallel (Tange, 2021).