T HE P OWER OF C ONTRAST FOR F EATURE L EARNING : A T HEORETICAL A NALYSIS
Anonymous authors
Paper under double-blind review
A BSTRACT
Contrastive learning has achieved state-of-the-art performance in various selfsupervised learning tasks and even outperforms its supervised counterpart.
Despite its empirical success, theoretical understanding of why contrastive learning works is still limited.
In this paper, under linear representation settings, (i) we provably show that contrastive learning outperforms autoencoder, a classical unsupervised learning method, for both feature recovery and downstream tasks; (ii) we also illustrate the role of labeled data in supervised contrastive learning.
This provides theoretical support for recent ﬁndings that contrastive learning with labels improves the performance of learned representations in the in-domain downstream task, but it can harm the performance in transfer learning.
We verify our theory with numerical experiments
I NTRODUCTION
Deep supervised learning has achieved great success in various applications, including computer vision (Krizhevsky et al., 2012), natural language processing (Devlin et al., 2018), and scientiﬁc computing (Han et al., 2018).
However, its dependence on manually assigned labels, which is usually difﬁcult and costly, has motivated research into alternative approaches to exploit unlabeled data.
Self-supervised learning is a promising approach that leverages the unlabeled data itself as supervision and learns representations that are beneﬁcial to potential downstream tasks.
At a high level, there are two common approaches for feature extraction in self-supervised learning: generative and contrastive (Liu et al., 2021).
Both approaches aim to learn latent representations of the original data, while the difference is that the generative approach focused on minimizing the reconstruction error from latent representations, and the contrastive approach targets to decrease the similarity between the representations of contrastive pairs.
Recent works have shown the beneﬁts of contrastive learning in practice (Chen et al., 2020a;
He et al., 2020; Chen et al., 2020b;c).
However, why the contrastive approach outperforms the generative approach remains mysterious.
Additionally, recent works aim to further improve contrastive learning by introducing the label information.
Speciﬁcally, Khosla et al.
(2020) proposed the supervised contrastive learning , where the contrasting procedures are performed across different classes rather than different instances.
With the help of label information, their proposed method outperforms self-supervised contrastive learning and classical cross entropy based supervised learning.
However, despite this improvement on in-domain downstream tasks, Islam et al.
(2021) found that such improvement in transfer learning is limited and even negative for such supervised contrastive learning.
This phenomenon motivates us to rethink the role of labeled data in the contrastive learning framework.
In this paper, we ﬁrst compare contrastive learning with a representative method in the generative approach – the autoencoders.
Speciﬁcally, we initialize the investigation in the linear representation setting, which has been widely adopted in theory to shed light upon complex machine learning phenomena such as in Du et al.
Tripuraneni et al.
We provide a theoretical analysis of their feature learning performances on the spiked covariance model (Bai & Yao, 2012;
Yao et al., 2015; Zhang et al., 2018) and theoretically justify why contrastive learning outperforms autoencoders—contrastive learning is able to remove more noises by constructing contrastive samples.
Then we investigate the role of label information in the contrastive learning framework and provide a theoretical justiﬁcation of why labeled data help to gain accuracy in same-domain classiﬁcation while can hurt multi-task transfer learning.
Related works The idea of contrastive learning was ﬁrst proposed in Hadsell et al.
(2006) as an effective method to perform dimension reduction.
Following this line of research, Dosovitskiy et al.
(2014) proposed to perform instance discrimination by creating surrogate classes for each instance and Wu et al.
(2018) further proposed to preserve a memory bank as a dictionary of negative samples.
Other extensions based on this memory bank approach include He et al.
; Misra &
Maaten (2020); Tian et al.
; Chen et al.
(2020c).
Rather than keeping a costly memory bank, another line of works exploits the beneﬁt of mini-batch training where different samples are treated as negative to each other Ye et al.
; Chen et al.
(2020a).
Moreover, Khosla et al.
(2020) explores the supervised version of contrastive learning where pairs are generated based on label information.
Despite its success in practice, theoretical understanding of contrastive learning is still limited.
Previous works provide provable guarantees for contrastive learning under conditional independence assumption (or its variants) (Arora et al., 2019; Lee et al., 2020;
Tosh et al., 2021;
Tsai et al., 2020).
Speciﬁcally, they assume the two contrastive views are independent conditioned on the label and show that contrastive learning can provably learn representations beneﬁcial for downstream tasks.
In addition to this line of research, Wang & Isola (2020); Graf et al.
(2021) investigated the representation geometry of supervised contrastive loss, and HaoChen et al.
(2021) provided analysis via a novel concept of augmentation graph with a new loss function that performs spectral decomposition on such graph.
Moreover, Wen & Li (2021) considered the representation learning under the sparse coding model and studied the optimization properties on shallow ReLU neural networks.
Different from all previous works, which aim to show that contrastive learning can learn useful representation, our paper aim to explain why contrastive learning outperforms other representation learning methods and also shed light on the role of labeled data in contrastive learning framework, which is under-explored in prior works.
P RELIMINARIES
Notations In this paper, we use O, Ω , Θ to hide universal constants and we write a k (cid:46) b k for two sequences of positive numbers { a k } and { b k } if and only if there exists an universal constant
C > 0 such that a k < Cb k for any k .
We use (cid:107) · (cid:107) , (cid:107) · (cid:107) 2 , (cid:107) · (cid:107) F to represent the (cid:96) 2 norm of vectors, spectral norm of matrices and Frobenius norm of matrices respectively.
Let O d,r be a set of d × r orthogonal matrices.
i.e., O d,r (cid:44) { U ∈ R d × r : U (cid:62) U = I r } .
We use | A | to denote the cardinality of a set A .
For any n ∈ N + , let [ n ] = { 1 , 2 , · · · , n } .
We use (cid:107) sin Θ( U 1 , U 2 ) (cid:107) F to refer to the sine distance between two orthogonal matrices U 1 , U 2 ∈
O d,r , which is deﬁned by: (cid:107) sin Θ ( U 1 , U 2 ) (cid:107) F (cid:44) (cid:13)(cid:13) U (cid:62) 1 ⊥ U 2 (cid:13)(cid:13) F .
More properties of sine distance can be found in Section A.1.
We use { e i } di =1 to denote the canonical basis in d -dimensional Euclidean space R d , that is, e i is the vector whose i -th coordinate is 1 and all the other coordinates are 0 .
Let I { A } be an indicator function that takes 1 when A is true, otherwise takes 0 .
We write a ∨ b and a ∧ b to denote max( a, b ) and min( a, b ) , respectively.
2. S ETUP
Given an input x ∈ R d , contrastive learning aims to learn a low dimensional representation h
= f ( x ; θ ) ∈ R r by contrasting different samples, i.e., maximizing the agreement between positive pairs, and minimizing the agreement between negative pairs.
Suppose we have n data points X
=
[ x 1 , x 2 , · · · , x n ] ∈ R d × n from the population distribution D .
The contrastive learning task can be formulated to be an optimization problem:
[Equation]
where (cid:96) ( · ) is a contrastive loss and λR ( θ ) is a regularization term; B Pos i , B Negi are the sets of positive samples and negative samples corresponding to x i , which we will describe in detail below.
Losses and Models.
We then present the model setup considered in this paper.
(a).
Linear representation and regularization term .
We consider the linear representation function f
( x, W ) =
Wx , where the parameter θ is a matrix W ∈ R r × d .
Since regularization techniques have been widely adapted in contrastive learning practice (Chen et al., 2020a;
He et al., 2020; Grill et al., 2020), we further consider penalizing the representation by a regularization term R ( W )
=
(cid:107) WW (cid:62) (cid:107) 2 F / 2 to encourage the orthogonality of W and therefore promote the diversity of w i to learn different representations.
(b).
Triplet Contrastive loss .
The contrastive loss is set to be the average similarity between positive pairs minus that between negative pairs:
[Equation]
where B Pos , B neg are sets of positive samples and negative samples corresponding to x .
This loss has been commonly used in constrastive learning (Hadsell et al., 2006) and metric learning (Schroff et al., 2015;
He et al., 2018).
In Khosla et al.
, the authors show that it is an approximation of the NT-Xent contrastive loss, which has been highlighted in recent contrastive learning practice (Sohn, 2016;
Wu et al., 2018;
Oord et al., 2018; Chen et al., 2020a).
(c).
Generation of positive and negative pairs .
There are two common approaches to generate such pairs, depending on whether or not label information is available.
When the label information is not available, the typical strategy is to generate different views of the original data via augmentation (Hadsell et al., 2006; Chen et al., 2020a).
Two views of the same data point serve as positive pair for each other while those of different data serve as negative pairs.
Deﬁnition 2.1 (Augmented pairs generation) .
Given two augmentation functions g 1 , g 2 : R d → R d and n training samples B = { x i } i ∈ [ n ] , the augmented views are given by: { ( g 1 ( x i ) , g 2 ( x i )) } i ∈ [ n ] .
Then for each view g v ( x i ) , v = 1 , 2 , the corresponding positive samples and negative samples are deﬁned by: B Posi,v = { g s ( x i ) :
s ∈ [2] \ { v }} and B Negi,v = { g s ( x j ) : s ∈ [2] , j ∈ [ n ] \ { i }} .
The loss function of self-supervised contrastive learning problem can be written as:
[Equation]
In particular, we adopt the following augmentation in our analysis.
Deﬁnition 2.2 (Random masking augmentation) .
The two views of the original data are generated by randomly dividing its dimensions to two sets, that is, g 1 ( x i ) =
Ax i , and g 2 ( x i ) =
( I − A ) x i , where A = diag( a 1 , · · · , a
d ) ∈ R d × d is the diagonal masking matrix with { a i } di =1 being i.i.d.
random variables sampled from a Bernoulli distribution with mean 1 / 2 .
A similar augmentation was considered in Wen & Li (2021).
However, our primary interest lies in comparing the performance of contrastive learning against autoencoders and analyzing the role of labeled data, while their work focuses on understanding the training process of neural networks in contrastive learning.
When the label information is available, Khosla et al.
(2020) proposed the following approach to generate pairs.
Deﬁnition 2.3 (Supervised pairs generation) .
In a K -class classiﬁcation problem, given n k samples for each class k ∈
[ K ] : { x ki : i ∈ [ n k ] } Kk =1 and let n = (cid:80) Kk
=
1 n k , the corresponding positive samples and negative samples for x ki are deﬁned by B Posi,k = { x kj : j ∈ [ n k ] \ i } and B Negi,k
= { x sj : s ∈
[ K ] \ k, j ∈ [ n s ] } .
That is, the positive samples are the remaining ones in the same class with x ki and the negative samples are the samples from different classes.
Correspondingly, the loss function of the supervised contrastive learning problem can be written as:
[Equation]
(d).
Spiked Covariance Model .
We consider the following spiked covariance model (Bai & Yao, 2012; Yao et al., 2015; Zhang et al., 2018) to study the power of contrastive learning:
[Equation]
where z ∈ R r and ξ ∈ R d are both zero mean sub-Gaussian random variables.
In particular,
U (cid:63) ∈
O d,r and Σ = diag( σ 21 , · · · , σ 2 d ) .
The ﬁrst term U (cid:63) z represents the signal of interest residing in a low-dimensional subspace spanned by the columns of U (cid:63) .
The second term ξ is the dense noise with heteroskedastic noise.
Given that, the ideal low-dimensional representation is to compress the observed x into a low-dimensional representation spanned by the columns of U (cid:63) .
In this paper, we aim to learn a good projection W ∈ R r × d onto a lower-dimensional subspace from observation x .
Since the information of W is invariant with the transformation W ← OW for any invertible matrix O ∈ R r,r , the essential information of W is contained in the right eigenvector of W .
Thus we quantify the goodness of the representation W by the sine distance (cid:107)
sin Θ
( U, U (cid:63) )
(cid:107) F , where U is the top- r right eigenspace of W .
S ELF - SUPERVISED CONTRASTIVE LEARNING VERSUS AUTOENCODER
Autoencoder and contrastive learning are two popular approaches for self-supervised learning.
Recent experiments have highlighted the improved performance of contrastive learning compared with autoencoders.
In this section, we rigorously demonstrate the advantage of contrastive learning over autoencoder by investigating the linear representation settings under the spiked covariance model
Eq.(2.5).
The investigation is conducted for both feature recovery and downstream tasks.
3. R ECOVER FEATURES FROM N OISY D ATA
Here we focus on the analysis of feature recovery to understand the beneﬁt of contrastive learning over autoencoders.
As mentioned above, our target is to recover the subspace spanned by the columns of U (cid:63) , which can further help us obtain information on the unobserved z that is important for downstream tasks.
However, the observed data has covariance matrix of ν 2 U (cid:63) U (cid:63) (cid:62) + Σ rather than the desired ν 2 U (cid:63) U (cid:63) (cid:62) , which brings difﬁculty to representation learning.
We demonstrate that contrastive learning can better exploit the structures of core features and obtain better estimation than an autoencoder in this setting.
We start with autoencoders.
Formally, an autoencoder consists of an encoder f AE : R d → R r and a decoder g DE : R r → R d .
While the encoder compresses the original data into low dimensional features, and the decoder recovers the original data from those features.
It can be formalized as the following optimization problem for samples { x i } ni =1 (Ballard, 1987;
Fan et al., 2019):
[Equation]
In the linear representation setting, where f AE ( x ) = W AE x + b AE and g DE ( y )
=
W DE y + b DE , previous works (Bourlard & Kamp, 1988; Plaut, 2018) have shown that the autoencoder can be reduced to principal component analysis (PCA).
That is, the optimal W AE is given by:
[Equation]
where U AE is the top- r eigenvectors of matrix M :
= X ( I − 1 n 1 (cid:62) n /n )
X (cid:62) , Σ AE is a diagonal matrix consists of eigenvalues of M and V AE =
[ v 1 , · · · , v n ] ∈ R r × r can be any orthonormal matrix.
In the noiseless case, the covariance matrix is ν 2 U (cid:63) U (cid:63) (cid:62) and autoencoder can perfectly recover the core features.
However, in noisy cases, the random noises sometimes perturb the core features, which make autoencoders fail to learn core features.
Such noisy cases are very common in real applications such as measurement errors and backgrounds in images such as grasses and sky.
Interestingly, we will later show that contrastive learning can better recover U (cid:63) despite the presence of large noise.
To provide rigorous analysis, we ﬁrst introduce the incoherent constant (Cand`es & Recht, 2009).
Deﬁnition 3.1 (Incoherent constant) .
We deﬁne the incoherence constant of U ∈
O d,r as
[Equation]
Intuitively, the incoherent constant measures the degree of the incoherence of the distribution of entries among different coordinates, or loosely speaking, the similarity between U and canonical basis { e i } di =1 .
For uncorrelated random noise, the covariance matrix is diagonal and its eigenspace is exactly spanned by the canonical basis { e i } di =
(if the diagonal entries in Σ are all different), which attains the maximum value of incoherent constant.
On the contrary, core features usually exhibit certain correlation structures and the corresponding eigenspace of the covariance matrix is expected to have a lower incoherent constant.
We then introduce a few assumptions where our theoretical results are built on.
Recall that in the spiked covariance model Eq.(2.5), x = U (cid:63) z
+ ξ , Cov( z ) =
ν 2 I r and Cov( ξ )
=
diag( σ 21 , · · · , σ 2 d ) .
Assumption 3.1 (Regular covariance condition) .
The condition number of covariance matrix Σ = diag( σ 21 , · · · , σ 2 d ) satisﬁes κ := σ 2(1) /σ 2( d ) < C, where σ 2( j ) represents the j -th largest number among σ 21 , · · · , σ 2 d and C > 0 is a universal constant.
Assumption 3.2 (Signal to noise ratio condition) .
Deﬁne the signal to noise ratio ρ := v/σ (1) , we assume ρ = Θ(1) , implying that the covariance of noise is of the same order as that of core features.
Assumption 3.3 (Incoherent condition) .
The incoherent constant of the core feature matrix U (cid:63) ∈ O d,r satisﬁes I ( U (cid:63) )
=
O ( r log d/d ) .
Remark 3.1 .
Assumption 3.1 implies that the variances of all dimensions are of the same order.
For Assumption 3.2, we focus on a large-noise regime where the noise may hurt the estimation signiﬁcantly.
Here we assume the ratio lies in a constant range, but our theory can easily adapt to the case where ρ has a decreasing order.
Assumption 3.3 implies a stronger correlation among coordinates of core features, which is the essential property to distinguish them from random noise.
Now we are ready to present our ﬁrst result, showing that the autoencoder is unable to recover the core features in the large-noise regime.
Theorem 3.1 (Recovery ability of autoencoder, lower bound) .
Consider the spiked covariance model Eq.(2.5), under Assumption 3.1-3.3 and n > d (cid:29) r , let W AE be the learned representation of autoencoder with singular value decomposition W AE
=
( U AE Σ AE V (cid:62) AE ) (cid:62) (as in Eq.(3.1)).
If we further assume { σ 2 i } di =1 are different from each other and σ 2(1) / ( σ 2( r ) − σ 2( r +1) )
<
C σ for some universal constant C σ .
Then there exist two universal constants C ρ > 0 , c ∈ (0 , 1) , such that when ρ < C ρ , we have
[Equation]
Remark 3.2 .
The additional assumptions { σ 2 i } di =1 are different from each other and σ 2(1) /
( σ 2( r ) − σ 2( r +1) )
<
C σ for some universal constant C σ are for technical consideration.
We need these conditions to guarantee the uniqueness of U AE .
As an extreme example, the top- r eigenspace of identity matrix can be any r -dimensional subspace and thus not unique.
To avoid discussing such arbitrariness of output, we make these assumptions to guarantee the separability of eigenspace.
Then we investigate the feature recovery ability of the self-supervised contrastive learning approach.
Theorem 3.2 (Recovery ability of contrastive learning, upper bound) .
Under the spiked covariance model Eq.(2.5), random masking augmentation in Deﬁnition 2.2, Assumption 3.1-3.3 and n > d (cid:29) r , let W CL be any solution that minimizes Eq.(2.3), and denote its singular value decomposition as
W CL =
( U CL Σ CL V (cid:62) CL ) (cid:62) , then we have
[Equation]
Remark 3.3 .
In Eq.(3.4), the ﬁrst term is due to the shift between the distributions of the augmented data and the original data.
Speciﬁcally, the random masking augmentation generates two views with disjoint non-zero coordinates and thus can mitigate the inﬂuence of random noise on the diagonal entries in the covariance matrix.
However, such augmentation slightly hurts the estimation of core features.
This bias, appearing as the ﬁrst term in Eq.(3.4), is measured by the incoherent constant
Eq.(3.2).
The second term corresponds to the estimation error of the population covariance matrix.
Theorem 3.1 and 3.2 characterize the difference of feature recovery ability between autoencoder and contrastive learning.
The autoencoder fails to recover most of the core features in the large-noise regime, since (cid:107) sin Θ( U, U (cid:63) ) (cid:107) F has a trivial upper bound √ r .
In contrast, with the help of data augmentation, the contrastive learning approach mitigates the corruption of random noise while preserving core features.
As n and d increase, it yields a consistent estimator of core features and further leads to better performance in the downstream tasks, as shown in the next section.
P ERFORMANCE ON THE D OWNSTREAM T ASK
In the previous section, we have seen that contrastive learning can recover the core feature effectively.
In practice, we are interested in using the learned features to downstream tasks.
He et al.
(2020) experimentally showed the overwhelming performance of linear classiﬁers trained on representations learned with contrastive learning against several supervised learning methods in downstream tasks.
Following the recent empirical studies, here we evaluate the downstream performance of simple predictors, which take a linear transformation of representation as an input.
Speciﬁcally, we consider the regression setting with a class of predictors δ W,w ( x ) =
w (cid:62)
Wx constructed upon the learned representations W = W CL and W AE respectively.
Given the observation ˇ x = U (cid:63) ˇ z + ˇ ξ independent of unsupervised data X , our prediction target is ˇ y generated from a linear model ˇ y = (cid:104) ˇ z, w (cid:63) (cid:105) /ν
+ ˇ (cid:15) , where ˇ z ∼ N (0 , ν 2 I r ) is the low-dimensional core feature, ν is the scale of ˇ z and ˇ (cid:15) is the error term independent of ˇ z with zero mean and ﬁnite variance.
w (cid:63) ∈ R r is a unit vector of coefﬁcients.
We can interpret this model as a principal component regression model (PCR) (Jolliffe, 1982) under standard error-in-variables settings 2 , where we assume that coefﬁcients lie in a low-dimensional subspace spanned by column vectors of U (cid:63) .
We either estimate or predict the signal based on observed samples contaminated by the measurement error ˇ ξ .
For details of PCR in error-in-variables settings, see, for example, ´Cevid et al.
Agarwal et al.
; Bing et al.
Now we state our result on the downstream performance in prediction task.
For any linear representation W ∈ R r × d , let R ( W ) :=
E X [inf w ∈ R r E ˇ y, ˇ x [ (cid:96) ( δ W,w )] be the risk of the best linear predictor, where (cid:96) ( δ ) =
(ˇ y − δ (ˇ x )) 2 .
Theorem 3.3 (Upper Bound for Downstream Excess Risk of W CL ) .
Suppose conditions in Theorem 3.2 are satisﬁed.
Then we have
[Equation]
This result shows that the price of estimating U (cid:63) by contrastive learning on a downstream prediction task can be made small in a case where the core feature lies in a relatively low-dimensional subspace, and the number of samples is relatively large compared to the ostensible dimension of data.
However, the downstream performance of autoencoders is not as good as contrastive learning.
We obtain the following lower bound for the downstream prediction risk with the autoencoder.
Theorem 3.4 (Lower Bound for Downstream Excess Risk of W AE ) .
Suppose the conditions in
Theorem 3.1 hold.
Assume r ≤ r c holds for some constant r c > 0 .
Additionally assume that ρ = Θ(1) is sufﬁciently small and n (
cid:29) d (cid:29) r .
Then,
[Equation]
where c (cid:48) > 0 is a constant independent of n and d .
Comparing theorems 3.3 and 3.4, we ﬁnd that even when r/d and (cid:112) d/n are small, the downstream performance of autoencoders is not satisfying and has a much larger downstream task error rate than that of contrastive learning.
We note that the constant lower bound of Theorem 3.4 can be obtained without the assumption r ≤ r c by assuming a slightly stronger conditions ρ = O (1 / log d ) and n (cid:29) dr .
We also illustrate this phenomenon via numerical simulation in Fig.1.
As predicted by Theorem 3.1 and 3.2, the downstream task risk of contrastive learning decreases as d increases (Fig. 1: Left ) and as n increases (Fig. 1: Center ) while that of autoencoder remains large when n and d increase.
Remark 3.4 .
A similar results of upper and lower bound hold for linear predictors δ ( x ) =
I { F ( w (cid:62) Wx ) > 1 / 2 } under binary classiﬁcation settings, where label is generated from a binary response model ˇ y | ˇ z
=
Ber ( F ( (cid:104)
ˇ z, w (cid:63) (cid:105) /ν )) with a known function F : R → [0 , 1] .
Our results imply that the learned representations by contrastive learning are also useful in downstream classiﬁcation tasks, compared to an autoencoder, thus supporting the empirical success of contrastive learning.
A detailed results and proofs are deferred to Appendix B.2.
T HE IMPACT OF LABELED DATA IN SUPERVISED CONTRASTIVE LEARNING
Recent works have explored adding label information to improve contrastive learning (Khosla et al., 2020).
Empirical results show that label information can signiﬁcantly improve the accuracy of the in-domain downstream tasks.
However, when domain shift (multiple sources) is considered, the label information hardly improves and even hurts transferability (Islam et al., 2021).
Motivated by those empirical observations, in this section, we aim to investigate the role of labeled data in contrastive learning and provide a theoretical foundation for such phenomena.
4. F EATURE MINING IN MULTI - CLASS CLASSIFICATION
We ﬁrst demonstrate the role of label in the single-sourced case in contrastive learning.
Suppose our samples are drawn from r + 1 different classes with probability p k for class k ∈
[ r + 1] , and (cid:80) r + k =1 p k
= 1 .
For each class, samples are generated from a class-speciﬁc Gaussian distribution:
[Equation]
To be consistent with the spiked covariance model Eq.(2.5), we assume (cid:107) µ k (cid:107) = √ rν, ∀ k ∈
[ r + 1] .
We further assume Σ k = diag( σ 2 1 ,k , · · · , σ 2 d,k ) , denote σ 2 (1) = max 1 ≤ i ≤ d, 1 ≤ j ≤ r +1 σ 2 i,j and assume (cid:80)
r +1 k =
p k µ k = 0 , where the last assumption is added to ensure identiﬁable since the classiﬁcation problem (4.1) is invariant under translation.
Denote Λ = (cid:80)
r
+1 k =
p k µ k µ (cid:62) k , we assume rank(Λ)
=
r and C 1 ν 2 < λ ( r ) (Λ) < λ (1) (Λ) <
C 2 ν 2 for two universal constants C 1 and C 2 .
We remark that this model is a labeled version of the spiked covariance model Eq.(2.5) since the core features and random noise are both sub-Gaussian.
We use r +1 classes to ensure that µ k ’s span an r -dimensional space, and denote its orthonormal basis as U (cid:63) .
Recall that our target is to recover U (cid:63) .
As introduced in Deﬁnition 2.3, Khosla et al.
(2020) proposed a novel approach named supervised contrastive learning, which allows us to discriminate instances across classes.
When we have both labeled data and unlabeled data, we can perform contrastive learning based on pairs that are generated separately for the two types of data.
Data Generation Process.
Formally, let us consider the case in which we draw n samples as unlabeled data X
=
[ x 1 , · · · , x n ] ∈ R d × n from the Gaussian mixture model Eq.(4.1) with p 1 = p 2 = · · · = p r +1 .
For the labeled data, we draw ( r + 1) m samples, i.e., m samples for each of the r +classes in the Gaussian mixture model, and denote them as ˆ X =
[ˆ x 1 , · · · , ˆ x ( r +1) m ] ∈ R d × ( r +1) m .
We discuss the above case for simplicity.
More general versions are considered in Theorem C.2 (in the appendix).
We study the following hybrid loss to illustrate how label information helps promote the performance over the self-supervised contrastive learning:
[Equation]
where α > 0 is the ratio between supervised loss and self-supervised contrastive loss.
We ﬁrst provide a high-level explanation of why label information help learn core features.
When the label information is unavailable, no matter how much (unlabeled) data we have, we can only take themselves (and their augmented views) as positive samples.
In such a scenario, performing augmentation leads to an unavoidable trade-off between estimation bias and accuracy.
However, if we have additional class information, we can contrast between data in the same class to extract more beneﬁcial features that help distinguish a particular class from others and therefore reduce the bias.
Theorem 4.1.
Suppose the labeled and unlabeled samples are generated as the process mentioned above.
If Assumption 3.1-3.3 hold, n > d (cid:29) r and let W CL be any solution that minimizes the supervised contrastive learning problem in Eq.(4.2), and denote its singular value decomposition as
W CL =
( U CL Σ CL V (cid:62) CL ) (cid:62) , then we have
[Equation]
Moreover, if we have m labeled data for each class and no unlabeled data, then
[Equation]
The ﬁrst bound in Theorem 4.1 demonstrates how the effect of labeled data changes against the ratio α in the hybrid loss in Eq.(4.2).
In addition, compared with Theorem 3.2, when we only have labeled data ( α → ∞ ), the second bound in Theorem 4.1 indicates that with labeled data being available, the supervised contrastive learning can yield consistent estimation as m → ∞ while the self-supervised contrastive learning consists of an irreducible bias term O ( r 3 / 2 log d/d ) .
At a high level, label information help gain accuracy by creating more positive samples for a single anchor and therefore extract more decisive features.
One should notice a caveat that when labeled data is extremely rare compared with unlabeled data, the estimation of supervised contrastive learning suffers from high variance.
In comparison, self-supervised contrastive learning, which can exploit a much larger number of samples, may outperform it.
4. I NFORMATION FILTERING IN MULTI - TASK TRANSFER L EARNING
Label information can tell us the beneﬁcial information for the downstream task, and learning with labeled data will ﬁlter out useless information and preserve the decisive parts of core features.
However, in transfer learning, the label information is sometimes found to hurt the performance of contrastive learning (Khosla et al., 2020; Islam et al., 2021).
In this section, we consider two regimes of transfer learning – tasks are insufﬁcient/abundant.
In both regimes, we provide theories to support the empirical observations and further demonstrate how to wisely combine the supervised and self-supervised contrastive learning to avoid those harms and reach better performance.
Speciﬁcally, we consider a transfer learning problem with regression settings.
Suppose we have T source tasks which share a common data generating model Eq.(2.5).
In order to study the case of transfer learning, the labels are generated in a different way, that is, for the t -th task, the labels are generated by y t = (cid:104) w t , z (cid:105) /ν, where w t ∈ R r is a unit vector and different across tasks.
To incorporate label information, we maximize the Hilbert-Schmidt Independence Criteria (HSIC)
(Gretton et al., 2005; Barshan et al., 2011), which has been widely used in literature (Song et al., 2007a;b;c; Barshan et al., 2011).
HSIC is deﬁned as HSIC ( X, y ; W ) =
X (cid:62) W (cid:62)
WXHyy (cid:62) H/ ( n − 1) 2 , where W ∈ R r × d is linear representation to be learned and H
=
I n − (1 /n )1 n 1 (cid:62) n is the centering matrix.
A detailed discussion about its background, motivation and its connec- tion to the mean squared loss is presented in Appendix A.3.
Suppose we have n unlabeled data
X =
[ x 1 , · · · , x n ] ∈ R d × n and m labeled data for each source task
ˆ X t =
[ˆ x t 1 , · · · , ˆ x tm ] , y t =
[ y t 1 , · · · , y tm ] , ∀ t = 1 , . .
, T where x i ’s
and ˆ x tj ’s are independently drawn from spiked covariance model Eq.
(2.5), we learn the linear representation via the joint optimization:
[Equation]
where α > 0 is a pre-speciﬁed ratio between the self-supervised contrastive loss and HSIC.
(A more general setting is considered in the appendix, see Section C.2 for details.)
We now present a theorem showing the recoverability of W by minimizing the hybrid loss function (4.3).
Theorem 4.2.
Suppose Assumption 3.1-3.3 hold for spiked covariance model Eq.(2.5) and n > d (cid:29) r , if we further assume that α > C for some constant C , T < r and w t ’s are orthogonal to each other, and let W CL be any solution that optimizes the problem in Eq.(4.3), and denote its singular value decomposition as W CL =
( U CL Σ CL V (cid:62) CL ) (cid:62) , then we have:
[Equation]
In Theorem 4.2, as α goes to inﬁnity (corresponding to the case where we only use the supervised loss), Eq.(4.4) is reduced to √ r − T + T 3 / 2 (cid:112) d/m , which is worse than the r 3 / 2 log d/d rate obtained by self-supervised contrastive learning (Theorem 3.2).
This implies that when the model focuses mainly on the supervised loss, the algorithm will extract the information only beneﬁcial for the source tasks and fail to estimate other parts of core features.
As a result, when the target task has a very different distribution, labeled data will bring extra bias and therefore hurt the transferability.
Additionally, one can minimize the right-hand side of Eq.(4.4) to obtain a sharper rate.
Speciﬁcally, we can choose an appropriate α such that the upper bound becomes (cid:112) r 2 ( r − T )
log d/d (when n, m → ∞ ), obtaining a smaller rate than that of the self-supervised contrastive learning.
These facts provide theoretical foundations for the recent empirical observations that smartly combining supervised and self-supervised contrastive learning achieves signiﬁcant improvement on transferability compared with performing each of them individually (Islam et al., 2021).
When the tasks are abundant enough then estimation via labeled data can recover core features completely.
Similar to Theorem 4.2, we have the following result.
Theorem 4.3.
Suppose Assumptions 3.1-3.3 hold for spiked covariance model Eq.(2.5) and n > d (cid:29) r , if we further assume that T > r and λ ( r ) ( (cid:80)
Ti =
1 w i w (cid:62) i ) >
c for some constant c > 0 , suppose W CL is the optimal solution of optimization problem eq.(4.3), and denote its singular value decomposition as W CL =
( U CL Σ CL V (cid:62) CL ) (cid:62) , then we have:
[Equation]
Similar to Theorem 4.2, Theorem 4.3 shows that in the case where tasks are abundant, as α goes to inﬁnity (corresponding to the case where we use the supervised loss only), Eq.(4.5) is reduced to
T (cid:112) rd/m .
This rate can be worse than the √ r 3 log d/d + (cid:112) rd/n rate obtained by self-supervised contrastive learning when m is small.
Recall that when the number of tasks is small, labeled data introduce extra bias term √ r − T (Theorem 4.2).
We note that when the tasks are abundant enough, the harm of labeled data is mainly due to the variance brought by the labeled data.
When m is sufﬁciently large, supervised learning on source tasks can yield consistent estimation of core features, whereas self-supervised contrastive learning can not.
We also illustrate the different behaviors of the two regimes via numerical simulations in Fig.
1 right panel.
Consistent with our theory, it is observed that when tasks are not abundant, the transfer performance exhibit a U -shaped curve, and the best result is achieved by choosing an appropriate α .
When tasks are abundant and labeled data are sufﬁcient, the error remains small when we take large α .
C ONCLUSION
In this work, we theoretically prove that contrastive learning, compared with autoencoders, can obtain a better low-rank representation under the spiked covariance model, which further leads to better performance in downstream tasks.
We also highlight the role of labeled data in supervised contrastive learning and multi-task transfer learning: labeled data can reduce the domain shift bias in contrastive learning, but it harms the learned representation in transfer learning.
To our knowledge, our result is the ﬁrst theoretical result to guarantee the success of contrastive learning by comparing it with existing representation learning methods.
However, in order to get tractable analysis, like many other theoretical works in representation learning (Du et al., 2020; Lee et al., 2020; Tripuraneni et al., 2021), our work starts with linear representations, which still provides important insights.
Extending the results to more complex models is an interesting direction of future work.
A B ACKGROUND A. D ISTANCE BETWEEN SUBSPACES
In this section
[Equation]
we will provide some basic properties of sin Θ distance between subspaces.
Recall that the deﬁnition is:
[Equation]
where U 1 , U 2 ∈
O d
[Equation]
r are two orthogonal matrices.
Similarly
[Equation]
we can also deﬁne:
[Equation]
We ﬁrst give two equivalent deﬁnition of this distance:
Proposition A
[Equation] [Equation]
Proof.
Write U =
[ U 1 , U 1 ⊥ ] ∈ O d,d we have :
[Equation]
then by deﬁnition of sin Θ distance we can obtain the desired equation.
Proposition A.2.
[Equation]
Proof.
Expand the right hand and use Proposition A.1 we have:
[Equation]
With Proposition A.1 and A.2, it’s easy to verify its properties to be a distance function.
Obviously, we have 0 ≤ (cid:107) sin Θ ( U 1 , U 2 ) (cid:107) F ≤ √ r
and (cid:107)
sin Θ ( U 1 , U 2 ) (cid:107) F = (cid:107) sin Θ ( U 2 , U 1 ) (cid:107) F by deﬁnition.
Moreover, we have the following results:
Lemma A.1 (Lemma 1 in Cai & Zhang (2018)) .
For any U, V ∈ O d,r ,
[Equation]
r
[Equation]
≤ √
[Equation]
and (cid:107)
[Equation]
Proposition A.3 (Identity of indiscernibles) .
[Equation]
Proof.
It’s a straightforward corollary by deﬁnition:
[Equation]
Proposition A.4 (Triangular inequality) .
[Equation]
Proof.
By the triangular inequality for Frobenius norm we have:
[Equation]
then apply Proposition A.2 to replace the Frobenius norm with sin Θ distance we can ﬁnish the proof
[Equation]
A. P RINCIPAL COMPONENT ANALYSIS AND AUTOENCODERS
Autoencoders are popular unsupervised learning methods to perform dimension reduction.
Its basic idea is to learn low dimensional representations for the original data while largely preserving its salient features.
To achieve this goal, autoencoders learn two functions: encoder f : R d → R r and decoder g : R r
→ R d .
While the encoder f compresses the high dimensional data to the low dimensional space, we try to recover the original data based on this low dimensional representation via the decoder g .
Formally, it can be formulated to be the following optimization problem:
[Equation]
Intuitively speaking, optimization problem (A.4) is trying to preserve the most essential features to recover the original data in the low dimensional representation.
In practice, a commonly used form is to optimize the empirical loss function and choose the loss function to be mean squared error, that is:
[Equation]
where f AE , g DE are usually two neural networks.
Now if we take f AE , g DE to be both linear transformations, i.e.,
[Equation]
where W AE ∈ R r × d , b AE ∈ R r , W DE ∈ R d × r , , b AE ∈ R d , and denote X =
[ x 1 , · · · , x n ] ∈ R d × n , the optimization problem (A.5) can be reduced to be:
[Equation]
It’s shown in Bourlard & Kamp (2004) that the bias term can be reduced by centeralize the data matrix X , that is, denote ¯ x = 1 n (cid:80) ni
=1 x i as the sample mean and X
=
X
− ¯ x 1 (cid:62) n as the centered sample matrix, then the optimization problem correspond to W AE and W DE can be transfromed to be:
[Equation]
By Theorem 2.4.8 in (Golub & Loan, 1996), the optimal solution is given by the singular value decomposition of matrix X 0 , i.e., the eigenspace of X ( I n − 1 n 1 n 1 (cid:62) n ) X (cid:62) , which is actually what
PCA does.
This fact indicates that PCA is actually a linear case of autoencoders, which is often known as undercomplete linear autoencoders .
(Bourlard & Kamp, 1988; Plaut, 2018; Fan et al., 2019)
A. H ILBERT -S CHMIDT I NDEPENDENT C RITERIA
In Gretton et al.
(2005), the Hilbert Schmidt Independent Criteria (HSIC) is proposed to measure the dependence between two random variables by computing the Hilbert-Schmidt norm of the crosscovariance operator associated with their Reproducing Kernel Hilbert Spaces (RKHSs).
Such measurement has been widely used as supervised loss function in feature selection(Song et al., 2007c), feature extraction(Song et al., 2007a), clustering(Song et al., 2007b) and supervised PCA (Barshan et al., 2011).
The basic idea behind HSIC is that two random variables, named X and Y , are independent if and only if any bounded continuous function of the two random variables are uncorrelated.
Let F be a separable RKHS containing all continuous bounded real-valued functions of x from X to R and
G be that of Y , likewise.
To each point x ∈ X , there corresponds an element φ ( x ) ∈ F such that
[Equation]
F = k ( x, x (cid:48) ) , where k : X × X
→ R is a unique positive deﬁnite kernel.
Likewise, deﬁne the kernel l ( · , · ) and feature map ψ for G .
Denote the joint measure of p x,y , then the empirical
HSIC is deﬁned to be:
Deﬁnition A.1 (Empirical HSIC (Gretton et al., 2005)) .
Let Z := { ( x 1 , y 1 ) , . . .
, ( x m , y m ) } ⊆
X × Y be a series of m independent observations drawn from p xy .
An estimator of HSIC, written
HSIC( Z, F , G ) , is given by
[Equation]
where H, K, L ∈
R m × m , K ij :
= k
( x i , x j ) , L ij :=
l ( y i , y j ) and H
=
I m − 1 m 1 m 1 (cid:62) m .
In our problems, we hope to maximize the dependency between learned features WX ∈ R r × n and label y ∈ R n via HSIC, equivalently maximize tr( KHLH ) where K is a kernel of WX (e.g.
X (cid:62) W (cid:62) WX ) and L is a kernel of y (e.g. yy (cid:62) ).
Then we obtain our supervised loss corresponding to parameter W :
[Equation]
A.3. C ONNECTION WITH MEAN SQUARED ERROR
In regression tasks, a more commonly used loss function is mean squared error:
[Equation]
where θ is model parameter.
In our contrastive learning framework, where we ﬁrst learn the representation via a linear transformation and then perform linear regression to learn a classiﬁer w ∈ R r with learned representation.
Assuming that both of X and y have been centered and we ignore the bias term, the model can be viewed as a two layer linear network:
[Equation]
Plug this formula back into mean squared error, we have:
[Equation]
Since the label is a scalar, the optimal W will be singular since we perform a 1-dimensional projection via w .
That is, the feature ﬁltering discussed in the main body and if we jointly optimize this loss with unsupervised contrastive loss, the optimal W should be full rank in general.
On the other hand, the classiﬁer w only involves in the supervised loss function and does not affect the contrastive loss, thus we can ﬁnd the optimal solution for w and W sequentially.
For any ﬁxed
W such that rank( WX ) =
d , which can be achieved via joint optimization, the optimal solution of w is:
[Equation]
And the optimal error is:
[Equation]
Ignoring the constant term y (cid:62) y and scalar, we can ﬁnd that the only difference between this loss function and HSIC (A.6) is the inverse matrix ( WXX (cid:62) W (cid:62) ) − 1 which can be viewed as normalization of W .
In our contrastive learning framework, this normalization can be achieved by the regularization term (cid:107) WW (cid:62) (cid:107) 2 F .
Thus we can use the HSIC to replace the standard regression error which helps us to avoid dealing with singularity and additional parameters in optimization.
B O MITTED PROOFS FOR S ECTIONB. P ROOFS FOR S ECTION 3.
In this section, we will prove Theorem 3.1 and 3.2 in Section 3.1. The restatement and proof of them can be found in B.1 and B.3. Before starting the proof, we ﬁrst provide a lemma to justify the order of I ( U (cid:63) ) in the Assumption 3.3.
Lemma B.1 (Expectation of incoherent constant over a uniform distribution) .
[Equation]
Before starting the proof, we give two technical lemmas to help the proof.
Lemma B.2 (Uniform distribution on the unit shpere (Marsaglia, 1972)) .
If x 1 , x 2 , · · · , x n i.i.d.
∼ N (0 , 1) , then ( x 1 / (cid:112)(cid:80) ni
=1 x 2 i , · · · , x n / (cid:112)(cid:80) ni
=1 x 2 i ) is uniformly distributed on the unit sphere
S d = { ( x 1 , · · · , x n ) ∈ R n : (cid:80) ni
=1 x 2 i = 1 } .
Lemma B.3.
If x 1 , x 2 , · · · , x n i.i.d.
∼ N (0 , 1) , then:
[Equation]
Proof.
Denote Y = max 1 ≤ i ≤ n x 2 i , then we have:
[Equation]
Note that the moment-generating function of chi square distribution with v degrees of freedom is:
[Equation]
Then combine this fact with equation B.1 we have:
[Equation]
which implies:
[Equation]
In particular, take t → 12 yields:
[Equation]
as desired.
Proof of Lemma B.1.
Denote the columns of U as U =
[ u 1 , · · · , u r ] ∈ O d,r , we have:
[Equation]
By Lemma B.2 we can transform this expectation on the uniform sphere distribution into normalized multivariate Gaussian variables:
[Equation]
where x 1 , x 2 , · · · , x d are i.i.d.
standard normal random variables.
Apply Chebyshev’s inequality we know that:
[Equation]
In particular, take (cid:15) =
1 we have:
[Equation]
Then take it back into equation B.2 and apply Lemma B.3 we obtain:
[Equation]
as desired.
Lemma B.1 demonstrates the expectation of incoherent constant over a uniform distribution on O d,r takes the order of O ( r d log d ) , thus in the main body we take the order to be the same as it.
Again we need to mention that the order can be chosen to be other functions that decrease to 0 when d → ∞ , and one can easily prove our results under more general assumptions.
Now, let’s start proving our main results.
In the mainbody and section A.2, we have shown that in the linear representation setting, the autoencoder can be deduced to PCA.
Here we brieﬂy review the results again, for the autoencoder with an encoder f AE : R d → R r and a decoder g DE : R r → R d , it can be formalized as solving the following optimization problem for samples { x i } ni =
[Equation]
In the linear representation setting, where f AE ( x ) = W AE x + b AE and g DE ( y )
=
W DE y + b DE , it has been shown that the optimal W AE is given by:
[Equation]
where U AE is top-r eigenspace of matrix M :
= X ( I − 1 n 1 (cid:62) n /n )
X (cid:62) , Σ AE is a diagonal matrix consists of eigenvalues of M and V AE =
[ v 1 , · · · , v n ] ∈ R r × r can be any orthonormal matrix.
Note that U AE is the top- r left eigenspace of the observed covariance matrix and U (cid:63) is that of core feature covariance matrix, and by Assumption 3.2 the observed covariance matrix is dominated by the covariance of random noise.
The Davis-Kahan theorem provides a technique to estimate the eigenspace distance via estimate the difference between target matrices.
We will adopt this technique to prove the lower bound of feature recovery ability of autoencoder in Theorem 3.1.
Theorem B.1 (Restatement of Theorem 3.1) . Consider the spiked covariance model Eq.(2.5), under Assumption 3.1-3.3 and n > d (cid:29) r , let W AE be the learned representation of autoencoder with singular value decomposition W AE =
( U AE Σ AE V (cid:62) AE ) (cid:62) (as in Eq.(3.1)).
If we further assume { σ 2 i } di =1 are different from each other and σ 2(1) / ( σ 2( r ) − σ 2( r +1) )
<
C σ for some universal constant
C σ .
Then there exist two universal constants C ρ > 0 , c ∈ (0 , 1) , such that when ρ < C ρ , we have
[Equation] Proof. Denote M = ν 2 U (cid:63) U (cid:63) (cid:62) to be the target matrix, x i = U (cid:63) z i + ξ i , i = 1 , 2 , · · · n to be the samples generated from model 2.5 and let X =
[ x 1 , · · · , x n ] ∈ R d × n , Z = [ z 1 , · · · , z n ] ∈ R r × n , E = [ ξ 1 , · · · , ξ n ] ∈ R d × n to be the corresponding matrices.
In addition, we write the column mean matrix ¯ X ∈ R n × d of a matrix X ∈ R n × d to be ¯ X = 1 n X 1 n 1 (cid:62) n , that is, each column of ¯ X is the column mean of X .
We denote the sum of variance σ 2 i as σ 2 sum =
(cid:80) di
=1 σ 2 i .
As shown in B.3, autoencoder ﬁnds the top-r eigenspace of the following matrix:
[Equation]
The rest of the proof is divided into three steps for the sake of presentation.
Step 1, bound the difference between ˆ M 1 and Σ .
In this step, we aim to show that the data recovery of autoencoder is dominated by the random noise term.
Note that Σ = Cov( ξ ) =
E ξξ (cid:62) , we just need to bound the norm of the following matrix:
[Equation]
and we will deal with these four terms separately.
For the ﬁrst term, note that E zz (cid:62) =
ν 2 I r , the ﬁrst term can then be divided into two terms
[Equation]
Then apply the concentration inequality of Wishart-type matrices (Lemma D.3) we have:
[Equation]
Plug it back into (B.6) we obtain the bound for the ﬁrst term:
[Equation]
For the second term, since Z and E are independent, we must have E U (cid:63) ZE (cid:62) = 0 , so apply Lemma D.2 twice we have:
[Equation]
For the third term, apply Lemma D.3 again yields:
[Equation]
For the last term, note that each columns of ¯ Z and ¯ E are the same, so we can rewrite is as:
[Equation]
where ¯ z = 1 n (cid:80) n i =1 z i and ¯ ξ = 1 n (cid:80) n i =1 ξ i .
Since z and ξ are independent zero mean sub-Gaussian random variables and Cov( z ) =
ν 2 I r , Cov( ξ ) =
Σ , we can conclude that:
[Equation]
To sum up, combine equations (B.7)(B.8)(B.9)(4) together we obtain the upper bound for the 2 norm expectation of matrix ˆ M − Σ :
[Equation]
Step 2, bound the sin Θ distance between eigenspaces.
As we have shown in step 1, the target matrix of autoencoder is close to the covariance matrix of random noise, i.e., Σ .
Note that Σ is assumed to be diagonal matrix with different elements, hence its eigenspace only consists of canonical basis e i .
Denote U Σ to be the top-r eigenspace of Σ and { e i } i ∈ C to be its corresponding basis vectors, apply the Davis-Kahan Theorem D.1 we can conclude that:
[Equation]
Step 3, obtain the ﬁnal result by triangular inequality.
By Assumption 3.3 we know that the distance between canonical basis and the eigenspace of core features can be large:
[Equation]
Then apply the triangular inequality of sin Θ distance (Proposition A.4)
we can obtain the lower bound of autoencoder.
[Equation]
By Assumption 3.
[Equation] it implies that when n and d is sufﬁcient large and ρ is sufﬁcient small (smaller than a given constant C ρ > 0 ) [Equation] there exist a universal constant c ∈ (0 , 1) such that:
[Equation]
On the other hand, recall that the optimization problem for self-supervised contrastive learning has been formulated to be:
[Equation]
|B Negi [Equation]   + λR ( W ) , (B.11) where f ( x, W ) =
Wx , R ( W ) =
(cid:107) WW (cid:62) (cid:107) 2 F / 2 .
To compare contrastive learning with autoencoder, we now derive the optimal solution of the optimization problem B.11.
Let’s start with the general result for self-supervised contrastive learning with augmented pairs generation (Deﬁnition 2.1), and then turn to the special case for random masking augmentation (Deﬁnition 2.2).
Theorem B.2.
For two ﬁxed augmentation function g 1 , g 2 : R d → R d , denote the augmented data matrices as X 1 =
[ g 1 ( x 1 ) , · · · , g 1 ( x n )] ∈ R d × n and X 2 =
[ g 2 ( x 1 ) , · · · , g 2 ( x n )] ∈ R d × n , when the augmented pairs are generated as in Deﬁnition 2.1, the optimal solution of contrastive learning problem (B.11) is given by:
[Equation]
where C > 0 is a positive constant, σ i is the i -th largest eigenvalue of the following matrix:
[Equation]
u i is the corresponding eigenvector and V =
[ v 1 , · · · , v n ] ∈ R r × r can be any orthonormal matrix.
Proof of Theorem B
[Equation] When augmented pairs generation 2.1 is applied [Equation] the contrastive loss can be written as:
[Equation]
Note that the last term only depends on X , and the ﬁrst term implies that when W CL is the optimal solution, λW CL W (cid:62) CL is the best rank- r approximation of [Equation]
) λ XHX (cid:62) .
Then apply Lemma D. we can conclude that W CL satisfy the desired conditions.
Theorem B.2 shows a general result for augmented pairs generation with any augmentation.
Specifically, if we apply the random masking augmentation 2.2, we can obtain a more precise result to characterize the optimal solution.
To formally state the result, we need additional notations: for any square matrix A ∈ R d × d , we denote D ( A ) to be A with all off-diagonal entries set to be zero and ∆( A ) =
A − D ( A ) to be A with all diagonal entries set to be zero.
Then we have the following corollary for random masking augmentation.
Corollary B.1.
Under the same conditions as in Theorem B.2, if we use random masking (Deﬁnition 2.2) as our augmentation function, then in expectation, the optimal solution of contrastive learning problem (B.11) is given by:
[Equation]
where C > 0 is a positive constant, σ i is the i -th largest eigenvalue of the following matrix:
[Equation]
u i is the corresponding eigenvector and V =
[ v 1 , · · · , v n ] ∈ R r × r can be any orthonormal matrix.
Proof of Corollary B.1.
Following the proof of Theorem B.2, now we only need to compute the expectation over the augmentation distribution deﬁned in 2.2:
[Equation]
Note that by the deﬁnition of random masking augmentation, we have X 1 = AX, X 2 =
( I − A ) X , which implies X 1 + X 2 = X .
On the other hand, X 1 and X 2 has disjoint nonzero dimensions, hence the matrix X 1 X (cid:62) 2 + X 2 X (cid:62) 1 only consists of off-diagonal entries and each of the off-diagonal entry, let’s say x ij , appears if and only if a i + a j = 1 .
Moreover, once it appears, we must have x ij equals to the ( i, j ) element of XX (cid:62) .
With this result, we can then compute the expectation in equation (B.14):
[Equation]
By similar argument as in the proof of Theorem B
[Equation] [Equation]
we can conclude that W CL satisfy the desired conditions
[Equation]
Remark B.1 .
Note that the two views generated by random masking augmentation have disjoint non-zero dimensions, hence contrasting such positive pairs yields correlation between different dimensions only.
That’s why the ﬁrst term in equation (B.13) appears to be ∆( XX (cid:62) ) where the diagonal entries are eliminated.
With Theorem B.2 and Corollary B.1 established, we can ﬁnd that the self-supervised contrastive learning equipped with augmented pairs generation and random masking augmentation can eliminate the effect of random noise on the diagonal entries of the observed covariance matrix.
Since
Cov( ξ ) =
Σ is a diagonal matrix, and by Assumption 3.3 we know that the diagonal entries
Cov( U (cid:63) z ) =
ν 2 U (cid:63) U (cid:63) (cid:62) only take a small proportion of the total Frobenius norm.
Thus contrasting augmented pairs will preserve the core features while eliminating most of the random noise, and give a more accurate estimation of core features.
To start the proof, we introduce a technical lemma ﬁrst.
Lemma B.4 (Lemma 4 in Zhang et al.
If M ∈ R p × p is any square matrix and ∆( M ) is the matrix M with diagonal entries set to 0 , then
[Equation]
Here, the factor ” 2 ” in the statement above cannot be improved.
Then we turn to prove Theorem 3.2. Theorem B.3 (Restatement of Theorem 3.2) . Under the spiked covariance model Eq.(2.5), random masking augmentation in Deﬁnition 2.2, Assumption 3.1-3.3 and n > d (cid:29) r , let W CL be any solution that minimizes Eq.(2.3), and denote its singular value decomposition as W CL = [Equation]
) (cid:62) , then we have
[Equation] Proof. The proof strategy is quite similar to that of Theorem 3.2 and we follow the notation deﬁned in the ﬁrst paragraph of that proof.
As we have shown in Corollary B.1, under our linear representation setting, the contrastive learning algorithm ﬁnds the top-r eigenspace of the following matrix:
[Equation]
To prove the theorem, ﬁrst we need to bound the difference between ˆ M 2 and M .
We aim to show that the contrastive learning algorithm is dominated by the core feature term.
Note that Σ =
E Uzz (cid:62) U (cid:62) , we just need to bound the norm of the following matrix:
[Equation]
and we will also deal with these ﬁve terms separately.
For the ﬁrst term, we can divide it into two parts:
[Equation]
[Equation]
Then apply Lemma B.4 and Lemma D.
we have:
[Equation]
Using the incoherent condition
I ( U ) =
O ( rd log d )
[Equation]
we know that:
[Equation]
Combine the two equations above together we obtain the bound for the ﬁrst term:
[Equation]
For the second term, apply equation (B.8) yields:
[Equation]
For the third term, apply equation (B.9) yields:
[Equation]
For the fourth term, apply equation (4) yields:
[Equation]
For the last term, by equations (B.7)( [Equation]
[Equation]
we know:
[Equation]
Thus we can conclude that:
[Equation]
To sum up
[Equation]
combine equations (B.18)(B.20)(B.21)(B.22)(B.23) together we obtain the upper bound for the 2 norm expectation of matrix ˆ M 2 − M :
[Equation]
With the upper bound for (cid:107) ˆ M 2 − M (cid:107)
[Equation]
simply apply Lemma D.1 we can obtain the desired bound for sin Θ distance:
[Equation]
Moreover, there exists an orthogonal matrix ˆ O ∈ O r × r depending on U CL such that:
[Equation]
which ﬁnishes the proof.
In the following
[Equation]
we will show that our results do not change if we applied the same augmentation (2.2) for autoencoders, which indicates that our comparison is fair.
As discussed in Section A.2, we can ignore the bias term in autoencoders for simplicity, which only serves as centralization of the data matrix.
In that case, we applied random augmentation t 1 ( x ) =
Ax and t 2 ( x ) =
( I − A )
x to the original data { x i } ni =1 , and the optimization problem can be formulated as follows:
[Equation]
Then, similar to Theorem B.2 for contrastive learning, we can also obtain an explicit solution for this optimization problem.
Theorem B.4.
The optimal solution of autoencoders with random masking augmentation (B.25) is given by:
[Equation]
where C > 0 is a positive constant, σ i is the i -th largest eigenvalue of the following matrix:
[Equation]
u i is the corresponding eigenvector and V =
[ v 1 , · · · , v n ] ∈ R r × r can be any orthonormal matrix.
Proof.
We ﬁrst derive the equivalent form for this objective function:
[Equation]
[Equation]
where M :=
AXX (cid:62) A (cid:62) +
( I − A ) XX (cid:62) ( I − A ) (cid:62) .
Note that A
= ( a 1 , · · · , a d ) and a i follows the
Bernoulli distribution, so we have:
[Equation]
Again, by Theorem 2.4.8 in (Golub & Loan, 1996), the optimal solution of B.25 is given by the eigenvalue decomposition of E A M = 12 ∆( XX (cid:62) )
+ D ( XX T ) , up to an orthogonal transformation, which ﬁnishes the proof.
With Theorem B.4 established, we can now derive the space distance for autoencoders with random masking augmentation.
Theorem B.5.
Consider the spiked covariance model Eq.(2.5), under Assumption 3.1-3.3 and n > d (cid:29) r , let W AE be the learned representation of augmented autoencoder with singular value decomposition W AE
=
( U AE Σ AE V (cid:62) AE ) (cid:62) (i.e., the optimal solution of optimization problem B.25).
If we further assume { σ 2 i } di =1 are different from each other and σ 2(1) / ( σ 2( r ) − σ 2( r +1) )
<
C σ for some universal constant C σ .
Then there exist two universal constants C ρ > 0 , c ∈ (0 , 1) , such that when ρ < C ρ , we have
[Equation]
Proof.
Step1, similar to the proof of Theorem B.1, we ﬁrst bound the difference between ˆ M :=
∆( XX (cid:62) ) + 2 D ( XX (cid:62) ) and Σ :=
Cov( ξξ (cid:62) ) .
Note that:
[Equation]
Since Σ is a diagonal matrix, then by Lemma B.4 we have:
[Equation]
Now, directly apply equation (B.7)(B.8)(B.9) we can obtain that:
[Equation]
Step 2, bound the sin Θ distance between eigenspaces.
As we have shown in step 1, the target matrix of autoencoder is close to the covariance matrix of random noise, i.e., Σ .
Note that Σ is assumed to be diagonal matrix with different elements, hence its eigenspace only consists of canonical basis e i .
Denote U Σ to be the top-r eigenspace of Σ and { e i } i ∈ C to be its corresponding basis vectors, apply the Davis-Kahan Theorem D.1 we can conclude that:
[Equation]
Step 3, obtain the ﬁnal result by triangular inequality.
By Assumption 3.3 we know that the distance between canonical basis and the eigenspace of core features can be large:
[Equation]
Then apply the triangular inequality of sin Θ distance (Proposition A.4)
we can obtain the lower bound of autoencoder.
[Equation]
By Assumption 3.2, it implies that when n and d is sufﬁcient large and ρ is sufﬁcient small (smaller than a given constant C ρ > 0 ), there exist a universal constant c ∈ (0 , 1) such that:
[Equation]
Compared with Theorem 3.1, we can ﬁnd that random masking augmentation makes no difference to autoencoders, which justiﬁes the fairness of our comparison.
B. P ROOFS FOR S ECTION 3.
In this section, we will provide the proof of Theorem 3.3 and 3.4 with both regression and classiﬁcation settings.
The statement and detailed proof can be found in Theorem B.6 and B.7.
Before going onto the proof, we clarify our models and assumptions.
Let W CL and W AE be the learned representations based on train data X ∈ R n × d .
We observe a new signal ˇ x = U (cid:63) ˇ z + ˇ ξ independent of X following the spiked covariance model 2.5.
For simplicity, assume ˇ z ∼ N (0 , ν 2 I r ) and ˇ ξ ⊥ ˇ z .
We consider the two major types of downstream tasks: classiﬁcation and regression.
For binary classiﬁcation task, we observe a new supervised sample ˇ y following the binary response model:
[Equation]
where F : R → [0 , 1] is a known monotone increasing function satisfying 1 − F ( u ) =
F ( − u ) for any u ∈ R .
Notice that our model B.33 includes logistic models (when F ( u )
=
1 / (1 + e − u ) ) and probit models (when F ( u ) =
Φ( u ) , where Φ is the cumulative distribution function of standard normal distribution.)
We can also interpret model B.33 as a shallow neural network model with width r for binary classiﬁcation.
For regression task, we observe a new supervised sample ˇ y following the linear regression model:
[Equation]
) where ˇ (cid:15) ∼ (0 , σ 2 (cid:15) ) is independent of ˇ z .
In classiﬁcation setting, we specify 0 - 1 loss, i.e., (cid:96) c
( δ ) (cid:44) I { ˇ y (cid:54) =
δ (ˇ x ) } for some predictor δ taking values in { 0 , 1 } .
For regression task, we employ squared error loss (cid:96) r ( δ ) (cid:44) (
ˇ y − δ (ˇ x )) 2 .
Based on some learned representation W , we consider a class of linear predictors, i.e., δ W,w (ˇ x ) (cid:44)
I { F ( w (cid:62) W ˇ x ) ≥ 1 / 2 } for classiﬁcation task and δ W,w (ˇ x ) (cid:44) w (cid:62) W ˇ x for regression task, where w ∈ R r is a weight vector w ∈ R r .
Note that the learned representation depends only on unsupervised samples X .
Let E D [ · ] and E E [ · ] the expectations with respect to ( X, Z ) and (ˇ y, ˇ x, ˇ z ) , respectively.
For notational simplicity, deﬁne the prediction risk of predictor δ for classiﬁcation and regression tasks as R c ( δ ) :
= E D [ (cid:96) c ( δ )] and R r ( δ ) :=
E D [ (cid:96) r ( δ )] , respectively.
Deﬁne Σ x :=
ν 2 U (cid:63) U (cid:63) (cid:62) + Σ .
Since any representation W can be decomposed into W = V Σ W U (cid:62) by singular value decomposition, where U ∈ O d,r , { δ W,w : w ∈ R r } = { δ U (cid:62) ,w : w ∈ R r } .
Thus we write δ U,w for δ W,w with a slight abuse of notation.
Our goal as stated above is to bound the prediction risk of predictors { δ W,w : w ∈ R r } constructed upon the learned representations W CL and W AE , i.e., the quantity inf w ∈ R r E E [ (cid:96)
[Equation]
δ W CL ,w )] and inf w ∈ R
[Equation]
E E [
[Equation]
AE ,w )] .
Now we state our result on the downstream task.
Theorem B.6 (Excess Risk for Downstream Task: Upper Bound) .
Suppose the conditions in Theorem 3.2 hold.
Then, for classiﬁcation task, we have
[Equation]
and for regression task,
[Equation]
We obtain the lower bound for the downstream predction risk with autoencoders.
Theorem B.7.
Suppose the conditions in Theorem 3.1 hold.
Suppose r ≤ r c for some constant r c > 0 . Additionally assume that ρ = Θ(1) is sufﬁciently small and n ( cid:29) d (cid:29) r .
For classiﬁcation task, assume F is differentiable at 0 and F (cid:48) (0) > 0 . Then,
[Equation]
For regression task,
[Equation]
For two matrices A and B of the same order, we deﬁne A (cid:23) B when A − B is positive semi-deﬁnite. The proofs of Theorem B.6 and B.7 relies on Lemma B.8, B.9, B.10, B.11 and B.12 which are proved later in this section. Proof of Theorem B.6: Classiﬁcation Task Part.
Lemma B.10 gives for any U ∈
O d,r ,
[Equation]
Substituting U ← U AE combined with Assumption 3.2 and κ = O (1) concludes the proof. Proof of Theorem B.6: Regression Part. Note that under Assumption 3.2 and κ = O (1) , (1 + ρ − 2 ) / [Equation]
{+1 + κ − 1 ρ − 2 ) 
+}
= O (1) .
Lemma B.12 gives for any U ∈
O d,r ,
[Equation]
Theorem 3.2 with substitution U ← U AE gives the desired result.
Proof of Theorem B.7: Classiﬁcation Part.
Lemma B.9 gives that for c 1 :=
1 − 1 / (2 κr c ) ∈ (0 , 1) , we can take n (cid:29) d (cid:29) r and sufﬁciently small ρ > 0 so that E D [ (cid:107) sin
Θ( U AE , U (cid:63) )
(cid:107) 2 F ] ≥ c
1 r holds.
By Lemma B.11,
[Equation]
where the last inequality follows since r ≤ r c .
If we further take ρ = Θ(1) < 1 / 2 , the right hand becomes a positive constant.
This concludes the proof.
Proof of Theorem B.7: Regression Part.
From proposition B.1, we have
[Equation]
Thus from Lemma B
[Equation]
[Equation]
Using Lemma B.9 and by the same argument in the proof of Theorem B
[Equation]
7: Classiﬁcation Part, we conclude the proof.
Lemma B.5.
For any U ∈ O d,r ,
[Equation]
Proof.
Since λ min ( AC ) ≥ λ min ( A ) λ min ( C ) for symmetric positive semi-deﬁnite matrices A and
C ,
[Equation]
[Equation]
where we used Weyl’s inequality λ min ( A + C ) ≥ λ min ( A ) − (cid:107) C (cid:107) 2 in the second inequality.
Lemma B.6.
For any U ∈ O d,r ,
[Equation]
Proof.
Since
[Equation]
where we used Weyl’s inequality λ min ( A + C ) ≥ λ min ( A ) − (cid:107) C (cid:107) 2 and λ min ( ν 2 I + U (cid:62) Σ U ) ≥ ν 2 + σ 2( d ) .
Lemma B.7.
For any U ∈ O d,r ,
[Equation]
Proof.
Observe that
[Equation]
[Table]
[Equation]
[Table]
:=
( T 1)
+ ( T 2) .
For the term ( T 1) ,
[Table]
[Equation]
[Table]
[Equation]
[Table]
Note
[Table]
[Equation]
[Equation]
Also we have λ min ( U (cid:62) Σ x U ) ≥ ν 2 (1 −(cid:107) sin Θ( U, U (cid:63) )
(cid:107) 22 )+ σ 2( d ) from the proof of Lemma B.6 and λ min ( U (cid:63) (cid:62) Σ x U (cid:63) ) ≥
ν 2 + σ 2( d ) .
Therefore ( T 1) ≤
[Equation]
For the term ( T 2) ,
[Equation]
From Lemma A.1, (cid:107) sin Θ( U, U (cid:63) )
(cid:107) 2 ≤ (cid:107) U − U (cid:63) (cid:107) 2 .
Finally from these results and (cid:107) U − U (cid:63) (cid:107) 2 2 ≤ [Equation] − [Equation]
(cid:107) 2 ,
[Equation]
Since LHS does not depend on the orthogonal transformation U ← UO where O ∈ O r,r , we obtain
[Equation]
Combined again with Lemma A.1, we obtain the desired result.
Lemma B.8.
For any U ∈ O d,r ,
[Equation]
Proof.
Observe
[Equation]
U [Equation] U + (1 /ν 2 ) U (cid:62) Σ U ) − 1 U [Equation]
U (cid:63) (cid:107) 2 . Since U (cid:62) U (cid:63) U (cid:63) (cid:62) U (cid:23) 0 , it follows that ( U (cid:62) U (cid:63) U (cid:63) (cid:62) U
+ (1 /ν 2 )
U
(cid:62) Σ U ) − 1 (cid:22) ν
( U (cid:62) Σ U ) − 1 .
Thus (cid:107)
[Equation]
U
[Equation]
U [Equation] where we used λ max (( U (cid:62) Σ U ) − 1 ) ≤ 1 /λ min ( U (cid:62) Σ U ) ≤ 1 /σ 2( d ) and (cid:107) sin Θ ( U 1 , U 2 ) (cid:107) 2 F = r − (cid:13)(cid:13) U (cid:62) 1 U 2 (cid:13)(cid:13) 2 F from Proposition A.1.
Combined with Lemma B.6, we obtain
[Equation]
Lemma B.9.
Suppose the conditions in Theorem 3.1 hold.
Fix c 1 ∈ (0 , 1) . There exists a constant c 2 > 0 such that if (cid:112) r log d/d ∨ ρ 2 ∨ d/n < c 2 , then
E D
[Equation]
[Equation]
, where c 1 ∈ (0 , 1) is a universal constant.
Proof.
By Cauchy-Schwartz inequality,
[Equation]
From Theorem 3.1, there exists a constant c 3 > 0 such that
[Equation]
Therefore combined with a trivial bound (cid:107)
[Equation]
where we used ρ (cid:112) d/n ≤ ρ 2 ∨ d/n ≤ ρ 2 ∨ (cid:112) d/n since d < n .
Thus we can take c 2 = 6(1 − c 1 ) /c 3 .
This concludes the proof.
Lemma B.10.
For any U ∈ O d,r ,
[Equation]
Proof.
Recall that we are considering the class of linear classiﬁers { δ U,w : w ∈ R r } , where δ U,w (ˇ x )
=
I { F (ˇ x (cid:62) Uw ) > 1 / 2 } .
For notational simplicity, write β :=
Uw and β (cid:63) :
= U (cid:63) w (cid:63) .
[Equation]
P E (ˇ y = 1 , F (ˇ x (cid:62) β ) ≤ 1 / 2) .
Since F (0) = 1 / 2 and F is monotone increasing, the false positive probability becomes
[Equation]
Write ω :
= ˇ x (cid:62) β and ω (cid:63) := ν − 1 ˇ z (cid:62) U (cid:63) (cid:62) β (cid:63) .
From assumption, ( ω (cid:63) , ω ) jointly follows a normal distribution with mean 0 .
Write v (cid:63) 2 :=
Var( ω (cid:63) )
=
w (cid:63) (cid:62) w (cid:63) , v 2 :=
Var( ω ) =
β (cid:62) Σ x β , where Σ x := ν 2 U (cid:63) U (cid:63) (cid:62) + Σ .
Let τ :=
Cor ( ω (cid:63) , ω ) = νw (cid:63) (cid:62) U (cid:63) (cid:62)
β/ ( v (cid:63) v ) .
By a formula for conditional normal distribution, we have ω | ω (cid:63) ∼ N ( τvω (cid:63) /v
(cid:63) , v 2 (1 − τ 2 )) .
This gives
P E (ˇ y = 0 , F (ˇ x (cid:62) β ) > 1 / 2)
[Equation]
[Equation]
I {
[Equation]
> 0 } ]
[Equation]
[Equation] [Equation]
<
[Equation]
[Equation]
where Φ is cumulative distribution function of N (0 , 1) and
α := τ/ (1 − τ 2 ) 1 / 2 .
We deﬁne Ψ F as Ψ F ( s 2 ) :
= 2 E u ∼ N (0 ,s 2 )
[ F ( u ) I { u > 0 } ] .
When F ( u ) = 1 / (1+ e − u ) , Ψ F ( s 2 ) is called the logisticnormal integral, whose analytical form is not known (Pirjol (2013)).
Since a random variable ω (cid:63) is symmetric about mean 0 and F ( u ) = 1 − F ( − u ) ,
[Equation]
( − ω (cid:63) ))(1 − Φ( αω (cid:63) /v (cid:63) ))
I { ω (cid:63) > 0 } ] =
E E [ F ( ω (cid:63) )(1 − Φ( αω (cid:63) /v (cid:63) ))
I { ω (cid:63) > 0 } ] .
Hence
P E (ˇ y = 0 , F (ˇ x (cid:62) β ) > 1 / 2) =
[Equation]
] =
[Equation]
Note that the true negative probability is exactly the same as the false positive probability under our settings:
[Equation]
Therefore
[Equation]
Let
[Equation]
From Cauchy-Schwartz inequality,
[Equation]
[Equation]
Deﬁne α max ,U := τ max ,U / (1 − τ 2max ,U ) 1 / 2 and α max ,U (cid:63) :
= τ max ,U (cid:63) / (1 − τ 2max ,U (cid:63) ) 1 / 2 .
Then, since on the event where ω (cid:63) > 0 , α (cid:55)→ Φ( αω (cid:63) /v (cid:63) ) is monotone increasing and 2 F ( w (cid:63) ) − 1 is non-negative, we have
[Equation]
This yields
[Equation]
Note that for any a, b ≥ 0 ,
[Equation]
where φ is a density function of standard normal distribution.
Observe
[Equation]
where we used sup u> 0 uφ ( u ) < ∞ .
Since ( a − b ) =
( a 2 − b 2 ) /
( a + b ) ≤ ( a 2 − b 2 ) /
( a ∧ b ) for a, b >
[Equation]
and Ψ F ≤
[Equation]
we obtain
[Equation]
When τ max ,U (cid:63) ≥ τ max ,U , since τ (cid:55)→ τ 2 / (1 − τ 2 ) is increasing in τ > 0 ,
[Equation]
From Lemma B.5 and B.
[Equation]
we have
[Equation]
Then, equation B.38 becomes
[Equation]
where the last inequality follows from Lemma B.7.
On the event where (cid:107) sin Θ
( U, U (cid:63) )
(cid:107) 22 ≤ 1 / 2 ,
[Equation]
When τ max ,U (cid:63) < τ max ,U , on the event where (cid:107) sin Θ
( U, U (cid:63) )
(cid:107) 2 ≤
[Equation]
2 , inf
[Equation]
inf
[Equation]
≤ (
[Equation]
where we used Lemma B.7 again.
In summary, on the event where (cid:107) sin Θ
( U, U
[Equation]
On the other hand, on the event where (cid:107) sin Θ
( U, U (cid:63) )
(cid:107) 2 > κ − 1 ρ − 2 / 2 ∧ 1 / 2 , we have a trivial inequality inf w ∈ R r R c ( δ U,w ) − inf w ∈ R r R c
( δ U (cid:63) ,w ) ≤ 1 .
This gives
[Equation]
where the last inequality follows from Markov’s inequality.
Lemma B.11.
Suppose U ∈ O d,r satisﬁes 1 / [Equation]
) ≥ 0 .
Then,
[Equation]
Proof.
We ﬁrstly bound the term τ 2 max ,U [Equation] − [Equation]
max ,
[Equation]
From Lemma B.8,
[Equation]
From assumption
[Equation]
RHS of equation B.40 is non-negative.
Then using the inequality a − b =
( a 2 − b 2 ) /
( a + b ) ≥ ( a 2 − b 2 ) / (2 a ) for a ≥ b ≥ 0 ,
[Equation]
From equation B.39 and equation B [Equation]
[Equation]
From the proof of Lemma B.10,
[Equation]
Note that for any b ≥ a ≥ 0 , Φ( b ) − Φ( a ) ≥ φ ( b )( b − a ) .
Since we assume RHS of equation B. is positive, α max ,U (cid:63) ≥ α max ,U .
Thus on the event where ω (cid:63) > 0 , α max ,U (cid:63) ω (cid:63) /v
(cid:63) ≥ α max ,U ω (cid:63) /v
(cid:63) .
Observe
[Equation]
[Table]
where in the last equality we transformed w (cid:63) →
(1 + α 2max ,U (cid:63) ) 1 / 2 w (cid:63) /v (cid:63) .
Since F ( u ) is differentiable at 0 and F (0) = 1 / 2 ,
F ( u ) − 1 / 2 = F (cid:48) (0) u + o ( u ) .
Thus there exists a constant (cid:15) > 0 only depending on F such that 2( F ( u ) − 1 / 2) ≥ F (cid:48) (0) u for all u ∈ [0 , (cid:15) ] since F (cid:48) (0) > 0 .
This gives
[Equation]
The last inequality follows since v (cid:63)
=
(cid:107) w (cid:63) (cid:107) = 1 by assumption.
It is noted that α 2max ,U (cid:63) ≤ ν 2 /σ 2( d ) from equation B.39.
Therefore with equation B.41,
[Equation]
Proposition B.1.
For any U ∈ O d,r ,
[Equation]
Proof of Proposition B.1.
Generate random variables (ˇ x, ˇ z, ˇ ξ, ˇ (cid:15) ) following the model equation B.34.
We calculate the prediction risk of δ U,w as:
[Equation]
where A := ν 2 U (cid:62) U (cid:63) U (cid:63) (cid:62) U + U (cid:62) Σ U and b := νU (cid:62) U (cid:63) w (cid:63) .
From this, we obtain inf w ∈ R r R r
( δ U,w ) =
w (cid:63) (cid:62) (cid:0) I − U (cid:63) (cid:62) U ( U (cid:62) U (cid:63) U (cid:63) (cid:62) U
+ (1 /ν 2 ) U
(cid:62) Σ U ) − 1 U (cid:62) U (cid:63) (cid:1) w (cid:63) + σ 2 (cid:15) .
Lemma B.12.
For any U ∈ O d,r ,
[Equation]
Proof of Lemma B.12.
From proposition B [Equation] 1, we have
[Equation]
Note that [Table] R
[Table]
for any [Table] O ∈ [Table]
[Table]
Take [Table] [Table]
such [Table] Θ( [Table]
[Table]
without loss of generality, since we can always take a sequence ( ˜ O m ) m [Table] 1 such that (cid:107) UO [Table]
U
[Table]
√ 2 (cid:107) sin Θ
[Table]
[Table]
2 +
[Table]
/m from Lemma A.1.
Lemma B.7 gives
[Equation]
On the event where (cid:107) sin Θ
( U, U (cid:63) )
(cid:107) 22 <
[Equation]
2 , inf
[Equation]
[Table]
(cid:19) . On the event where (cid:107) sin Θ
( U, U (cid:63) )
(cid:107) 22 ≥ 1 / 2 , we utlize the trivial upper bound
[Equation]
Combining these results
[Equation]
[-Note that under Assumption 3.2 and κ = O (1) , (1 + ρ − 2 ) / (1 + κ − 1 ρ − 2 ) 
-]
we have
[Equation]
where the last inequality follows from Markov’s inequality
[Equation]
C O MITTED PROOFS FOR S ECTIONC. P ROOFS FOR S ECTION 4.
In this section, we will provide the proof of a generalized version of Theorem 4.1 to cover the imbalanced setting, the statement and the detailed proof can be found in Theorem C.2.
In the main body, we assume the unlabeled data and labeled data are both balanced for the sake of clarity and simplicity.
Now we allow them to be imbalanced and provide a more general analysis.
Suppose we have n unlabeled data X =
[ x 1 , · · · , x n ] ∈ R d × n and n k labeled data X k =
[ x 1 k , · · · , x n k k ] ∈
R d × n k for class k , the contrastive learning task can be formulated as:
[Equation]
In addition, we write a generalized version of supervised contrastive loss function to cover the imbalanced cases:
[Equation]
where α k > 0 is the weight for supervised loss of class k .
Again we ﬁrst provide a theorem to give the optimal solution of contrastive learning problem.
Theorem C.1.
The optimal solution of supervised contrastive learning problem (C.1) is given by :
[Equation]
where C > 0 is a positive constant, σ i is the i -th largest eigenvalue of the following matrix:
[Equation]
u i is the corresponding eigenvector and V =
[ v 1 , · · · , v n ] ∈ R r × r can be any orthonormal matrix
[Equation]
Proof.
Under this setting [Equation] combine with the result obtained in Corollary B
[Equation]
1, the contrastive loss can be rewritten as:
[Equation]
Then we deal with the last term independently [Equation] note that:
[Equation]
Thus we have:
[Equation]
Then by similar argument as in the proof of Theorem B.2, we can conclude that the optimal solution
W CL must satisfy the desired conditions.
With optimal solution obtained in Theorem C.1, we can provide a generalized version of Theorem 4.1 to cover the imbalance cases.
Theorem C.2 (Generalized version of Theorem 4.1) .
If Assumption 3.1-3.3 hold, n > d (cid:29) r and let W CL be any solution that minimizes the supervised contrastive learning problem in Eq.(C.1), and denote its singular value decomposition as W CL =
( U CL Σ CL V (cid:62) CL ) (cid:62) , then we have
[Equation]
where T (cid:44) 14 (cid:80) r +1 k =1 p i µ k µ (cid:62) k + 1 r +1 (cid:80) r +1 k =1 α k ( µ k µ
[Equation]
k −
[Equation] s (cid:54) = k n
[Equation] =
k
[Equation]
[Equation]
k )) .
Proof.
For labeled data X
=
[ x 1 , · · · , x n ] , we write it to be X
=
M + E , where M = [ µ 1 , · · · , µ n ] and E = [ ξ 1 , · · · , ξ n ] are two matrices consisting of class mean and random noise.
To be more speciﬁc, if x i subject to the k -th cluster, then µ i = µ k and ξ i ∼ N (0 , Σ k ) .
Since the data is randomly drawn from each class, µ i follows the multinomial distribution over µ 1 , · · · , µ r with probability p 1 , · · · , p r +1 .
Thus µ i follows a subgaussian distribution with covariance matrix
[Equation]
As shown in Theorem C.1, the optimal solution of contrastive learning is equivalent to PCA of the following matrix:
[Equation]
Again we will deal with these terms separately, 1.
For the ﬁrst term, as we have discussed, X can be divided into two matrices M and E , each of them consists of subgaussian columns.
Again we can obtain the result as in (B.24) (the proof is totally same):
[Equation]
For the second term, notice that:
[Equation]
and that:
[Equation]
n s
[Equation]
ki
[Equation]
sTj ] .
(C.5)
Since ξ ki ∼ N (0 , Σ k )
[Equation]
we can conclude that:
[Equation]
Moreover, we have
[Equation]
Take equation (C.6) and (C.7) back into (C.4) we can conclude:
[Equation]
On the other hand [Equation] by equation (C.6) we know:
[Equation]
Notice that:
[Equation]
Thus take equations (C.9) and (C.10) back into equation (C.5) we have:
[Equation]
Then combine equations (C.3)(C.8)(C.11) together, we can obtain the following result:
[Equation]
Since we have assumed that rank( (cid:80) r +1 k =1 p k µ k µ (cid:62) k ) =
r we can ﬁnd that the top-r eigenspace of matrix:
[Equation]
  is spanned by U (cid:63) , then apply Lemma D.1 again we have:
[Equation]
Roughly speaking, since (cid:107) µ k (cid:107) = √ rν
and (cid:80)
r
+1 k =
p k µ k = 0 , approximately we have ν 2 λ r ( N ) ≈min
k ∈ [ r ] [1+ α k ] .
Although we can not obtain the closed-form eigenvalue in general, in a special case, where α = α 1 = · · · = α r +1 , m = n 1 = n 2 = · · · = n r +1 and 1 r +
=
p 1 = p 2 = · · · = p r +1 , it’s easy to ﬁnd that:
[Equation]
which further implies that:
[Equation]
and we can obtain the result in Theorem 4.1.
C. P ROOFS FOR S ECTION 4.
In this section, we will provide the proof of generalized version of Theorem 4.2 and 4.3 to cover the imbalanced setting, the statement and detailed proof can be found in Theorem C.4 and C.5.
First we prove a useful lemma to illustrate that supervised loss function only yields estimation along a 1-dimensional space.
Consider a single source task, where the data x = U (cid:63) z + ξ is generated by spiked covariance model and the label is generated by
[Equation]
suppose we have collect n labeled data from this task, denote the data as X
=
[ x 1 , x 2 , · · · , x n ] ∈ R d × n and the label y =
[ y 1 , y 2 , · · · , y n ] ∈ R n , then we have the following result.
Lemma C.1.
Under the conditions similar to Theorem 3.2, we can ﬁnd an event A such that
P ( A C ) =
O ( (cid:112) d/n ) and:
[Equation]
The proof strategy is to estimate the difference between the two rank-1 matrices via bounding the difference of the corresponding the vector component.
We ﬁrst provide a simple lemma to illustrate the technique:
Lemma C.2.
Suppose α, β ∈ R d are two vectors, then we have:
[Equation]
Proof.
Denote α
=
( α 1 , · · · , α d ) , β = ( β 1 , · · · , β d ) , then we have:
[Equation]
Take square root on both side we can ﬁnish the proof.
Now we can prove the Lemma C.
[Equation]
Proof of Lemma C.1.
Clearly, we have:
[Equation]
thus we can replace the 1 ( n − 1) 2 with 1 n in equation (C.13) and conclude the proof.
Denote ˆ N (cid:44) 1 n 2 XHyy (cid:62) HX (cid:62) , note that both of ˆ N and Uw (cid:63) w (cid:63) (cid:62) U (cid:62) are rank-1 matrices.
We ﬁrst bound the difference between 1 n XHy and Uw (cid:63) :
[Equation]
We deal with the four terms in (C.14) separately:
For the ﬁrst term, apply Lemma D.
we have:
[Equation]
For the second term, apply Lemma D.2 twice we have
[Equation]
For the third term and fourth term, from equation (4) we know:
[Equation]
Combine these three equations (C.15)(C.16)(C.17) together we have:
[Equation]
With equation (C.18)
[Equation]
we can now turn to the difference between ˆ N and Uw (cid:63) w [Equation] (cid:62) U
[Equation]
By Lemma
C.2 we know that:
[Equation]
Using Markov’s inequality
[Equation]
we can conclude from (C.18) that:
[Equation]
Then denote A = { ω : (cid:107) 1 n XHy − ν 2 U (cid:63) w (cid:63) (cid:107) 2 < ν } we have:
[Equation]
which ﬁnished the proof.
In the main body, we assume the number of labeled data and the ratio of loss function is both balanced.
Now we will provide a more general result to cover the imbalance occasions.
Formally, suppose we have n unlabeled data X =
[ x 1 , · · · , x n ] ∈ R d × n and n i labeled data S
i
X i =
[ x 1 i , · · · , x n i i ] , y i =
[ y 1 i , · · · , y n 1 i ] , ∀ i = 1 , · · · T for source task , we learn the linear representation via joint optimization:
[Equation]
To investigate its feature recovery ability, we ﬁrst give the following result.
Theorem C.3.
For optimization problem C.19, if we apply augmented pairs generation 2.1 with random masking augmentation 2.2 for unlabeled data, then the optimal solution is given by:
[Equation]
where C > 0 is a constant, σ i is the i-largest eigenvalue of the following matrix:
[Equation]
u i is the corresponding eigenvector, V =
[ v 1 , · · · , v r ] ∈ R r × r can be any orthogonal matrix and
[Equation] n i is the centering matrix.
Proof.
Under this setting
[Equation] combine with the result obtained in B. [Equation] the loss function can be rewritten as:
[Equation]
F . Then by similar argument as in the proof of Theorem B.2, we can conclude that the optimal solution
W CL must satisfy the desired conditions.
Then we can give the proofs of Theorem 4.2 and Theorem 4.3 under our generalized setting, one can easily obtain those under balanced setting by simply setting α
= α 1 = · · · = α T and m = n 1 = · · · = n T , which is consistent with Theorem 4.2 and Theorem 4.3 in the mainbody.
Theorem C.4 (Generalized version of Theorem 4.2) .
Suppose Assumption 3.1-3.3 hold for spiked covariance model Eq.(2.5) and n > d (cid:29) r , if we further assume that T < r and w t ’s are orthogonal to each other, and let W CL be any solution that optimizes the problem in Eq.(C.19), and denote its singular value decomposition as W CL =
( U CL Σ CL V (cid:62) CL ) (cid:62) , then we have:
[Equation]
Proof.
As shown in Theorem C.3, optimizing loss function (C.19) is equivalent to ﬁnd the top- r eigenspace of matrix
[Equation]
Again denote ˆ M 2 (cid:44) 1 n (∆( XX (cid:62) ) − 1 n − 1 X
[Equation]
n 1 (cid:62) n − I n ) X (cid:62) ) and ˆ N i (cid:44)
[Equation]
By equation (B.24) we know that:
[Equation]
By Theorem C.1 we know that for each task S
i , we can ﬁnd an event
A i such that P ( A i ) =
O ( (cid:113) dn ) :
[Equation]
The target matrix is N = ν
[Equation]
w T i U
[Equation]
, and we can obtain the upper bound for the difference between N and ˆ N :
[Equation]
We divide the top-r eigenspace U CL of W CL W (cid:62) CL into two parts: the top- T eigenspace U (1) CL and top- ( T + 1) to top- r eigenspace U (2) CL .
Similarly, we also divide the top-r eigenspace U (cid:63) of N into two parts: U (cid:63) (1) and U (cid:63) (2) . Then apply Lemma D.1 we have we can bound the sin Θ distance for each parts: on the one hand,
[Equation]
) (cid:17) (cid:107) F I {∪ Ti =1 A Ci } ≤ √ T E (cid:107) ˆ N
[Equation] N (cid:107)
[Equation]
I {∩ Ti =1 A i } λ ( T ) ( N )
[Equation]
( T +1) ( N )
[Equation]
=1 A Ci ) (cid:46) √ T min
i ∈
[ T ] α i ν(cid:32)
ν 2 r d log d + σ 2(1) (cid:114) d n +
T (cid:88) i
= α
i (cid:114) d n i σ (1) ν (cid:33) + √ T T (cid:88) i = (cid:114) d n i (cid:46) √ T min i ∈
[ T ] α i (cid:32) r d log d + (cid:114) d n (cid:33) + √ T T (cid:88) i
= α i + min i ∈
[ T ] α i min i ∈
[ T ] α i (cid:114) d n i .
On the other hand,
E
[Equation]
[Equation]
F = E
[Equation]
[Equation]
F
[Equation]
∩
[Equation]
A
[Equation]
[Equation]
[Equation]
F
[Equation]
∪
[Equation]
=1 A Ci } ≤
[Equation]
} min
[Equation]
+
[Equation]
(cid:46) √ r − T min i ∈
[ T ] { α i , 1 } ν(cid:32) ν 2 r d log d + σ 2(1) (cid:114) d n +
T (cid:88) i
= α
i (cid:114) d n i σ (1) ν (cid:33) + √ r − T T (cid:88) i = (cid:114) d n i (cid:46) √ r − T min i ∈
[ T ] { α i , 1 } (cid:32) r d log d + (cid:114) d n (cid:33) + √ r − T T (cid:88)
i = (cid:18)
α i min i ∈
[ T ] { α i , 1 } +(cid:19)(cid:114) d n i .
Note that:
[Equation]
and the sin Θ distance has trivial upper bounds:
[Equation]
Thus we can conclude:
[Equation]
Theorem C.5 (Generalized version of Theorem 4.3) .
Suppose Assumptions 3.1-3.3 hold for spiked covariance model Eq [Equation] (2.5) and n > d (cid:29) r , if we further assume that T ≥ r and (cid:80) Ti =
α i w i w (cid:62) i is full rank, suppose W CL is the optimal solution of optimization problem eq.(C.19), and denote its singular value decomposition as W CL =
( U CL Σ CL V (cid:62) CL ) (cid:62) , then we have:
[Equation]
Proof.
The proof strategy is similar to that of Theorem 4.2, here the difference is that each direction can be accurately estimated by the labeled data and we don’t need to separate the eigenspace
[Equation]
[-λ ( r ) ( (cid:80) Ti =
 α i w i w (cid:62) i ) (cid:32) r [Equation]-]Directly applying Lemma D.1 and equation (C.20) we have:
[Equation]
D
U SEFUL LEMMAS
In this section, we list some of the main techniques that have been used in the proof of the main results.
Lemma D.1 (Theorem 2 in Yu et al.
Let Σ , ˆΣ ∈ R p
×
p be symmetric, with eigenvalues λ 1 ≥ . . .
≥ λ p and ˆ λ 1 ≥ . .
≥ ˆ λ p respectively.
Fix 1 ≤ r ≤ s ≤ p and assume that min ( λ r − 1 − λ r , λ s − λ s +1 ) > 0 where λ 0 := ∞ and λ p
+1 :=
−∞ .
Let d := s − r + 1 , and let
V = ( v r , v r +1 , . .
. , v s ) ∈ R p ×
d and ˆ V =
(ˆ v r , ˆ v r +1 , . . .
, ˆ v s ) ∈
R p × d have orthonormal columns satisfying Σ v j = λ
j v j and ˆΣˆ v j = ˆ λ j ˆ v j for j = r, r + 1 , . . s.
Then [Equation]
Moreover, there exists an orthogonal matrix ˆ O ∈ R d × d such that
[Equation]
Lemma D.2 (Lemma 2 in Zhang et al.
Assume that E ∈ R p 1 × p 2 has independent subGaussian entries, Var ( E ij ) =
[Equation]
ij ,
[Equation]
Assume that
[Equation]
Let V ∈
O p 2 ,r be a ﬁxed orthogonal matrix.
Then
[Equation]
Lemma D.3 (Theorem 6 in Cai et al.
Suppose Z is a p 1 -by- p 2 random matrix with independent mean-zero sub-Gaussian entries.
If there exist σ 1 , . .
σ p ≥ 0 such that (cid:107)
Z ij /σ
i (cid:107) ψ 2 ≤ C K for constant C K > 0 , then
[Equation]
Lemma D.4 (The Eckart-Young-Mirsky Theorem (Eckart & Young, 1936)) .
Suppose that A =
U Σ V T is the singular value decomposition of A .
Then the best rank- k approximation of the matrix
A w.r.t the Frobenius norm, (cid:107) · (cid:107) F , is given by
[Equation]
that is, for any matrix B of rank at most k
[Equation]
E N UMERICAL EXPERIMENTS E. S UPPORTING EMPIRICAL RESULTS IN RELATED WORKS
In this section, we list some recent empirical evidence that provides sound support to our theory: Contrastive learning outperforms generative self-supervised learning. In Figure 1 of Chen et al. (2020a) and Figure 5 of Liu et al.
(2021), it’s observed that contrastive learning has superior performance compared to the generative approach. These results are consistent with our theory in Theorem 3.1-3.4, where we show that contrastive learning achieves better performance on both feature recovery and downstream tasks compared with autoencoder, a representative generative selfsupervised learning method. Supervised contrastive learning improves downstream accuracy. In Table 2 of Khosla et al.
(2020) and the ﬁrst column in Table 4 of Islam et al.
(2021), supervised contrastive learning shows signiﬁcant improvement with 7%-8% accuracy increase on ImageNet and Mini-ImageNet.
This observation is consistent with our ﬁnding in Theorem 4.1, where we prove that supervised contrastive learning can achieve a better upper bound in feature recovery compared with self-supervised contrastive learning.
Label information may hurt transferability in contrastive learning.
In Table 4 of Khosla et al.
(2020) and
Table 4 of Islam et al.
(2021), where the supervised contrastive learning hardly increases the predictive accuracy compared to the self-supervised contrastive learning (the difference of mean accuracy is less than 1%) and can harm signiﬁcantly on some datasets (e.g. 5.5% lower for SUN 397 in Table 3 of Khosla et al.
These results indicate that some mechanisms in supervised contrastive learning hurt model transferability since the improvement on source tasks is signiﬁcant.
Moreover, in Table 4 of Islam et al. (2021), it is observed that combining supervised learning and self-supervised contrastive learning together achieves the best transfer learning performance compared with each of them individually.
These ﬁndings are consistent with our theory in Theorem 4.2, where we show that the error can increase as α (the ratio between supervised learning loss and self-supervised learning loss) grows and the optimal error is achieved when choosing a moderate α , i.e., combining self-supervised learning and supervised learning together.
E. S IMULATION WITH SYNTHETIC DATA
To verify our theory, we conducted numerical experiments on the spiked covariance model (2.5) under a linear representation setting.
As we have explicitly formulated the loss function and derive its equivalent form in the main body and appendix, we simply minimize the corresponding loss by gradient descent to ﬁnd the optimal linear representation W .
For self-supervised contrastive learning with random masking augmentation, we independently draw the augmentation function by
Deﬁnition 2.2 and apply them to all of the samples in each iteration.
To ensure the convergence, we set the maximum number of iteration for it (typically 10000 or 50000 depends on dimension d ).
We report two criteria to evaluate the quality of the representation, downstream error and sine distance.
To obtain the sine distance for a learned representation W , we perform singular value decomposition to get W
=
( U Σ V (cid:62) )
(cid:62) and then compute (cid:107) sin Θ( U, U (cid:63) )
(cid:107) F .
To obtain the downstream task performance, in the comparison between autoencoder and contrastive learning, we ﬁrst draw n labeled data from spiked covariance model (2.5) with labels generated as in Section 3.2, then we train the model by using the data without labels to obtain the linear representation W , and learn a linear predictor w using the data with labels and compute the regression error.
In the transfer learning setting, we draw some labeled data on the source tasks and additional unlabeled data.
The number of labeled data is set m = 1000 and the number of unlabeled data is set n = 1000 .
Then train with them to obtain the linear representation W , and draw labeled data from a new source task to learn a linear predictor w to compute the regression error.
In particular, we subtract the optimal regression error obtained by the best representation U (cid:63) (cid:62) for each regression error and report the difference, or more precisely, the excess risk as downstream performance.
The results are reported in Fig.
1, 2 and Table 1, 2.
As predicted by Theorem 3.1 and 3.2, the feature recovery error and downstream task risk of contrastive learning decreases as d increases (Fig. 1:
Left ) and as n increases (Fig. 1: Center ) while that of autoencoder is insensible to the changes in d and n .
The performance of transfer learning exhibits a U-shape type curve when the number of tasks is insufﬁcient, which implies that the supervised training may hurt the transferability and we need to choose an appropriate ratio α to obtain the best performance.
When tasks are abundant enough, the performance of transfer learning becomes better as we increase the weight of supervised loss.
[Table] [Table]