Learning Contrastive Embedding in Low-Dimensional Space
Anonymous Author(s)
Abstract
Contrastive learning (CL) pretrains feature embeddings to scatter instances in thefeature space so that the training data can be well discriminated.
Most existing
CL techniques usually encourage learning such feature embeddings in the highdimensional space to maximize the instance discrimination.
However, this practicemay result in the curse of dimensionality where the scattering instances are sparselydistributed in the high-dimensional feature space, making it difficult to capturethe underlying similarity between pairwise instances.
To this end, we propose anovel framework called contrastive learning with low-dimensional reconstruction(CLLR), which adopts a regularized projection layer to reduce the dimensionalityof the feature embedding.
In CLLR, we build the sparse / low-rank regularizer toadaptively reconstruct a low-dimensional projection space while preserving the basic objective for instance discrimination, and thus successfully learning contrastiveembeddings that alleviate the curse of dimensionality.
Theoretically, we prove atighter error bound for CLLR; empirically, the superiority of CLLR is demonstratedacross multiple domains, i.e. , image classification, sentence representation, andreinforcement learning.
Both theoretical and experimental results emphasize thesignificance of learning low-dimensional contrastive embeddings.
Introduction
Recently, unsupervised learning approaches have been greatly promoted by the contrastive learning(CL), which shows encouraging performance compared to fully supervised approaches [8, 21].
CLpretrains deep neural networks with unlabeled instances and the learned feature embeddings can bedirectly used to extract features from the raw data [35].
Thereby, CL has been successfully applied inmany downstream recognition tasks such as classification [28], retrieval [41] and clustering [3].
As an unsupervised learning problem setting where the human annotation is not available, CLapproaches usually consider building the pseudo supervision in their learning objectives [36, 19] andthus CL is also regarded as a self-supervised learning approach.
Originally, the pseudo supervisionof CL is to push away each pair of instances to scatter data points in the feature space, by which allinstances in the training data can be well discriminated ( i.e. , the instance discrimination ) [14, 40].
This original design has been empirically validated to be particularly effective in the representationlearning [28, 6], and has also been theoretically proved to approximate an unbiased supervisedlearning objective [32, 11].
Many recent efforts have increasingly focused on two different directionsto further improve the performance of CL.
The first one is to introduce plentiful data-augmentationto generate the positive pair which consists of each instance and its perturbation [35, 28].
Then,any two instances in the training data are regarded as the negative pair , and the objective of metriclearning [33] can be used to learn feature embeddings that distinguish positive pairs and negativepairs.
Nevertheless, the negative pairs built in CL are inherently noisy because they contain false negatives consisted of semantically similar instances [30].
Therefore, the second way to improve theperformance of CL is to reduce the impact of false negative pairs.
To this end, some recent worksconvert it to positive-unlabeled learning [11, 27] and clustering problems [3, 43] to reweight theimportance of negative pairs, and thus constraining the undesired repelling of negative pairs [3, 43].
Although the existing methods have achieved promising results to some extent, their reliabilitieshighly depend on the effectiveness of instance discrimination [20, 32].
However, recent worksusually encourage learning contrastive embedding in high-dimensional space to maximally discriminate instances, so that the dimensionality of self-supervised contrastive embedding [7, 9] isset to be much larger than the dimensionality of traditional fully supervised embedding [10, 15].
This practice makes data points sparsely distribute in the feature space ( i.e. , the curse of dimensionality [18]), and thereby the corresponding CL methods may fail to capture the intrinsic similarity between pairwise instances.
Such a problem can hardly be solved by simply setting a lowdimensionality for the output layer, as it will cause the dimensional collapse with insufficient instance discrimination [20].
Some popular compression approaches such as distillation techniques[5, 44] enable us to train small networks under the supervision of original contrastive embeddings, yet the improper similarity predictions can still be inherited from the original networks.
Therefore, a new CL method is desired to effectively learn the low-dimensional feature embedding.
[Figure]
-dimensional latent space.
In this paper, we propose a novel framework dubbed contrastive learning with lowdimensional reconstruction (CLLR) to explicitlyaddress the above issue caused by the high dimensionality in CL.
Specifically, we introducea new sparse projection layer to reconstruct thefeatures of instances in low-dimensional spaceand meanwhile scatter all instances in the original high-dimensional space (see Fig. 1).
Then,we obtain the low-dimensional contrastive embedding which can also effectively distinguishinstances in the training data.
Theoretically, weprove a lower bound for the min-max distance ratio of the learned contrastive embedding, whichensures that CLLR can better capture the instance similarity than the existing CL models.
Experimentally, our approach consistently improves the state-of-the-art methods on vision, language, and reinforcement learning benchmarks.
Tothe best of our knowledge, we are the first to propose learning the original contrastive embedding inlow-dimensional space.
The proposed method is very generic, so it can be applied in many existing
CL models (as demonstrated in experiments).
Our main contributions are summarized below:
• We propose a novel framework to enhance the generalization ability of contrastive learning
via introducing a sparse / low-rank regularized projection layer to adaptively reduce the high
dimensionality of contrastive embedding.
• We establish complete theoretical guarantees for our method by analyzing the error bound of
distance predictions and the convergence of the learning algorithm, respectively.
• We conduct extensive experiments on real-world datasets to validate the superiority of our
method over the state-of-the-art CL approaches, and the results consistently emphasize the
necessity / significance of learning low-dimensional contrastive embeddings.
Notations.
We write matrices and vectors as bold uppercase characters and bold lowercase characters, respectively.
We denote the training dataset X
= { x i ∈
R m | i = 1 , 2 , . . .
, N } where m is the data dimensionality and N is the sample size.
Operators ∥·∥ 2 , ∥·∥
, 1 , and ∥·∥ * denote the vector/matrixℓ 2 -norm, ℓ 2 , 1 -norm ( i.e. , the sum of ℓ 2 -norm for columns), and nuclear-norm, respectively.
1.1 Background & Related Work
In this subsection, we briefly review the background of contrastive learning and the related work.
Instance Discrimination & Contrastive Learning.
As a popular unsupervised / self-supervisedlearning approach, the basic goal of contrastive learning (CL) algorithm is to learn a generic featureembedding Φ : R m (cid:55)→ R H , which transforms the data point from m -dimensional sample space to
H -dimensional embedding space.
The primitive CL method called instance discrimination learnssuch an embedding by directly enlarging the following distance between each pair of two instancesx i and x j in the training data [14, 40]
[Equation]
where H is the dimensionality of the learned feature embedding.
The design philosophy for instancediscrimination is that when we scatter all instances in the feature space, the characteristic of eachinstance are captured and thus the training data can be well memorized by the neural network [20].
When we further generate the positive pairs ( x , x + ) by combining each single instance x and itsperturbation x + , we are able to use the noise contrastive estimation (NCE) loss [16] to learn a featureembedding Φ from positive and negative pairs. In this paper, we focus on such a NCE loss whichhas the form of L NCE ( Φ )= E x , x − j ∈X [ − log(e Φ ( x ) ⊤ Φ ( x + ) /(e Φ ( x ) ⊤ Φ ( x + ) + (cid:80) nj =1 e Φ ( x ) ⊤ Φ ( x − j ) )) ] .
Hereinstances x and { x − j } nj =1 are uniformly sampled from the training data X , and n is the batch size.
Admittedly, as the original prototype of CL, instance discrimination is very critical to ensuring theeffectiveness of most CL methods.
However, the feature dimensionality settings in existing CLmethods are usually very high ( e.g. , 2048-dimension and 4096-dimension in [7, 9] ), which are muchlarger than the feature dimensionality in most fully supervised learning methods ( e.g. , 512-dimensionand 1024-dimension in [10, 15]).
We demonstrate that learning contrastive embeddings in suchhigh-dimensional space can be weak in capturing the similarity between pairwise instances.
Toaddress this issue, in this paper, we propose a novel framework to learn contrastive embedding inlow-dimensional space, which uses a sparse / low-rank regularized projection layer for reconstruction.
PCA & Autoencoder.
As a classical unsupervised / self-supervised learning method, principalcomponent analysis (PCA) has shown promising results in many machine learning tasks [39, 42, 4].
Actually, PCA shares very similar motivation with the instance discrimination of CL.
It is well-knownthat PCA seeks for a vector p ∈ R m to scatter instances in the projection space by maximizing thevariance E x ∈X [( x − x ) ⊤ pp ⊤ ( x − x )] , where x ∈ R m is the mean of all instances in the training data X .
Enlarging such a variance is quite similar to the instance discrimination of CL which also pushes awaydata pairs to scatter instances.
PCA has another reconstruction based form E x ∈X [ ∥ P P ⊤ x − x ∥ 22 ]which is equivalent to the variance maximization ( P ∈ R m × l is the projection matrix and l ∈ Z
+ isthe dimensionality of orthogonal space). To further improve the fitting ability of PCA for complexdata, the non-linear extension Autoencoder introduces the non-linear activation function σ and twodifferent projection matrices P and P ′ to reconstruct the training data in the by minimizing theobjective E x ∈X [ ∥ σ ( P ′⊤ σ ( P ⊤ x ))
− x ∥ 22 ] .
Some further extensions such as masked autoencoder(MAE) [17] achieved very promising results in several downstream tasks.
In this paper, we are inspired by PCA / Autoencoder to reduce the dimensionality of contrastiveembedding based on a sparse / low-rank regularized reconstruction loss.
Interestingly, from thisperspective, our method can also be regarded as a natural combination of two main existing selfsupervised learning approaches.
Methodology
In this section, we first investigate the distribution of instances scattered by CL in the high-dimensionalfeature space.
After that, we propose a novel framework dubbed contrastive learning with lowdimensional reconstruction by introducing a new sparse projection layer.
The learning objective andthe corresponding optimization algorithm are finally designed with convergence guarantee.
2.1 Motivation
As we mentioned before, the contrastive embedding Φ maps an m -dimensional instance into the
H -dimensional feature space.
Now we want to investigate the distribution of data points in such an
H -dimensional space.
We consider the H -dimensional hypercube and its inscribed-suprasphere .
Wesuppose that the edge length of the H -dimensional hypercube is 2 r , and the radius of its inscribed- suprasphere will be r .
Then their corresponding volumes in the high-dimensional space are
V cube
[Equation]
=
[Equation]
and
[Equation]
respectively, where Γ( · ) is the gamma function [37] having a form of Γ( z )
= (cid:82) ∞
t z − 1 e − t dt .
Wefurther study the ratio of the suprasphere volume to the hypercube volume.
We let H → ∞ and theformulation lim H →∞ V sphere ( H ) / V cube
( H ) equals to
[Equation]
and thus we have lim H →∞ V sphere ( H ) / V cube ( H )
=
by using the fact that V sphere ( H ) / V cube ( H ) ≥ [Equation]
This result of volume ratio clearly reveals that the proportion of the inscribed-suprasphere in thehypercube will gradually converge to 0 with the increase of the dimensionality H .
It means that,in the high-dimensional hypercube, a random given data point is less likely to appear inside of theinscribed-suprasphere ( i.e. , in the central area of the hypercube) but it will usually exist outside ofthe inscribed-suprasphere ( i.e. , in the corner area of the hypercube).
However, the learning objective of CL expects to scatter all instances in the H -dimensional hypercube,and thus making the N instances sparsely distribute in the (cid:98) N = 2 H corners .
Specifically, for thecommon dimensionality setting H = 2048 in popular CL methods, we have that
[Equation]
which implies that the corner number (cid:98) N is significantly larger than the sample number N .
Inthis case, the distribution of instances in the feature space will be very sparse, and all instancesare far away from each other.
Thereby the learning algorithm can hardly capture the intrinsicsimilarity between intra-class instances, and the downstream recognition tasks will be affected.
[Figure]
156 To be more religious, we consider the min-maxdistance ratio to investigate the distance contrastin the high-dimensional space.
For the independent and identically distributed (i.i.d.) instancesx , x i ∈ R m
( i =1 , 2 , . . . , n ), their embeddingsΦ ( x ) and Φ ( x i ) are also i.i.d.
no matter howthe embedding is learned [13].
The following
Theorem 1 reveals that the minimal distance
D min Φ ( H ) and the maximal distance D max Φ ( H )tend to be equivalent in the high-dimensionalspace [1] (provided that Φ ( x ) and Φ ( x i ) arei.i.d.
to each other), so the similarity between pairwise instances can hardly provide contrast todiscriminate the intra-class and inter-class (as shown by the distance distributions in Fig. 2).
Theorem 1.
For any given i.i.d.
random data points x , x 1 , x 2 , . . .
, x n ∈ R m , we denote
D max Φ ( H )=
max {D Φ ( x , x i ) | i =1 , . . . , n } and D min Φ ( H )=
min {D Φ ( x , x i ) | i =1 , . . . , n } .
Then wehave that
lim H →∞ { var [ D Φ ( x , x i ) /
E
( D Φ ( x , x i ))] } = 0 and
P
[Equation]
) where the distance function D Φ ( · , · ) is defined in Eq.
(1) and the feature embedding Φ is learnedfrom the training data and independent to the data points x , x 1 , x 2 , . . .
, x n .
In summary, from the above analytical results, we can clearly find that it is very necessary to constrainthe dimensionality of existing CL approaches in a reasonable range.
Motivated by this, in thenext subsection, we provide the formulation of our proposed framework CLLR which reduces thedimensionality of contrastive embedding by a sparse projection layer.
2.2 Formulation
As we discussed in the previous subsection, the feature embedding Φ transforms the raw data fromm -dimensional space into H -dimensional space, where Φ is learned by the NCE loss.
To avoidhigh-dimensional features, we may directly reduce the dimensionlaity of the output layer, but thiswill cause the dimensional collapse with insufficient instance discrimination (as we discussed in
Section 1).
Thereby, we consider to use an additional matrix L ∈ R H
× H to transform the featureembedding result Φ ( x ) into the latent vector L · Φ ( x ) , and then we minimize ∥ L ⊤ L · Φ ( x ) − Φ ( x ) ∥ 22 ,encouraging the latent vector to preserve the useful information in Φ ( x ) .
When we further introducethe low-rank constraint for L , we can obtain a low-dimensional latent space for contrastive learning.ℓ 2 , 1 -Norm based Regularization.
The row (column) sparsity is a long-standing concept which aimsto maintain very few non-zero columns for a matrix.
When we employ the well-known ℓ 2 , 1 -norm torestrict the projection matrix L , we can certainly have a column-sparse L which selects the importantfeatures in Φ ( x ) ∈ R H corresponding to the non-zero columns.
Then we use the selected features toreconstruct the original feature embedding, i.e. ,
[Equation]
where L ∈ R H
× H and α> 0 is tuned by users.
Note that the column sparsity is just a special case ofthe low-rank, but considering its good usability, here we can easily obtain a low-dimensional featureembedding L · Φ ( x ) if the above L is column sparse.
We also provide the following nuclear-normbased formulation to consider the more general case of low-dimensional space.
Nuclear-Norm based Regularization.
To ensure the projection result L · Φ ( x ) in a low-dimensionalspace, a more general way is directly restricting the projection matrix L to be low-rank.
Then, thecolumn vectors of L will be linearly dependent so that we can remove the redundant column toachieve a low-dimensional projection space.
The realized formulation can be written as
[Equation]
where L ∈ R H
× H and α > 0 is tuned by users.
When we obtain the learned projection matrix L ∗ ,we need to further compute its maximal linearly independent set A , and then we calculate the finalprojection matrix (cid:98) L ∈ R H
× H by setting 1 the redundant columns of L ∗ to 0 .
For the above two different formulations in Eq.
(6) and Eq.
(7), it is hard to say which one istheoretically better.
Actually, their final performance may also be influenced by the non-convexity ofthe learning objectives.
Therefore, in our experiments, we evaluate both the two regularizations onmultiple domains.
Now we want to summarize our final learning objective as follows.
Learning Objective of CLLR.
Based on the realized formulation in Eq.
(6) and Eq.
(7), we caneasily deploy the proposed two regularizers in the learning objective of conventional CL methods.
Without loss of generality, for most existing CL methods equipped with NCE loss, we build thefollowing framework of contrastive learning with low-dimensional reconstruction (CLLR)
[Equation]
(8) where the regularization parameter λ > 0 is tuned by users and the regularizer R ( Φ , L ) can berealized by R 2 , 1 ( Φ , L ) and R nuclear ( Φ , L ) in Eq.
(6) and Eq.
(7), respectively.
As a regularizedlearning objective, CLLR is very generic because here the loss term L NCE ( Φ ) can be implemented bymany existing CL methods.
In the next subsection, we provide iteration algorithm to solve Eq.
2.3 Optimization
Minimizing the objective function in Eq.
(8) is a typical batch optimization problem [45], where boththe loss function L NCE ( Φ ) and the regularizer R ( Φ , L ) involve all training data.
Therefore, we adoptthe stochastic gradient descent (SGD) method [22] to solve it, and here we demonstrate the stochasticgradient for the objective function F ( Φ , L ) .
Specifically, for n + 1 ( i.e. , the batch size) randomlyselected data points { x b j | x b j ∈X , b j ∈ B } n +1 j =1 , the NCE loss already has a stochastic form 2 , so herewe need to demonstrate the stochastic loss for the regularizer in the mini-batch, i.e. ,
[Equation]
Here we denote NCE loss L NCE ( φ ) =
E [ ℓ ( φ ; { x b j } n +1 j =1 )] , where the function ℓ ( φ ; { x b j } n +1 j =1 ) = − log ( exp ( φ ( x b n +1 )
⊤ φ ( x + b n +1 )) /
( exp ( φ ( x b n +1 ))
⊤ φ ( x + b n +1 ))
+ (cid:80) nj
=1 exp ( φ ( x b j )) ⊤ φ ( x − b j )))) .
The index vector set B = { b =( b 1 , . . .
, b n +1 )
⊤ | b i , b j =1 , . .
, N, b i ̸ =
b j , i, j =1 , . . . , n + 1 } .
where (cid:98) R ( L ) indicates the penalty ∥ L ∥ 2 , 1 or ∥ L ∥ ∗ for different regularizations.
Here we use thesubgradients [2] of ℓ 2 , 1 -norm and nuclear-norm for optimziation.
Then the learning objective
F ( Φ , L ) in Eq.
(8) has the stochastic form ℓ ( Φ ; { x b j } n +1 j =1 )+ λ R B ( Φ , L ; { x b j } n +1 j =1 ) .
Based on sucha stochastic loss, we further provide the SGD iteration steps in Algorithm 1 to solve Eq.
In summary, introducing the projection layer (i.e., the projection matrix L ) merely incurs an additionalstochastic gradient in Eq.
It means that our method can be easily implemented in most existing
CL methods and only introduces very little computational overheads.
In the next section, we provethat the iteration sequence Φ (1) , . . . , Φ ( T ) in Algorithm 1 converges to a stationary point of thelearning objective F with a convergence rate O (1 / √ T ) , where T is the number of iterations.
Theoretical Analyses
In this section, we further provide in-depth theoretical analyses for our proposed method.
Weinvestigate the convergence of learning algorithm and the lower bound of min-max distance ratio todemonstrate the effectiveness of our method.
All proofs are given in supplementary materials.
3.1 Convergence Analysis
As we described before, the learning objective of CLLR is a regularized empirical loss which isdifferent from the traditional empirical loss solved by SGD, so here we provide careful convergenceanalysis for the SGD based iterations, i.e. , the Algorithm 1.
Specifically, we suppose the learningobjective has δ -bounded gradient, and then we have the following Theorem 2.
Theorem 2.
If the function F ( Φ , L ) has δ -bounded gradient ( i.e. , ∥∇F ( Φ , L ) ∥ 2 < δ ), then we letη = (cid:113) 2( F ( Φ (0) , L (0) )
− F ( Φ ∗ , L ∗ )) /
( Sδ 2 T ) , and for the iterations in Algorithm 1 we have that
[Equation]
where S > 0 is a lipschitz constant such that ∥∇F ( Φ , L ) −∇F ( Φ ′ , L ′ )
∥ 2 ≤ S ∥ [ Φ , L ] − [ Φ ′ , L ′ ] ∥ 2 .
The above Eq.
(11) clearly reveals that the iteration results in Algorithm 1 can gradually convergeto a stationary point with a convergence rate O (1 / √ T ) when setting the proper learning rate η andincreasing the iteration number T .
Therefore, the convergence of our learning algorithm is guaranteedthough the additional projection layer and regularization term are introduced.
3.2 Lower Bound of Min-Max Distance Ratio
Now, we further analyze the distance between pairwise instances in the low-dimensional space.
Aswe mentioned before, in the high-dimensional space, the min-max distance ratio trends to be 0 andthus the distance function will lose its discriminatory.
Therefore, we want to investigate the value ofmin-max distance ratio in low-dimensional feature space learned by our method.
Our method explicitly constrain the dimensionality of the feature space, so it is intuitive that the minmax distance ratio ( D max Φ ( H ) −D min Φ ( H )) /
D min Φ ( H ) in Eq.
(5) should certainly be lower-bounded.
To be religious, we have the following Theorem 3 to reveal the lower bound of distance ratio.
Theorem 3.
For any given n + 1 i.i.d.
random data points x , x 1 , x 2 , . . .
, x n ∈ R m , we denotethat D max (cid:98) Φ , (cid:98) L = max {D (cid:98) Φ , (cid:98) L ( x , x i ) | i = 1 , 2 , . . . , n } and D min (cid:98) Φ , (cid:98) L = min {D (cid:98) Φ , (cid:98) L ( x , x i ) | i
= 1 , 2 , . .
. , n } ,and then we have that
[Equation]
(12) where D (cid:98) Φ , (cid:98) L ( x , x i ) =
∥ (cid:98) L (cid:98) Φ ( x ) − (cid:98) L (cid:98) Φ ( x i ) ∥ 2 / rank ( (cid:98) L ) , and (cid:98) Φ and (cid:98) L are learned from Eq.
From the above Eq.
(12), we can easily observe that the min-max distance ratio has an explicit lowerbound which is mainly determined by the two regularization parameters α and λ (given the trainingdata X ).
It means that the low-rank reconstruction terms ( i.e. , Eq.
(6) and Eq.
(7)) make the min-maxdistance ratio be controllable, and the larger regularization parameters can produce the better lowerbound.
When the min-max distance ratio is lower-bounded, our CLLR predicts low similarities forinter-cluster and high similarities for intra-cluster, so that the learned embedding effectively capturesthe intrinsic similarities / features and thus improving the performance of downstream tasks.
Experimental Results
In this section, we show experimental results on real-world datasets to validate the effectivenessof our proposed method.
In detail, we first conduct ablation study to reveal the usefulness of ourintroduced new block and new regularizers.
Then, we compare our proposed learning algorithmwith existing state-of-the-art models on vision and language tasks.
Finally, we test our method onthe CL based reinforcement learning task.
Further experiments such as parametric sensitivity andrunning time comparison are given in supplementary materials .
The training process is implementedon Pytorch [29] with NVIDIA TeslaV100 GPUs.
We adopt the projection result LΦ ( x ) for featureextraction, where regularization parameters λ
and α are fixed to 0 . 1 and 10 , resepectively.
The hyperparameters of compared methods are set to the recommended values according to their original papers.
[Table]
277 4.1 Ablation Study
In this subsection, we conduct ablation study on the superiority of thelow-dimensional contrastive embedding ( i.e. , our method) over the traditional contrastive embedding ( i.e. , thebaseline method).
We use the STL-10and CIFAR-10 datasets to train the baseline SimCLR [7] and two implementations of CLLR, i.e. , the ℓ 2 , 1 -normbased regularization and nuclear-normbased regularization.
We train all models with 100 and 400 epochs with thesame batch size and learning rate, respectively, and we record the test accuracy of all methods by finetuning a linear softmax .
The baseline method learns contrastive embeddings in the high-dimensionalspace (dimension = 2048 , 3072 , and 4096 ) and the simply fixed low-dimensional space (dimension =256 and 512 ).
We also include the baseline results that do not use the ℓ 2 , 1 -norm and nuclear normconstraints ( i.e. , α = 0 ).
Our method learns embeddings in low-dimensional space, where we use theregularizer to maintain the corresponding non-zero columns in the projection matrix L .
We record the test accuracy (mean ± std, 5 random trials) of compared methods at the 100 - th epochand 400 - th epoch in Tab. 1.
We can observe that the baseline method is better than our method in thefirst 100 epochs, but the two implementations of our method can outperform the baseline method withthe increase of iterations.
This is because that the baseline method only emphasizes on the instancediscrimination, so it can quickly discriminate the training data in the early epochs.
However, in thelatter epochs, the low-rank reconstruction in our method becomes useful in capturing the similaritybetween pairwise instances.
Meanwhile, we can find that the average accuracy of nuclear-norm based
[Figure]
[Figure]
[Figure]
regularization is slightly higher than the ℓ 2 , 1 -norm based one on both two datasets.
Furthermore,we also perform the t -test at significance level 0 .
in the last column, and “ ✓ ” indicates that ourmethod is significantly better than the best baseline result.
In our following experiments, we employthe 256 -dimensional latent features for multiple domain tasks.
4.2 Experiments on Sentence Representation
In this subsection, we employ the BookCorpus dataset [23] to evaluate the performance of allcompared methods on six text classification tasks, including movie review sentiment ( MR ), productreviews ( CR ), subjectivity classification ( SUBJ ), opinion polarity ( MPQA ), question type classification( TREC ), and paraphrase identification ( MSRP ).
We follow the experimental settings in the baselinemethod quick-thought (QT) [26], which chooses the neighboring sentences as positive pairs.
Herethe 10 -fold cross validation is adopted, and the average classification accuracy is listed in Tab.
[Table]
74.7 HCL [30] 77.
73.5 CLLR (DCL+ ℓ 2 , 1 -norm) 77.
75.2 CLLR (DCL+nuclear-norm) 78.
315 For the six classification tasks, ourmethod improves the classification accuracy of baseline method QT for at leastone percentage on most classificationbenchmarks.
The distance histograms of
QT, debiased contrastive learning (DCL)[11], hard negative based contrastivelearning (HCL) [30], and our CLRR areshown in Fig.
We clearly observe that our method obtains the more accurate distance determinationthan baseline methods, and this reveals that our method is effective for the text classification task.
Experiments on Image Classification
[Figure]
In this subsection, we validate the effectiveness of our method on the image classificationtask.
Here we select contrastive multiview coding (CMC) [35] as baseline methods, and implement our method CLLR under such a classicalframework.
We also compare our method withthree additional state-of-the-art methods including DCL, HCL, SwAV [3], and CO2 [38] on STL10 [12], CIFAR-10 [24], and ImageNet-100 [31]datasets.
All methods are fairly implemented bythe ResNet50 with the same training epoch 100 .
For STL-10 and CIFAR-10 datasets, we recordthe classification accuracy of all compared methods with varying numbers of negative sample.
From Fig.
4, we can clearly observe thatour method CLLR successfully improves the baseline for at least 1% and 2% on CIFAR-10 dataset and STL-10 dataset, respectively.
Similar experiments are conducted on ImageNet-100dataset, and Tab. 3 shows that our method consistently improves all baseline methods, whereour method improves the baseline CMC from 73 .
58% to 76 .
For different negative sample sizes, the accuracy rates of our method are also higher than all compared methods, and itclearly demonstrates the effectiveness of our method.
Since CLLR is implemented on differentbaselines, our method has good compatibility with existing CL algorithms on the image classification task.
In supplementary materials , we further compare our method with the distillation based CL models [5, 44] ( i.e. , the low-dimensional small networks supervised by the original contrastive embeddings), and the results clearly demonstrate the superiority of our method.
352 4.4 Experiments on Reinforcement Learning
[Table]
This subsection further extends our experimentson reinforcement learning task, which is anotherapplication scenario of contrastive learning.
Herethe contrastive unsupervised representations forreinforcement learning (CURL) [25] method isemployed to perform image-based policy controlon representation learned by the CL algorithm.
All methods are tested on the DeepMind controlsuite [34], which consists of six control taskslisted in Tab. 4.
By following the experimental settings in CURL, the positive pair is builtby simply cropping a single image, and the negative pair is composed of each two images inthe control sequence.
All methods are retrainedfor 3 times, and the corresponding means and standards of 100K scores are shown in Tab. 4.
[Table]
For the six control tasks, our methodconsistently outperforms the baselinemethod CURL with higher means.
Whencompared to DCL and HCL methods,our method almost achieves the best results in all six scenarios.
Although ourmethod CLRR (CURL+nuclear-norm)has slightly lower scores than CURL or
DCL on the Run / Walk tasks, our methodshows smaller variance.
Moreover, whenwe incorporate our method to DCL and HCL, our method could further improve the overall scores ofcompared methods on the six tasks.
This also reveals that our method is compatible with existing CLalgorithms on the reinforcement learning task.
Conclusion and Future Work
In this paper, we considered the curse of dimensionality existing in the current contrastive learningmethod.
To overcome such an issue, we proposed a novel framework called contrastive learningwith low-dimensional reconstruction (CLLR), which uses a sparse projection layer to reduce thedimensionality of the feature embedding.
We reconstructed the original high-dimensional features inthe low-dimensional projection space while preserving the basic objective for instance discrimination,and thus successfully learning low-dimensional contrastive embeddings.
To the best of our knowledge,this is the first work in CL that considers reducing the feature dimensionality.
We conducted intensivetheoretical analyses to guarantee the effectiveness of our method.
Comparison experiments on realworld datasets across multiple domains indicated that our learning algorithm acquires more reliablefeature embedding than state-of-the-art methods.
Both the theoretical and experimental results clearlydemonstrated the necessity / significance of learning low-dimensional contrastive embeddings.
Our approach mainly focuses on the mainstream CL models which use both positive and negativepairs.
The effectiveness of negative-free CL has also been shown by recent works such as BYOL and
SimSiam.
When the negative pairs ( i.e. , the instance discrimination) are unavailable, exploring thecorresponding optimal (low-dimensional) projection space would be interesting future work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]( b) Did you describe the limitations of your work?
[Yes] See Section 5. c)
Did you discuss any potential negative societal impacts of your work
[N/A] (d)
Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Theo-
rem1/2/3.
(b) Did you include complete proofs of all theoretical results?
[Yes] See the Appendix in the supplementary material.
If you ran experiments...
a) Did you include the code, data, and instructions needed to reproduce the main experi mental
results
(either in the supplemental
material or as a URL [N/A] We will release
the code if the paper is accepted.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 4.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Section 4.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4 and the supplyment
material.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]