Learning Contrastive Embedding in Low-Dimensional Space
Abstract
Contrastive learning (CL) pretrains feature embeddings to scatter instances in the feature space so that the training data can be well discriminated.
Most existing
CL techniques usually encourage learning such feature embeddings in the highdimensional space to maximize the instance discrimination.
However, this practice may lead to undesired results where the scattering instances are sparsely distributed in the high-dimensional feature space, making it difficult to capture the underlying similarity between pairwise instances.
To this end, we propose a novel framework called contrastive learning with low-dimensional reconstruction (CLLR), which adopts a regularized projection layer to reduce the dimensionality of the feature embedding.
In CLLR, we build the sparse / low-rank regularizer to adaptively reconstruct a low-dimensional projection space while preserving the basic objective for instance discrimination, and thus successfully learning contrastive embeddings that alleviate the above issue.
Theoretically, we prove a tighter error bound for CLLR; empirically, the superiority of CLLR is demonstrated across multiple domains.
Both theoretical and experimental results emphasize the significance of learning low-dimensional contrastive embeddings.
Introduction
Recently, unsupervised learning approaches have been greatly promoted by the contrastive learning (CL), which shows encouraging performance compared to fully supervised approaches [8, 21].
CL pretrains deep neural networks with unlabeled instances, and the learned feature embeddings can be directly used to extract features from the raw data [35].
Thereby, CL has been successfully applied in many downstream recognition tasks such as classification [28], retrieval [41], and clustering [3].
As an unsupervised learning problem setting where the human annotation is not available, CL approaches usually consider building the pseudo supervision in their learning objectives [36, 19], and thus CL is also regarded as a self-supervised learning approach.
Originally, the pseudo supervision of CL is to push away each pair of instances to scatter data points in the feature space, by which all instances in the training data can be well discriminated ( i.e. , the instance discrimination ) [14, 40].
This original design has been empirically validated to be particularly effective in the representation learning [28, 6], and has also been theoretically proved to approximate an unbiased supervised learning objective [32, 11].
Many recent efforts have increasingly focused on two different directions to further improve the performance of CL.
The first one is to introduce plentiful data-augmentation to generate the positive pair which consists of each instance and its perturbation [35, 28].
Then, any two instances in the training data are regarded as the negative pair , and the objective of metric learning [33] can be used to learn feature embeddings that distinguish positive pairs and negative pairs.
Nevertheless, the negative pairs built in CL are inherently noisy because they contain false negatives consisted of semantically similar instances [30].
Therefore, the second way to improve the performance of CL is to reduce the impact of false negative pairs.
To this end, some recent works convert it to positive-unlabeled learning [11, 27] and clustering problems [3, 43] to reweight the importance of negative pairs, and thus constraining the undesired repelling of negative pairs [3, 43].
Although the existing methods have achieved promising results to some extent, their reliabilities highly depend on the effectiveness of instance discrimination [20, 32].
However, recent works usually encourage learning contrastive embedding in high-dimensional space to maximally discriminate instances, so that the dimensionality of self-supervised contrastive embedding [7, 9] is set to be much larger than the dimensionality of traditional fully supervised embedding [10, 15].
This practice makes data points sparsely distribute in the feature space (which is similar to the curse of dimensionality [18]), and thereby the corresponding CL methods may fail to capture the intrinsic similarity between pairwise instances.
Such a problem can hardly be solved by simply setting a low dimensionality for the output layer, as it will cause the dimensional collapse with insufficient instance discrimination [20].
Some popular compression approaches such as distillation techniques [5, 44] enable us to train small networks under the supervision of original contrastive embeddings, yet the improper similarity predictions can still be inherited from the original networks.
Therefore, a new CL method is desired to effectively learn the low-dimensional feature embedding.
[Figure]
, we discriminate all instances in high-dimensional space and introduce a sparse projec- tion layer (the red part) to reconstruct the features of instances in the low-dimensional latent space.
In this paper, we propose a novel framework dubbed contrastive learning with lowdimensional reconstruction (CLLR) to explicitly address the above issue caused by the high dimensionality in CL.
Specifically, we introduce a new sparse projection layer to reconstruct the features of instances in low-dimensional space and meanwhile scatter all instances in the original high-dimensional space (see Fig. 1).
Then, we obtain the low-dimensional contrastive embedding which can also effectively distinguish instances in the training data.
Theoretically, we prove a lower bound for the min-max distance ratio of the learned contrastive embedding, which ensures that CLLR can better capture the instance similarity than the existing CL models.
Experimentally, our approach consistently improves the state-of-the-art methods on vision, language, and reinforcement learning benchmarks.
To the best of our knowledge, we are the first to propose learning the original contrastive embedding in low-dimensional space.
The proposed method is very generic, so it can be applied in many existing CL models.
Our main contributions are summarized as: I).
we propose a novel framework to enhance the generalization ability of contrastive learning via introducing a sparse / low-rank regularized projection layer to adaptively reduce the high dimensionality of contrastive embedding; II).
we establish complete theoretical guarantees for our method by analyzing the error bound of distance predictions and the convergence of the learning algorithm, respectively; III).
we conduct extensive experiments on real-world datasets to validate the superiority of our method over the state-of-the-art CL approaches, and the results consistently emphasize the necessity / significance of learning low-dimensional contrastive embeddings.
Notations.
We write matrices and vectors as bold uppercase characters and bold lowercase characters, respectively.
We denote the training dataset X
= { x i ∈
R m | i = 1 , 2 , . . .
, N } where m is the data dimensionality and N is the sample size.
Operators
1.1 Background & Related Work
In this subsection, we briefly review the background of contrastive learning and the related work.
Instance Discrimination & Contrastive Learning.
As a popular unsupervised / self-supervised learning approach, the basic goal of contrastive learning (CL) algorithm is to learn a generic feature embedding Φ : R m (cid:55)→ R H , which transforms the data point from m -dimensional sample space to
H -dimensional embedding space.
The primitive CL method called instance discrimination learns such an embedding by directly enlarging the following distance between each pair of two instances x i and x j in the training data [14, 40]
[Equation]
where H is the dimensionality of the learned feature embedding.
The design philosophy for instance discrimination is that when we scatter all instances in the feature space, the characteristic of each instance are captured and thus the training data can be well memorized by the neural network [20].
When we further generate the positive pairs ( x , x + ) by combining each single instance x and its perturbation x + , we are able to use the noise contrastive estimation (NCE) loss [16] to learn a feature embedding Φ from positive and negative pairs.
In this paper, we focus on such a NCE loss which has the form of L NCE ( Φ )=
E x , x − j ∈X [ − log(e Φ ( x ) ⊤ Φ ( x + ) /(e
Φ ( x ) ⊤ Φ ( x [Equation] ) + (cid:80) nj =1 e Φ ( x ) ⊤ Φ ( x − j ) )) ] .
Here instances x and { x − j } nj =1 are uniformly sampled from the training data X , and n is the batch size.
Admittedly, as the original prototype of CL, instance discrimination is very critical to ensuring the effectiveness of most CL methods.
However, the feature dimensionality settings in existing CL methods are usually very high ( e.g. , 2048-dimension and 4096-dimension in [7, 9] ), which are much larger than the feature dimensionality in most fully supervised learning methods ( e.g. , 512-dimension and 1024-dimension in [10, 15]).
We demonstrate that learning contrastive embeddings in such high-dimensional space can be weak in capturing the similarity between pairwise instances.
To address this issue, in this paper, we propose a novel framework to learn contrastive embedding in low-dimensional space, which uses a sparse / low-rank regularized projection layer for reconstruction.
PCA & Autoencoder.
As a classical unsupervised / self-supervised learning method, principal component analysis (PCA) has shown promising results in many machine learning tasks [39, 42, 4].
Actually, PCA shares very similar motivation with the instance discrimination of CL.
It is well-known that PCA seeks for a vector p ∈ R m to scatter instances in the projection space by maximizing the variance E x ∈X [( x − x ) ⊤ pp ⊤ ( x − x )] , where x ∈ R m is the mean of all instances in the training data X .
Enlarging such a variance is quite similar to the instance discrimination of CL which also pushes away data pairs to scatter instances.
PCA has another reconstruction based form E x ∈X [ ∥ P P ⊤ x − x ∥ 22 ] which is equivalent to the variance maximization ( P ∈ R m × l is the projection matrix and l ∈ Z + is the dimensionality of orthogonal space).
To further improve the fitting ability of PCA for complex data, the non-linear extension Autoencoder introduces the non-linear activation function σ and two different projection matrices P and P ′ to reconstruct the training data in the by minimizing the objective E x ∈X [ ∥ σ ( P ′⊤ σ ( P ⊤ x )) − x ∥ 22 ] .
Some further extensions such as masked autoencoder (MAE) [17] achieved very promising results in several downstream tasks.
In this paper, we are inspired by PCA / Autoencoder to reduce the dimensionality of contrastive embedding based on a sparse / low-rank regularized reconstruction loss.
Interestingly, from this perspective, our method can also be regarded as a natural combination of two main existing selfsupervised learning approaches.
Methodology
In this section, we first investigate the distribution of instances scattered by CL in the high-dimensional feature space.
After that, we propose a novel framework dubbed contrastive learning with lowdimensional reconstruction by introducing a new sparse projection layer.
The learning objective and the corresponding optimization algorithm are finally designed with convergence guarantee.
2.1 Motivation
As we mentioned before, the contrastive embedding Φ maps an m -dimensional instance into the
H -dimensional feature space.
Now we want to investigate the distribution of data points in such an
H -dimensional space.
We consider the H -dimensional hypercube and its inscribed-suprasphere .
We suppose that the edge length of the H -dimensional hypercube is 2 r , and the radius of its inscribed- suprasphere will be r .
Then their corresponding volumes in the high-dimensional space are
V cube
[Equation]
=
[Equation]
and
[Equation]
respectively, where Γ( · ) is the gamma function [37] having a form of Γ( z )
= (cid:82) ∞
t z − 1 e − t dt .
We further study the ratio of the suprasphere volume to the hypercube volume.
We let H → ∞ and the formulation lim H →∞ V sphere ( H ) / V cube ( H ) equals to
[Equation]
and thus we have lim H →∞ V sphere ( H ) / V cube ( H )
=
by using the fact that V sphere ( H ) / V cube ( H ) ≥ 0 .
This result of volume ratio clearly reveals that the proportion of the inscribed-suprasphere in the hypercube will gradually converge to 0 with the increase of the dimensionality H .
It means that, in the high-dimensional hypercube, a random given data point is less likely to appear inside of the inscribed-suprasphere ( i.e. , in the central area of the hypercube) but it will usually exist outside of the inscribed-suprasphere ( i.e. , in the corner area of the hypercube).
However, the learning objective of CL expects to scatter all instances in the H -dimensional hypercube, and thus making the N instances sparsely distribute in the (cid:98) N = 2 H corners .
Specifically, for the common dimensionality setting H = 2048 in popular CL methods, we have that
[Equation]
which implies that the corner number (cid:98) N is significantly larger than the sample number N .
In this case, the distribution of instances in the feature space will be very sparse, and all instances are far away from each other.
Thereby the learning algorithm can hardly capture the intrinsic similarity between intra-class instances, and the downstream recognition tasks will be affected.
[Figure]
To be more religious, we consider the min-max distance ratio to investigate the distance contrast in the high-dimensional space.
For the independent and identically distributed (i.i.d.) instances x , x i ∈
R m ( i =1 , 2 , . . . , n ), their embeddings Φ ( x ) and Φ ( x i ) are also i.i.d.
no matter how the embedding is learned [13].
The following
Theorem 1 reveals that the minimal distance
D min Φ ( H ) and the maximal distance D max Φ ( H ) tend to be equivalent in the high-dimensional space [1] (provided that Φ ( x ) and Φ ( x i ) are i.i.d.
to each other), so the similarity between pairwise instances can hardly provide contrast to discriminate the intra-class and inter-class (as shown by the distance distributions in Fig. 2).
Theorem 1.
For any given i.i.d.
random data points x , x 1 , x 2 , . . .
, x n ∈ R m , we denote
D max Φ ( H )=
max {D Φ ( x , x i ) | i =1 , . . . , n } and D min Φ ( H )=
min {D Φ ( x , x i ) | i =1 , . . . , n } .
Then we have that
lim H →∞ { var [ D Φ ( x , x i ) /
E
( D Φ ( x , x i ))] } = 0 and
[Equation]
where the distance function D Φ ( · , · ) is defined in Eq.
(1) and the feature embedding Φ is learned from the training data and independent to the data points x , x 1 , x 2 , . . .
, x n .
In summary, from the above analytical results, we can clearly find that it is very necessary to constrain the dimensionality of existing CL approaches in a reasonable range.
Motivated by this, in the next subsection, we provide the formulation of our proposed framework CLLR which reduces the dimensionality of contrastive embedding by a sparse projection layer.
2.2 Formulation
As we discussed in the previous subsection, the feature embedding Φ transforms the raw data from m -dimensional space into H -dimensional space, where Φ is learned by the NCE loss.
To avoid high-dimensional features, we may directly reduce the dimensionlaity of the output layer, but this will cause the dimensional collapse with insufficient instance discrimination (as we discussed in
Section 1).
Thereby, we consider to use an additional matrix L ∈ R H
× H to transform the feature embedding result Φ ( x ) into the latent vector L · Φ ( x ) , and then we minimize ∥ L ⊤ L · Φ ( x ) − Φ ( x ) ∥ 22 , encouraging the latent vector to preserve the useful information in Φ ( x ) .
When we further introduce the low-rank constraint for L , we can obtain a low-dimensional latent space for contrastive learning. ℓ 2 , 1 -Norm based Regularization.
The row (column) sparsity is a long-standing concept which aims to maintain very few non-zero columns for a matrix.
When we employ the well-known ℓ 2 , 1 -norm to restrict the projection matrix L , we can certainly have a column-sparse L which selects the important features in Φ ( x ) ∈ R H corresponding to the non-zero columns.
Then we use the selected features to reconstruct the original feature embedding, i.e. ,
[Equation]
where L ∈ R H
× H and α> 0 is tuned by users.
Note that the column sparsity is just a special case of the low-rank, but considering its good usability, here we can easily obtain a low-dimensional feature embedding L · Φ ( x ) if the above L is column sparse.
We also provide the following nuclear-norm based formulation to consider the more general case of low-dimensional space.
Nuclear-Norm based Regularization.
To ensure the projection result L · Φ ( x ) in a low-dimensional space, a more general way is directly restricting the projection matrix L to be low-rank.
Then, the column vectors of L will be linearly dependent so that we can remove the redundant column to achieve a low-dimensional projection space.
The realized formulation can be written as
[Equation]
where L ∈ R H
× H and α > 0 is tuned by users.
When we obtain the learned projection matrix L ∗ , we need to further compute its maximal linearly independent set A , and then we calculate the final projection matrix (cid:98) L ∈ R H
× H by setting 1 the redundant columns of L ∗ to 0 .
For the above two different formulations in Eq.
(6) and Eq.
(7), it is hard to say which one is theoretically better.
Actually, their final performance may also be influenced by the non-convexity of the learning objectives.
Therefore, in our experiments, we evaluate both the two regularizations on multiple domains.
Now we want to summarize our final learning objective as follows.
Learning Objective of CLLR.
Based on the realized formulation in Eq.
(6) and Eq.
(7), we can easily deploy the proposed two regularizers in the learning objective of conventional CL methods.
Without loss of generality, for most existing CL methods equipped with NCE loss, we build the following framework of contrastive learning with low-dimensional reconstruction (CLLR)
[Equation]
(8) where the regularization parameter λ > 0 is tuned by users and the regularizer R ( Φ , L ) can be realized by R 2 , 1 ( Φ , L ) and R nuclear ( Φ , L ) in Eq.
(6) and Eq.
(7), respectively.
As a regularized learning objective, CLLR is very generic because here the loss term L NCE ( Φ ) can be implemented by many existing CL methods.
In the next subsection, we provide iteration algorithm to solve Eq.
2.3 Optimization
Minimizing the objective function in Eq.
(8) is a typical batch optimization problem [45], where both the loss function L NCE ( Φ ) and the regularizer R ( Φ , L ) involve all training data.
Therefore, we adopt the stochastic gradient descent (SGD) method [22] to solve it, and here we demonstrate the stochastic gradient for the objective function F ( Φ , L ) .
Specifically, for n + 1 ( i.e. , the batch size) randomly selected data points { x b j | x b j ∈X , b j ∈ B } n +1 j =1 , the NCE loss already has a stochastic form 2 , so here we need to demonstrate the stochastic loss for the regularizer in the mini-batch, i.e. ,
[Equation]
Here we denote NCE loss L NCE ( φ ) =
E [ ℓ ( φ ; { x b j } n +1 j =1 )] , where the function ℓ ( φ ; { x b j } n +1 j =1 )
= − log ( exp ( φ ( x b n +1 )
⊤ φ ( x + b n +1 )) /
( exp ( φ ( x b n +1 ))
⊤ φ ( x + b n +1 ))
+ (cid:80) nj
=1 exp ( φ ( x b j ))
⊤ φ ( x − b j )))) .
The index vector set B = { b =( b 1 , . . .
, b n +1 )
⊤ | b i , b j =1 , . .
, N, b i ̸ =
b j , i, j =1 , . . . , n + 1 } .
where (cid:98) R ( L ) indicates the penalty ∥
L ∥ 2 , 1 or ∥ L ∥ ∗ for different regularizations.
Here we use the subgradients [2] of ℓ 2 , 1 -norm and nuclear-norm for optimziation.
Then the learning objective
F ( Φ , L ) in Eq.
(8) has the stochastic form ℓ ( Φ ; { x b j } n +1 j =1 )+ λ R B ( Φ , L ; { x b j } n +1 j =1 ) .
Based on such a stochastic loss, we further provide the SGD iteration steps in Algorithm 1 to solve Eq.
In summary, introducing the projection layer (i.e., the projection matrix L ) merely incurs an additional stochastic gradient in Eq.
It means that our method can be easily implemented in most existing
CL methods and only introduces very little computational overheads.
In the next section, we prove that the iteration sequence Φ (1) , . . . , Φ ( T ) in Algorithm 1 converges to a stationary point of the learning objective F with a convergence rate O (1 / √ T ) , where T is the number of iterations.
Theoretical Analyses
In this section, we further provide in-depth theoretical analyses for our proposed method.
We investigate the convergence of learning algorithm and the lower bound of min-max distance ratio to demonstrate the effectiveness of our method.
All proofs are given in supplementary materials.
3.1 Convergence Analysis
As we described before, the learning objective of CLLR is a regularized empirical loss which is different from the traditional empirical loss solved by SGD, so here we provide careful convergence analysis for the SGD based iterations, i.e. , the Algorithm 1.
Specifically, we suppose the learning objective has δ -bounded gradient, and then we have the following Theorem 2.
Theorem 2.
If the function F ( Φ , L ) has δ -bounded gradient ( i.e. , ∥∇F ( Φ , L ) ∥ 2 < δ ), then we let η = [Equation]
[Equation] F ( Φ [Equation] ) , L
[Equation]
− F ( Φ ∗ , L ∗ )) /
( Sδ 2 T ) , and for the iterations in Algorithm 1 we have that
[Equation]
where S > 0 is a lipschitz constant such that ∥∇F ( Φ , L ) −∇F ( Φ ′ , L ′ )
∥ 2 ≤ S ∥
[ Φ , L ] − [ Φ ′ , L ′ ] ∥ 2 .
The above Eq.
(11) clearly reveals that the iteration results in Algorithm 1 can gradually converge to a stationary point with a convergence rate O (1 / √ T ) when setting the proper learning rate η and increasing the iteration number T .
Therefore, the convergence of our learning algorithm is guaranteed though the additional projection layer and regularization term are introduced.
3.2 Lower Bound of Min-Max Distance Ratio
Now, we further analyze the distance between pairwise instances in the low-dimensional space.
As we mentioned before, in the high-dimensional space, the min-max distance ratio trends to be 0 and thus the distance function will lose its discriminatory.
Therefore, we want to investigate the value of min-max distance ratio in low-dimensional feature space learned by our method.
Our method explicitly constrain the dimensionality of the feature space, so it is intuitive that the minmax distance ratio ( D max Φ ( H ) −D min Φ ( H )) /
D min Φ ( H ) in Eq.
(5) should certainly be lower-bounded.
To be religious, we have the following Theorem 3 to reveal the lower bound of distance ratio.
Theorem 3.
For any given n + 1 i.i.d.
random data points x , x 1 , x 2 , . . .
, x n ∈ R m , we denote that D max (cid:98) Φ , (cid:98) L = max {D (cid:98) Φ , (cid:98) L ( x , x i ) | i = 1 , 2 , . . . , n } and D min (cid:98) Φ , (cid:98) L = min {D (cid:98) Φ , (cid:98) L ( x , x i ) | i = 1 , 2 , . . . , n } , and then we have that
[Equation]
(12) where D (cid:98) Φ , (cid:98) L ( x , x i ) =
∥ (cid:98) L (cid:98) Φ ( x )
− (cid:98) L (cid:98) Φ ( x i )
∥ 2 / rank ( (cid:98) L ) , and (cid:98) Φ and (cid:98) L are learned from Eq.
From the above Eq.
(12), we can easily observe that the min-max distance ratio has an explicit lower bound which is mainly determined by the two regularization parameters α and λ (given the training data X ).
It means that the low-rank reconstruction terms ( i.e. , Eq.
(6) and Eq.
(7)) make the min-max distance ratio be controllable, and the larger regularization parameters can produce the better lower bound.
When the min-max distance ratio is lower-bounded, our CLLR predicts low similarities for inter-cluster and high similarities for intra-cluster, so that the learned embedding effectively captures the intrinsic similarities / features and thus improving the performance of downstream tasks.
Experimental Results
In this section [Table] we show experimental results on real-world datasets to validate the effectiveness of our proposed method.
In detail [Table] we first conduct ablation study to reveal the usefulness of our introduced new block and new regularizers.
Then, we compare our proposed learning algorithm with existing state-of-the-art models on vision and language tasks.
Finally, we test our method on the CL based reinforcement learning task.
Further experiments such as parametric sensitivity and running time comparison are given in supplementary materials .
The training process is implemented on Pytorch [29] with NVIDIA TeslaV100 GPUs.
We adopt the projection result LΦ ( x ) for feature extraction, where regularization parameters λ and α are fixed to 0 .
1 and 10 , resepectively.
The hyperparameters of compared methods are set to the recommended values according to their original papers
[Table]
[Table]
4.1 Ablation Study
In this subsection
[Table] we conduct ablation study on the superiority of the low-dimensional contrastive embedding ( i [Table] e. , our method) over the traditional contrastive embedding ( i
[Table] e. , the baseline method). We use the STLand CIFAR-10 datasets to train the baseline SimCLR [7] and two implementations of CLLR, i.e. , the ℓ 2 , 1 -norm based regularization and nuclear-norm based regularization.
We train all models with 100 and 400 epochs with the same batch size and learning rate, respectively [Table] and we record the test accuracy of all methods by finetuning a linear softmax .
The baseline method learns contrastive embeddings in the high-dimensional space (dimension = 2048 , 3072 , and 4096 ) and the simply fixed low-dimensional space (dimension = 256 and 512 ).
We also include the baseline results that do not use the ℓ 2 , 1 -norm and nuclear norm constraints ( i.e. , α = 0 ).
Our method learns embeddings in low-dimensional space, where we use the regularizer to maintain the corresponding non-zero columns in the projection matrix L .
We record the test accuracy (mean ± std, 5 random trials) of compared methods at the 100 - th epoch and 400 - th epoch in Tab. 1.
We can observe that the baseline method is better than our method in the first 100 epochs, but the two implementations of our method can outperform the baseline method with the increase of iterations.
This is because that the baseline method only emphasizes on the instance discrimination, so it can quickly discriminate the training data in the early epochs.
However, in the latter epochs, the low-rank reconstruction in our method becomes useful in capturing the similarity between pairwise instances.
Meanwhile, we can find that the average accuracy of nuclear-norm based
[Figure]
regularization is slightly higher than the ℓ 2 , 1 -norm based one on both two datasets.
Furthermore, we also perform the t -test at significance level 0 .
in the last column, and “ ✓ ” indicates that our method is significantly better than the best baseline result.
In our following experiments, we employ the 256 -dimensional latent features for multiple domain tasks.
4.2 Experiments on Sentence Representation
In this subsection, we employ the BookCorpus dataset [23] to evaluate the performance of all compared methods on six text classification tasks, including movie review sentiment ( MR ), product reviews ( CR ), subjectivity classification ( SUBJ ), opinion polarity ( MPQA ), question type classification ( TREC ), and paraphrase identification ( MSRP ).
We follow the experimental settings in the baseline method quick-thought (QT) [26], which chooses the neighboring sentences as positive pairs.
Here the 10 -fold cross validation is adopted, and the average classification accuracy is listed in Tab. 2.
[Table]
[Table]
[Table] [Table] [Table] [Table]
5 CLLR (DCL+ ℓ [Table] [Table] -norm) 77.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
CLLR (
[Table]
+nuclear [Table] norm) 78. [Table]
[Table]
[Table] [Table] [Table] For the six classification tasks, our method improves the classification accuracy of baseline method QT for at least one percentage on most classification benchmarks.
The distance histograms of
QT, debiased contrastive learning (DCL) [11], hard negative based contrastive learning (HCL) [30], and our CLRR are shown in Fig.
We clearly observe that our method obtains the more accurate distance determination than baseline methods, and this reveals that our method is effective for the text classification task.
4.3 Experiments on Image Classification
[Figure]
In this subsection, we validate the effectiveness of our method on the image classification task.
Here we select contrastive multiview coding (CMC) [35] as baseline methods, and implement our method CLLR under such a classical framework.
We also compare our method with three additional state-of-the-art methods including DCL, HCL, SwAV [3], and CO2 [38] on STL10 [12], CIFAR-10 [24], and ImageNet-100 [31] datasets.
All methods are fairly implemented by the ResNet50 with the same training epoch 100 .
For STL-10 and CIFAR-10 datasets, we record the classification accuracy of all compared methods with varying numbers of negative sample.
From Fig.
4, we can clearly observe that our method CLLR successfully improves the baseline for at least 1% and 2% on CIFAR-  dataset and STL-10 dataset, respectively.
Similar experiments are conducted on ImageNetdataset, and Tab. 3 shows that our method consistently improves all baseline methods, where our method improves the baseline CMC from 73 .
58% to 76 .
For different negative sample sizes, the accuracy rates of our method are also higher than all compared methods, and it clearly demonstrates the effectiveness of our method.
Since CLLR is implemented on different baselines, our method has good compatibility with existing CL algorithms on the image classification task.
In supplementary materials , we further compare our method with the distillation based CL models [5, 44] ( i [Table] e. , the low-dimensional small networks supervised by the original contrastive embeddings) [Table]
and the results clearly demonstrate the superiority of our method
[Table]
4.4 Experiments on Reinforcement Learning
[Table]
This subsection further extends our experiments on reinforcement learning task, which is another application scenario of contrastive learning.
Here the contrastive unsupervised representations for reinforcement learning (CURL) [25] method is employed to perform image-based policy control on representation learned by the CL algorithm.
All methods are tested on the DeepMind control suite [34], which consists of six control tasks listed in Tab. 4.
By following the experimental settings in CURL, the positive pair is built by simply cropping a single image, and the negative pair is composed of each two images in the control sequence.
All methods are retrained for 3 times, and the corresponding means and standards of 100K scores are shown in Tab. 4.
[Table]
For the six control tasks, our method consistently outperforms the baseline method CURL with higher means.
When compared to DCL and HCL methods, our method almost achieves the best results in all six scenarios.
Although our method CLRR (CURL+nuclear-norm) has slightly lower scores than CURL or
DCL on the Run / Walk tasks, our method shows smaller variance.
Moreover, when we incorporate our method to DCL and HCL, our method could further improve the overall scores of compared methods on the six tasks.
This also reveals that our method is compatible with existing CL algorithms on the reinforcement learning task.
Conclusion and Future Work
In this paper [Table] we considered the issue of high-dimensional features existing in the current contrastive learning method.
To overcome such an issue, we proposed a novel framework called contrastive learning with low-dimensional reconstruction (CLLR), which uses a sparse projection layer to reduce the dimensionality of the feature embedding.
We reconstructed the original high-dimensional features in the low-dimensional projection space while preserving the basic objective for instance discrimination, and thus successfully learning low-dimensional contrastive embeddings.
To the best of our knowledge, this is the first work in CL that considers reducing the feature dimensionality.
We conducted intensive theoretical analyses to guarantee the effectiveness of our method.
Comparison experiments on real-world datasets across multiple domains indicated that our learning algorithm acquires more reliable feature embedding than state-of-the-art methods.
Both the theoretical and experimental results clearly demonstrated the necessity / significance of learning low-dimensional contrastive embeddings.
Our approach mainly focuses on the mainstream CL models which use both positive and negative pairs.
The effectiveness of negative-free CL has also been shown by recent works such as BYOL and SimSiam.
When the negative pairs are unavailable, exploring the corresponding optimal (low-dimensional) projection space would be interesting future work. Acknowledgment
S.C., G.N., and M.S. were supported by JST AIP Acceleration Research Grant Number JPMJCR20U3,
Japan.
M.S. was also supported by the Institute for AI and Beyond, UTokyo.
C.G., J.L., and J.Y. were supported by NSF of China (Nos: U1713208, 61973162, 62072242), NSF of Jiangsu Province (No: BZ2021013), NSF for Distinguished Young Scholar of Jiangsu Province (No: BK20220080), and the Fundamental Research Funds for the Central Universities (Nos: 30920032202, 30921013114). Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Theorem1/2/3.
(b) Did you include complete proofs of all theoretical results?
[Yes] See the Appendix in the supplementary material.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?[Yes] See Sectionand supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 4.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section 4.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4 and the supplyment material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]