Optimal Positive Generation via Latent Transformation for Contrastive Learning
Abstract
Contrastive learning, which learns to contrast positive with negative pairs of samples, has been popular for self-supervised visual representation learning.
Although great effort has been made to design proper positive pairs through data augmentation, few works attempt to generate optimal positives for each instance.
Inspired by semantic consistency and computational advantage in latent space of pretrained generative models, this paper proposes to learn instance-specific latent transformations to generate Contrastive Optimal Positives (COP-Gen) for self-supervised contrastive learning.
Specifically, we formulate COP-Gen as an instance-specific latent space navigator which minimizes the mutual information between the generated positive pair subject to the semantic consistency constraint.
Theoretically, the learned latent transformation creates optimal positives for contrastive learning, which removes as much nuisance information as possible while preserving the semantics.
Empirically, using generated positives by COP-Gen consistently outperforms other latent transformation methods and even real-image-based methods in self-supervised contrastive learning.
Introduction
Learning effective visual representation without explicit human annotation is attractive but challenging in machine learning and computer vision.
One way to achieve this goal is through self-supervised learning, which leverages handcrafted pretext tasks to create the learning objectives, such as context prediction [1], colorization [2], rotation prediction [3] and generative modeling [4, 5].
Recently, contrastive learning [6, 7, 8, 9, 10, 11] has achieved great success in self-supervised visual representation learning.
The goal of contrastive learning is to pull semantically similar samples (positive pairs) together in the feature space while pushing dissimilar samples (negative pairs) apart.
Without access to labels, negative pairs are sampled randomly and in large amount [10, 11] to obtain effective representations [12, 13].
A line of works are proposed to calibrate the impact of negative samples [14, 15, 16, 17, 18, 19].
On the other hand, the construction of positive pairs is also very crucial.
Tian et al.
[20] demonstrates that optimal positive pairs are task-dependent.
Since the labels of downstream tasks and pretraining dataset are not accessible in self-supervised learning, a common way to create positive pairs is to use semantic-preserving data space transformation (augmentation) to obtain two or more different views of the same sample, as shown in Figure 1a.
Great effort has been made to find the optimal augmentation policy [11, 20].
Nevertheless, data augmentation cannot create instance-specific optimal positives, which are key in contrastive learning to preserve semantic and discard nuisance information for each sample.
Some works try to address this problem by learning adversarial and x v v ′
h h
′ t x ∼ T x t ′ x ∼ T x f ( · ) f ( · )
Max agreement (a) SimCLR [11] z z ′ x x ′ v v ′
h h ′ t
z ∼ T z
G ( · )
G ( · ) t x ∼ T x t ′ x ∼ T x f ( · ) f ( · )
Max agreement (b) GenRep [28]
z z ′ x x ′ v v ′ h h ′
T z ( · )
Min agreement
G ( · )
G ( · ) t x ∼ T x t ′ x ∼ T x f ( · ) f ( · )
Max agreement
[Figure] [Figure]
perturbations in data space [21, 22, 23] or using nearest neighbors in the learned representation space [24, 25, 26, 27].
However, the former limits the transformation diversity and the latter suffers from a high false positive rate [25].
Instead of creating optimal positives for each instance in data space or representation space, a more proper way to address this problem is through transformation in the latent space of well-trained generative models.
The main reasons include (1) semantic consistency between the positive pair can be achieved in the latent space (in contrast to representation space transformation and non-domainknowledge-guided data augmentation) by controlling the transformation distance within a small range; (2) latent space transformation is continuous and differentiable (in contrast to data augmentation), which is computationally friendly for searching instance-wise optimal transformations.
In addition, it may bring more diversity beyond data augmentation.
Jahanian et al.
[28] have made a first step towards leveraging latent space transformation together with data augmentation to create positive pairs, as shown in Figure 1b.
However, the transformation is still designed for the whole dataset.
How to find the optimal latent transformation for each sample remains an open issue.
In this work, to address the above problems, we propose an instance-specific Contrastive Optimal
Positive Generation (COP-Gen) approach via latent space transformation.
Specifically, COP-Gen learns a navigator in the latent space, whose objective is to transform each input anchor to the optimal position.
To this end, subjecting to the semantic constraint, the navigator is simply learnt to minimize the mutual information between the generated positive pair.
Therefore, the training process is implemented by adversarially training a mutual information estimator together with the latent space navigator, as illustrated in Figure 1c.
We summarize our contributions as follows:
• We find the latent space of well-trained generative model fits exactly for generating instance-specific optimal positives in fully unsupervised setting and thus propose COP-Gen. •
We theoretically demonstrate that the learned latent transformation generates optimal positive pairs for contrastive learning, which can preserve semantic-relevant information while discarding nuisance information.
• Extensive experiments show that using COP-Gen to generate positives outperforms other latent transformation methods and even real-image-based methods in self-supervised contrastive learning.
We also extend COP-Gen to supervised contrastive learning.
Preliminaries
Contrastive Learning.
Given a training sample x , state-of-the-art contrastive methods like SimCLR [11], as illustrated in Figure 1a, use data augmentations t x ∼ T x and t ′ x ∼ T x to create a positive pair ( v , v ′ ) , and then learn an encoder f to maximize the agreement between them by contrasting with K randomly sampled negatives, using the InfoNCE loss [9]:
[Equation]
where sim denotes the cosine similarity between two vectors.
It is proved that minimizing InfoNCE loss maximizes a lower bound on mutual information [9, 29]: I ( v ; v ′ ) ≥ log( K ) − L f NCE , where I ( v ; v ′ ) represents mutual information between v and v ′ .
Optimal Positives for Contrastive Learning.
Intuitively, the data augmentation T x that creates positive pairs should preserve the semantic characteristics while changing the nuisance aspects [30].
To find the right balance, Tian et al.
[20] introduce an “InfoMin principle”, which demonstrates that optimal positives for contrastive learning are task-dependent.
Definition 2.1 (Optimal Positives for Contrastive Learning, Proposition A.1 in [20]) .
Given a downstream task T with label y , the optimal positive pair ( v ∗ , v ′∗ ) created from the data x satisfies
[Equation]
subject to I ( v ; y )
=
I ( v ′ ;
y )
=
I ( x ; y ) .
Definition 2.1 indicates that when we have access to the downstream task, the optimal positive pair should only capture the task-relevant information I ( x ; y ) , while leaving the nuisance information
I ( v ∗ ; v ′∗ | y ) to be zero.
Unfortunately, y is not accessible in unsupervised contrastive learning.
The construction of positive pairs is commonly guided by domain knowledge or experimental results instead [11, 20] and could not be instance-wise optimal.
To address this problem, we will next develop an optimal positive generation approach that uses pretrained generative models without accessing the labels.
Contrastive Optimal Positive Generation via Latent Transformation
In this section, we first introduce our contrastive optimal positive generation (COP-Gen) approach in Section 3.1.
Next, we give a theoretical analysis in Section 3.2.
In Section 3.3 we describe and explain the practical details.
3.1 Motivation and COP-Gen Approach
Before describing our approach, we explain the inspiration behind it by formulating a prominent property of pretrained generative models.
It has been observed that using pretrained generative models can create images with interpretable transformations [31, 32, 33], which has several interesting applications such as face attribute editing [34] and data augmentation [35].
These generative models [31, 36, 37] are commonly trained on annotated or one-class datasets.
Yet it is worth noting that even they are trained on unlabeled large scale dataset like ImageNet [38], nearby latent vectors can map to semantically similar images, which is a remarkable property for unsupervised representation learning [28].
See Figure 2 for an empirical observation.
We formulate this property as follows.
Proposition 3.1 (Semantic Consistency in Latent Space) .
Given a well-trained unconditional generative model G , z and z ′ are two latent vectors, if the distance d ( z , z ′ ) ≤ δ , then G ( z ) and G ( z ′ ) will have the similar semantic label y .
In the form of mutual information, we have
[Equation]
4) where ε stands for tolerable semantic difference, and δ is the maximum shifted distance to maintain semantic consistency (approximately corresponding to the second column of G ( z ′ ) in Figure 2).
The above observation and Proposition 3.1 inspire us to find optimal positives in the latent space of well-trained generative models without access to the labels.
Therefore, we propose the following and general.
COP-Gen approach.
Our goal is to learn a latent space navigator T z ( z ) , which can map each sampled anchor z to the right position z ′ , where the constructed pair ( G ( z ) , G ( z ′ )) is the optimal for contrastive learning.
COP-Gen. Our idea, inspired by the InfoMin principle, is to reduce as much shared information between G ( z ) and G ( T z ( z )) as possible while preserving the semantics: [Equation] where
[Equation]
In practice, a shared encoder f trained with InfoNCE loss is applied to estimate a lower bound of the mutual information between the positive samples, as illustrated in Figure 1c.
Specifically, Eq.
(5) is implemented as the following adversarial training form:
[Equation]
After learning T z , we freeze it and use it together with G to generate positive pairs offline.
After that, we perform the standard contrastive learning by training an encoder that maximizes (a lower bound on) the mutual information between generated positive pairs.
As will be proved in Section 3.2, there is no nuisance information remaining in any generated positive pairs to be learned by the encoder.
Note that the synthetic images are usually object-centric and color-biased [40].
As a consequence, when using these synthetic images for contrastive learning, data space augmentations T x like random cropping and color jittering cannot be replaced even when the latent transformation is optimal.
Thus,
T x is also adopted in COP-Gen to keep consistency with the contrastive learning stage.
3.2 Theoretical Analysis
In this subsection, we theoretically demonstrate that under the guarantee of semantic consistency,
COP-Gen will find optimal latent transformations for each sampled latent vector.
We first give the following assumption and lemma.
Assumption 3.2 (Semantic-preserving T x ) .
The data space augmentation T x : X → X should be designed to be semantic-preserving.
In the form of mutual information,
[Equation]
where x ∈ X is the data, y ∈ Y is the semantic label of x .
In practice, we take the well-trained BigBiGAN [39] generator as the data source, which was trained jointly with an additional encoder to learn the inverse mapping as BiGAN [41] and ALI [42].
It has been proved that these models have the following property.
Lemma 3.3 (Theorem 2 in [41]) .
Assume discriminator D , generator G and encoder E are optimal, if G and E are deterministic, then G − 1 = E almost everywhere.
Lemma 3.3 ensures the invertibility of the pretrained generator G . In COP-Gen (Figure 1c), the invertibility ensures G ( T z ( z )) can be mapped back to a deterministic place T z ( z ) in the latent space. Thus training a navigator T z : z → T z ( z ) can make sense. Otherwise the navigator would be confused if there were more than one destination.
The formal proof will be given in Theorem 3.4.
We also use a pretrained flow model [43], which is naturally invertible, as the data source and conduct a toy experiment in Appendix D.
Theorem 3.4 (Optimal T z ) .
Under the above assumption and lemma, given specific G ( z ) , T ∗ z obtained by optimizing Eq.
(5) generates optimal positives G ( T ∗ z ( z )) for contrastive learning.
Proof.
Below we use the notations in Figure 1c, i.e., z ′ =
T z ( z ) , x = G ( z ) , x ′ =
G ( z ′ ) , v = t x
x ) , v ′ =
t ′ x
( x ′ ) , for brevity.
Using the chain rule of mutual information, the training objective in Eq.
(5) can be written as
[Equation]
We first prove that v introduces no extra semantic information when v ′ is known, i.e., the above second item
[Equation]
Strictly, the equality approximately holds within the error range of ε as shown below.
Using the chain rule of mutual information, we have
[Equation]
Since t ′ x preserves the semantics of x ′ , and the semantic consistency between z and z ′ =
T z ( z ) holds under the condition of d ( z , T z ( z )) ≤ δ from Proposition 3.1, we have
[Equation]
by combining Eq.
(7) and (4).
In addition, under Lemma 3.3, x ′ and thus v ′ =
t ′ x
( x ′ ) can be expressed as a function of x , so that
[Equation]
By plugging Eq.
(11) and (12) into Eq.
(10), we find I ( y ; x | v ′ ) is approximately zero within the error range of ε .
And v = t x ( x ) is a function of x .
Therefore, y → v ′ → x → v forms a Markov chain approximately.
Therefore, Eq.
(9) holds within the error range of ε .
Now, Eq.
(8) becomes
[Equation]
Since t x is semantic-preserving, the first item I ( v ; y ) in Eq.
(13) equals to the constant I ( x ; y ) .
Therefore, minimizing Eq.
(13) will make the second item, which represents the nuisance information, tend to zero.
Formally, under the semantic constraint, T ∗ z obtained by minimizing Eq.
(13) satisfies
[Equation]
within the semantic error ε defined in Proposition 3.1, where v ′∗ = t ′ x
( G ( T ∗ z ( z ))) .
According to
Definition 2.1, T ∗ z creates optimal positives for contrastive learning in latent space.
3.3 Practical Details
Choice of T z .
The latent space navigator T z is implemented as a nonlinear two-layer neural network
T z ( z ) =
W 2 ( σ ( W 1 z + b 1 ))
+ b 2 by default, where σ is the ReLU [44] function.
We adopt nonlinear trajectories in latent space since linear trajectories may cause generated images distorted or meaningless [45, 46].
Linear T z is also experimented as an ablation in Section 4.3.
Implementation of d ( z , z ′ ) ≤ δ .
In practice, we model z ′ as z ′ =
z + T z ( z ) instead of z ′ =
T z ( z ) , since random initializing the parameters of T z may not guarantee d ( z , z ′ ) ≤ δ thus violate the semantic constraint.
Take L2 distance as an example, d ( z , z ′ ) now becomes d 2 ( z , z ′ )
= (cid:113)(cid:80)
Kk =
( T z ( z ) k ) 2 .
Since z ∈ R K is usually sampled from an isotropic Gaussian
N ( 0 , I ) , T z ( z ) approximately (due to nonlinear T z ) follows Gaussian distribution with marginals
T z ( z ) k ∼ N ( µ k , σ 2 k ) , where µ k and σ k are determined by W 1 , W 2 , b 1 , b 2 .
Thus E [ d 2 ( z , z ′ ) 2 ] = (cid:80)
Kk =1 E [( T z ( z ) k ) 2 ] =
(cid:80)
Kk
=1 ( µ 2 k + σ 2 k ) .
Therefore, in order to satisfy d ( z , z ′ ) ≤ δ experimentally, we control the initialization of the parameters in T z and monitor the mean and standard deviation of
T z ( z ) as well as the quality of generated images during training.
A summary of the implementation of the semantic consistency constraint during the whole training process is discussed in Appendix B.
Experiments
In this section we conduct experiments on the proposed COP-Gen approach and compare it with other latent transformation and real-image-based methods.
We use Pytorch [47] for all experiments.
4.1 Experimental Settings
Pretrained Generative Models.
Most of our self-supervised experiments are conducted using ImageNet ILSVRC-2012 [38] pretrained BigBiGAN [39, 48].
We do not use the encoder of BigBiGAN, but only use the generator as a black-box data source.
Besides, some additional self-supervised experiments leveraging MNIST [49] pretrained Residual Flow [43] model can be found in Appendix D.
We also extend our approach to supervised contrastive learning [50] in Appendix E.
COP-Gen Training.
The latent space navigator T z is implemented as a nonlinear two-layer MLP by default.
The sizes of the input, hidden and output layers are the same, all equal to the latent dimension (120-d) of BigBiGAN.
Data space SimCLR [11] augmentation is adopted, which is implemented to be differentiable by applying torchvision.transforms to tensors x on-the-fly, to let gradients flow from v =
T x ( x ) to x .
The backbone of the mutual information estimator f is ResNet-18 [51] to save GPU memory.
We use InfoNCE loss [9] as the adversarial training objective for f and T z , with the temperature set to 0 .
Adam [52] ( β 1 = 0 .
5 , β 2 = 0 .
999 ) is used as the optimizer, where the learning rate is set to 3 × 10 − 5 for f and 1 × 10 − 5 for T z .
We train over 200 K generated sampleswith batch size of 176 , which takes about 1 hour on 4 NVIDIA 2080 Ti GPUs.
Methods for Comparison.
We compare our COP-Gen with the following positive generation methods.
The baseline SimCLR [11], which leverages data space transformation (e.g., random cropping, color jittering, etc.) to create positive pairs.
InfoMin Aug. [20], which is inspired by the
InfoMin principle and adopts additional (cf.
SimCLR Aug.) data augmentations.
NNCLR [25], which uses nearest neighbors in the learned representation space as the positives.
Negative Feature
Transformation [53], which creates hard negatives through feature interpolation.
Two latent space transformation methods introduced in GenRep [28].
The one is through Gaussian transformation, with standard deviation set to 0 .
2 determined by grid searching.
The other is a “ steering ” method introduced in [54], which learns a shared latent trajectory over 160 K samples to mimic data space transformations (including zoom, shifts, rotations, and color transformations), and then randomly samples the length of the learned trajectory to create positives.
More implementation details are given in Appendix C.1.
Comparison of Computational Overhead. When comparing with non-learning-based latent transformation methods (GenRep Gaussian [28]), our COP-Gen introduces additional navigator training overhead. This cost also happens in the previous learning-based latent “steering” method [28]. When comparing with real-image-based baseline [11], all computations including navigator training and positive sampling in the latent space of generative models are considered.
All these computations are not too much (several GPU hours).
But if we take training generative models into consideration, the extra training cost would be expensive (typically several GPU weeks).
Nevertheless, the community has released many well-trained generative models for researchers to use.
The detailed computational overhead is listed in Appendix F.
Contrastive Learning.
We conduct contrastive learning on both real and synthetic datasets.
For both of them in all experiments, we use 128 × 128 image size, which is the output resolution of the pretrained BigBiGAN generator.
The real image encoder is trained on unlabeled ImageNet [38] which has ∼ 1 .
28 M images of 1000 classes (ImageNet-1K), and the synthetic image encoder is trained on 1 .
3 M randomly sampled anchor images from BigBiGAN.
We use InfoNCE as the loss, which is optimized using SGD with momentum of 0 .
9 , the learning rate of 0 .
03 × BatchSize / 256 , and weight decay of 10 − 4 .
We train with batch size of 224 for 100 epochs and decay the learning rate using the cosine decay schedule [55], which takes ∼ 3 days for ResNet-50 on 2 NVIDIA 2080 Ti GPUs. [Table]
4.2 Experimental Results
Linear Evaluation on ImageNet-1K. We train a fully connected linear classifier on top of the frozen 2048-d embeddings from the pretrained ResNet-50 encoder.
It is trained using SGD with momentum of 0 .
9 , batch size of 224 for 60 epochs with cosine learning rate decay.
Following [28], the initial learning rate is set as 0 .
3 × BatchSize / 256 for the real and 2 × BatchSize / 256 for the synthetic respectively.
We report Top-1 and Top-5 classification accuracies on ImageNet-1K validation set.
The comparison result is presented in Table 1.
4 First, COP-Gen achieves the best performance compared to other synthetic-image-based methods, demonstrating the effectiveness of our latent space optimal positive generation approach.
In addition, it is worth noting that synthetic-image-based contrastive representation learning can outperform the real-image-based method, as long as the proper latent transformation method is adopted.
That is because the latent transformation brings more diversity to and could achieve instance-wise optimal positives.
Although there is still a gap in the resolution and quality between synthetic and real images, this phenomenon inspires us to further develop black-box generative models as datasets to perform visual representation learning.
Object Detection on PASCAL VOC.
We test the pretrained encoders on the downstream object detection task to evaluate the feature transferability.
Following the protocol in [10], we use detectron [56] to train a Faster-RCNN [57] with the R50-C4 backbone.
We fine-tune all layers end-to-end with batch size of 4 for 96 K iterations ( ∼ 23 epochs) on the PASCAL VOC [58] trainval07+ set and evaluate on the test07 set.
The results are in Table 1.
It can be seen that COP-Gen achieves consistent gains over other synthetic encoders and competitive performance compared with real-image-based baselines.
Transfer Learning and Semi-Supervised Learning Evaluations. Following the procedure in [11, 25], we evaluate the pretrained encoders on transfer learning and semi-supervised learning tasks. For transfer learning task, we train a linear classifier on following datasets: Food101 [59], CIFAR10 [60], CIFAR100 [60], SUN397 [61], Pets [62], Caltech-101 [63], and Flowers [64].
For semi-supervised learning task, we follow the setting in [11], i.e., sample 1% or 10% of the labeled
ImageNet training set in a class-balanced way, and fine-tune the whole network on the labeled data.
These two experiments are implemented based on the open-source codebase VISSL [65] using its standard set of hyper-parameters and evaluation protocols.
We compare our method with the baseline
SimCLR and two of the most competitive methods: NNCLR on the real and latent “steering” method.
Table 2 shows the results.
It is worth noting that synthetic-image-trained encoders outperform real-image-trained encoders on nearly all transfer and semi-supervised benchmarks.
And our method achieves competitive performances when compared with the previous latent “steering” method.
[Table]
4.3 Ablation Study
In this section we present a thorough analysis of COP-Gen and its influence on contrastive learning.
Default Setting.
We conduct ablation experiments on ImageNet-100, a subset of ImageNet-1K,
[Table]
as done in [66, 15, 28].
In this setting, the synthetic encoder is trained on 130 K anchor images sampled from pretrained BigBiGAN.
Note that these sampled images actually come fromclasses since the fixed BigBiGAN was trained on unlabeled ImageNet-1K.
Therefore, we here only perform contrastive learning on these synthetic images and report ImageNet-100 linear evaluation results, since training a real encoder on ImageNet-100 will be an unfair comparison.
Here, the number of contrastive pretraining epoch is set to 200 , and other training hyperparameters and process are the same as those in Section 4.1 by default.
The ImageNetlinear evaluation results of the default setting are presented in the upper part of Table 3.
Do We Still Need Data Augmentation T x ?
Having obtained the optimal latent space transformation
T z , a natural question is that do we still need data augmentation T x when performing contrastive learning .
We answer this question by removing T x and using only T z to create positive pairs.
The results are shown in the bottom part of Table 3.
It can be seen that in the absence of T x , the performance drops significantly, even though T z obtained by our COP-Gen is optimal.
And best result is achieved by combining instance-specific latent (COP-Gen) and data space transformations.
Further, we conduct ablations by removing color, crop, and all transformations in the data space respectively at both navigator learning and contrastive learning stages , to see whether the learned T z ( · ) (together with the frozen G ) could cover these data space transformations.
As shown in Table 4, the large performance drops imply that G ( T z ( · )) has poor ability to cover color and crop augmentations, which are crucial when creating contrastive pairs [11]. [Table]
The reason behind this is probably because images generated by GANs are usually objectcentric and color-biased [40].
To overcome this, one might involve these complex T x when training generative models G , or train G together with our navigator T z , so that G ( T z ( · )) could cover more T x .
Number of Training Samples for T z .
By default, we decide when to terminate COP-Gen training process by monitoring the quality of generated positives.
This allows efficient experiments in Section 4. without operations like grid searching.
Now, we increase the number of training samples to investigate its influence on the generated positives and contrastive learning performance.
For random anchor images, we show how the generated positives evolve over the training time in the middle part of Figure 3. training samples.
A smoother evolution with more examples is given in Figure 4 in Appendix by other latent-transformation-based positive methods
It can be seen that: (1)
In the first row, the clothes of the person change and the background becomes simpler and simpler.
On the contrary, for an anchor with simple background (row 2), the training process makes it gradually more complex.
This phenomenon shows that the learned positives are instancespecific.
In addition, the COP-Gen training process only changes the above nuisance information between the positive pair, while keeping the semantics (the person and the obelisk) unchanged, so that the following contrastive learning stage will not focus on the semantic-irrelevant information.
As for the last positive in row 4, the semantics of the subject is changed (a human face appearing on the cup), which indicates that the semantic consistency constraint is violated and the training process should have been terminated. [Table]
The contrastive learning results using these positives generated at different training periods are shown in Table 5.
Note that although the best performance may not be achieved through simply and conservatively monitoring the quality of generated positives, the results only fluctuate slightly and all of them outperform other noninstance-specific latent transformation methods.
Linear T z .
We also conduct experiments on modeling T z as a linear trajectory.
With such design, we achieve 61 .
18% Top-1 accuracy, which falls behind non-linear T z and other latent transformation methods.
This may be due to the limited modeling capability of linear transformations.
4.4 Visualization Analysis
We give a visual comparison on the generated positives with other latent transformation methods.
As shown in Figure 3, the “steering” method [54, 28], which learns a shared latent trajectory with random scales to mimic data augmentations, causes unrealistic in a few images.
This is partly because the pretrained GANs could not achieve these strong data space transformations.
Relatively, the Gaussian transformation constructs positives in a conservative way after grid searching its standard deviation [28], which only ensures the semantic consistency but lacks diversity.
In contrast, our COP-Gen is aware of the instance-specific nuisance information and removes that between the positives.
Related Work
Contrastive Learning.
The goal of contrastive learning is to contrast positive with negative pairs [6].
Without access to labels, positive pairs are typically created from a single image through data augmentation while negative pairs are sampled randomly from the whole dataset [10, 11].
Many works have been proposed to improve this strategy by correcting the sampling bias or mining hard samples.
For negative pairs, some works [14, 19] eliminate the false negatives in randomly sampled pairs.
Some works construct hard negatives from existing samples [15, 24, 17, 16, 53], or adversarial perturbations [21, 18].
For positive pairs, Chen et al.
[11] and Tian et al.
[20] revisit common data augmentations and Reed et al.
[67] searches augmentation policies.
To construct hard positives, feature extrapolation [53] and neighbor discovery [24, 25, 26, 27, 68] are adopted.
Our goal is more than mining hard but to create optimal positives.
The most relevant works to ours are [69, 20] that share the same high level idea to construct positives with minimum sufficient information.
However, there are several fundamental differences.
The work [69] focuses on graph contrastive learning and the model design relies on domain knowledge, while our target is general visual representation learning without any supervision.
Besides proposing the InfoMin principle,
Tian et.
al [20] also conducted a view learning experiment to find optimal positives in data space.
However, the construction is under the color-channel splitting framework and requires the labels of downstream task.
In contrast, COP-Gen finds instance-specific optimal positives in the latent space of pretrained unconditional generative models, which is a fully unsupervised setting.
Besides, the objective in [70, 71] simply pulls positives together without explicitly contrasting with negative samples.
We check if it is possible to combine our contrastive optimal positive generation method with these non-contrastive frameworks in Appendix C.4.
Using Generative Models as Data Source.
Deep generative models have been developed to produce photo realistic images [36, 37, 72, 73], thus they can be used as great tools for generating desired image datasets.
Some works investigate intermediate GAN representations to construct part segmentation datasets [74, 75], while some treat pretrained GANs as black-box models and use them to augment data for robustness [76, 35] or ensembling [77].
The pretrained generative models can be data source alone.
Some works [78, 79] leverage class-conditional generative models to train a classifier, while GenRep [28] and this paper focus on a more general setting of visual representation learning.
Latent Space Navigation.
A key technique for using pretrained generative models to produce interested images is latent space navigation [32, 33, 80, 81, 46, 82, 83], which manipulates images by discovering interpretable directions in the latent space.
Different from these works, our goal is to learn not only the direction but also the magnitude of the trajectory.
In addition, the trajectory is instance-specific, i.e., the learned transformations applied to different images have different meanings.
Refer to Figure 3 for the visualization result.
Discussion
In this paper, we propose an instance-specific contrastive positive generation approach based on welltrained generative models.
Thanks to the attractive property of the latent space, we simply optimize a latent trajectory that reduces as much information as possible within a reasonable magnitude.
Theoretical analysis and experimental results demonstrate the effectiveness of our approach.
Limitations and Broader Impact.
Since this work uses synthetic data to perform contrastive learning, the performance naturally and partly depends on the quality of pretrained generative models.
Nevertheless, having seen that learning from synthetic dataset has outperformed learning from the real, which might impact representation learning community as a whole, we are inspired to further develop more proper generative models (e.g., to reduce the reliance on data space augmentations) as dataset sources to perform contrastive learning.
Besides, learning from synthetic data alleviates several societal issues of learning from real datasets, which might be private [84] or biased [85].
On the other hand, pretrained generative models may still leak [86] or inherit the biases from the training data [87].
These risks and impacts on representation learning need to be considered when leveraging pretrained generative models as the data source.
Take a class-imbalanced setting for example.
Although vanilla contrastive learning can learn more balanced feature space than its supervised counterpart since it does not use labels [88], it still could not immune to imbalance.
It is hypothesized that the high-frequency “classes” might dominate the learning process and leaves the low-frequency “classes” under-learned [89].
We here hypothesize that the learning of samples from low-frequency “classes” might benefit from COP-Gen generated positives.
The training process of COP-Gen has seen the overall imbalanced distribution thus has the potential to take it into consideration to create more proper positives.
Acknowledgements
This work is partially supported by National Key R&D Program of China (No. 2017YFA0700800), and National Natural Science Foundation of China (NSFC): 61976203, 61876171.
We thank Dr.
Xiaoyi Yin for valuable suggestions on the theoretical part of this work.
We also thank anonymous reviewers for their constructive feedback.