RényiCL: Contrastive Representation Learning with Skew Rényi Divergence
Abstract
Contrastive representation learning seeks to acquire useful representations by estimating the shared information between multiple views of data.
Here, the choice ofdata augmentation is sensitive to the quality of learned representations: as harderthe data augmentations are applied, the views share more task-relevant information,but also task-irrelevant one that can hinder the generalization capability of representation.
Motivated by this, we present a new robust contrastive learning scheme,coined RényiCL, which can effectively manage harder augmentations by utilizing
Rényi divergence.
Our method is built upon the variational lower bound of Rényidivergence, but a naïve usage of a variational method is impractical due to thelarge variance.
To tackle this challenge, we propose a novel contrastive objectivethat conducts variational estimation of a skew Rényi divergence and provide atheoretical guarantee on how variational estimation of skew divergence leads tostable training.
We show that Rényi contrastive learning objectives perform innatehard negative sampling and easy positive sampling simultaneously so that it canselectively learn useful features and ignore nuisance features.
Through experimentson ImageNet, we show that Rényi contrastive learning with stronger augmentationsoutperforms other self-supervised methods without extra regularization or computational overhead.
Moreover, we also validate our method on other domains suchas graph and tabular, showing empirical gain over other contrastive methods.
Introduction
Recently, many AI studies are enamored by the power of contrastive methods in learning usefulrepresentations without any human supervision, e g., image [1, 2, 3, 4, 5, 6], text [7, 8], audio [9, 10]and multimodal video [11, 12, 13].
The key components of contrastive learning can be summarizedinto two-fold: data augmentation and contrastive objectives.
The data augmentation generatesdifferent views of data where the views share information that is relevant to the downstream tasks.
The contrastive objective then enforces the representation to capture the shared information betweenthe views by bringing the views from the same data together and pushing the views from different dataaway in the representation space.
Therefore, the choice of data augmentation is critical in contrastivelearning, cf., see [3, 14, 15].
If the views share insufficient information, the representations cannotlearn sufficient features for downstream tasks.
On the other hand, if there is too much informationbetween the views, the views might share nuisance features that hamper the generalization.
In this paper, we propose a new contrastive learning objective using Rényi divergence [16], coined
RényiCL , which can effectively handle the case when the views share excessive information.
The
Rényi divergence is a family of probabilistic divergence that is known for being "less forgiving";they penalize more when two distributions have more differences.
Thus by modeling contrastive learning with Rényi divergence, we can learn better representations under the harder augmentations Specifically, we show that RényiCL intrinsically gives more weight to easy positives and hardnegatives, emphasizing the ability of RényiCL to focus more on informative views and less onnuisance views.
To implement RényiCL, one can consider variational lower bound of Rényi divergence [17].
However,we show that a variational lower bound of Rényi divergence does not suffice for contrastive learningdue to their large variance.
To tackle this challenge, we first made a key observation that the existingcontrastive objectives are a variational form of a skew KL divergence, and show that variationalestimation of skew divergence induces low variance property.
Inspired by this, we consider avariational lower bound of a skew Rényi divergence and use it to implement RényiCL.
Through experiments under ImageNet [18], RényiCL (with harder data augmentations) achieves75 .
3% linear evaluation accuracy with only 200 training epochs, 1 outperforming other self-supervisedmethods without using large batch size or extra computational overhead.
Moreover, we also showthat RényiCL achieves better performance on transfer learning and few-shot classification, comparedto baselines.
Finally, we demonstrate that RényiCL provides empirical gains over original contrastivelearning methods on various domains including images, graphs, and tabular.
Related works.
Since Chen et al.
[3] emphasized that the usage of data augmentation is crucial forcontrastive learning, a series of works proposed various contrastive learning frameworks that showedgreat empirical success on visual representation learning [2, 3, 4, 5, 6, 19].
Inspired by the empiricalsuccess, recent works focused on optimal view generation for both positive and negatives.
Tian etal.
[14] proposed the InfoMin principle for view generation, which states that the views should shareminimal information that is relevant to the downstream tasks.
Another line of research focuses onhard negative samples [20, 21, 22] to enhance the performance of contrastive learning. Especially,
Robinson et al.
[20] proposed to use importance sampling to learn with hard negative samples, whichcan expedite the contrastive learning.
Meanwhile, others are interested in searching for differentcontrastive objectives by modeling with probabilistic divergences other than KL divergence.
Hjelm etal.
[23] proposed DeepInfoMax which used Jensen-Shannon divergence for variational estimationand maximization of mutual information.
Zbontar et al.
[24] proposed Barlow-Twins, which is shownto be a Hilbert-Schmidt Independence Criterion that approximates a Maximum Mean discrepancymeasure between positives and negatives [25].
Our work is similar to that of Tsai et al.
[26], wherethey proposed relative predictive coding that uses skew χ 2 divergence to model contrastive learning.
Preliminaries
2.1 Background on contrastive learning
Let X be a set of natural data (without any data augmentation) and denote p be a density of datadistribution. To formalize data augmentation, let v = t ( x ) be a view from a data x ∈ X by stochasticdata transformation t : X → X with transition probability A ( v | x ) . Let V = t ( X ) be a randomvariable for views with distribution P V , then the density of a random variable V can be written asp ( v ) = E p ( x ) [ A ( v | x )] .
To perform multiview contrastive learning, let V ′ be another random variableof views from data transformation t ′ .
Then given a pair of views ( v, v ′ ) ∼ V × V ′ , the joint densitycan be written by p ( v, v ′ )
= E p ( x ) [ A ( v | x ) A ( v ′ | x )] . A pair of views ( v, v ′ ) is called a positive pairif they are augmented from same data, i.e., v ∼ A ( ·| x ) and v ′ ∼ A ( ·| x ) for some x ∈ X . If a pair ofviews ( v, v ′ ) are augmented from different data, we call it a negative pair and we have
[Equation]
| x )] = p ( v ) p ( v ′ ) . Throughout the paper, we denote v + for positive counterpart and v − for negative counterpart of v , i.e.( v, v + ) ∼ p ( v, v + ) and ( v, v − ) ∼ p ( v ) p ( v − ) .
Also, denote P V,V ′ and P V × P V ′ be distribution ofpositive and negative pairs respectively.
Contrastive representation learning. The goal of contrastive representation learning is to trainan encoder g : X → R d that is a useful representation for downstream tasks by enforcing g todiscriminate between positive pairs and negative pairs.
From our definition, the KL divergencebetween the positives and negatives coincides with the mutual information (MI) between the views:
[Equation]
Then the contrastive representation learning follows InfoMax principle [27, 28, 23], which can beformulated into following optimization problem:
[Equation]
here the inequality always hold due to the data-processing inequality, and equality holds iff g isinjective.
Generally, we use a neural network for encoder g (such as ResNet50 [29]).
If two viewsshare sufficient task-relevant information, the encoder learns useful information that is invariantagainst nuisance features. Therefore, the choice of data augmentation for view generation greatlyaffects the performance of contrastive learning [14].
Contrastive learning objectives. Since the MI cannot be computed unless the densities are known,many works resort to the variational lower bounds of MI [30] associated with neural networks [31].
Among them, the contrastive predictive coding (CPC) objective (also as known as InfoNCE)
[32] is alower bound to the MI that is widely used for contrastive representation learning [2, 4, 3].
Given Bbatch of views { v i } Bi =1 , assume we have a single positive v
+ i (i.e., v + i ∼ p ( v + i | v i ) ) and K negatives{ v − ij } Kj =1 (i.e., ( v i , v − ij ) ∼ p ( v i ) p ( v − ij ) ) for each v i .
Then the CPC objective is given by following:
[Equation]
Then we have I CPC ( f ) ≤
I ( V ; V ′ ) for any bounded measurable function f : X × X → R [32].
However, as I CPC ( f ) ≤ log( K + 1) , the CPC objective becomes high-bias MI estimator if the true
MI is larger than log( K + 1) .
To address this, Song et al.
[33] proposed α -CPC which can controlthe bias by inserting α ∈ [0 , 1] into CPC as following:
[Equation]
The α -CPC also admits a variational lower bound to the MI for any α ∈ [0 , 1] (equivalent to theoriginal CPC when α = 1 K +1 ), and we can have smaller bias by using small α .
Furthermore, theyintroduced α - Multi Label CPC (MLCPC) which is a tighter lower bound to the MI, defined by
[Equation]
One can easily see that ∀ α ∈ [0 , 1] , I ( α ) CPC ( f ) ≤ I ( α ) MLCPC ( f ) ≤ I ( V ; V ′ ) by using Jensen’s inequality.
2.2 Rényi divergence
Let P and Q be two probability distributions with densities p and q with respect to a base measuredx on domain X . Suppose P is absolutely continuous with respect to Q , denoted by P ≪ Q(i.e. , the Radon-Nikodym derivative dP/dQ exists).
Then the Rényi divergence [16] of orderγ ∈ (0 , 1) ∪ (1 , ∞ ) between P and Q is defined as [Equation]
Remark that Rényi divergence is intertwined with f -divergences, e.g., it becomes KL divergence asγ → 1 , and Rényi divergence with γ = 2 corresponds to χ 2 divergence. We refer to [34] for detailedcharacteristics of Rényi divergence. Variational representation of Rényi divergence. To perform contrastive learning with Rényidivergence, we consider variational methods similar to CPC for MI.
The following lemma introducesa variational representations of the Rényi divergence.
Lemma 2.1 ([17]) .
Let P and Q be two probability distributions with densities p and q with respectto a base measure dx on domain X . Let F be a set of bounded measurable functions on X . Suppose
P ≪ Q , then the Rényi divergence of order γ ∈ (0 , 1) ∪ (1 , ∞ ) admits following variational form:
[Equation]
where the optimum f ∗ ∈ F satisfies f ∗ ∝ log dPdQ .
Note that (3) is a generalization of Donsker-Varadhan (DV) [35] objective, which is a variationalrepresentation of KL divergence defined as following: [Equation] Note that the optimum f ∗ ∈ F of (4) satisfies f ∗ ∝ log dPdQ and (3) is equivalent to (4) if γ → 1 .  Rényi Contrastive Learning
3.1 Challenges in contrastive learning with Rényi divergence
Similar to the InfoMax principle, we use Rényi divergence to characterize the shared informationbetween the views. Thus, the goal of Rényi contrastive learning is to solve following:
[Equation]
To that end, one may consider the variational lower bound of Rényi divergence in (3).
However, thecontrastive learning with (3) is impractical due to the large variance.
That large-variance problem of(3) resembles that of the variational MI estimator observed in [30]. Song et al.
[36] showed that thevariational MI estimators such as DV objective, require a large number of samples proportional tothe exponential of ground-truth MI to achieve low variance.
In the following theorem, we provide asimilar result for the Rényi variational objective in (3).
Theorem 3.1.
Assume P ≪ Q , and Var Q [ dP/dQ ] < ∞ .
Let P m and Q n be the empiricaldistributions of m i.i.d samples from P and n i.i.d samples from Q , respectively.
Define
[Equation]
and assume we have optimum f ∗ ∝ log( dP/dQ ) .
Then ∀ γ > 1 , ∀ m ∈ N , we have
[Equation]
The proof is in the appendix.
Theorem 3.1 indicates that even though we achieve the optimal function,the variance of (3) could explode exponentially with the ground-truth MI.
In the appendix, we showthat even for a simple synthetic Gaussian dataset, (3) is not able to estimate Rényi divergence betweentwo correlated Gaussian distributions if the correlation coefficient is large.
Note that our resultgeneralizes that of [36] for KL divergence if we take γ → 1 .
In this section, we claim that the CPC and MLCPC objectives are variational lower bounds of askew KL divergence, and show that variational estimation of skew divergence induces a low varianceestimator.
For P and Q with P ≪ Q and for any α ∈ [0 , 1] , the α -skew KL divergence between Pand Q is defined by the KL divergence between P and the mixture αP + (1 − α ) Q :
[Equation]
Then the DV objective (4) for α -skew KL divergence can be written as
[Equation]
Note that the α -CPC objective in (1) can be written as following generalized form:
[Equation]
Then one can observe that (6) is equivalent to the expectation of the variational DV objective (5) ofα -skew KL divergence between p ( v + | v ) and p ( v − ) . Thus we have
[Equation]
showing that the α -CPC is a DV variational lower bound of α -skew KL divergence between the
P V ′ ,V and P V ′ ×
P V . Similarly, the generalized form of α -MLCPC objective (2) is given by
[Equation]
and in the appendix, we show that α -MLCPC also admits a variational form of α -skew KL divergence.
Low variance properties of CPC and MLCPC objectives.
Now we show that the variationalestimation of empirical skew divergence has low variance, explaining how CPC and MLCPC objectives become a low-variance estimator of MI.
The following theorem shows that the variance of thevariational estimator can be adjusted with the right choice of α .
Theorem 3.2.
Assume P ≪ Q and Var Q [ dP/dQ ] < ∞ .
Let P m and Q n be empirical distributionsof m i.i.d samples from P and n i.i.d samples from Q .
Then define
[Equation]
and assume that there is (cid:98) f ∈ F that |I ( α ) KL ( (cid:98) f ) − D ( α ) KL ( P m ∥ Q n ) | < ε f for some ε f > 0 .
Then for∀ α < 1 / 8 , the variance of estimator satisfies
[Equation]
for some constants c 1 , c 2 , c 3 , c 4 , c 5 > 0 independent of n and m .
The proof is in the appendix.
Theorem.
3.2 demonstrates that if we can find a sufficiently close criticfor empirical skew KL divergence, the variance of the estimator is asymptotically bounded unless wechoose large enough α = α m = ω ( m − 1 / 2 ) .
Also, the ε f can be further formalized with universalapproximation theorem [37], if we use a neural network for f .
3.3 Rényi contrastive learning objectives
Now we present Rényi contrastive learning (RényiCL) objectives based on variational estimation ofskew Rényi divergence. As shown in Section 3.2, we consider the variational lower bound of skew Rényi divergence to guarantee stable optimization.
Remark that the analogous version of Theorem 3.2can be achieved for Rényi divergence, which we refer to in the appendix.
For any α ∈ [0 , 1] andγ ∈ (0 , 1) ∪ (1 , ∞ ) , the α -skew Rényi divergence of order γ is defined as:
[Equation]
Then we define ( α, γ ) - Rényi MLCPC (RMLCPC) objective by using variational lower bound (3) forα -skew α -skew Rényi divergence between P V,V ′ and P V P V ′ :
[Equation]
which satisfies sup f I ( α,γ ) RMLCPC ( f )
=
R ( α ) γ
( P V,V ′ ∥
P V × P V ′ )
for any α ∈ [0 , 1] and γ ∈ (0 , 1) ∪ (1 , ∞ ) .
Note that the RMLCPC objective can be used for contrastive learning with multiple positive views.
For example, when using multi-crops data augmentation [38], instead of computing contrastiveobjective for each crop, we gather all positive pairs and negative pairs and compute only once for thefinal loss.
The PyTorch style pseudo-code for the RMLCPC objective can be found in the appendix.
Gradient analysis for Rényi contrastive learning.
We provide an analysis of how RényiCL canmanage harder views based on the gradient of contrastive objectives.
Without loss of generality, welet α = 0 .
Then note that the MLCPC objective becomes the DV objective in (4).
Let f θ be a neuralnetwork parameterized by θ , then the gradient of I MLCPC ( f θ ) is
[Equation]
where q θ ( v, v − ) ∝ p ( v ) p ( v − )
e f θ ( v,v − ) is a importance weights.
Thus the MLCPC objective isequivalent to following in terms of gradient:
[Equation]
which is same as contrastive divergence that is used for training energy-based models [39].
Thisshows that the original contrastive objective performs innate hard negative sampling with importanceweight q θ [20], as the gradient of negative pairs becomes larger as the value of f θ becomes larger.
The general argument for α -MLCPC and α -CPC is in the appendix.
On the other hand, for RMLCPCobjective, by letting α = 0 , and taking gradient with respect to θ , we have
[Equation]
where q θ ( v, v + ; γ − 1) ∝ p ( v, v + ) e ( γ − 1) f θ ( v,v + ) and q θ ( v, v − ; γ ) ∝ p ( v ) p ( v − ) e γf θ ( v,v − ) .
Thusthe RMLCPC objective is equivalent to following in terms of gradient: [Equation] v [Equation]
γ
[Equation]
[ f θ
[Equation]
v [Equation] v − )] When comparing MLCPC and RMLCPC, it occurs two differences:
• Hard negative sampling : the gradient weights more on harder negatives, i.e., ( v, v − ) with
high value of f θ ( v, v − ) , as γ increases [20].
• Easy positive sampling : the gradient weights more on easier positives,
i
e., ( v, v + ) with high value of f θ ( v, v + ) as γ ∈ (1 , ∞ ) increases.
Therefore, when there are positive views that share less information, the RMLCPC objective resistsupdating, and it regularizes the model to ignore task-irrelevant information.
The hard negativesampling with importance weight was proposed in [20], but our analysis shows that CPC and MLCPCconduct intrinsic hard negative sampling.
Also, the Rényi contrastive learning can control the level ofhard negative sampling by choosing the appropriate γ .
3.4 MI estimation with contrastive objectives
By interpreting contrastive objectives with a variational estimator of skew KL divergence, we canexplain the large bias of CPC and MLCPC.
Note that ∀ α ∈ (0 , 1) , [Equation]
from the convexity of KL divergence.
Therefore, CPC and MLCPC are intrinsically strictly-biased MIestimators, and similarly, the RMLCPC is a strictly-biased estimator of Rényi divergence.
However,we provide a general method to recover the MI by using critic f learned from CPC, MLCPC, and
RMLCPC objectives.
From the optimal condition of (4) and (3), the optimal critic f ∗ of α -CPC,α -MLCPC and ( α, γ ) -RMLCPC of any γ satisfies
[Equation]
where r ( v, v ′ )
= p ( v, v ′ ) /
( p ( v ) p ( v ′ )) is a density ratio.
Thus, we have exp( f ∗ ( v, v ′ )) /Z
= r ( v,v ′ ) αr ( v,v ′ ) +1 − α for some normalization constant Z > 0 , and we use Monte-Carlo estimation (cid:98) Z toapproximate Z .
Then the approximate density ratio ˆ r ( v, v ′ ) is given by ˆ r ( v, v ′ ) = [Equation]
v, [Equation] Lastly, we estimate the MI by the empirical mean of the log-density ratio:
[Equation]
In Section 4.3 and the appendix, we empirically show that (8) can effectively estimate the true MIwith low variance and low bias.
[Table]
[Table] [Table]
[Table]
BYOL
[Table]
[Table]
Experiments
4.1 ImageNet results
Setup.
For ImageNet [18] experiments, we use ResNet-50 [29] for encoder g .
We use MLP projectionhead with momentum encoder [4], which is updated by EMA, and we use predictor, following thepractice of [41, 6].
Then the critic f is implemented by the cosine similarity between the output ofmomentum encoder and base encoder with predictor, divided by temperature τ = 0 .
We maximize
RMLCPC objective defined in (7) with γ = 2 . 0 .
For optimization, we use LARS [42] optimizer witha batch size of 512, and train for 200 epochs.
Data augmentation.
Given the popular data augmentation baseline [41, 3], we further use RandAugment [43] and multi-crop augmentation [38] to implement harder data augmentation.
When usingmulti-crop, we gather all positives and negatives, then directly compute the RMLCPC objective,which differs from the original method in [38] (see the appendix for details).
Linear evaluation.
We follow the linear evaluation protocol, where we report the Top-1 ImageNetvalidation accuracy (%) of a linear classifier trained on the top of frozen features.
Table 1 comparesthe performance of different self-supervised methods using linear evaluation protocol.
First, RényiCLachieves better performance than other self-supervised methods with only 200 epochs of training.
Meanwhile, RényiCL enjoys better computational efficiency in that it does not require a large batchsize [3, 41, 38, 6] or extra memory queue [19].
Semi-supervised learning.
We evaluate the usefulness of the learned feature in a semi-supervisedsetting with 1% and 10% subsets of ImageNet dataset [3, 38].
The results are in Table 2.
We observethat RényiCL achieves the best performance comparable with SwAV [38] on fine-tuning with 10%subset, and slightly less than NNCLR [19] on fine-tuning with 1% subset, validating that our methodhas decent generalization capacity in low-shot learning scenarios.
Transfer learning.
We evaluate the learned representation through the performance of transfer learning on fine-grained object classification datasets by using linear evaluation.
We use CIFAR10/100 [44],
Food101 [45], Flowers [46], Cars [47], Aircraft [48], DTD [49], SUN397 [50], and VOC2007 [51].
In Table 3, we compare it with other self-supervised methods.
Note that RényiCL achieves the bestperformance in 7 out of 9 datasets, especially showing superior performance on SUN397 (3.4%) and
VOC2007 (5.2%) datasets.
[Table] [Table]
Comparison among self-supervised methods using hard augmentations.
To demonstrate theeffectiveness of RényiCL on learning with harder augmentations, we compare with InfoMin [14] and
CLSA [52] where they also use harder augmentations for contrastive representation learning.
Table 4demonstrates
the performance of representations learned with different self-supervised methodsby linear evaulation on ImageNet [18], Pets [53], Caltech101 [54], and few-shot classification on
FC100 [55], Caltech-UCSD Birds (CUB200) [43], and Plant Disease [56] datasets.
While RényiCLachieves slightly lower performance than InfoMin on ImageNet validation accuracy, it outperforms
InfoMin and CLSA on other transfer learning tasks with a large margin, which indicates the superiorityof RényiCL on learning better representations under the hard data augmentations.4.2 RényiCL for representation learning on various domains
Setup.
We perform ablation studies of RényiCL on various domains such as images, tabular, andgraph datasets.
For images, we consider CIFAR-10/100 [44] and ImageNet-100 [57], which is a 100class subset of ImageNet [18].
For the base data augmentation, we use a popular benchmark from [3],and for the harder augmentation, we further apply RandAugment [43] and RandomErasing [58].
For ImageNet-100, we use the default settings of [6] with ResNet-50 backbone with only changein learning objective.
For CIFAR-10 and CIFAR-100, we follow the settings in [3] with ResNet-18backbone adjusted for the CIFAR dataset.
We all use same α for CPC, MLCPC, RMLCPC, and useγ = 2 . 0 for RMLCPC.
For tabular experiments, we use Forest Cover Type (CovType) and Higgs Boson (Higgs) [59] datasetfrom UCI repository [60].
Due to its massive size, we consider a subset of 100K for the Higgsexperiments.
Since the tabular dataset has limited domain knowledge, we do not use any dataaugmentation for the baseline and when using data augmentation, we use random masking noise [61]for the Higgs dataset and random feature corruption [62] for the CovType dataset.
We follow thesettings of MoCo v3 [6] except that a 5-layer MLP is used for the backbone.
We use γ = 1 . 1 for
CovType and γ = 1 . 2 for Higgs.
For evaluation, we use linear evaluation for CovType and wecompute the linear regression layer by pseudo-inverse between the feature matrix and label matrix.
Lastly, we examine RényiCL on graph TUDataset [63], which contains numerous graph machinelearning benchmarks of different domains such as bioinformatics, molecules, and social network.
Wefollow the experimental setup of [15]: we use graph isomorphism network [64] backbone and usenode dropout, edge perturbation, attribute masking, and subgraph sampling for data augmentations.
For the graph experiments, we do not change the strength of data augmentation nor add new dataaugmentation.
The detailed hyper-parameters for training and evaluation for the whole dataset can befound in the appendix.
[Table]
Results.
Table 5 compares the Top-1 linear evaluation accuracy (%) of representations with differentcontrastive learning objectives.
We observe that RMLCPC obtains the best performance when usingharder data augmentations on all images and tabular datasets.
Also, RMLCPC has the largest gainon using harder augmentation, showing that the effectiveness of RényiCL is amplified when weuse stronger augmentation.
Table 6 compare the Top-1 linear evaluation accuracy (%) of graph
RényiCL with other graph contrastive learning methods.
Even without harder data augmentation,
RényiCL achieves the best performance in 7 out of 8 benchmarks.
Especially, RényiCL outperforms
JOAO [66], where the views are learned to boost the performance of graph contrastive learning.
4.3 MI estimation and view selection
288 MI estimation on synthetic Gaussian.
We first consider estimating mutual information betweentwo correlated synthetic Gaussian distributions following the procedure in [30].
In the appendix, weshow that by using the critique learned from each α -CPC, α -MLCPC, and ( α, γ ) -RMLCPC objective,we can estimate the true MI with low variance and low bias by using (8).
: RandomErasing [58], MC: multi-crops [
InfoMin principle and RényiCL.
By using (8),we plot the MI versus linear evaluation accuracy (%)by applying different data augmentations on the
ImageNet-100 dataset. In Figure 1, we observethat for harder data augmentations, the RMLCPCinduces lower MI than that of CPC and MLCPC,while showing better downstream performances.
This is because of the effect of easy positive andhard negative sampling explained in Section 3.3. Note that InfoMin [14] principle argues that theoptimal views must share sufficient and minimalinformation about the downstream tasks. Here, onecan observe that RényiCL intrinsically follows the InfoMin principle as it extracts minimal and sufficient information between the views.
Conclusion and Discussion
We propose Rényi Contrastive Learning (RényiCL) which utilizes Rényi divergence to deal withstronger data augmentations.
Since the variational lower bound of Rényi divergence is insufficientfor contrastive learning due to large variance, we introduce a variational lower bound of skew Rényidivergence, namely Rényi-MLCPC.
Indeed, we show that CPC and MLCPC are variational forms ofskew KL divergence, and provide theoretical analysis on how they achieve low variance.
Throughexperiments, we validate the effectiveness of RényiCL by using harder data augmentations.
Limitations and future works.
Even though RényiCL is beneficial when using harder dataaugmentations, it cannot identify which data augmentation is actually beneficial.
We think thatthe proposed scheme based on Rényi divergence could be useful for other tasks, e.g., informationbottleneck [67], which is an interesting direction to explore in the future.
Negative societal impacts.
Since contrastive learning often requires long epochs of training, it raisesenvironmental concerns, e.g. carbon generation.
Nevertheless, RényiCL is shown to be effective even
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section.•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes](d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes] The proofs of theoretical results are in the appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] We provide
code and instructions in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] They are in the appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] They are in the appendix.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] They are in the appendix.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[-C 
-]
[-Note that C ∗ 
-]
[-− 1 + α ≥ C 2 = 1 / 2 , and C 
-]
[-We train for 
-]
[N/A]