RényiCL: Contrastive Representation Learning with Skew Rényi Divergence
Abstract
Contrastive representation learning seeks to acquire useful representations by estimating the shared information between multiple views of data.
Here, the choice ofdata augmentation is sensitive to the quality of learned representations: as harderthe data augmentations are applied, the views share more task-relevant information,but also task-irrelevant one that can hinder the generalization capability of representation.
Motivated by this, we present a new robust contrastive learning scheme,coined RényiCL, which can effectively manage harder augmentations by utilizing
Rényi divergence.
Our method is built upon the variational lower bound of Rényidivergence, but a naïve usage of a variational method is impractical due to thelarge variance.
To tackle this challenge, we propose a novel contrastive objectivethat conducts variational estimation of a skew Rényi divergence and provide atheoretical guarantee on how variational estimation of skew divergence leads tostable training.
We show that Rényi contrastive learning objectives perform innatehard negative sampling and easy positive sampling simultaneously so that it canselectively learn useful features and ignore nuisance features.
Through experimentson ImageNet, we show that Rényi contrastive learning with stronger augmentationsoutperforms other self-supervised methods without extra regularization or computational overhead.
Moreover, we also validate our method on other domains suchas graph and tabular, showing empirical gain over other contrastive methods.
Introduction
Recently, many AI studies are enamored by the power of contrastive methods in learning usefulrepresentations without any human supervision, e g., image [1, 2, 3, 4, 5, 6], text [7, 8], audio [9, 10]and multimodal video [11, 12, 13].
The key components of contrastive learning can be summarizedinto two-fold: data augmentation and contrastive objectives.
The data augmentation generatesdifferent views of data where the views share information that is relevant to the downstream tasks.
The contrastive objective then enforces the representation to capture the shared information betweenthe views by bringing the views from the same data together and pushing the views from different dataaway in the representation space.
Therefore, the choice of data augmentation is critical in contrastivelearning, cf., see [3, 14, 15].
If the views share insufficient information, the representations cannotlearn sufficient features for downstream tasks.
On the other hand, if there is too much informationbetween the views, the views might share nuisance features that hamper the generalization.
In this paper, we propose a new contrastive learning objective using Rényi divergence [16], coined
RényiCL , which can effectively handle the case when the views share excessive information.
The
Rényi divergence is a generalization of KL divergence that is defined with an additional parametercalled an order.
As the order becomes higher, the Rényi divergence penalizes more when twodistributions mismatch.
Therefore, since the goal of contrastive representation learning is to train an encoder to discriminate between positives and negatives, we hypothesize that maximizing the Rényidivergence between positives and negatives could lead to more discriminative representations whenthe data augmentations are aggressive.
Furthermore, our theoretical analysis shows that RényiCLweighs importance on easy positives and hard negatives by using a higher order.
To implement RényiCL, one can consider variational lower bound of Rényi divergence [17].
However,we show that a variational lower bound of Rényi divergence does not suffice for contrastive learningdue to their large variance.
To tackle this challenge, we first made a key observation that the existingcontrastive objectives are a variational form of a skew KL divergence, and show that variationalestimation of skew divergence induces low variance property.
Inspired by this, we consider avariational lower bound of a skew Rényi divergence and use it to implement RényiCL.
Through experiments under ImageNet [18], RényiCL (with harder data augmentations) achieves76 .
2% linear evaluation accuracy with only 300 training epochs, 1 outperforming other self-supervisedmethods without using large batch size or extra computational overhead.
Moreover, we also showthat RényiCL achieves better performance on transfer learning and few-shot classification, comparedto baselines.
Finally, we demonstrate that RényiCL provides empirical gains over original contrastivelearning methods on various domains including images, graphs, and tabular datasets.
Related works.
Since Chen et al.
[3] emphasized that the usage of data augmentation is crucialfor contrastive learning, a series of works proposed various contrastive learning frameworks thatshowed great empirical success on visual representation learning [2, 3, 4, 5, 6, 19].
Inspired bythe empirical success, recent works focused on optimal view generation for both positive andnegatives.
Tian et al.
[14] proposed the InfoMin principle for view generation, which states that theviews should share minimal information that is relevant to the downstream tasks.
Another line ofresearch focuses on hard negative samples [20, 21, 22] to enhance the performance of contrastivelearning.
Especially, Robinson et al.
[20] proposed to use importance sampling to learn with hardnegative samples.
Meanwhile, others are interested in searching for different contrastive objectivesby modeling with probabilistic divergences other than KL divergence.
Hjelm et al.
[23] proposed
DeepInfoMax which used Jensen-Shannon divergence for variational estimation and maximizationof mutual information.
Zbontar et al.
[24] proposed Barlow-Twins, which is shown to be a HilbertSchmidt Independence Criterion that approximates a Maximum Mean discrepancy measure betweenpositives and negatives [25].
Our work is similar to that of Tsai et al.
[26], where they proposedrelative predictive coding that uses skew χ 2 divergence to model contrastive learning.
Preliminaries2.1 Rényi divergence
We first formally introduce Rényi divergence, which is a family of probability divergences includingthe popular Kullback-Leibler (KL) divergence as a special case.
Let P , Q be two probabilitydistributions such that P is absolutely continuous with respect to Q , denoted by P ≪ Q (i.e.,the Radon-Nikodym derivative dP/dQ exists).
Then, the Rényi divergence [16, 27] of order γ ∈(0 , 1) ∪ (1 , ∞ ) is defined by
[Equation]
It is intertwined with various f -divergences, e.g., Rényi divergence with γ → 1 and γ = 2 becomes
KL divergence D KL ( P ∥ Q ) and a monotonic transformation of χ 2 divergence, respectively.
Remarkthat the Rényi divergence with higher order γ , penalizes more when the probability mass of P doesnot overlap with Q [28].
This property allows Rényi divergence to utilized in various machinelearning tasks such as variational inference [29, 30] and training [31, 32] or evaluation of generativemodels [33].
2 Variational estimation of divergence and mutual information
We now review the variational estimators of probability divergences associated with neural networks.
Donsker-Varadhan (DV) [34] objective is a variational representation of KL divergence defined as follows:
[Equation]
where F is a set of bounded measurable functions on the support of P and Q .
Remark that theoptimum f ∗ ∈ F of (1) satisfies f ∗ ∝ log dPdQ .
One often considers variational estimator of mutual information where it is defined by the KLdivergence between the joint distribution and the product of marginal distributions.
Formally, giventwo random variables X, Y , let P XY be the joint distribution of X × Y , and P X P Y be the productof marginal distributions over X × Y .
Then, we call the pair of samples ( x, y ) ∼ P XY as a ‘positivepair’, and ( x, y ) ∼ P X P Y as a ‘negative pair’.
Then, the mutual information is defined by
[Equation]
Since the mutual information is intractable to compute unless the densities of X and Y are explicitlyknown, many works resort to optimize its variational lower bounds [35] associated with neuralnetworks [36].
Belghazi et al.
[36] used the DV objective (1) to compute mutual information, i.e.,
[Equation]
However, Song et al.
[37] showed that the above objective might have large variance unless one useslarge number of samples, and Tsai et al.
[26] also empirically evidenced that contrastive learningwith the DV objective suffers from training instability.
To address the issue, the contrastive predictive coding (CPC) objective (also as known as InfoNCE)
[38] is a popular choice for various practices including contrastive representation learning [2, 4, 3].
Given B batch of samples { x i } Bi =1 from X , assume we have a single positive y + i and
K negatives { y − ij } Kj =1 for each x i .
Then, the CPC objective is defined as follows:
[Equation]
where it is known that I CPC ( f ) ≤
I ( X ; Y ) for any bounded measurable function f [38].
However, as
I CPC ( f ) ≤ log( K + 1) for any f , the CPC objective becomes a high-bias estimator if the true value
I ( X ; Y ) is larger than log( K + 1) .
To address this, Poole et al.
[35] proposed α -CPC which controlsthe bias by inserting α ∈ [0 , 1] into CPC as follows:
[Equation]
The α -CPC objective also admits a variational lower bound to the mutual information I ( X ; Y ) forany α ∈ [0 , 1] (recovers the original CPC when α = 1 K +1 ), and it achieves smaller bias when usingsmaller value of α .
Furthermore, Song et al.
[39] proposed α - Multi Label CPC (MLCPC) whichprovides even a tighter lower bound to the mutual information defined as follows:
[Equation]
Variational Estimation of Skew Rényi Divergence
Similar to variational estimators of D KL ( P XY ∥
P X P Y ) in the previous section, we consider a variational estimator of Rényi divergence R γ ( P XY ∥
P X P Y ) , in particular, considering its application tocontrastive representation learning.
To that end, we first introduce the following known lemma whichstates variational representation of Rényi divergence similar to the DV objective in (1).
Lemma 3.1 ([17]) .
For distributions P , Q such that P ≪ Q , let F be a set of bounded measurablefunctions.
Then Rényi divergence of order γ ∈ (0 , 1) ∪ (1 , ∞ ) admits following variational form:
[Equation]
where the optimum f ∗ ∈ F satisfies f ∗ ∝ log dPdQ .
By continuity, I ( γ ) Rényi ( f ) in (4) is a generalization of the DV objective (1) by taking γ → 1 .
3.1 Challenges in variational Rényi divergence estimation
To perform contrastive representation learning, one can use the variational estimator (4) for
R γ ( P XY ∥
P X P Y ) , similarly as did for KL divergence.
However, the contrastive learning with(4) is impractical due to the large variance.
In Appendix C.2, we show that even for a simple synthetic
Gaussian dataset, (4) is not able to estimate Rényi divergence between two correlated Gaussiandistributions if the correlation coefficient is large.
The large-variance issue of (4) resembles thatof the variational mutual information estimator well observed in the literature [26, 35, 37].
In thefollowing theorem, we provide an analogous result for the Rényi variational objective (4).
Theorem 3.1.
Assume P ≪ Q , and Var Q [ dP/dQ ] < ∞ .
Let P m and Q n be the empiricaldistributions of m i.i.d samples from P and n i.i.d samples from Q , respectively.
Define
[Equation]
and assume we have f ∗ ∝ log( dP/dQ ) .
Then ∀ γ > 1 , ∀ m ∈ N , we have
[Equation]
n
[Equation]
The proof of Theorem 3.1 is in Appendix A.1.
Theorem 3.1 implies that even though one achievesthe optimal function for variational estimation, the variance of (4) could explode exponentially withrespect to the ground-truth mutual information.
This result is coherent with [40], which identified theproblems in variational estimation of mutual information.
On the other hand, we recall that CPC and MLCPC are empirically shown to be low varianceestimators [35, 39].
Then, the natural question arises: what makes CPC and MLCPC fundamentallydifferent from the DV objective?
In the following section, we answer to this question by showing thatthe CPC and MLCPC objectives are variational lower bounds of a skew KL divergence, and providea theoretical evidence that variational estimators of skew divergence can have low variance.
Thisinsight will be used later for designing a low variance estimator for the desired Rényi divergence.
3.2 Low variance of variational skew divergence estimators
We first define α -skew KL divergence.
For distributions P , Q with P ≪ Q and for any α ∈ [0 , 1] ,the α -skew KL divergence between P and Q is defined by the KL divergence between P and themixture αP + (1 − α ) Q :
[Equation]
One can see that the DV objective (1) for α -skew KL divergence can be written as following:
[Equation]
We claim that (5) is equivalent to α -CPC and α -MLCPC.
Remark that the generalized form of α -CPCobjective (2) can be written as follows:
[Equation]
Then, one can observe that (6) is equivalent to the expectation of the variational DV objective (5) ofα -skew KL divergence between P Y | X and P Y over P X , which shows that α -CPC is a DV objectiveof α -skew KL divergence between P XY and P X P Y :
[Equation]
Similarly, the generalized form of α -MLCPC objective (3) is given by
[Equation]
which also becomes the DV objective for α -skew KL divergence D ( α ) KL ( P XY
∥
P X P Y ) .
Further, we formally show that the variational estimation of empirical skew divergence has lowvariance, explaining why CPC and MLCPC objectives become low-variance estimators of the mutualinformation.
The following theorem shows that the variance of the variational estimator can beadjusted with the right choice of α .
Theorem 3.2.
Assume P ≪ Q and Var Q [ dP/dQ ] < ∞ .
Let P m and Q n be empirical distributionsof m i.i.d samples from P and n i.i.d samples from Q .
Then define
[Equation]
and assume that there is ˆ f ∈ F that |I ( α ) KL ( ˆ f ) − D ( α ) KL ( P m ∥ Q n ) | < ε f for some ε f > 0 .
Then for∀ α < 1 / 8 , the variance of estimator satisfies
[Equation]
for some constants c 1 , c 3 , c 4 , c 5 > 0 that are independent of n , m , α , and D KL ( P ∥ Q ) , and c 2 ( α )satisfies c 2 ( α )
= min { 1 α , χ 2 ( P ∥ Q ) 1 − α } , where χ 2 ( P ∥ Q ) =
E Q
[( dP/dQ ) 2 ] .
The proof is in Appendix A.2.
Since we have χ 2 ( P ∥ Q ) ≥ e D KL ( P ∥ Q ) − 1 , if α is too small, thebound in Theorem 3.2 is loose as in Theorem 3.1.
Therefore, Theorem 3.2 implies that one shoulduse sufficiently large α to achieve low variance.
Theorem.
3.2 demonstrates that if one canfind a sufficiently close critic for empirical skew KL divergence, the variance of the estimator isasymptotically bounded unless we choose large enough α = α n = ω ( n − 1 / 2 ) .
In addition, we remarkthat ε f can be further formalized with universal approximation theorem [41], if we use a neuralnetwork for f .
3.3 Variational estimation of skew Rényi divergence
Inspired by the theoretical guarantee that skew divergence can achieve bounded variance, we nowpresent a variational estimator of skew Rényi divergence.
Remark that the analogous version of
Theorem 3.2 can be achieved for Rényi divergence (see Appendix A.3).
For any α ∈ [0 , 1] andγ ∈ (0 , 1) ∪ (1 , ∞ ) , the α -skew Rényi divergence of order γ is defined as:
[Equation]
Then, we define ( α, γ ) - Rényi Multi-Label CPC (RMLCPC) objective by using the variational lowerbound (4) for α -skew Rényi divergence between P XY and P X P Y :
[Equation]
which satisfies sup f I ( α,γ ) RMLCPC ( f )
=
R ( α
[Equation]
γ ( P XY ∥ P X P Y ) for any α ∈ [0 , 1] and γ ∈ (0 , 1) ∪ (1 , ∞ ) .
Note that the RMLCPC objective can be used for contrastive learning with multiple positive views.
For example, when using multi-crops data augmentation [42], instead of computing contrastiveobjective for each crop, we gather all positive pairs and negative pairs and compute only once for thefinal loss.
The PyTorch style pseudo-code for the RMLCPC objective is in Algorithm 1.
Rényi Contrastive Representation Learning
In this section, we present Rényi contrastive learning (RényiCL) where we use the RMLCPC objectivefor contrastive representation learning.
As we discussed earlier, the Rényi divergence penalizes morewhen two distributions differ.
Thus, when using harder data augmentations in contrastive learning,one can expect that RényiCL can learn more discriminative representation.
4.1 Rényi contrastive representation learning
We begin with backgrounds for contrastive representation learning.
Given a dataset X , the goal ofrepresentation learning is to train an encoder g : X → R d that is a useful feature of X .
Especially,contrastive representation learning enforces g to discriminate between positive pairs and negativepairs [2], where positive pairs are generated by applying data augmentation on the same data andnegative pairs are augmented data from different source data.
Formally, let V, V ′ be random variablesfor augmented views from dataset X .
Then denote P V V ′
be the joint distribution of a positive pairand P V P V ′
be the distribution of negative pairs.
For the sake of brevity, we denote ( v, v + ) ∼ P V V ′be a positive pair, and ( v, v − ) ∼ P V P V ′ be a negative pair.
The InfoMax principle [23] for representation learning aims to find g that preserves the maximalmutual information between V and V ′ by following:
[Equation]
i.e., finds a neural encoder g which discriminates between positive and negative pairs as much aspossible.
Here, α -CPC or α -MLCPC are used for plug-in variational estimators of mutual information,where the objective becomes following:
[Equation]
where I ( α ) MLCPC ( f, g ) is a variational estimator of D ( α ) KL
( P g ( V ) g ( V ′ )
∥
P g ( V ) P g ( V ′ ) ) .
Then, we present Rényi contrastive representation learning, which considers the following optimization problem by using skew Rényi divergence:
[Equation]
where I ( α,γ ) RMLCPC ( f, g ) is a variational estimator of R ( α ) γ ( P g
[Equation]
V ) g
[Equation]
V ′ )
∥ P g [Equation] ) P g
[Equation]
V
[Equation]
4.2 Gradient analysis for RényiCL
In this section, we provide an analysis on the effect of γ in RényiCL based on the gradient ofcontrastive objectives.
For simplicity, we consider the case when α = 0 , and we provide generalanalysis for α > 0 in Appendix B.1.
Let f θ be a neural network parameterized by θ .
By using thereparametrization trick, the gradient of I MLCPC ( f θ ) becomes
[Equation]
where q θ ( v, v − )
∝ e f θ ( v,v − ) is a self-normalized importance weights [43].
Thus, the MLCPCobjective is equivalent to following in terms of gradient [44]:
[Equation]
where sg is a stop-gradient operator.
This shows that the original contrastive objective performsinnate hard negative sampling with importance weight q θ [20], as the gradient of negative pairsbecomes larger as the value of f θ becomes larger.
On the other hand, for RMLCPC objective, byletting α = 0 , and taking gradient with respect to θ , we have
[Equation]
where q θ ( v, v + ; γ − 1) ∝ e ( γ − 1)
f θ ( v,v + ) and q θ ( v, v − ; γ ) ∝ e γf θ ( v,v − ) .
Hence, the RMLCPCobjective is equivalent to the following in terms of gradient:
[Equation]
When comparing MLCPC and RMLCPC, it occurs two differences:
• Hard negative sampling : the gradient weights more on harder negatives, i.e., ( v, v − ) with
high value of f θ ( v, v − ) , as γ increases [20].
• Easy positive sampling : the gradient weights more on easier positives,
i
e., ( v, v + ) withhigh value of f θ ( v, v + ) as γ ∈ (1 , ∞ ) increases.
Therefore, when there are positive views that share less information, the RMLCPC objective resistsupdating, and it regularizes the model to ignore task-irrelevant information.
The hard negativesampling with importance weight was proposed in [20], but our analysis shows that CPC and MLCPCconduct intrinsic hard negative sampling.
Also, the Rényi contrastive learning can control the level ofhard negative sampling by choosing the appropriate γ .
Lastly, remark that our analysis is given forα = 0 .
In practice, we use nonzero α
so it affects contrastive learning.
In Appendix B.1, we showthat it requires sufficiently small values of α to have the effect of easy positive sampling for harderdata augmentations. [Table] [Table] [Table]
[Table] [Table] SimCLR [3] [Table]
BYOL [45]
[Table]
NNCLR [
[Table]
[Table]
Experiments
5.1 RényiCL for visual representation learning on ImageNet
Setup.
For ImageNet [18] experiments, we use ResNet-50 [47] for encoder g .
We use MLP projectionhead with momentum encoder [4], which is updated by EMA, and we use predictor, following thepractice of [45, 6].
Then the critic f is implemented by the cosine similarity between the output ofmomentum encoder and base encoder with predictor, divided by temperature τ = 0 .
We maximize
RMLCPC objective with γ = 2 . 0 .
Data augmentation.
Given the popular data augmentation baseline [45, 3], we further use RandAugment [48] and multi-crop augmentation [42] to implement harder data augmentation.
When usingmulti-crop, we gather all positives and negatives, then directly compute the RMLCPC objective,which differs from the original method in [42] (see 1 in Appendix B.1).
Linear evaluation.
We follow the linear evaluation protocol, where we report the Top-1 ImageNetvalidation accuracy (%) of a linear classifier trained on the top of frozen features.
Table 1 comparesthe performance of different self-supervised methods using linear evaluation protocol.
First, RényiCLoutperforms other self-supervised methods with a large margin by only training for 300 epochs.
Meanwhile, RényiCL enjoys better computational efficiency in that it does not require a large batchsize [3, 45, 42, 6] or extra memory queue [19].
Semi-supervised learning.
We evaluate the usefulness of the learned feature in a semi-supervisedsetting with 1% and 10% subsets of ImageNet dataset [3, 42].
The results are in Table 2.
We observethat RényiCL achieves the best performance on fine-tuning with 10% subset, and runner-up onfine-tuning with 1% subset, showing the generalization capacity in low-shot learning scenarios.
Transfer learning.
We evaluate the learned representation through the performance of transfer learning on fine-grained object classification datasets by using linear evaluation.
We use CIFAR10/100 [49],
Food101 [50], Flowers [51], Cars [52], Aircraft [53], DTD [54], SUN397 [55], and VOC2007 [56].
In Table 3, we compare it with other self-supervised methods.
Note that RényiCL achieves the bestperformance in 7 out of 9 datasets, especially showing superior performance on SUN397 (3.4%) and
VOC2007 (5.2%) datasets.
[Table]
[Table]
[Table]
Comparison among self-supervised methods using hard augmentations.
To demonstrate theeffectiveness of RényiCL on learning with harder augmentations, we compare with InfoMin [14] and
CLSA [57] where they also use harder augmentations for contrastive representation learning.
Table 4demonstrates
the performance of representations learned with different self-supervised methodsby linear evaulation on ImageNet [18], Pets [58], Caltech101 [59], and few-shot classification on
FC100 [60], Caltech-UCSD Birds (CUB200) [48], and Plant Disease [61] datasets.
While RényiCLachieves slightly lower performance than InfoMin on ImageNet validation accuracy, it outperforms
InfoMin and CLSA on other transfer learning tasks with a large margin, which indicates the superiorityof RényiCL on learning better representations under the hard data augmentations.
5.2 RényiCL
for representation learning on various domains
Setup.
We perform ablation studies of RényiCL on various domains such as images, tabular, andgraph datasets.
For images, we consider CIFAR-10/100 [49] and ImageNet-100 [62], which is a 100class subset of ImageNet [18].
For the base data augmentation, we use a popular benchmark from [3],and for the harder augmentation, we further apply RandAugment [48] and RandomErasing [63].
For ImageNet-100, we use the default settings of [6] with ResNet-50 backbone with only changein learning objective.
For CIFAR-10 and CIFAR-100, we follow the settings in [3] with ResNet-18backbone adjusted for the CIFAR dataset.
We all use same α for CPC, MLCPC, RMLCPC, and useγ = 2 . 0 for RMLCPC.
For tabular experiments, we use Forest Cover Type (CovType) and Higgs Boson (Higgs) [64] datasetfrom UCI repository [65].
Due to its massive size, we consider a subset of 100K for the Higgsexperiments.
Since the tabular dataset has limited domain knowledge, we do not use any dataaugmentation for the baseline and when using data augmentation, we use random masking noise [66]for the Higgs dataset and random feature corruption [67] for the CovType dataset.
We follow thesettings of MoCo v3 [6] except that a 5-layer MLP is used for the backbone.
We use γ = 1 . 1 for
CovType and γ = 1 . 2 for Higgs.
For evaluation, we use linear evaluation for CovType and wecompute the linear regression layer by pseudo-inverse between the feature matrix and label matrix.
Lastly, we examine RényiCL on graph TUDataset [68], which contains numerous graph machinelearning benchmarks of different domains such as bioinformatics, molecules, and social network.
Wefollow the experimental setup of [15]: we use graph isomorphism network [69] backbone and usenode dropout, edge perturbation, attribute masking, and subgraph sampling for data augmentations.
For the graph experiments, we do not change the strength of data augmentation nor add new dataaugmentation.
The detailed hyper-parameters for training and evaluation for the whole dataset can befound in Appendix B.3. [Table]
Results.
Table 5 compares the Top-1 linear evaluation accuracy (%) of representations with differentcontrastive learning objectives.
We observe that RMLCPC obtains the best performance when usingharder data augmentations on all images and tabular datasets.
Also, RMLCPC has the largest gainon using harder augmentation, showing that the effectiveness of RényiCL is amplified when weuse stronger augmentation.
Table 6 compare the Top-1 linear evaluation accuracy (%) of graph
RényiCL with other graph contrastive learning methods.
Even without harder data augmentation,
RényiCL achieves the best performance in 7 out of 8 benchmarks.
Especially, RényiCL outperforms
JOAO [71], where the views are learned to boost the performance of graph contrastive learning.
5.3 Mutual information estimation and view selection
Remark that CPC and MLCPC are intrinsically high-bias mutual information estimator as α -skew
KL divergence is strictly smaller than original KL divergence for 0 < α < 1 , i.e.,
[Equation]
∥ Q ) , from the convexity of KL divergence.
Similarly, RMLCPC is also a strictly-biased estimator of
Rényi divergence.
However, one can recover the original mutual information by using the optimalcondition of function f from all CPC, MLCPC, and RMLCPC objectives [26].
Here, we describethe sketch and present detailed method in Appendix C.1.
From the optimality condition of (1)and (4), the optimal critic f ∗ of α -CPC, α -MLCPC and ( α, γ ) -RMLCPC of any γ satisfies f ∗ ∝log dP XY αdP XY +(1 − α ) dP X dP Y , then we Monte-Carlo method [43] to approximate log-normalizationconstant, and use it to approximate the log-density ratio ˆ r = log dP XY dP X dP Y .
Then we estimate themutual information by ˆ I ( X ; Y )
= 1 B (cid:80) Bi
=1 log ˆ r
( x i , y i ) , where ( x i , y i ) ∼ P XY are positive pairs.
Mutual information estimation of synthetic Gaussian.
Following [35, 39], we conduct experiments on estimating the mutual information of multivariate Gaussian distributions.
In Appendix C.2,we show that while α -CPC, α -MLCPC, and ( α, γ ) -RMLCPC objectives are high biased objectives,one can achieve low bias, low variance estimator by using the approximation method above.
Conclusion and Discussion
We propose Rényi Contrastive Learning (RényiCL) which utilizes Rényi divergence to deal withstronger data augmentations.
Since the variational lower bound of Rényi divergence is insufficientfor contrastive learning due to large variance, we introduce a variational lower bound of skew Rényidivergence, namely Rényi-MLCPC.
Indeed, we show that CPC and MLCPC are variational forms ofskew KL divergence, and provide theoretical analysis on how they achieve low variance.
Throughexperiments, we validate the effectiveness of RényiCL by using harder data augmentations.
Limitations and future works.
While RényiCL is beneficial when using harder data augmentations,we did not identified the optimal data augmentation strategy.
One can use policy search method ondata augmentation [72] with our RényiCL similar to that of [73].
Also, we think that the proposedscheme based on Rényi divergence could be useful for other tasks, e.g., information bottleneck [74].
Lastly, many works focused on the diverse perspectives of contrastive learning [44, 75, 76], wherethose approaches could be complementary to our work.
We leave them for future works.
Negative societal impacts.
Since contrastive learning often requires long epochs of training, it raisesenvironmental concerns, e.g. carbon generation.
Nevertheless, RényiCL is shown to be effective evenunder a smaller number of epochs, compared to existing schemes.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section.
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes] The proofs of theoretical results are in the appendix.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] We provide
code and instructions in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] They are in the appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] They are in the appendix.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] They are in the appendix.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Proofs
A.1 Proof of Theorem 3.1
Proof.
Without loss of generality, assume e f ∗ =
dP/dQ .
Then we have
[Equation]
since E Q [ dP/dQ ] = 1 .
Then the variance of n i.i.d random variable gives us
[Equation]
For a sequence of random variable X n defined with respect to distribution Q , and assume we have
[Equation]
X n = E X , then the following comes from the delta method:
[Equation]
Thus, by applying f ( t ) =
t γ and E Q [ dP/dQ ] = 1 gives us
[Equation]
(9) where the last two equations are from Jensen’s inequality and the definition of KL divergence.
Now,by applying f ( x ) = log x in (8) , we have
[Equation]
from the definition of Rényi divergence and (9).
Thus, we have the following:
[Equation]
A.2 Proof of Theorem 3.2
In this section, we state and define the regularity assumptions to derive the asymptotic upper boundsfor the variance of α -skew KL divergence in Theorem 3.2.
We first review the definition of fdivergence.
Let f : (0 , ∞ )
→ R be a convex function with f (1) = 0 .
Let P and Q be distributionswith respect to a base measure dx on domain X , and assume P ≪ Q .
Then the f -divergencegenerated by f is defined by
[Equation]
Note that for any c ∈ R , D f c ( P ∥ Q ) =
D f ( P ∥ Q ) , where f c ( t ) =
f ( t )+ c
( t − 1) .
Hence, w.l.o.g, weassume f ( t ) ≥ 0 for all t ∈ (0 , ∞ ) .
The conjugate of f is a function f ∗ : (0 , ∞ ) → [0 , ∞ ) definedby f ∗ ( t )
= tf (1 /t ) , where f ∗ (0) =
lim t → 0 + f ∗ ( t ) for convenience.
The conjugate f ∗ of convexfunction f is also convex.
Also, one can see that f ∗ (1) = 0 and f ∗ ( t ) ≥ 0 for all t ∈ (0 , ∞ ) , thus itinduces another divergence D f ∗ .
The conjugate divergence D f ∗ satisfies D f ∗ ( P ∥ Q ) =
D f ( Q ∥ P ) .
The KL divergence is a f -divergence generated by f ( t ) =
t log t − t + 1 , and the α -skew KLdivergence is a f -divergence generated by
[Equation]
From [77], we state following
regularity assumptions on the functions f and f ∗ .
Assumption A.1.
The generator f is twice continuously
differentiable with
f ′ (
=
Moreover
(A1 )
We have C 0 :
= f (0) < ∞ and C ∗ 0 :
= f ∗ (0) < ∞ .(A2 )
There exist constants C 1 , C ∗ 1 < ∞ such that for any t ∈ (0 , 1) , we have,| f ′ ( t ) | ≤ C 1 max { 1 , log(1 /t ) } , and | ( f ∗ ) ′
( t ) | ≤ C ∗ 1 max { 1 , log(1 /t ) } .
(A3 )
There exist constants C 2 , C ∗ 2 < ∞ such that for every t ∈ (0 , ∞ ) , we have,t 2 f ′′ ( t ) ≤ C 2 , and t 2( f ∗ ) ′′ ( t ) ≤ C ∗ 2 .
We refer authors [77] for the detailed discussion on Assumption A.1.
Then one can observe that KLdivergence does not satisfy Assumption A.1, because KL divergence can be unbounded.
On the otherhand, the α -skew KL divergence satisfies Assumption A.1 from following proposition.
Proposition A.1 ([77]) .
The α -skew KL divergence generated by f ( α ) satisfies Assumption A.1 with
C
= 1 − α,
[Equation]
For the general f -divergences which satisfy Assumption A.1, the following concentration boundholds.
Proposition A.2 ([77]) .
Assume f satisfies Assumption A.1, and let P and Q be two distributionswith P ≪ Q .
Let P m be m i.i.d samples from P and Q n be n i.i.d samples from Q .
Then thef -divergence
D
f satisfies following:
[Equation]
where c 1 = max { C ∗ 0 , C 2 } and c 2 = max { C 0 , C ∗ 2 } .
Thus, the following lemma derives a concentration bound for the α -skew KL divergence by pluggingthe constants in Proposition A.1 to Proposition A.2.
Lemma A.1.
For α < 18 , the following holds:
[Equation]
Proof.
Note that C ∗
= log(1 /α )
− 1 + α ≥ C 2 = 1 / 2 , and C
= 1 − α ≤ C ∗ 2 = 1 − α 8 α for α < 18 .
Then the concentration bound follows from Proposition A.2.
Lastly, we present following upper bound on the bias of empirical estimator of KL divergence:
Proposition A.3 ([78]) .
Suppose P ≪ Q , and Var [ dP/dQ ] < ∞ .
Then we have
[Equation]
From proposition A.3, we have
[Equation]
where χ 2 ( P ∥ αP + (1 − α ) Q )
= (cid:82)
[Equation]
≤
[Equation]
, or
[Equation]
1 1 − α (cid:82) d 2 P dQ = χ 2 ( P ∥ Q ) 1 − α .
Therefore, we have
[Equation]
Now we present the proof of Theorem 3.2 in the main paper.
Proof of Theorem 3.2.
Define
[Equation]
From Proposition A.3, we have B 2 ≤ c 2 min { n,m } for some constant c ( α ) > 0 , and from the assumption,we have B 3 ≤ ε f .
By using triangle inequality twice, we have
[Equation]
Therefore, it follows that
[Equation]
where the last inequality comes from Proposition A.2.
Since the following holds for any random variable X ,
[Equation]
we have
[Equation]
which proves our result.
A.3 Proof Sketch for Rényi Divergence
In this section, we provide a proof sketch for the upper bound for the variance of RMLCPC objective.
We use similar proof technique that we used in previous section.
Recall that the Rényi Divergence isrelated to γ -divergence 2 D γ , a f -divergence generated by f ( γ ) ( t )
=
t γ − 1 γ ( γ − 1) .
The KL divergence isrecovered from γ -divergence if we let γ → 1 .
Then we have following equation for Rényi divergenceand γ -divergence:
[Equation]
One can observe that f ( x ) =
[Equation]
− 1) log( γ ( γ − 1) x + 1) is of 1-Lipschitz function for any γ ̸ = 0 , 1 .
Thus, if we can derive concentration bound for α -skew γ -divergence, we can derive concentrationbound for Rényi divergence.
Here, we provide an example when γ = 2 .
Remark that when γ = 2 , the γ -divergence is equivalent toχ 2 -divergence, which is generated by f ( t ) =
( t − 1) 2 .
Then the α -skew χ 2 -divergence is generatedby following generators:
[Equation]
Then from [77], the α -skew χ 2 -divergence satisfies Assumption A.1 with
[Equation]
Then by using Proposition A.2, for α < min { 23 √ 3 , 1 − 23 √ 3 } , we have
[Equation]
( P m
[Equation]
n
[Equation]
E
[Equation]
P
[Equation]
[Equation]
≤
[Equation]
exp
[Equation]
(cid:33) , for some constant c 1 , c 2 , c 3 , c 4 > 0 , then from the Lipschitz continuity, we have
P [ |
[Equation]
( α )
[Equation]
P m
[Equation]
Thus, by deriving the asymptotic bound for the bias | E [ R ( α ) 2 ( P m ∥ Q n )] − R ( α ) 2 ( P ∥ Q ) | , and if the
Rényi variational objective has sufficiently small error, we can derive upper bound for the variance of
RMLCPC objective.
B Details for Experiments
B.1 Implementation
As explained in Section 3.3, we derive the full equivalent form of MLCPC and RMLCPC withnon-zero α .
Recall that the α -MLCPC objective with neural network f θ is given by
[Equation]
10) then the gradient of (10) with respect to parameter θ is given by
[Equation]
where
[Equation]
are importance weights for positive and negative pairs in the second term of (10).
For the generalized( α, γ ) -RMLCPC, recall that the RMLCPC with neural network f θ is given by
[Equation]
Then the gradient of (11) with respect to θ is given by
[Equation]
where
[Equation]
are importance weights for positive and negative pairs for the first and second term in (11).
Asdiscussed in Section 3.3, regardless of the insertion of parameter α , MLCPC and RMLCPC explicitlyconduct hard-negative sampling, and especially RMLCPC, conducts easy-positive sampling.
The PyTorch style pseudo-code for our implementation on RényiCL is demonstrated in Algorithm 1.
In our default implementation, we use multi-crop data augmentation [42] with two global views andmultiple local views.
In Algorithm 1, we implement the negative of ( α, γ ) -RMLCPC, and α -MLCPCwhich is equivalent to the case when γ → 1 for RMLCPC, for the contrastive losses.
Note that ourpseudo-code for contrastive objectives are based on the approach that uses importance weights.
Effect of α in RényiCL.
One can observe that the gradient of positive pairs in the second termis multiplied by α , thus the gradient of positive pairs is affected by α and negative pairs.
However,since we use a small value of α , the gradient of the second term can be ignored in practice.
To verifyour claim, we experimented with different values of α on both basic and hard data augmentations:
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
In Table 7, one can observe that as α becomes smaller, the gap between using hard augmentation andthe base augmentation becomes larger.
Thus, there is a tradeoff in the choice of α : the α should belarge enough to evade large variance (Thm 3.2), but small α is preferred to have the effect of easypositive sampling.
B.2 Further ablation studies
In this section, we provide further ablation studies to validate the effectiveness of RényiCL.
[Table] [Table]
[Table]
Fair comparison with other self-supervised methods.
To see the effectiveness of RényiCL, weconduct additional experiments for fair comparison with other self-supervised methods.
MoCo v3 [6]is a state-of-the-art method in contrastive self-supervised learning.
For fair comparison of RényiCLwith MoCo v3, we apply harder data augmentations such as RandAugment [48] and multi-crops [42]on MoCo v3.
We first reimplemented MoCo v3 with smaller batch size (original 4096 to 1024) andin our setting, we achieved 69.6%, which is better than their original reports (68.9%).
Then we useharder data augmentations RandAugment (RA), RandomErasing (RE), and multi-crops (MC) whentraining MoCo v3.
Second, we compare RényiCL with DINO [79].
Note that DINO uses defaultmulti-crops, therefore we further applied RandAugment (RA) and RandomErasing (RE) for a faircomparison.
For each model, we train for 100 epochs.
The results are shown in Table 8.
One can observe that while MoCo v3 and DINO benefit from usingharder data augmentation, RenyiCL shows the best performance when using harder data augmentation.
Also, compared to the Base data augmentation, RényiCL attains the most gain by using harder dataaugmentation.
Robustness to data augmentations.
To see the robustness of RényiCL on the choice of dataaugmentation, we conduct additional ablation studies on the robustness of RényiCL on various dataaugmentation schemes.
We first show that RényiCL is more robust to the noisy data augmentation, assimilar to that in [80], we apply Noisy Crop, which additionally perform RandomResizedCrop ofsize 0.2.
Then the generated views might contain only nuisance information (e.g. background), thusit can hurt the generalization of contrastive learning.
We train each model with CPC, MLCPC, and
RMLCPC objectives and data augmentations with and without Noisy Crop.
We experimented on
CIFAR-10, CIFAR-100, and ImageNet-100 datasets and the results are shown in Table 9.
One canobserve that while the performance of using CPC and MLCPC degrades as we apply the noisy crop,
RMLCPC conversely improves the performance.
Hence, RMLCPC is robust in the choice of dataaugmentation.
Second, we experiment when there is limited data augmentation.
For each CIFAR-10, CIFAR-100,and STL-10 dataset, we remove data augmentations such as color jittering and grayscale and onlyapply RandomResizedCrop and HorizontalFlip.
The results are in Table 10.
Note that RMLCPCachieves the best performance among CPC and MLCPC, when we do not apply data augmentation.
Therefore we show that the RMLCPC objective is not only robust to the hard data augmentation butis also robust when there is limited data augmentation.
B.3 ImageNet experiments
Data augmentation.
For the baseline, we follow the good practice in existing works [2, 3, 45, 6],which includes random resized cropping, horizontal flipping,
color jittering [2], grayscale conversion [2], Gaussian blurring [3], and solarization [45].
Given the baseline data augmentation, we addthe following data augmentation methods
• RandAugment [48]: a strong data augmentation strategy that includes 14 image-specific
transformations such as AutoContrast, Posterize, etc.
We use default settings that were usedin [48] (2 operations at each iteration and a magnitude scale of 10).• RandomErasing [63]: a simple data augmentation strategy that randomly masks a box in animage with a random value.
The RandomErasing was applied with probability 0 .
• Multi-crop [42]: a multi-view data augmentation strategy for self-supervised learning, weuse two global crops of size 224 and 2 or 6 local crops of size 96.
For global crops, we scalethe image with (0 . 2 , 1 . 0) and for the local crops, we scale the image with (0 . 05 , 0 . 2) , whichis slightly different from that in [42].
For the implementation of RandAugment and RandomErasing, we use the implementation in [81].
Note that when using Multi-crop, we only use RandAugment on the global views, and we do not use
RandomErasing, which slightly degrades performance when used for Multi-crops.
Otherwise, if wedo not use Multi-crop, we use both RandAugment and RandomErasing.
Model.
We use ResNet-50 [47] for backbone.
For our base encoder, we further use a projectionhead [3], and a prediction head [45].
The momentum encoder is updated by the moving average ofthe base encoder, except that we do not use the prediction head for momentum encoder [6].
Themomentum coefficient is initialized by 0 .
99 and gradually increased by the half-cycle cosine scheduleto 1.
We use 2-layer MLP with dimensions 2048-4096-256 for the projection head.
We attach thebatch-normalization (BN) layer and ReLU activation layer at the end of each fully-connected layer,except for the last one, to which we only attach the BN layer.
For the prediction head, we use2-layer MLP with dimensions of 256-4096-256, and we attach BN and ReLU at the end of each fullyconnected layer except the last one.
Then the critic is implemented by the temperature-scaled cosinesimilarity: f ( x, y ; τ ) =
x ⊤ y/ ( τ ∥ x ∥ 2 ∥ y ∥ 2 ) , with τ = 0 . 5 .
For RMLCPC objective, we use γ = 2 . 0 ,and α = 1 / 65536 .
Optimization.
We use LARS [82] optimizer with a batch size of 512, weight decay of 1 . 5 × 10 − 6 ,and momentum of 0 .
9 for all experiments.
We use root scaling for learning rate [3], which waseffective in our framework.
When training for 100 epochs, we use the base learning rate of 0.15, andfor training for 200 epochs, we use the base learning rate of 0.075.
The learning rate is graduallyannealed by a half-cycle cosine schedule.
We also used automatic mixed precision for scalability.
Linear evaluation.
For ImageNet linear evaluation accuracy, we report the validation accuracy ofthe linear classifier trained on the top of frozen features.
We use random resized cropping of size 224and random horizontal flipping for training data augmentation, and resizing to the size of 256 andcenter-crop of 224 for validation data augmentation.
We use cross-entropy loss for training, wherewe train for 90 epochs with SGD optimizer, batch size of 1024, and base learning rate of 0.1, wherewe use linear learning rate scaling and learning rate follows cosine annealing schedule.
We do notuse weight decay.
Semi-supervised learning.
For semi-supervised learning on ImageNet, we follow the protocol asin [3, 45, 42].
We add a linear classifier on the top of frozen representation, then train both backboneand classifier using either 1% or 10% of the ImageNet training data.
For a fair comparison, we use thesplits proposed in [3].
We use the same data augmentation and loss function as in linear evaluation.
As same as previous works, we use different learning rates for each backbone and classificationhead.
For the 1% setting, we use a base learning rate of 0.01 for backbone and 0.2 for classificationhead.
For the 10% setting, we use a base learning rate of 0.08 for the backbone and 0.03 for theclassification head.
The learning rate is decayed with a factor of 0.2 at 16 and 20 epochs.
We donot use weight decay or other regularization technique.
We train for 20 epochs with SGD optimizer,batch size of 512 for both 1% and 10% settings, and do not use weight decay.
[Table]
Transfer learning.
For transfer learning, we train a classifier on the top of frozen representationsas done in many previous works [3, 45].
In Table
[Table]
we list the information of the datasets weused; the number of classes, number of samples for each train/val/test split, and evaluation metric.
We use the same data augmentation that we used for linear evaluation on the ImageNet dataset.
For optimization, we ℓ 2 -regularized L-BFGS, where the regularization parameter is selected from arange of 45 logarithmically spaced values from 10 − 6 to 10 5 using the validation split.
After the besthyperparameter is selected, we train the linear classifier using both training and validation splits andreport the test accuracy using the metric instructed in Table .
The maximum number of iterationsin L-BFGS is 5000 and we use the previous solution as an initial point, i.e., a warm start, for thenext step.
For few-shot learning experiments, we perform logistic regression on the top of frozenrepresentations and use N × K support samples without fine-tuning and data augmentation in a
N -way K -shot episode.
Extended version of Table 4.
We provide additional experimental results in the comparison withother contrastive representation learning methods that use stronger augmentations.
The InfoMin [14]used RandAugment [48] and Jigsaw cropping for data augmentation and trained for 800 epochs.
The CLSA [57] used both RandAugment [48] and pyramidal multi-crop data augmentation, whichgenerates additional crops of size 192 × 192 , 160 × 160 , 128 × 128 , and 96 × 96 .
For comparison,we use the checkpoints from their official github repositories 3 4 .
In Table 12, we report the few-shotclassification accuracies (%) on FC100 [60], CUB200 [48], and Plant Disease [61] datasets, whichis the extended version of Table 3 in the main paper.
In Table 13, we report the linear evaluationaccuracies (%) on various object classification datasets extending the results in Table 3 in the mainpaper.
B.4 CIFAR experiments
For CIFAR experiments, we use random resized cropping with a size of 32 × 32 , random horizontalflipping, color jittering, and random grayscale conversion with probability 0.2 for the base dataaugmentation.
For harder data augmentation, we apply RandAugment [48], where we use 3 operationsat each iteration with a magnitude scale of 5, and RandomErasing [63].
We use modified ResNet18 [47], where the kernel size of the first convolutional layer is converted from 7 × 7 to 3 × 3 , andwe do not use Max pooling at the penultimate layer [3].
We attach a 2-layer projection head of size512-2048-128 with ReLU and BN layer attached at each FC layer, except for the last one.
We donot use a prediction head, and the temperature is set to be 0.5 for every CIFAR experiment.
Weuse γ = 1 .
and α = 1 / 4096 for both CIFAR-10 and CIFAR-100 experiments.
We train for
[Table] [Table]
epochs with the SGD optimizer, learning rate of 0.5 (without linear scaling), weight decay of 5e-4,and momentum of 0.9.
For evaluation, we train a linear classifier on the top of frozen features with100 epochs with an SGD optimizer, learning rate of 0.3, without using weight decay.
We only usedrandom resized crop with a size of 32, and random horizontal flipping for both training and evaluation.
B.5 Tabular experiments
For the data augmentation of tabular experiments, we consider simple random masking (RM) noisethat was used in [66], and random feature corruption (FC) proposed in [67].
The random maskingnoise is analogous to RandomErasing [63] that we used in image contrastive learning.
The FC dataaugmentation randomly selects a subset of attributes on each tabular data and mixes it with theattributes from another data.
Note that the random corruption is performed in the whole datasetscale [67].
We refer to [67] for detailed implementation of FC data augmentation.
We use a probabilityof 0.2 for both random masking noise and FC data augmentation.
For the backbone, we use a 5-layer
MLP with a dimension of 2048-2048-4096-4096-8192, where all layers have a BN layer followedby RELU, except for the last one.
We use MaxOut activation for the last layer with 4 sets, and onthe top of the backbone, we attach 2-layer MLP with a dimension of 2048-2048-128 as a projectionhead.
For Higgs-100K, we set the τ = 0 .
1 , γ = 1 . 2 , and α = 1 / 4096 .
For CovType, we set τ = 0 .
2 ,γ = 1 . 1 , and α = 1 / 4096 .
We train for 500 epochs with an SGD optimizer of momentum 0.9, baselearning rate of 0.3, and the learning rate is scheduled by half-cycle cosine annealing.
For evaluationof the CovType experiments, we train a linear classifier on the top of frozen representation for 100epochs with a learning rate in { 1 , 3 , 5 } and report the best validation accuracy (%).
For evaluationof the Higgs-100K experiments, we use linear regression on the top of frozen representations bypseudo-inverse and report the validation accuracy (%).
In Table 14, we provide the full experimentalresults on CovType and Higgs-100K. One can notice that the RM data augmentation is effective forthe Higgs dataset, and FC data augmentation is effective for the CovType dataset.
B.6 Graph experiments
For graph contrastive learning experiments on TUDataset [68], we follow the same experimentalsetup in [15].
Thus, we adopt the official implementation 5 , and experiment over the hyper-parameterγ ∈ { 1 . 5 , 2 . 0 , 3 . 0 } and choose the best parameter by evaluating on the validation dataset.
[Table]
C Mutual Information Estimation
[Table]
C.1 Implementation Details
Assume we have B batches of samples { x i } Bi =1 from x i ∼ X and B batches of samples { y i } Bi =
1from y i ∼ Y .
Assume ( x i , y i ) ∼ P XY , i.e. positive pair, and ( x i , y j ) ∼ P X P Y for i ̸
=
j .
Then letf ∗ ( x, y ) be optimal critic learned by maximizing CPC, MLCPC
[Table]
or RMLCPC objectives.
From theoptimality condition, we have
[Equation]
where r ( x, y ) =
dP XY dP X dP Y is a true density ratio.
Then our idea is to recover r by using f ∗ andapproximate the true mutual information.
To do that, we first compute the log-normalization constant
Z , i.e. e f ∗ ( x,y ) Z = r ( x,y ) αr ( x,y )
+1 − α .
We use Monte-Carlo [43] methods by computing empirical mean asfollowing:
[Equation]
Then we have following approximate density ratio ˆ r :
[Equation]
Since the mutual information is defined by the empirical mean of log-density ratio, we have
[Equation]
C.2 MI estimation between correlated Gaussians
Setup.
We follow the general procedure in [35, 39].
We sample a pair of vectors (
x, y ) ∈ R
d × dfrom random variable X × Y , where X and Y are unit normal distribution, and are correlatedwith correlation coefficient ρ ∈ (0 , 1) .
Then the true mutual information (MI) between X and Ycan be computed by I ( X ; Y )
= − d 2 log (cid:0) 1 − ρ 2 (cid:1) .
In our experiments, we set the initial mutualinformation to be 2 , and we increase the MI by × 2 for every 4K iterations.
We consider joint critic,which concatenates the inputs x, y and then passes through a 2-layer MLP with a dimension ofd − 256 − 1 , and the output of the fully connected layers are followed by the ReLU activation layer.
For optimization, we use the Adam optimizer with learning rate of 10 − 3 , β 1 = 0 .
9 , β 2 = 0 .
999 ,batch size of 128.
Results.
In Figure , we depict the curve of training curves and MI estimation that we proposed in
Section 3.4.
We consider α -CPC, α -MLCPC, ( α, γ ) -RMLCPC with γ = 2 . 0 .
Remark that whenα = 0 . 0 , the MI estimators exhibit large variance as we proved in Theorem 3.1.
But for largeenough α > 0 , the training objectives tend to have low variance, and thus we have a low variance MIestimator.
[Figure]
augmentation that generates views # x: batch of samples # f: critic that takes a pair of inputs #
[Figure]
_pos_and_neg: a function that extracts positive and negative pairs (Alg. 1)
[Figure]
MI_
[Figure]
(x, f, aug, alpha, gamma): x1, x2 = aug(x),
[Figure]
(x) # two views
[Figure]
neg =
[Figure]
pos
[Figure]
x
[Figure] [Figure]
From Theorem 4.1, since there are total 128 × 127 negative pairs, we need to select α ∝ 1 / 128 .
Therefore, we divide the α by 128 for each experiment in Figure 2.
One can observe that even thoughthe optimal α varies across the objectives, choosing α around 1/128 gives a fairly good estimation of
MI.
Note that the skew divergence is not only applicable to DV objectives, but also for the NWJ [84]objective, which is another variational lower bound of KL divergence defined as following:
[Equation]
and we have I ( X ; Y ) =
D KL ( P XY ∥
P X P Y ) =
sup f
[Equation]
NWJ ( f ) [84].
Note that the NWJ objective isproven to have large variance as the true MI is too large [37].
However, we show that by introducingthe NWJ objective for α -skew KL divergence, one can lower the variance of the training objective,and perform MI estimation as we explained in Section 3.4.
We refer the NWJ variational lower boundof α -skew KL divergence as α -NWJ, and is defined by following:
[Equation]
[Equation]
and we have D ( α ) KL ( P XY ∥
P X P Y ) =
sup f I ( α ) NWJ ( f ) .
Lastly, in Figure , we show that one can performstable MI estimation through α -NWJ objective by choosing appropriate α .
C. Mutual Information between the views and InfoMin principle.
Implementation.
We present the pseudocode for estimation of MI between the views in Algorithm 2.
In Figure , we compute the MI for every iteration, and report the mean value of MI for the last epochof training.
InfoMin principle and RényiCL.
Following [14], we plot the mutual information versus linearevaluation accuracy (%) by training with different data augmentations and contrastive objectives.
In
Figure 1, we observe that for harder data augmentations, the RMLCPC induces lower MI than that of
CPC and MLCPC, while showing better downstream performances.
This is because of the effect ofeasy positive and hard negative sampling explained in Section 3.3.
Note that InfoMin [14] principleargues that the optimal views must share sufficient and minimal information about the downstreamtasks.
Here, one can observe that RényiCL intrinsically follows the InfoMin principle as it extractsminimal and sufficient information between the views.
Therefore, RényiCL intrinsically follows
InfoMin principle [14] that it extracts minimal and sufficient information between the views.
Here,we establish the orthogonal relationship between the InfoMin principle [14] and RényiCL: InfoMinprinciple aims to find views that share minimal and sufficient information, while RényiCL can extractminimal and sufficient information from given views.
[Figure] [Figure]
V a l u
[Figure] [Figure]
b j e c t i v e V a l u
[Figure]
[Figure] [Figure]
D Hyperparameter Sensitivity Analysis
The temperature is a hyperparameter that greatly affects the performance of contrastive learning [3,85].
Here, we show that RMLCPC objective is robust to the choice of temperature hyperparameter.
We conduct simple ablation studies on CIFAR-10 and CIFAR-100 with temperature τ = 0 .
andγ
= 1 . 5 , 2 . 0 .
We use the same experimental setup described in B.4 except the value of temperature.
For comparison, we use same data augmentation setup; base data augmentation and hard augmentationusing RandAugment and RandomErasing.
In Table 15, we report the linear evaluation accuracyof each setting trained on CIFAR-10 and CIFAR-100 datasets.
When using large temperature, i.e.,τ = 1 . 0 , CPC and MLCPC degrades their performance when using harder data augmentation.
Onthe other hand, RMLCPC consistently achieves better performance when using harder augmentationson both τ = 0 .
5 and τ = 1 . 0 .
Especially for CIFAR-100 dataset, using RMLCPC objective withτ = 1 . 0 shows similar performance with using MLCPC objective with τ = 0 .
Thus, our RényiCLdoes not only provide empirical gain, but also it could be a good starting point to search for effectivehyperparameter due to their robustness to temperature.
[Table]