Original: Energy-Based Contrastive Learning of Visual Representations
Revised: Energy-Based Contrastive Learning of Visual Representations

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress
Revised: AffiliationAddress

Original: Abstract
Revised: Abstract

Original: Contrastive learning is a method of learning visual representations by trainingDeep
Revised: Contrastive learning is a method of learning visual representations by training

Original:  Neural Networks (DNNs) to increase the similarity between representations ofpositive pairs  and reduce the similarity between representations of negative pairs.
Revised: Deep Neural Networks (DNNs) to increase the similarity between representationsof positive pairs (transformations of the same image) and reduce the similaritybetween representations of negative pairs (transformations of different images).

Original: Unfortunately, contrastive methods usually require large datasets with significantnumber of negative pairs per iteration to achieve reasonable performance on downstream tasks.

Original: To address this problem, here we explore Energy-Based Contrastive Learning (EBCLR) that combines contrastive learning with Energy-Based Models (EBMs) and can be theoretically interpreted as learning the joint distributionof positive pairs.
Revised: Here we explore Energy-Based Contrastive Learning (EBCLR) that leveragesthe power of generative learning by combining contrastive learning with EnergyBased Models (EBMs).

Original: EBCLR  shows promising results on small and medium-scaledatasets such as MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100.
Revised: EBCLR can be theoretically interpreted as learning thejoint distribution of positive pairs, and it shows promising results on small andmedium-scale datasets such as MNIST, Fashion-MNIST, CIFAR-10, and CIFAR100.

Original: Specifically,  EBCLR demonstrates from × 4 up to × 20 acceleration compared to SimCLRand MoCo v2 in terms of training epochs.
Revised: Specifically, we find EBCLR demonstrates from × 4 up to × 20 accelerationcompared to SimCLR and MoCo v2 in terms of training epochs.

Original: Furthermore, in contrast to SimCLR,  EBCLR achieves nearly the same performance with 254 negative pairs (batch size128 ) and 30 negative pairs (batch size 16 ) per positive pair, demonstrating therobustness of EBCLR to small  number of negative pairs.
Revised: Furthermore, incontrast to SimCLR, we observe EBCLR achieves nearly the same performancewith 254 negative pairs (batch size 128 ) and 30 negative pairs (batch size 16 ) perpositive pair, demonstrating the robustness of EBCLR to small numbers of negativepairs. Hence, EBCLR provides a novel avenue for improving contrastive learningmethods that usually require large datasets with a significant number of negativepairs per iteration to achieve reasonable performance on downstream tasks.

Original:  Introduction
Revised:  Introduction

Original: In computer vision, supervised learning requires a large-scale human-annotated dataset of images totrain accurate deep neural networks (DNNs).
Revised: In computer vision supervised learning requires a large-scale human-annotated dataset of images totrain accurate deep neural networks (DNNs).

Original: However, acquiring labels for millions of images can bedifficult or impossible in practice.
Revised: However, acquiring labels for millions of images can bedifficult or impossible in practice.

Original: This has led to the rise of self-supervised learning, which learnsuseful visual representations by forcing DNNs to be invariant or equivariant to image transformations.
Revised: This has led to the rise of self-supervised learning, which learnsuseful visual representations by forcing DNNs to be invariant or equivariant to image transformations.

Original: Among self-supervised learning algorithms, contrastive methods are rapidly gaining popularity fortheir superb performance.
Revised: Among self-supervised learning algorithms, contrastive methods are rapidly gaining popularity fortheir superb performance.

Original: Specifically, contrastive learning methods [1, 2, 3, 4, 5] train DNNs by increasing the similaritybetween representations of positive pairs (transformations of the same image) and decreasing thesimilarity between representations of negative pairs (transformations of different images).
Revised: Specifically, contrastive learning methods [1, 2, 3, 4, 5] train DNNs by increasing the similaritybetween representations of positive pairs (transformations of the same image) and decreasing thesimilarity between representations of negative pairs (transformations of different images).

Original: Thenegative pairs prevent DNNs from collapsing to the trivial solution, i e., the constant function.
Revised: Thenegative pairs prevent DNNs from collapsing to the trivial solution, i.e., the constant function.

Original: Thereare numerous variants of contrastive learning methods, such as SimCLR [4], Momentum Contrast(MoCo) [5], etc.
Revised: Thereare numerous variants of contrastive learning methods, such as SimCLR [4], Momentum Contrast(MoCo) [5], etc.

Original: Despite this flurry of research in contrastive learning, contrastive methods require large datasets anda large number of negative pairs per positive pair to achieve reasonable performance on downstreamtasks.
Revised: Despite this flurry of research in contrastive learning, contrastive methods require large datasets anda large number of negative pairs per positive pair to achieve reasonable performance on downstreamtasks.

Original: Although there are recently proposed non-contrastive methods such as BYOL [6] and SimSiam[7] that do not rely on negative pairs
Revised: Although there are recently proposed non-contrastive methods such as BYOL [6] and SimSiam[7] that do not rely on negative pairs

Original: , they require heuristic techniques such as stop-gradient to avertcollapsing to the trivial solution.
Revised: , they require heuristic techniques such as stop-gradient to avertcollapsing to the trivial solution.

Original: There has been an effort to explain the dynamics of non-contrastivemethods with linear neural networks [3], but it is unclear how the analyses generalize to DNNs.
Revised: There has been an effort to explain the dynamics of non-contrastivemethods with linear neural networks [3], but it is unclear how the analyses generalize to DNNs.

Original: 𝑣 + 𝑣 −
Revised: 𝑣 + 𝑣 −

Original: Shared Params 𝑣 𝜏log1/𝑝 𝑣,𝑣 − 𝜏log1/𝑝(𝑣,𝑣 + ) 𝑓 𝜃 𝑓
Revised: Shared Params 𝑣 ∝1/𝑝(𝑣,𝑣 + ) 𝑓 𝜃 𝑓

Original: 𝜃 𝑓 𝜃 𝑧 𝑧 + 𝑧 −
Revised: 𝜃 𝑓 𝜃 𝑧 𝑧 + 𝑧 −

Original: Projection Space 𝑧
Revised: Projection Space 𝑧

Original: + 𝑧 𝑧 − 𝑝 𝑣,𝑣 + ≫ 𝑝(𝑣,𝑣 − )
Revised: + 𝑧 𝑧 − 𝑝 𝑣,𝑣 + ≫ 𝑝(𝑣,𝑣 − )

Original: Projections 
Revised: Projections ∝1/𝑝(𝑣,𝑣 − )

Original: In this paper, we approach this issue by proposing Energy-Based Contrastive Learning (EBCLR)which combines contrastive learning with energy-based models (EBMs).
Revised: In this paper, we explore a novel avenue in visual representation learning: Energy-Based Contrastive Learning (EBCLR) which leverages the power of generative learning [8, 9, 10] by combiningcontrastive learning with energy-based models (EBMs).

Original: EBCLR complementsthe contrastive learning loss with a generative loss, and it can be interpreted as learning the jointdistribution of positive pairs.
Revised: EBCLR complements the contrastivelearning loss with a generative loss, and it can be interpreted as learning the joint distribution ofpositive pairs.

Original: In fact, we demonstrate that the existing contrastive loss is a specialcase of the EBCLR loss if the generative term is not used.
Revised: In fact, we demonstrate that the existing contrastive loss is a special case of the EBCLRloss if the generative term is not used.

Original: Although EBMs are notorious for beingdifficult to train due to its reliance on Stochastic Gradient Langevin Dynamics (SGLD) [8], anotherimportant contribution of this work is that we overcome this by appropriate modifications to SGLD.
Revised: Although EBMs are notorious for being difficult to traindue to their reliance on Stochastic Gradient Langevin Dynamics (SGLD) [11], another importantcontribution of this work is that we overcome this by appropriate modifications to SGLD.

Original: Extensive experiments on a variety of small and medium-scale datasets demonstrate that EBCLR isrobust to small number of negative pairs, and it outperforms SimCLR and MoCo v2 [9] in terms ofsample efficiency and linear evaluation accuracy. 
Revised: Extensive experiments on a variety of small and medium-scale datasets demonstrate that EBCLR isrobust to small numbers of negative pairs, and it outperforms SimCLR and MoCo v2 [12] in terms ofsample efficiency and linear evaluation accuracy. Hence, EBCLR opens up a new research directionfor alleviating the dependence of contrastive methods on large datasets and large batches.

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: • A novel contrastive learning method called EBCLR is proposed by learning the jointdistribution of positive pairs.
Revised: • We propose a novel contrastive learning method called EBCLR which learns the joint distribution of positive pairs.

Original: We show that EBCLR loss is equivalent to a combination of a contrastive term and a generative term.
Revised: We show that EBCLR loss is equivalent to a combination of acontrastive term and a generative term (Section 3).

Original: To the best of our knowledge, this is the first work toapply EBMs to contrastive learning of visual representations.
Revised: To the best of our knowledge, this is the first work to apply EBMs to contrastive learning of visual representations.

Original: • To accelerate the training of EBCLR, we show that it is necessary to use an appropriatevariance schedule in SGLD.• By exploring the sensitivity of EBCLR to the generative term, we propose a further improvement by appropriately weighing the generative term.

Original: •  EBCLR is several times more sample-efficient than contrastive and non-contrastive methods. This leads to a large performance advantage for EBCLR given the same number of training
Revised: • We show that EBCLR offers two advantages over conventional contrastive learning methods: EBCLR is several times more sample efficient (Section 4.1) and robust to small batch sizes

Original: epochs.
Revised: (Section 4.2).

Original: • Unlike SimCLR, EBCLR is shown to be robust to small number of negative pairs.
Revised: These factors lead to a non-trivial performance gain for EBCLR.

Original: Empiri- cally, we observe little difference between linear evaluation accuracies for EBCLR with
Revised: • We perform thorough ablation studies of the components of EBCLR: effect of changing the

Original: negative pairs (batch size 128 ) and 30 negative pairs (batch size 16 ) per positive pair.
Revised: weight of the generative term (Section 4.3), effect of projection space dimension (Section4.3), and the effect of the proposed SGLD modifications (Section 4.4).

Original:  Related Works
Revised:  Related Works

Revised: In this section, we go over related works necessary for understanding EBCLR.

Revised: In Appendix A, wegive a more extensive review of relevant works for those not familiar with EBMs, contrastive learning,or generative models.

Original: 2.1 Contrastive Learning
Revised: 2.1 Contrastive Learning

Original: For a given batch of images { x n } Nn =1 and two image transformations t , t ′ , contrastive learningmethods first create two views v n = t ( x n ) , v ′ n = t ′ ( x n ) of each instance x n .
Revised: For a given batch of images { x n } Nn =1 and two image transformations t , t ′ , contrastive learningmethods first create two views v n = t ( x n ) , v ′ n = t ′ ( x n ) of each instance x n .

Original: Here, the pair ( v n , v ′ m )is called a positive pair if n = m and a negative pair if n ̸ = m .
Revised: Here, the pair ( v n , v ′ m ) is called a positive pair if n = m and a negative pair if n ̸ = m .

Original: Given a DNN f θ , the views are thenembedded into the projection space by passing the views through f θ and normalizing.
Revised: Given a DNN f θ , the views are thenembedded into the projection space by passing the views through f θ and normalizing.

Original: Contrastive methods train f θ to increase agreement between projections of positive pairs, and decreaseagreement between projections of negative pairs.
Revised: Contrastive methods train f θ to increase agreement between projections of positive pairs and decreaseagreement between projections of negative pairs.

Original: Specifically, f θ is trained to maximize the InfoNCEobjective [1].
Revised: Specifically, f θ is trained to maximize the InfoNCEobjective [1].

Original: After training, outputs from the final layer or an intermediate layer of f θ are used fordownstream tasks.
Revised: After training, outputs from the final layer or an intermediate layer of f θ are used fordownstream tasks.

Original: There are numerous variants of contrastive methods.
Revised: There are numerous variants of contrastive methods.

Original: For instance, SimCLR [4] uses a compositionof random cropping, random flipping, color jittering, color dropping, and blurring as the imagetransformation.
Revised: For instance, SimCLR [4] uses a compositionof random cropping, random flipping, color jittering, color dropping, and blurring as the imagetransformation.

Original: Negative pairs are created by transforming different images within a batch.
Revised: Negative pairs are created by transforming different images within a batch.

Original: On theother hand, MoCo [9] maintains a queue of negative samples, so negative samples are not limited toviews of images from the same batch.
Revised: On theother hand, MoCo [12] maintains a queue of negative samples, so negative samples are not limited toviews of images from the same batch.

Original: 2.2 Energy-Based Models
Revised: 2.2 Energy-Based Models

Original: Given a scalar-valued energy function E θ ( v ) with parameter θ , an energy-based model (EBM) [10]defines a distribution by the formula
Revised: Given a scalar-valued energy function E θ ( v ) with parameter θ , an energy-based model (EBM) [13]defines a distribution by the formula

Original: [Equation]
Revised: [Equation]

Original:  where Z ( θ ) is the partition function which guarantees q θ integrates to 1 .
Revised: 1) where Z ( θ ) is the partition function which guarantees q θ integrates to 1 .

Original: Since there are essentiallyno restrictions on the choice of the energy function, EBMs have great flexibility in modelingdistributions.
Revised: Since there are essentially norestrictions on the choice of the energy function, EBMs have great flexibility in modeling distributions.

Original: Hence, EBMs have been applied to a wide variety of machine learning tasks suchas learning generative classifiers [11, 12], generating images [13], and training regression models[14, 15].
Revised: Hence, EBMs have been applied to a wide variety of machine learning tasks, such as dimensionalityreduction via autoencoding [27], learning generative classifiers [8, 9, 10, 14], generating images [15],and training regression models [16, 17]. Wang et al. [26] have explored connections between EBMsand InfoNCE to enhance generative performance of EBMs.

Original: However, to the best of our knowledge, this paper is the firstwork to combine EBMs withcontrastive learning.
Revised: However, to the best of our knowledge,this paper is the first to combine EBMs with contrastive learning for representation learning.

Original: Given a target distribution, an EBM can be used to estimate its density p when we can only samplefrom p .
Revised: Given a target distribution, an EBM can be used to estimate its density p when we can only samplefrom p .

Original: One way of achieving this is by minimizing the Kullback-Leiber (KL) divergence betweenq θ and p that maximizes the expected log-likelihood of q θ under p [16]: [Equation]
Revised: One way of achieving this is by minimizing the Kullback-Leibler (KL) divergence betweenq θ and p that maximizes the expected log-likelihood of q θ under p [18]:

Original: [Equation]
Revised: [Equation]

Original: Stochastic gradient ascent can be used to solve (2) [16].
Revised: Stochastic gradient ascent can be used to solve (2) [18].

Original: Specifically, the gradient of the expectedlog-likelihood with respect to the parameters θ can be shown to be
Revised: Specifically, the gradient of the expectedlog-likelihood with respect to the parameters θ can be shown to be

Original: [Equation]
Revised: [Equation]

Original: Hence, updating θ with (3) amounts to pushing up on the energy for samples from q θ and pushingdown on the energy for samples from p . 
Revised: Hence, updating θ with (3) amounts to pushing up on the energy for samples from q θ and pushingdown on the energy for samples from p . This optimization method is also known as contrastivedivergence [19].

Original: While the second term in (3) can be easily calculated as we have access to samples from p , the firstterm requires sampling from q θ .
Revised: While the second term in (3) can be easily calculated as we have access to samples from p , the firstterm requires sampling from q θ .

Original: Previous works [13, 17, 11, 12] have used Stochastic Gradient
Revised: Previous works [15, 20, 10, 14] have used Stochastic Gradient

Original: Langevin Dynamics (SGLD) [8] to generate samples from q θ .
Revised: Langevin Dynamics (SGLD) [11] to generate samples from q θ .

Original: Specifically, given a sample v 0 fromsome proposal distribution q 0 , the iteration
Revised: Specifically, given a sample v 0 fromsome proposal distribution q 0 , the iteration

Original: [Equation]
Revised: [Equation]

Original: guarantees that the sequence { v t } converges to a sample from q θ assuming { α t } decays at a polynomial rate [8].
Revised: guarantees that the sequence { v t } converges to a sample from q θ assuming { α t } decays at a polynomial rate [11].

Original: However, SGLD requires an infinite number of steps until samples from the proposal distributionconverge to samples from the target distribution.
Revised: However, SGLD requires an infinite number of steps until samples from the proposal distributionconverge to samples from the target distribution.

Original: This is unfeasible, so in practice, only a finitenumber of steps along with constant step size, i.e. α t = α and constant noise variance σ t =
Revised: This is unfeasible, so in practice, only a finitenumber of steps along with constant step size, i.e. α t = α and constant noise variance σ t =

Original: σ 2 areused [13, 17, 11, 12].
Revised: σ 2 areused [15, 20, 10, 14].

Original: Moreover, Yang and Ji [12] noted SGLD often generates samples with extremepixel values that cause EBMs to diverge during training.
Revised: Moreover, Yang and Ji [14] noted SGLD often generates samples with extremepixel values that cause EBMs to diverge during training.

Original: Hence, they have proposed proximal SGLDwhich clamps gradient values into an interval [ − δ, δ ] for a threshold δ > 0 .
Revised: Hence, they have proposed proximal SGLDwhich clamps gradient values into an interval [ − δ, δ ] for a threshold δ > 0 .

Original: Then, the update equationbecomes
Revised: Then, the update equationbecomes

Original:  [Equation]
Revised: [Figure] [Equation]

Original: 5) for t = 0 , . . .
Revised: for t = 0 , . . .

Original: , T − 1 , where ϵ ∼ N (0 , σ 2 ) and clamp {· , δ } clamps each element of the input vectorinto [ − δ, δ ] .
Revised: , T − 1 , where ϵ ∼ N (0 , σ 2 ) and clamp {· , δ } clamps each element of the input vectorinto [ − δ, δ ] .

Original: In our work, we introduce additional modifications to SGLD which accelerate theconvergence of EBCLR.
Revised: In our work, we introduce additional modifications to SGLD which accelerate theconvergence of EBCLR.

Original: 𝑡 ∼ 𝒯 𝑣 𝑛 ′ 𝑣 𝑛 𝑧 𝑛′ ෤𝑣 𝑛 𝑧 𝑛 ǁ𝑧 𝑛
Revised: 𝑡 ∼ 𝒯 𝑣 𝑛 ′ 𝑣 𝑛 𝑧 𝑛′ ෤𝑣 𝑛 𝑧 𝑛 ǁ𝑧 𝑛

Original: Contrastive Loss 𝑥
Revised: Contrastive Loss 𝑥

Original: 𝑛 −𝜆 ⋅𝐸 𝜃 ( ǁ𝑧 𝑛 ;{𝑧 𝑚′ }) 𝜆 ⋅𝐸 𝜃 (𝑧 𝑛 ;{𝑧 𝑚 ′ })
Revised: 𝑛 −𝜆 ⋅𝐸 𝜃 ( ǁ𝑧 𝑛 ;{𝑧 𝑚′ }) 𝜆 ⋅𝐸 𝜃 (𝑧 𝑛 ;{𝑧 𝑚 ′ })

Original: Replay Buffer
Revised: Replay Buffer

Original: Update
Revised: Update

Original: Shared Parameters 𝑓 𝜃
Revised: Shared Parameters 𝑓 𝜃

Original: 𝑓
Revised: 𝑓

Original: 𝜃 𝑓 𝜃
Revised: 𝜃 𝑓 𝜃

Original: Update ෤𝑣 𝑛 via SGLD using ∇ ෤𝑣 𝑛 𝐸 𝜃 ( ǁ𝑧 𝑛 ;{𝑧 𝑚′ })𝑡′ ∼ 𝒯 [Figure] 𝜌
Revised: Update ෤𝑣 𝑛 via SGLD using ∇ ෤𝑣 𝑛 𝐸 𝜃 ( ǁ𝑧 𝑛 ;{𝑧 𝑚′ })

Original: [Figure]
Revised: [Figure]

Original: (𝑣) 1−𝜌

Original: [Figure]

Original:  Theory
Revised:  Theory

Original: 3.1 Energy-Based Contrastive Learning
Revised: 3.1 Energy-Based Contrastive Learning

Original: Let D be a distribution of images and T a distribution of stochastic image transformations.
Revised: Let D be a distribution of images and T a distribution of stochastic image transformations.

Original: Given x ∼ D and i.i.d.
Revised: Given x ∼ D and i.i.d.

Original: t, t ′ ∼ T , our goal is to approximate the joint distribution of the views p ( v, v ′ ) , where v = t ( x ) , v ′ =
Revised: t, t ′ ∼ T , our goal is to approximate the joint distribution of the views p ( v, v ′ ) , where v = t ( x ) , v ′ =

Original: t ′ ( x ) using the model distribution
Revised: t ′ ( x ) using the model distribution

Original: [Equation]
Revised: [Equation]

Original:  where Z ( θ ) is a normalization constant, τ  is a temperature hyper-parameter, and z and z ′ areprojections computed by passing the views v and v ′ through the DNN f θ and then normalizing.
Revised: 6) where Z ( θ ) is a normalization constant, τ > 0 is a temperature hyper-parameter, and z and z ′ areprojections computed by passing the views v and v ′ through the DNN f θ and then normalizing tohave unit norm.

Original:  Our key idea isthen to use p ( v, v ′ ) as a measure of semantic similarity of v and v ′ .
Revised: We now explain the intuitive meaning of matching q θ to p . Our key idea is to use p ( v, v ′ ) as a measure of semantic similarity of v and v ′ .

Original: Specifically,p ( v, v ′ ) will be high when v and v ′ are semantically similar and low otherwise.
Revised: If two images v and v ′are semantically similar, they are likely to be transformations of similar images. So, p ( v, v ′ ) will behigh when v and v ′ are semantically similar and low otherwise.

Original: Thus, if q θ successfullyapproximates p , by the definition of q θ  in (6), the distance between z and z ′ will be controlled by theinverse of semantic similarity of v and v ′ . 
Revised: Suppose q θ successfully approximatesp . If we equate p ( v, v ′ ) to q θ ( v, v ′ ) in (6) and solve for ∥ z − z ′ ∥ , we see that the distance between zand z ′ will become a monotone increasing function of 1 /p ( v, v ′ ) , which is the inverse of semanticsimilarity of v and v ′ . So, semantically similar images will have nearby projections, and dissimilarimages will have distant projections.

Original: This idea is illustrated in Figure 1.
Revised: This idea is illustrated in Figure 1.

Original: To approximate p using q θ , we train f θ to maximize the expected log-likelihood of q θ under p :
Revised: To approximate p using q θ , we train f θ to maximize the expected log-likelihood of q θ under p :

Original: [Equation]
Revised: [Equation]

Original: In order to solve this problem with stochastic gradient ascent, we could naively extend (3) to thesetting of joint distributions to obtain the following result.
Revised: In order to solve this problem with stochastic gradient ascent, we could naively extend (3) to thesetting of joint distributions to obtain the following result.

Original: Proposition 1.
Revised: Proposition 1.

Original: The the joint distribution (6) can be formulated as an EBM
Revised: The the joint distribution (6) can be formulated as an EBM

Original: [Equation]
Revised: [Equation]

Original: and the gradient of the objective of (7) is given by
Revised: and the gradient of the objective of (7) is given by

Original: [Equation]
Revised: [Equation]

Original: However, computing the first expectation in (9) requires sampling pairs of views ( v, v ′ ) from q θ ( v, v ′ )via SGLD, which could be expensive.
Revised: However, computing the first expectation in (9) requires sampling pairs of views ( v, v ′ ) from q θ ( v, v ′ )via SGLD, which could be expensive.

Original: To avert this problem, we use Bayes’ rule to decompose
Revised: To avert this problem, we use Bayes’ rule to decompose

Original: E p [Equation] log q 
Revised: E p [log q θ ( v, v ′ )]

Original: [Equation]
Revised: =

Original: E p [log q θ ( v  )] where q θ ( v ) = (cid:90) q θ ( v, v ′ ) dv ′ .
Revised: E p [log q θ ( v ′ | v )] + E p [log q θ ( v )] where q θ ( v ) = (cid:90) q θ ( v, v ′ ) dv ′ .

Original: In the first equation of (10), the first and second terms at the RHS will be referred to as discriminativeand generative terms, respectively, throughout the paper.
Revised: In the first equation of (10), the first and second terms at the RHS will be referred to as discriminativeand generative terms, respectively, throughout the paper.

Original: Similar decomposition was used by
Revised: A similar decomposition was used by

Original: Grathwohl et.
Revised: Grathwohl et al.

Original: al [11] in the setting of learning generative classifiers.
Revised: [10] in the setting of learning generative classifiers.

Original: Furthermore, we add a hyper-parameter λ to balance the strength of the discriminative term and thegenerative term.
Revised: Furthermore, we add a hyper-parameter λ to balance the strength of the discriminative term and thegenerative term.

Original: The advantage of this modification will be discussed in Section 4.3.
Revised: The advantage of this modification will be discussed in Section 4.3.

Original: This yields our
Revised: This yields our

Original: Energy-Based Contrastive Learning (EBCLR) objective
Revised: Energy-Based Contrastive Learning (EBCLR) objective

Original: [Equation] 
Revised: [Equation] [Equation]

Original: The discriminative term can be easily differentiated since the partition function Z ( θ ) cancels outwhen
Revised: The discriminative term can be easily differentiated since the partition function Z ( θ ) cancels outwhen

Original: q θ ( v, v ′ ) is divided by q θ ( v ) .
Revised: q θ ( v, v ′ ) is divided by q θ ( v ) .

Original: However, the generative term still contains Z ( θ ) .
Revised: However, the generative term still contains Z ( θ ) .

Original: We now presentour key result which addresses this problem by relating the generative term to EBMs.
Revised: We now presentour key result, which addresses this problem by relating the generative term to EBMs.

Original: The proof isdeferred to Appendix ?? in the Supplementary Material.
Revised: The proof isdeferred to Appendix C.1.

Original: Theorem 2.
Revised: Theorem 2.

Original: The marginal distribution in (10) can be formulated as an EBM
Revised: The marginal distribution in (10) can be formulated as an EBM

Original: [Equation]
Revised: [Equation]

Original: where Z ( θ ) is the partition function in (6) , and the gradient of the generative term is given by
Revised: where Z ( θ ) is the partition function in (6) , and the gradient of the generative term is given by

Original: [Equation]
Revised: [Equation]

Original: Thus, the gradient of the EBCLR objective is
Revised: Thus, the gradient of the EBCLR objective is

Original: [Equation]
Revised: [Equation]

Original: Theorem 2 suggests that the EBM for the joint distribution can be learned by computing the gradientsof the discriminative term and the EBM for the marginal distribution.
Revised: Theorem 2 suggests that the EBM for the joint distribution can be learned by computing the gradientsof the discriminative term and the EBM for the marginal distribution.

Original: Moreover, we only need tosample v from q θ ( v ) to compute the second expectation in (14).
Revised: Moreover, we only need tosample v from q θ ( v ) to compute the second expectation in (14).

Original: 3.2 Approximating the EBCLR Objective
Revised: 3.2 Approximating the EBCLR Objective

Original: To implement EBCLR, we need to approximate expectations in (11) with their empirical means.
Revised: To implement EBCLR, we need to approximate expectations in (11) with their empirical means.

Original: Suppose samples { ( v n , v ′ n ) }
Revised: Suppose samples { ( v n , v ′ n ) }

Original: Nn =1 from p ( v, v ′ ) are given, and let { ( z n , z ′ n ) }
Revised: Nn =1 from p ( v, v ′ ) are given, and let { ( z n , z ′ n ) }

Original: Nn =1 be the correspondingprojections.
Revised: Nn =1 be the correspondingprojections.

Original: As the learning goal is to make q θ ( v n , v ′ n ) approximate the joint probability densityfunction p ( v n , v ′ n ) , the empirical mean (cid:98) q θ ( v n ) can be defined as:
Revised: As the learning goal is to make q θ ( v n , v ′ n ) approximate the joint probability densityfunction p ( v n , v ′ n ) , the empirical mean (cid:98) q θ ( v n ) can be defined as:

Original: [Equation]
Revised: [Equation]

Original: where the sum is over the collection of v ′ m defined as
Revised: where the sum is over the collection of v ′ m defined as

Original: [Equation]
Revised: [Equation]

Original:  and N ′ :
Revised: 16) and N ′ :

Original: = |{ v ′ m : v [Equation] m ̸
Revised: = |{ v ′ m : v ′ m ̸

Original: = v n }| = 2 N − 1 .
Revised: = v n }| = 2 N − 1 .

Original: One could also use a simpler form of the empirical mean:
Revised: One could also use a simpler form of the empirical mean:

Original: [Equation] 
Revised: [Equation] [Equation]

Original: Similarly, q θ ( v ′ | v ) in (11), which should approximate the conditional probability density p ( v ′ | v ) , canbe represented in terms of q θ ( v n , v ′ n ) .
Revised: Similarly, q θ ( v ′ | v ) in (11), which should approximate the conditional probability density p ( v ′ | v ) , canbe represented in terms of q θ ( v n , v ′ n ) .

Original: Specifically, we have
Revised: Specifically, we have

Original: [Equation]
Revised: [Equation]

Original: It is then immediate that the empirical form of the discriminative term using (18) is a particularinstance of the contrastive learning objective such as InfoNCE and SimCLR.
Revised: It is then immediately apparent that the empirical form of the discriminative term using (18) isa particular instance of the contrastive learning objective such as InfoNCE and SimCLR.

Original: Hence, EBCLR canbe interpreted as complementing contrastive learning with a generative term defined by an EBM.
Revised: Hence, EBCLR can be interpreted as complementing contrastive learning with a generative term defined byan EBM.

Original: We will demonstrate in Section 4.1 that the generative term offers significant advantages over othercontrastive learning methods.
Revised: We will demonstrate in Section 4.1 that the generative term offers significant advantagesover other contrastive learning methods.

Original: For the second term, we use the simpler form of  empirical mean in (17):
Revised: For the second term, we use the simpler form of the empirical mean in (17):

Original: [Equation]
Revised: [Equation]

Original: We could  use (15) asfor the empirical mean, but either choice showed identical performance (c.f.
Revised: We could also use (15) as the empirical mean, but either choice showed identical performance (c.f.

Original: Appendix D.3).
Revised: Appendix E.3).

Original: So, we have found (15) to be not worth the additional complexity, and have resortedto the simpler approximation (17) instead.
Revised: So, we have found (15) to be not worth the additional complexity, and have resortedto the simpler approximation (17) instead.

Original:  If we compare (19) with (12), we can see that this approximation of q θ ( v ) yields the energy function(after ignoring theirrelevant constant log N )
Revised: In Appendix C.2, we theoretically justify that EBCLR willwork as intended even with the approximations (15) or (17). If we compare (19) with (12), we cansee that this approximation of q θ ( v ) yields the energy function (after ignoring the constant log N )

Original: [Equation]
Revised: [Equation]

Original: 3.3 Modifications to SGLD
Revised: 3.3 Modifications to SGLD

Original: According to Theorem 2, we need samples from the marginal q θ ( v ) to calculate the second expectationin (14).
Revised: According to Theorem 2, we need samples from the marginal q θ ( v ) to calculate the second expectationin (14).

Original: Hence, we apply proximal SGLD (5) with the energy function (20) to sample from q θ ( v ) asfollows:
Revised: Hence, we apply proximal SGLD (5) with the energy function (20) to sample from q θ ( v ) as

Original: [Equation] 
Revised: [Equation] [Equation]

Original:  where ϵ ∼ N (0 , σ 2 ) and the update is repeated T times.
Revised: for t = 0 , . . . , T − 1 , where ϵ ∼ N (0 , σ 2 ) .

Original: We make three additional modifications toproximal SGLD to expedite the training process.
Revised: We make three additional modifications to proximal SGLD to expedite the training process.

Original: From here on, we will be referring to proximal SGLD in (5) when we say SGLD.
Revised: From here on, we will be referring to proximal SGLD in (5)when we say SGLD.

Original: First, we initialize SGLD from generated samples from previous iterations, and with probabilityρ , we reinitialize SGLD chains from samples from a proposal distribution q 0 .
Revised: First, we initialize SGLD from generated samples from previous iterations, and with probabilityρ , we reinitialize SGLD chains from samples from a proposal distribution q 0 .

Original: This is achieved bykeeping a replay buffer B of SGLD samples from previous iterations.
Revised: This is achieved bykeeping a replay buffer B of SGLD samples from previous iterations.

Original: This technique of maintaininga replay buffer has also been used in previous works and has proven to be crucial for stabilizing andaccelerating the convergence of EBMs [13, 11, 12].
Revised: This technique of maintaininga replay buffer has also been used in previous works and has proven to be crucial for stabilizing andaccelerating the convergence of EBMs [15, 10, 14].

Original: Second, the proposal distribution q 0 is set to be the data distribution p ( v ) .
Revised: Second, the proposal distribution q 0 is set to be the data distribution p ( v ) .

Original: This choice differs fromthose of previous works [13, 11, 12] which have either used the uniform distribution or a mixture of
Revised: This choice differs fromthose of previous works [15, 10, 14] which have either used the uniform distribution or a mixture of

Original: Gaussians as the proposal distribution.
Revised: Gaussians as the proposal distribution.

Original: Finally, we use multi-stage SGLD (MSGLD) , which adaptively controls the magnitude of noise addedin SGLD.
Revised: Finally, we use multi-stage SGLD (MSGLD) , which adaptively controls the magnitude of noise addedin SGLD.

Original: For each sample ˜ v in the replay buffer B , we keep a count κ ˜ v of number of times it hasbeen used as the initial point of SGLD.
Revised: For each sample ˜ v in the replay buffer B , we keep a count κ ˜ v of number of times it hasbeen used as the initial point of SGLD.

Original: For samples with a low count, we use noise of high variance,and for samples with a high count, we use noise of low variance.
Revised: For samples with a low count, we use noise of high variance,and for samples with a high count, we use noise of low variance.

Original: Specifically, in (5), we set
Revised: Specifically, in (5), we set

Original: [Equation]
Revised: [Equation]

Original: ) where [ · ] + := max { 0 , ·} , σ 2max and σ 2min are the upper and lower bounds on the noise variance,respectively, and K controls the decay rate of noise variance.
Revised: where [ · ] + := max { 0 , ·} , σ 2max and σ 2min are the upper and lower bounds on the noise variance,respectively, and K controls the decay rate of noise variance.

Original: The purpose of this technique is tofacilitate quick exploration of the modes of q θ and still guarantee SGLD generates samples withsufficiently low energy.
Revised: The purpose of this technique is tofacilitate quick exploration of the modes of q θ and still guarantee SGLD generates samples withsufficiently low energy.

Original: The pseudocodes for MSGLD and EBCLR are given in given in Algorithms1 and 2, respectively, in the Supplementary Material, and the overall learning flow of EBCLR isdescribed in Figure 2.
Revised: The pseudocodes for MSGLD and EBCLR are given in Algorithms 1 and 2,respectively, in Appendix B, and the overall learning flow of EBCLR is described in Figure 2.

Original:  Experiments
Revised:  Experiments

Original: We now describe the experiment settings.
Revised: We now describe the experimental settings.

Original: A complete description is deferred to Appendix C.
Revised: A complete description is deferred to Appendix D.

Original: Baseline methods and datasets.
Revised: Baseline methods and datasets.

Original: The baseline methods are SimCLR, MoCo v2, SimSiam, and
Revised: The baseline methods are SimCLR, MoCo v2, SimSiam, and

Original: BYOL.
Revised: BYOL.

Original: The hyper-parameters are chosen closely following the original works [4, 9, 7, 6].
Revised: The hyper-parameters are chosen closely following the original works [4, 12, 7, 6].

Original: We usefour datasets: MNIST [18], Fashion MNIST (FMNIST) [19], CIFAR10, and CIFAR100 [20].
Revised: We usefour datasets: MNIST [21], Fashion MNIST (FMNIST) [22], CIFAR10, and CIFAR100 [23].

Original: DNN architecture.
Revised: DNN architecture.

Original: We decompose f θ = π θ ◦ ϕ θ where ϕ θ is the encoder network and π θ is theprojection network.
Revised: We decompose f θ = π θ ◦ ϕ θ where ϕ θ is the encoder network and π θ is theprojection network.

Original: Rather thandirectly using the output of f θ for downstream tasks, we followprevious works [4, 5, 1, 2, 3, 6, 7] and use the output of ϕ θ instead.
Revised: Rather than using the output of f θ for downstream tasks, we follow previousworks [4, 5, 1, 2, 3, 6, 7] and use the output of ϕ θ instead.

Original: In our experiments, we set ϕ θ to be a ResNet-18 [21] up to the global average pooling layer. Thearchitecture of π θ and π θ to be a 2-layer MLP with output dimension 128 .
Revised: In our experiments, we set ϕ θ to bea ResNet-18 [24] up to the global average pooling layer and π θ to be a 2-layer MLP with outputdimension 128 .

Original: However, we removebatch normalization because batch normalization hurts SGLD [13].
Revised: However, we remove batch normalization because batch normalization hurts SGLD[15].

Original: We also replace ReLU activationswith leaky ReLU to expedite the convergence of SGLD.
Revised: We also replace ReLU with leaky ReLU to expedite the convergence of SGLD.

Original: For the baseline methods, we use settingsproposed in the original works while keeping the backbone fixed to be ResNet-18.[Table]
Revised: For the baselines,we use settings proposed in the original works while keeping the backbone fixed to be ResNet-18.

Original: Evaluation.
Revised: Evaluation.

Original: We assess the quality of the representations by training a linear classifier on top of frozenϕ θ .
Revised: We evaluate the representations by training a linear classifier on top of frozen ϕ θ .

Original: The linear classifier is trained with Adam for 200 epochs with batch size 512 . The learning rate ηis found by grid search for log 10 η in [ − 4 , − 1] .
Revised: [Table]

Original: 4.1 Comparison with Baselines
Revised: 4.1 Comparison with Baselines

Original: [Table]
Revised: [Table]

Original: the arrow is the dataset that linear evaluation was performed on. We use the following abbreviations. M : MNIST, FM : FMNIST, C10 : CIFAR10, C100 : CIFAR100.
Revised: [Equation]

Original: We use batch size 128 for EBCLR and batch size256 for the baseline methods following Wanget.
Revised: We use batch size 128 for EBCLR andbatch size 256 for the baseline methodsfollowing Wang et.

Original: al [22] and train each method for 100 epochs.
Revised: al [25] and train eachmethod for 100 epochs.

Original: Table 1 shows the result of training each methodfor 100 epochs.
Revised: Table 1 showsthe result of training each method for 100epochs.

Original: Observe that EBCLR consistently outperforms all baseline methods in termsof linear evaluation accuracy.
Revised: Observe that EBCLR consistentlyoutperforms all baseline methods in termsof linear evaluation accuracy.

Original: Moreover, relativeefficiency indicates EBCLR is capable of achieving the same level of performance as the baselinemethods with much fewer training epochs.
Revised: Moreover,relative efficiency indicates EBCLR is capable of achieving the same level of performance as the baseline methods with muchfewer training epochs.

Original: Concretely, we observe at least × 4 acceleration interms of epochs compared to contrastive methods.
Revised: Concretely, we observe at least × 4 acceleration in terms ofepochs compared to contrastive methods.

Original: Hence, EBCLR is a much more desirablechoice than SimCLR or MoCo v2 for learningvisual representations when we have a smallnumber of training samples.
Revised: Hence, EBCLR is a much more desirablechoice than SimCLR or MoCo v2 for learning visual representations when we have a small number of training samples.

Original: We also investigate the transfer learning performance of EBCLR.
Revised: We also investigate the transfer learning performance of EBCLR.

Original: Table 2 compares the transfer learning accuracies.
Revised: Table 2 compares the transferlearning accuracies.

Original: EBCLR always outperformsthe baseline methods, and the performance gap is especially large on CIFAR10 and CIFAR100.
Revised: EBCLR always outperforms the baseline methods, and the performance gap isespecially large on CIFAR10 and CIFAR100.

Original: Thisindicates EBCLR learns visual representations that generalize well across datasets.
Revised: This indicates EBCLR learns visual representationsthat generalize well across datasets.

Original: Repeating theabove experiments with longer training  led to similar conclusions (c.f.Supplementary Material
Revised: Repeating the above experiments with longer training or KNNclassification led to similar conclusions (c.f.

Original: Section D.1).
Revised: Appendixes E.1 and E.4, respectively).

Original: 4.2 Effect of Reducing Negative Pairs
Revised: 4.2 Effect of Reducing Negative Pairs

Original: We compared the performances of EBCLR and SimCLR as we reduced the number of negativepairs per positive pair.
Revised: We compared the performances of EBCLR and SimCLR as we reduced the number of negativepairs per positive pair.

Original: For MoCo v2, the negative samples are provided by a queue updated by amomentum encoder.
Revised: For MoCo v2, the negative samples are provided by a queue updated by amomentum encoder.

Original: On the other hand, for EBCLR and SimCLR, negative samples come from thesame batch as the positive pair.
Revised: On the other hand, for EBCLR and SimCLR, negative samples come from thesame batch as the positive pair.

Original: So, we did not have a way of fairly comparing EBCLR and SimCLRwith MoCo v2.
Revised: So, we did not have a way of fairly comparing EBCLR and SimCLRwith MoCo v2.

Original: Hence, we excluded MoCo v2 from this experiment.
Revised: Hence, we excluded MoCo v2 from this experiment.

Original: We note that according to (18), given a batch of size N , we obtain 2 N − 2 negative pairs foreach positive pair.
Revised: We note that, according to (18), given a batch of size N , we obtain 2 N − 2 negative pairs foreach positive pair.

Original: SimCLR also has 2 N − 2 negative pairs for each positive pair.
Revised: SimCLR also has 2 N − 2 negative pairs for each positive pair.

Original: Hence, we canconveniently compare the sensitivity of EBCLR and SimCLR to the number of negative pairs byvarying the batch size.
Revised: Hence, we canconveniently compare the sensitivity of EBCLR and SimCLR to the number of negative pairs byvarying the batch size.

Original: Table 3 shows the result of training each method for 100 epochs with batch sizes in { 16 , 64 , 128 } .
Revised: Table 3 shows the result of training each method for 100 epochs with batch sizes in { 16 , 64 , 128 } .

Original: We make three important observations.
Revised: We make three important observations.

Original: First, EBCLR consistently beats SimCLR in terms of linearevaluation accuracy for every batch size.
Revised: First, EBCLR consistently beats SimCLR in terms of linearevaluation accuracy for every batch size.

Original: Second, EBCLR is invariant to the choice of batch size.
Revised: Second, EBCLR is invariant to the choice of batch size.

Original: This contrasts with SimCLR whose performance degrades as batch size decreases.
Revised: This contrasts with SimCLR whose performance degrades as batch size decreases.

Original: Consequently,
Revised: Consequently,

Original: EBCLR with batch size 16 beats SimCLR with batch size 128 .
Revised: EBCLR with batch size 16 beats SimCLR with batch size 128 .

Original: Finally, as a byproduct of the second [Table] observation, the efficiency of EBCLR relative to SimCLR increases as batch size decreases.
Revised: Finally, as a byproduct of the secondobservation, the efficiency of EBCLR relative to SimCLR increases as batch size decreases.

Original: Theseproperties make EBCLR suitable for situations where we cannot use large batch sizes, e.g., when wehave a small number of GPUs.
Revised: Theseproperties make EBCLR suitable for situations where we cannot use large batch sizes, e.g., when we [Table] have a small number of GPUs.

Original: Repeating the experiments with longer training again led to similarconclusions (c.f. Supplementary Material Section D.2).
Revised: Repeating the experiments with longer training or KNN classificationagain led to similar conclusions (c.f. Appendixes E.2 and E.4, respectively).

Original: 4.3 Effect of λ and Projection Dimension
Revised: 4.3 Effect of λ and Projection Dimension

Original: [Figure]
Revised: [Figure]

Original: We explored the effect of changing the hyperparameter λ which controls the importance ofthe generative term relative to the discriminativeterm (c.f. Equation (11)).
Revised: We explored the effect of changing the hyperparameter λ which controls the importance ofthe generative term relative to the discriminativeterm (c.f. Equation (11)).

Original: Figure 3a shows theperformance of EBCLR with various values of λas training progresses.
Revised: Figure 3a shows theperformance of EBCLR with various values of λas training progresses.

Original: We observe that naivelyusing λ = 1 . 

Revised: We observe that naivelyusing λ = 1 .

Original: leads to poor results.
Revised: leads to poor results.

Original: The performance peaks at λ = 0 .
Revised: The performance peaks at λ = 0 .

Original: 1 , and then degrades aswe further decrease λ .
Revised: 1 , and then degrades aswe further decrease λ .

Original: This result has two crucial implications.
Revised: This result has two crucial implications.

Original: First,the generative term plays a non-trivial role in
Revised: First,the generative term plays a non-trivial role in

Original: EBCLR.
Revised: EBCLR.

Original: Second, we need to strike a right balance between the discriminative term and thegenerative term to achieve good performance ondownstream tasks.
Revised: Second, we need to strike a right balance between the discriminative term and thegenerative term to achieve good performance ondownstream tasks 1 .

Original: We also investigated the effect of varying the output dimension of π θ .
Revised: We also investigated the effect of varying the output dimension of π θ .

Original: Figure 3b shows linearevaluation results for projection dimensions in { 128 , 256 , 512 } .
Revised: Figure 3b shows linearevaluation results for projection dimensions in { 128 , 256 , 512 } .

Original: We observe that the projectiondimension has essentially no influence on the training process.
Revised: We observe that the projectiondimension has essentially no influence on the training process.

Original: In this respect, EBCLR resembles
Revised: In this respect, EBCLR resembles

Original: SimCLR which is also invariant to the output dimension (c.f. Figure 8 in the work by Chen et.
Revised: SimCLR which is also invariant to the output dimension (c.f. Figure 8 in the work by Chen et.

Original: al [4]).
Revised: al [4]).

Original: 4.4 Effect of SGLD Modifications
Revised: 4.4 Effect of SGLD Modifications

Original: We now study the roles of the three SGLD modifications proposed in Section 3.3.
Revised: We now study the roles of the three SGLD modifications proposed in Section 3.3.

Original: Figure 4 shows theresults of varying one parameter of MSGLD while keeping  others fixed.
Revised: Figure 4 shows theresults of varying one parameter of MSGLD while keeping the others fixed.

Original: Effect of reintialization frequency ρ .
Revised: Effect of reintialization frequency ρ .

Original: Figure 4a displays linear evaluation results for ρ ∈{ 0 . 0 , 0 . 2 , 1 . 0 } .
Revised: Figure 4a displays linear evaluation results for ρ ∈{ 0 . 0 , 0 . 2 , 1 . 0 } .

Original: We note that setting ρ = 1 . 0 is equivalent to removing the replay buffer.
Revised: We note that setting ρ = 1 . 0 is equivalent to removing the replay buffer.

Original: Also,setting ρ = 0 . 0 is equivalent to never reinitializing SGLD chains.
Revised: Also,setting ρ = 0 . 0 is equivalent to never reinitializing SGLD chains.

Original: Initially, ρ = 0 . 0 shows the best performance, as SGLD quickly reaches samples of lower energy.
Revised: Initially, ρ = 0 . 0 shows the best performance, as SGLD quickly reaches samples of lower energy.

Original: However, learning then slows down because of the lack of diversity of samples in the replay buffer B .
Revised: However, learning then slows down because of the lack of diversity of samples in the replay buffer B .

Original: This implies  it is necessary to set ρ > 0 in order to learn good representations.
Revised: This implies that it is necessary to set ρ > 0 in order to learn good representations.

Original: On the other hand, ρ = 1 . 0 shows slow convergence in the beginning because samples in the replaybuffer are not given enough iterations to reach low energy.
Revised: On the other hand, ρ = 1 . 0 shows slow convergence in the beginning because samples in the replaybuffer are not given enough iterations to reach low energy.

Original: Although it does beat ρ = 0 . 

Revised: Although it does beat ρ = 0 .

Original: at latterepochs, it still often performs worse than ρ = 0 .
Revised: at latterepochs, it still often performs worse than ρ = 0 .

Original: Moreover, it is not sample-efficient compared toρ = 0 .
Revised: Moreover, it is not sample-efficient compared toρ = 0 .

Original: 2 since we have to provide an entire batch of new samples for reinitializing SGLD chains ateach iteration.
Revised: 2 since we have to provide an entire batch of new samples for reinitializing SGLD chains ateach iteration.

Original: Given the above observations, it is clear why the intermediate value 0 .
Revised: Given the above observations, it is clear why the intermediate value 0 .

Original: 2 is the best choice out ofρ ∈ { 0 . 0 , 0 . 2 , 1 . 0 } .
Revised: 2 is the best choice out ofρ ∈ { 0 . 0 , 0 . 2 , 1 . 0 } .

Original: ρ = 0 .
Revised: ρ = 0 .

Original: 2 allows enough time for samples in the replay buffer to reach low energywhile still maintaining  diversity of samples in B .
Revised: 2 allows enough time for samples in the replay buffer to reach low energywhile still maintaining the diversity of samples in B .

Original: Also, it is sample-efficient compared to ρ = 1 . 0 .
Revised: Also, it is sample-efficient compared to ρ = 1 . 0 .

Original: Effect of proposal distribution q 0 .
Revised: Effect of proposal distribution q 0 .

Original: Figure 4b compares linear evaluation accuracies with q 0 asthe uniform distribution and q 0 = p ( v ) .
Revised: Figure 4b compares linear evaluation accuracies with q 0 as theuniform distribution and q 0 = p ( v ) .

Original: We observe prominent acceleration in the initial epochsfor q 0{+epochs forq 
+} = p ( v ) .
Revised: We observe prominent acceleration in the initial {+epochs forq 
+} = p ( v ) .

Original: Hence, we can conclude that this choice of proposal distribution is crucial for  highefficiency of EBCLR compared to the baseline methods in Tables 1 and 3.
Revised: Hence, we can conclude that this choice of proposal distribution is crucial for the highefficiency of EBCLR compared to the baseline methods in Tables 1 and 3.

Original: [Figure]
Revised: [Figure]

Original: We believe this acceleration effect can be explainedthrough the work of Hinton [23].
Revised: We believe this acceleration effect can be explainedby the work of Hinton [19].

Original: Specifically, letus observe that the EBM update equation (3) pushesup the energy on the model distribution q θ .
Revised: Specifically, let us observe that the EBM update equation (3) pushes upthe energy on the model distribution q θ .

Original: In theimplementation of EBCLR with q 0 = p ( v ) , however,q θ is replaced by the distribution of samples createdby a finite number of (noisy) gradient steps on realdata points (c.f. Section 3.3).
Revised: In the implementation of EBCLR with q 0 = p ( v ) , however,q θ is replaced by the distribution of samples createdby a finite number of (noisy) gradient steps on realdata points (c.f. Section 3.3).

Original: Hence, the modified
Revised: Hence, the modified

Original: EMB update equation contains the curvature information of the data manifold.
Revised: EMB update equation contains the curvature information of the data manifold.

Original: This curvature informationmay expedite the training process of EBCLR.
Revised: This curvature informationmay expedite the training process of EBCLR.

Original: For adetailed discussion on this, we refer the readers to
Revised: For adetailed discussion on this, we refer the readers to

Original: Section 3 of the work by Hinton [23].
Revised: Section 3 of the work by Hinton [19].

Original: Comparison of SGLD and MSGLD.
Revised: Comparison of SGLD and MSGLD.

Original: Figure 4cshows results with SGLD with σ ∈ { 0 . 01 , 0 . 05 } and
Revised: Figure 4cshows results with SGLD with σ ∈ { 0 . 01 , 0 . 05 } and

Original: MSGLD with σ min = 0 .
Revised: MSGLD with σ min = 0 .

Original: and σ max = 0 . 05 .
Revised: and σ max = 0 . 05 .

Original: Wenote that setting σ min = σ max reduces MSGLD to
Revised: Wenote that setting σ min = σ max reduces MSGLD to

Original: SGLD.
Revised: SGLD.

Original: We observe σ = 0 .
Revised: We observe σ = 0 .

Original: 01 initially shows fast convergence but then saturates due to the lack of diversityof generated samples.
Revised: 01 initially shows fast convergence but then saturates due to the lack of diversityof generated samples.

Original: On the other hand, σ = 0 .
Revised: On the other hand, σ = 0 .

Original: 05initially has the worst performance but eventuallybeats σ = 0 .
Revised: 05initially has the worst performance but eventuallybeats σ = 0 .

Original: 01 since σ = 0 .
Revised: 01 since σ = 0 .

Original: 05 quickly explores themodes of q θ .
Revised: 05 quickly explores themodes of q θ .

Original: MSGLD inherits the best of both settings.
Revised: MSGLD inherits the best of both settings.

Original: Specifically, MSGLD is as fast as σ = 0 .
Revised: Specifically, MSGLD is as fast as σ = 0 .

Original: 01 inthe beginning, and it does not suffer from the saturation problem.
Revised: 01 inthe beginning, and it does not suffer from the saturation problem.

Original:  Limitations and Societal Impacts
Revised:  Limitations and Societal Impacts

Original: Limitations.
Revised: Limitations.

Original: The main limitation of our work is of scale.
Revised: The main limitation of our work is of scale.

Original: While EBCLR demonstrates superiorsample efficiency, it requires inner SGLD iterations (which cannot be parallelized) and a replay buffer
Revised: While EBCLR demonstrates superiorsample efficiency, it requires inner SGLD iterations (which cannot be parallelized) and a replay buffer

Original: B .
Revised: B .

Original: These two components increase the computational burden of EBCLR.
Revised: These two components increase the computational burden of EBCLR.

Original: So, we found it difficult toapply EBCLR to large-scale data such as ImageNet.
Revised: So, we found it difficult toapply EBCLR to large-scale data such as ImageNet.

Original: However, we note that inner SGLD iterationsand the replay buffer are not particular limitations of EBCLR, butare limitations of EBMs in general.
Revised: However, we note that inner SGLD iterationsand the replay buffer are not particular limitations of EBCLR, but limitations of EBMs in general.

Original: Given the increasing efforts to overcome these limitations such as Proximal-YOPO-SGLD, we believe EBCLR will eventually be applicable to larger data.
Revised: Given the increasing efforts to overcome these limitations such as Proximal-YOPO-SGLD (for morediscussion, see Appendix F), we believe EBCLR will eventually be applicable to larger data.

Original: Social Impacts.
Revised: Social Impacts.

Original: We generally expect positive outcomes from this research.
Revised: We generally expect positive outcomes from this research.

Original: Further developmentof EBCLR can mitigate the need for large amount of data and large batch sizes to learn goodrepresentations, and ultimately lead to  reduction of resource consumption.
Revised: Further developmentof EBCLR can mitigate the need for large amount of data and large batch sizes to learn goodrepresentations and ultimately lead to a reduction in resource consumption.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we proposed EBCLR which combines contrastive learning with EBMs.
Revised: In this work, we proposed EBCLR which combines contrastive learning with EBMs.

Original: This amalgamation of ideas have led to both theoretical and practical contributions.
Revised: This amalgamation of ideas has led to both theoretical and practical contributions.

Original: Theoretically, EBCLRassociates distance on the projection space with the density of positive samples.
Revised: Theoretically, EBCLRassociates distance in the projection space with the density of positive samples.

Original: Since the distributionof positive samples reflects  semantic similarity of images, EBCLR is capable of learning goodvisual representations.
Revised: Since the distributionof positive samples reflects the semantic similarity of images, EBCLR is capable of learning goodvisual representations.

Original: Practically, EBCLR is several times more sample-efficient than conventionalcontrastive and non-contrastive learning approaches and  robust to small number of negative pairs.
Revised: Practically, EBCLR is several times more sample-efficient than conventionalcontrastive and non-contrastive learning approaches and is robust to small numbers of negative pairs.

Original: Hence, EBCLR is applicable even in the scenario with limited data or devices.
Revised: Hence, EBCLR is applicable even in scenarios with limited data or devices.

Original: We believe that EBCLR makes representation learning available to a wider range of machine learning practitioners.
Revised: We believe that EBCLRmakes representation learning available to a wider range of machine learning practitioners.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu-
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu-

Original: tions and scope?
Revised: tions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Appendix B.
Revised: [Yes] See Appendix C.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental

Original: results (either in the supplemental material or asa URL)? [Yes]
Revised: results (either in the supplemental material or as

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes]
Revised: [Table]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experimentsmultiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments [Table] multiple times)?

Original: [No] Each EBM was trained once due to computation costs.
Revised: [No] Each EBM was trained once due to computation costs.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,internal cluster, or cloud provider)?

Original: [Yes] See Appendix C.4.
Revised: [Yes] See Appendix D.

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: a) If your work uses existing assets, did you cite the creators

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: b) you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Table] c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re us-
Revised: d) Did you discuss whether and how consent was obtained from people whose data you’re us-

Original: ing/curating?
Revised: ing/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: e) Did you discuss whether the data you are using/curating contains personally identifiable informa- [Table] tion or offensive content?

Original: [N/A]5.
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: applicable?

Original: [Table] 
Revised: [Table] N/A]

Original: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: you describe any potential participant , with to Institutional Review Board (IRB)

Original: [Table]
Revised: [Table]

Original:  [N/A] 
Revised: approvals, if applicable? [N/A] c) Did you include the estimated hourly wage paid to participants and the total amount spent on

Original: [Table]
Revised: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)approvals, if applicable?
Revised: participant compensation?

Original: [N/A](c)
Revised: [N/A]

Original: Did you include the estimated hourly wage paid to participants and the total amount spent onparticipant compensation?
Revised: Contrastive divergence, initially proposed by Hinton et al.

Original: [N/A]
Revised: [19], is a particular method for solving the maximum- 448 likelihood estimation problem for EBMs. Intuitively, it proceeds by minimizing the energy function on dataand maximizing the energy function on (possibly short-run) MCMC samples of the current EBM distribution 450 through gradient ascent. In fact, we use an instance of contrastive divergence to optimize the generative term inour EBCLR objective (11).

Original: [Table]
Revised: We also disambiguate contrastive learning from contrastive divergence. They are similar in the sense that they 453 contrast certain pairs of data. Contrastive learning contrasts positive pairs and negative pairs. Contrastivedivergence contrasts data and MCMC samples.

Revised: However, they differ in the aspect that contrastive learning 455 is inherently a discriminative learning method (in contrastive learning, the model “predicts” the positive pairamong negative pairs) and contrastive divergence is a generative learning method.

Revised: In the context of unsupervised 457 visual representation learning, to the best of our knowledge, there are no works which use discriminative andgenerative losses simultaneously.

Revised: Our work, for the first time, explores the synergy between contrastive learning 459 (the discriminative term in (11)) and contrastive divergence (the generative term in (11)) for learning visualrepresentations.

Revised: A.2 Relation to Contrastive Energy-Based Models (CEMs) [26]

Revised: According to the terms introduced in the work of Wang et al.

Revised: [26], our EBCLR model distribution (6) with 463 τ = 1 is identical to the model distribution of Non-Parametric CEM ((4) in [26]).

Revised: This is because the squaredℓ 2 norm between two unit-norm vectors z and z ′ is equivalent to the inner product between z and z ′ up to anadditive constant.

Revised: However, our work differs from the work of Wang et al.

Revised: [26] in three aspects.

Revised: Objective function.

Revised: Wang et al.

Revised: [26] optimizes the log-likelihood of the joint model distribution q θ ( v, v ′ )

Revised: directly whereas we use Bayes’ rule to decompose the objective into two terms, the discriminative term and the 468 generative term.

Revised: This allows us to reweight the generative term to learn better visual representations (as exploredin Section 4.3).

Revised: Optimization method.

Revised: Wang et al.

Revised: [26] maximizes the log-likelihood of the joint model distribution directly 471 via adversarial training.

Revised: On the other hand, we use gradient ascent to optimize the discriminative term andcontrastive divergence to optimize the generative term.

Revised: Computation costs are similar, as adversarial training 473 also requires inner iterations to generate adversarial examples at every model update step.

Revised: Application of interest.

Revised: Wang et al.

Revised: [26] focuses on enhancing image generation, while we focus on learning 475 visual representations useful for downstream tasks.

Revised: Wang et al.

Revised: [26] also establishes a connection between contrastive divergence and InfoNCE (Section 5.1 in 477 [26]).

Revised: Specifically, Wang et al.

Revised: [26] show that when we approximate the model marginal q θ ( v ) with the datadistribution p ( v ) , MLE gradient for q θ ( v, v ′ ) becomes the InfoNCE loss gradient.

Revised: Surprisingly, we find thisholds in our framework as well.

Revised: If we set q θ ( v ) = p ( v ) in (11), the generative term in (11) becomes independent 480 of θ , so we end up optimizing only the discriminative term.

Revised: We have shown that the discriminative term can beapproximated by a contrastive learning objective, so we recover the result of Wang et al.

Revised: In this sense, the 482 derivations in Section 3 of our paper generalizes the connection discovered in the work of Wang et al.

Revised: [26], aswe work with weaker assumptions (we do not set q θ ( v ) =

Revised: p ( v ) in (11)).

Revised: A.3 EBMs for Learning Useful Representations

Revised: Ranzato et al.

Revised: [27] proposes to train an EBM with an autoencoder structure by minimizing the sum of an 486 encoding loss and a decoding loss to learn an energy surface.

Revised: A byproduct is that the EBM learns a compressedrepresentation of the data, which can be used for, e.g., denoising.

Revised: Our work is similar in the aspect that we use 488 a particular EBM architecture (a triplet network) and training the EBM with our proposed method results ina compressed representation of the data which is useful for downstream tasks, e.g., classification and transfer 490 learning.

Revised: In that sense, our work is also roughly related to Denoising Autoencoders (DAEs) [28] and Restricted

Revised: Boltzmann Machines (RBMs) as well.

Revised: Both are EBMs whose goal is to extract useful representations from data.

Revised: However, we cannot say Ranzato et al.

Revised: [2], RBMs, or DAEs precede our work in relating contrastive learning to

Revised: EBMs, because they mention anything about contrasting positive pairs against negative pairs.

Revised: A.4 Sample Efficiency of Generative Models

Revised: We motivate the sample efficiency of EBCLR through the lens of discriminative and generative learning.

Revised: We 496 note that standard contrastive learning (e.g., SimCLR) optimizes a discriminative objective (the model mustpredict positive pairs) and EBCLR optimizes a generative objective (given a pair of images, the model must output how likely it is to be a positive pair).

Revised: So, we can compare the performance of discriminative models and 499 generative models.

Revised: There is a long line of works which show (theoretically and / or empirically) that generative models can 501 outperform discriminative ones.

Revised: For instance, Efron [29] has theoretically shown that Normal Discriminant

Revised: Analysis can be more efficient than logistic regression.

Revised: Ng et al.

Revised: [30] has also theoretically shown that in the 503 small sample regime, naïve Bayes can be more resistant to overfitting than logistic regression (i.e., find bettersolutions when given a small number of training samples).

Revised: There are also works [8, 9, 10] that consider optimizing a weighted combination of a discriminative loss and a 506 generative loss, just like our EBCLR objective (11).

Revised: They find that with an appropriate weight, the model canoutperform discriminative models.

Revised: In particular, Larochelle et al.

Revised: [9] finds this hybrid approach to be beneficial 508 in the limited training data setting.

Revised: Grathwohl et al.

Revised: [10] uses this hybrid approach to train a classifier and findthat the classifier exhibits excellent calibration, unlike other calibration methods that require additional training 510 data.

Revised: We can also address the efficiency of generative models from the perspective of generative pre-training.

Original: {+Bengio 
+}
Revised: {+Bengio 
+}

Revised: et al.

Revised: [31] finds that pre-training a network with a generative loss brings the network weights towards a goodlocal minimum, thus bringing better generalization.

Revised: Erhan et al.

Revised: [32] finds that generative pre-training acts as a 514 regularizer and that this regularization effect can be beneficial when we have a small training set.

Revised: Another explanation specific to EBCLR is that the generative loss has an effect orthogonal to the effect of the 516 discriminative loss.

Revised: To minimize the generative loss, the model needs to place low energy on data and highenergy elsewhere.

Revised: This can be achieved only when the model has features that accurately capture the manifold 518 of the data.

Revised: When we have a small training set, the model may easily solve the discriminative objective andoverfit.

Revised: By providing an auxiliary generative task, the overfitting can be mitigated.

Revised: An analogous effect with 520 Joint Energy-Based Models is demonstrated by Zhao et al.

Revised: B Psuedocodes

Revised: C Missing Proofs

Revised: C.1 Proof of Theorem 2

Revised: Proof.

Revised: We first observe that

Revised: [Equation]

Revised: which proves the relation (12).

Revised: We also have

Revised: [Equation]

Revised: and so

Revised: [Equation]

Revised: Taking expectation w.r.t.

Revised: p ( v ) establishes the relation (13).

Revised: C.2 Another Theoretical Justification for EBCLR

Revised: Here we provide another theoretical justification on how solving the EBCLR objective (11) with λ = 1 causes 531 q θ ( v, v ′ ) to approximate p ( v, v ′ ) .

Revised: It is known that optimizing the first term of (11) with (18) in place of q θ ( v ′ | v )will cause (18) to approximate p ( v ′ | v ) [34].

Revised: Also, optimizing the second term of (11) with (15) or (17) in placeof q θ ( v ) will cause (15) or (17) to be proportional to p ( v ) (indeed, in Table 10, we see SGLD samples of the 534 EBCLR marginal q θ ( v ) approximated by (17) achieve a non-trivial FID score).

Revised: So, the product of (15) or (17) 535 with (18) will be proportional to p ( v, v ′ ) .

Revised: Moreover, by construction, the product of (15) or (17) with (18) is(approximately) q θ ( v, v ′ ) .

Revised: Hence, optimizing the EBCLR objective (11) will cause q θ ( v, v ′ ) to model p ( v, v ′ ) .

Revised: D Complete Training and Evaluation Details

Revised: Baseline methods and datasets.

Revised: The baseline methods are SimCLR, MoCo v2, SimSiam, and BYOL.

Revised: The 540 hyper-parameters are chosen closely following the original works [4, 12, 7, 6].

Revised: We use four datasets: MNIST[21], Fashion MNIST (FMNIST) [22], CIFAR10, and CIFAR100 [23].

Revised: Data transformation.

Revised: For fair comparison, we use the stochastic data transformation proposed by Chen et.

Revised: al [4] 543 for all methods.

Revised: In the case of grayscale images, we only use the random cropping augmentation.

Revised: For EBCLR,we also add small Gaussian noise N (0 , 0 .

Revised: 03 2 ) to stabilize training [15, 10, 14].

Revised: We remark that Gaussian noiseof standard deviation 0 .

Revised: 03 is nearly invisible to the human eye.

Revised: DNN architecture.

Revised: We decompose f θ = π θ ◦ ϕ θ where ϕ θ is the encoder network and π θ is the projection 547 network.

Revised: Rather than directly using the output of f θ for downstream tasks, we follow previous works [4, 5, 1, 2,3, 6, 7] and use the output of ϕ θ instead.

Revised: In our experiments, we set ϕ θ to be a ResNet-18 [24] up to the global average pooling layer.

Revised: The architecture of 550 π θ and π θ to be a 2-layer MLP with output dimension 128 .

Revised: However, we remove batch normalization because 551 batch normalization hurts SGLD [15].

Revised: We also replace ReLU activations with leaky ReLU to expedite theconvergence of SGLD.

Revised: For the baseline methods, we use the settings proposed in the original works while 553 keeping the backbone fixed to be ResNet-18.

Revised: SGLD settings.

Revised: The default parameters for MSGLD are |B| =

Revised: 50 k , ρ = 0 .

Revised: 2 , T = 10 , α = 0 .

Revised: 05 , δ = 1 . 0 , 555 K = 3 , σ min = 0 .

Revised: 01 , and σ max = 0 .

Revised: We use ρ = 0 .

Revised: 4 for MNIST, ρ = 0 .

Revised: 6 for FMNIST, and ρ = 0 . 2 for 556 CIFAR10 and CIFAR100.

Revised: At the beginning of training, we fill the buffer B entirely with proposal distributionsamples.

Revised: Training.

Revised: The temperature is τ = 0 .

Revised: 1 for EBCLR.

Revised: For EBCLR, as we cannot use batch normalization, it 559 is difficult to apply SGD with momentum.

Revised: Hence, we use the Adam optimizer [35].

Revised: The learning rate is0 .

Revised: 0002 if the batch size is 128 and 0 .

Revised: if the batch size is smaller than 128 .

Revised: We also weakly regularize 561 the squared ℓ 2 -norm of the outputs of f θ with weight 0 .

Revised: For the baseline methods, we closely follow the 562 settings of previous works [4, 12, 25].

Revised: Specifically, we use SGD with momentum 0.9 and learning rate of0 .

Revised: 03 · ( batch size / 256) and weight decay 0 . 0001 .

Revised: Each model is trained with a single GeForce RTX 3090.

Revised: Evaluation.

Revised: We assess the quality of the representations by training a linear classifier on top of frozen ϕ θ .

Revised: The 565 linear classifier is trained with Adam for 200 epochs with batch size 512 .

Revised: The learning rate η is found by gridsearch for log 10 η in [ − 4 , − 1] .

Revised: E Additional ExperimentsE.1 Section 4.1 with Longer Training

Revised: We repeat the experiments in Section 4.1 by training the models for 200 epochs.

Revised: For EBCLR, the number of 570 SGLD iterations T is increased by 5 when SGLD is unable to generate samples with sufficiently high energy.

Revised: Tables 4 and 5 describe the results.

Revised: We again note that EBCLR consistently beats all baseline methods and 572 demonstrates high efficiency.

Revised: [Table]

Revised: [Table]

Revised: E.2 Section 4.2 with Longer Training

Revised: We repeat the experiments in Section 4.2 by training the models for 200 epochs.

Revised: For EBCLR, the number of 575 SGLD iterations T is increased by 5 when SGLD is unable to generate samples with sufficiently high energy.

Revised: Table 6 describes the results.

Revised: We observe similar patterns as in the case of training for 100 epochs.

Revised: In the 577 case of CIFAR10, SimCLR seems to have a pattern agnostic to the batch size.

Revised: However, we note that there isa significant gap between the performance of SimCLR with batch size 256 (c.f.

Revised: Table 4) and batch sizes in 579 { 16 , 64 , 128 } .

Revised: Interestingly, for SimCLR with small batch sizes, we observe that training longer with small batch sizes leads to 581 worse performance than training only for 100 epochs.

Revised: This is particularly prominent on CIFAR10 and CIFAR100(compare Tables 3 and 6).

Revised: We speculate that this is because, with small batches, SimCLR is unable to provide 583 hard negatives, and this leads to overfitting of the DNN.

Revised: On the other hand, EBCLR does not suffer from thisphenomenon.

Revised: This again demonstrates the importance of the generative term.

Revised: [Table]

Revised: EBCLR is calculated by the following formula: (number of epochs used by EBCLR with the same batch size to reach the final accuracy of SimCLR) /

Revised: (total number of training epochs).

Revised: E.3 Comparison of Approximations

Revised: We can either use Equation (15) or Equation (17) when approximating the generative term.

Revised: Figure 5 shows 587 results for both approximations on CIFAR10.

Revised: We observe identical performance for both approximations, whichimplies that the generative term is robust to the number of samples used in the calculation of the empirical mean.

Revised: This justifies our choice of using the simpler Equation (17) to approximate the marginal distribution q ( v ) .

Revised: E.4 Evaluation with KNN Classifiers

Revised: We also reproduce Tables 1,

Revised: [Table]

Revised: and 3 using K-nearest neighbour (KNN) classifiers.

Revised: We followed the evaluation 592 protocol of Caron et al.

Revised: [36] and used a weighted KNN classifier with K = 20 .

Revised: The results are given below.

Revised: In

Revised: Table 7, except for the case of BYOL on MNIST, we see that EBCLR beats all baselines again by a non-trivial 594 margin.

Revised: EBCLR is also shown to be sample efficient.

Revised: In Table 8, we see EBCLR outperforms all baselineson the task of transfer learning.

Revised: In Table 9, we see EBCLR is robust to small batch sizes.

Revised: This leads to the 596 lower efficiency of SimCLR compared to EBCLR.

Revised: On CIFAR10 and CIFAR100, relative efficiency does notdecrease below 0.05 since we saved checkpoints every 5 epochs.

Revised: We can reasonably expect SimCLR to have 598 lower relative efficiency at smaller batch sizes if we had saved checkpoints every epoch.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: E.5 Generative Performance of EBCLR

Revised: In this section, we explore the effect of the discriminative loss on the generative performance of EBCLR.

Revised: To this 601 end, we solved

Revised: [Equation]

Revised: (23) for various values of γ on FMNIST.

Revised: Recall that in the original EBCLR objective, weight λ comes in front of 603 the generative term.

Revised: We then measured the Fréchet Inception Distance (FID, lower is better) between samplesfrom the true marginal p ( v ) and SGLD samples from the EBM marginal q θ ( v ) approximated by (19) / (17).

Revised: The 605 results are shown in Table 10.

Revised: Interestingly, we found using an appropriate γ > 0 led to improvements in FID over γ = 0 .

Revised: In other words, 607 contrastive learning did help improve the generative performance of the model.

Revised: However, using an excessivelylarge γ led to a deterioration of the performance.

Revised: This is analogous to the result of Section 4.3 where we observed 609 using an appropriate weight on the generative term led to improvements over using only the discriminativeterm (contrastive loss).

Revised: The difference is that to achieve optimal generative performance, the generative term 611 weight needs to be larger (i.e., γ < 1 ) whereas to achieve optimal discriminative performance, the discriminativeterm weight needs to be larger (i.e., λ < 1 in (11)).

Revised: We speculate that using a larger model could mitigate this 613 trade-off, but this topic is beyond the scope of this work.

Revised: [Table]

Revised: F A Discussion on Making EBCLR Scalable

Revised: We could consider three alternative losses / methods for training EBCLR: noise contrastive estimation (NCE) 616 [37], VERA [38], and score matching [39].

Revised: Although the NCE loss does not depend on MCMC, it requiresspecification of a noise distribution whose normalized density can be easily evaluated.

Revised: In most cases, such noise 618 distribution samples are easily distinguishable from natural image samples and lead to the density chasm problem[40], making optimization difficult.

Original: {+Telescoping density-ratio estimation [40] attempts to mitigate this problem, 
+}
Revised: {+Telescoping density-ratio estimation [40] attempts to mitigate this problem, 
+}

Revised: but it does not seem to scale well to large data.

Revised: Flow contrastive estimation [41] uses a flow model to define thenoise distribution and trains a flow model along with the EBM.

Revised: Though flow contrastive estimation requires an 622 auxiliary flow model, we believe this is the most promising alternative to MCMC.

Revised: In a similar sense, VERA,which uses a generator to sample from the EBM, seems promising as well.

Revised: Yet, flow contrastive estimation and 624 VERA also did not demonstrate scalability to large-scale datasets such as ImageNet.

Revised: Score matching matchesthe gradient of model log-density to the gradient of data log-density, so it does not rely on MCMC.

Revised: However, it 626 requires calculation of the Hessian, so in the deep learning setting, this approach is also quite expensive.