Original: Energy-Based Contrastive Learning of Visual Representations
Revised: Energy-Based Contrastive Learning of Visual Representations

Original: Anonymous Author(s)

Original: AffiliationAddress

Original: Abstract
Revised: Abstract

Original: Contrastive learning is a method of learning visual representations by training
Revised: Contrastive learning is a method of learning visual representations by training

Original: Deep Neural Networks (DNNs) to increase the similarity between representationsof positive pairs (transformations of the same image) and reduce the similaritybetween representations of negative pairs (transformations of different images).
Revised: Deep Neural Networks (DNNs) to increase the similarity between representations of positive pairs (transformations of the same image) and reduce the similarity between representations of negative pairs (transformations of different images).

Original: Here we explore Energy-Based Contrastive Learning (EBCLR) that leveragesthe power of generative learning by combining contrastive learning with EnergyBased Models (EBMs).
Revised: Here we explore Energy-Based Contrastive Learning (EBCLR) that leverages the power of generative learning by combining contrastive learning with EnergyBased Models (EBMs).

Original: EBCLR can be theoretically interpreted as learning thejoint distribution of positive pairs, and it shows promising results on small andmedium-scale datasets such as MNIST, Fashion-MNIST, CIFAR-10, and CIFAR100.
Revised: EBCLR can be theoretically interpreted as learning the joint distribution of positive pairs, and it shows promising results on small and medium-scale datasets such as MNIST, Fashion-MNIST, CIFAR-10, and CIFAR100.

Original: Specifically, we find EBCLR demonstrates from × 4 up to × 20 accelerationcompared to SimCLR and MoCo v2 in terms of training epochs.
Revised: Specifically, we find EBCLR demonstrates from × 4 up to × 20 acceleration compared to SimCLR and MoCo v2 in terms of training epochs.

Original: Furthermore, incontrast to SimCLR, we observe EBCLR achieves nearly the same performancewith 254 negative pairs (batch size 128 ) and 30 negative pairs (batch size 16 ) perpositive pair, demonstrating the robustness of EBCLR to small numbers of negativepairs.
Revised: Furthermore, in contrast to SimCLR, we observe EBCLR achieves nearly the same performance with 254 negative pairs (batch size 128 ) and 30 negative pairs (batch size 16 ) per positive pair, demonstrating the robustness of EBCLR to small numbers of negative pairs.

Original: Hence, EBCLR provides a novel avenue for improving contrastive learningmethods that usually require large datasets with a significant number of negativepairs per iteration to achieve reasonable performance on downstream tasks. 
Revised: Hence, EBCLR provides a novel avenue for improving contrastive learning methods that usually require large datasets with a significant number of negative pairs per iteration to achieve reasonable performance on downstream tasks. Code: https://github.com/1202kbs/EBCLR

Original:  Introduction
Revised:  Introduction

Original: In computer vision supervised learning requires a large-scale human-annotated dataset of images totrain accurate deep neural networks (DNNs).
Revised: In computer vision, supervised learning requires a large-scale human-annotated dataset of images to train accurate deep neural networks (DNNs).

Original: However, acquiring labels for millions of images can bedifficult or impossible in practice.
Revised: However, acquiring labels for millions of images can be difficult or impossible in practice.

Original: This has led to the rise of self-supervised learning, which learnsuseful visual representations by forcing DNNs to be invariant or equivariant to image transformations.
Revised: This has led to the rise of self-supervised learning, which learns useful visual representations by forcing DNNs to be invariant or equivariant to image transformations.

Original: Among self-supervised learning algorithms, contrastive methods are rapidly gaining popularity fortheir superb performance.
Revised: Among self-supervised learning algorithms, contrastive methods are rapidly gaining popularity for their superb performance.

Original: Specifically, contrastive learning methods [1, 2, 3, 4, 5] train DNNs by increasing the similaritybetween representations of positive pairs (transformations of the same image) and decreasing thesimilarity between representations of negative pairs (transformations of different images).
Revised: Specifically, contrastive learning methods [1, 2, 3, 4, 5] train DNNs by increasing the similarity between representations of positive pairs (transformations of the same image) and decreasing the similarity between representations of negative pairs (transformations of different images).

Original: Thenegative pairs prevent DNNs from collapsing to the trivialsolution, i.e., the constant function.
Revised: The negative pairs prevent DNNs from collapsing to the trivial constant function.

Original: Thereare numerousvariants of contrastive learning methods, such as SimCLR [4], Momentum Contrast(MoCo) [5], etc.
Revised: There are numerous contrastive learning methods, such as SimCLR [4], Momentum Contrast (MoCo) [5], etc.

Original: Despite this flurry of research in contrastive learning, contrastive methods require large datasets anda large number of negative pairs per positive pair to achieve reasonable performance on downstreamtasks.
Revised: Despite this flurry of research in contrastive learning, contrastive methods require large datasets and a large number of negative pairs per positive pair to achieve reasonable performance on downstream tasks.

Original: Although there are recently proposed non-contrastive methods such as BYOL [6] and SimSiam[7] that do not rely on negative pairs , they require heuristic techniques such as stop-gradient to avertcollapsing to the trivial solution.
Revised: Although there are recently proposed non-contrastive methods such as BYOL [6] and SimSiam [7] that do not rely on negative pairs, they require heuristic techniques such as stop-gradient to avert collapsing to the trivial solution.

Original: There has been an effort to explain the dynamics of non-contrastivemethods with linear neural networks [3], but it is unclear how the analyses generalize to DNNs.
Revised: There has been an effort to explain the dynamics of non-contrastive methods with linear neural networks [3], but it is unclear how the analyses generalize to DNNs.

Original: 𝑣 + 𝑣 −
Revised: 𝑣 + 𝑣 −

Original: Shared Params 𝑣 ∝1/𝑝(𝑣,𝑣 + ) 𝑓 𝜃 𝑓
Revised: Shared Params 𝑣 ∝1/𝑝(𝑣,𝑣 + ) 𝑓 𝜃 𝑓

Original: 𝜃 𝑓 𝜃 𝑧 𝑧 + 𝑧 −
Revised: 𝜃 𝑓 𝜃 𝑧 𝑧 + 𝑧 −

Original: Projection Space 𝑧
Revised: Projection Space 𝑧

Original: + 𝑧 𝑧 − 𝑝 𝑣,𝑣 + ≫ 𝑝(𝑣,𝑣 − )
Revised: + 𝑧 𝑧 − 𝑝 𝑣,𝑣 + ≫ 𝑝(𝑣,𝑣 − )

Original: Projections ∝1/𝑝(𝑣,𝑣 − )
Revised: Projections ∝1/𝑝(𝑣,𝑣 − )

Original: In this paper, we explore a novel avenue in visual representation learning: Energy-Based Contrastive
Revised: In this paper we explore a novel avenue in visual representation learning: Energy-Based Contrastive

Original: Learning (EBCLR) which leverages the power of generative learning [8, 9, 10] by combiningcontrastive learning with energy-based models (EBMs).
Revised: Learning (EBCLR) which leverages the power of generative learning [8, 9, 10] by combining contrastive learning with energy-based models (EBMs).

Original: EBCLR complements the contrastivelearning loss with a generative loss, and it can be interpreted as learning the joint distribution ofpositive pairs.
Revised: EBCLR complements the contrastive learning loss with a generative loss and it can be interpreted as learning the joint distribution of positive pairs.

Original: In fact, we demonstrate that the existing contrastive loss is a special case of the EBCLRloss if the generative term is not used.
Revised: In fact we demonstrate that the existing contrastive loss is a special case of the EBCLR loss if the generative term is not used.

Original: Although EBMs are notorious for being difficult to traindue to their reliance on Stochastic Gradient Langevin Dynamics (SGLD) [11], another importantcontribution of this work is that we overcome this by appropriate modifications to SGLD.
Revised: Although EBMs are notorious for being difficult to train due to their reliance on Stochastic Gradient Langevin Dynamics (SGLD) [11], another important contribution of this work is that we overcome this by appropriate modifications to SGLD.

Original: Extensive experiments on a variety of small and medium-scale datasets demonstrate that EBCLR isrobust to small numbers of negative pairs, and it outperforms SimCLR and MoCo v2 [12] in terms ofsample efficiency and linear evaluation accuracy.
Revised: Extensive experiments on a variety of small and medium-scale datasets demonstrate that EBCLR is robust to small numbers of negative pairs, and it outperforms SimCLR and MoCo v2 [12] in terms of sample efficiency and linear evaluation accuracy.

Original: Hence, EBCLR opens up a new research directionfor alleviating the dependence of contrastive methods on large datasets and large batches.
Revised: Hence, EBCLR opens up a new research direction for alleviating the dependence of contrastive methods on large datasets and large batches.

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: • We propose a novel contrastive learning method called EBCLR which learns the joint distribution of positive pairs.
Revised: • We propose a novel contrastive learning method called EBCLR which learns the joint distribution of positive pairs.

Original: We show that EBCLR loss is equivalent to a combination of acontrastive term and a generative term (Section 3).
Revised: We show that EBCLR loss is equivalent to a combination of a contrastive term and a generative term (Section 3).

Original: To the best of our knowledge, this is the first work to apply EBMs to contrastive learning of visual representations.
Revised: To the best of our knowledge, this is the first work to apply EBMs to contrastive learning of visual representations.

Original: • We show that EBCLR offers two advantages over conventional contrastive learning methods: EBCLR is several times more sample efficient (Section 4.1) and robust to small batch sizes (Section 4.2).
Revised: • We show that EBCLR offers two advantages over conventional contrastive learning methods: EBCLR is several times more sample efficient (Section 4.1) and robust to small batch sizes (Section 4.2).

Original: These factors lead to a non-trivial performance gain for EBCLR.
Revised: These factors lead to a non-trivial performance gain for EBCLR.

Original: • We perform thorough ablation studies of the components of EBCLR: effect of changing the weight of the generative term (Section 4.3), effect of projection space dimension (Section4.3), and the effect of the proposed SGLD modifications (Section 4.4).
Revised: • We perform thorough ablation studies of the components of EBCLR: effect of changing the weight of the generative term (Section 4.3), effect of projection space dimension (Section 4.3), and the effect of the proposed SGLD modifications (Section 4.4).

Original:  Related Works
Revised:  Related Works

Original: In this section, we go over related works necessary for understanding EBCLR.
Revised: In this section, we go over related works necessary for understanding EBCLR.

Original: In Appendix A, wegive a more extensive review of relevant works for those not familiar with EBMs, contrastive learning,or generative models.
Revised: In Appendix A, we give a more extensive review of relevant works for those not familiar with EBMs, contrastive learning, or generative models.

Original: 2.1 Contrastive Learning
Revised: 2.1 Contrastive Learning

Original: For a given batch of images { x n } Nn =1 and two image transformations t , t ′ , contrastive learningmethods first create two views v n = t ( x n ) , v ′ n = t ′ ( x n ) of each instance x n .
Revised: For a given batch of images { x n } Nn =1 and two image transformations t , t ′ , contrastive learning methods first create two views v n = t ( x n ) , v ′ n = t ′ ( x n ) of each instance x n .

Original: Here, the pair ( v n , v ′ m ) is called a positive pair if n = m and a negative pair if n ̸ = m .
Revised: Here, the pair ( v n , v ′ m ) is called a positive pair if n = m and a negative pair if n ̸ = m .

Original: Given a DNN f θ , the views are thenembedded into the projection space by passing the views through f θ and normalizing.
Revised: Given a DNN f θ , the views are then embedded into the projection space by passing the views through f θ and normalizing.

Original: Contrastive methods train f θ to increase agreement between projections of positive pairs and decreaseagreement between projections of negative pairs.
Revised: Contrastive methods train f θ to increase agreement between projections of positive pairs and decrease agreement between projections of negative pairs.

Original: Specifically, f θ is trained to maximize the InfoNCEobjective [1].
Revised: Specifically, f θ is trained to maximize the InfoNCE objective [1].

Original: After training, outputs from the final layer or an intermediate layer of f θ are used fordownstream tasks.
Revised: After training, outputs from the final layer or an intermediate layer of f θ are used for downstream tasks.

Original: There are numerous variants of contrastive methods.
Revised: There are numerous variants of contrastive methods.

Original: For instance, SimCLR [4] uses a compositionof random cropping, random flipping, color jittering, color dropping, and blurring as the imagetransformation.
Revised: For instance, SimCLR [4] uses a composition of random cropping, random flipping, color jittering, color dropping, and blurring as the image transformation.

Original: Negative pairs are created by transforming different images within a batch.
Revised: Negative pairs are created by transforming different images within a batch.

Original: On theother hand, MoCo [12] maintains a queue of negative samples, so negative samples are not limited toviews of images from the same batch.
Revised: On the other hand, MoCo [12] maintains a queue of negative samples, so negative samples are not limited to views of images from the same batch.

Original: 2.2 Energy-Based Models
Revised: 2.2 Energy-Based Models

Original: Given a scalar-valued energy function E θ ( v ) with parameter θ , an energy-based model (EBM) [13]defines a distribution by the formula
Revised: Given a scalar-valued energy function E θ ( v ) with parameter θ , an energy-based model (EBM) [13] defines a distribution by the formula

Original: [Equation]
Revised: [Equation]

Original: 1) where Z ( θ ) is the partition function which guarantees q θ integrates to 1 .
Revised: where Z ( θ ) is the partition function which guarantees q θ integrates to 1 .

Original: Since there are essentially norestrictions on the choice of the energy function, EBMs have great flexibility in modeling distributions.
Revised: Since there are essentially no restrictions on the choice of the energy function, EBMs have great flexibility in modeling distributions.

Original: Hence, EBMs have been applied to a wide variety of machine learning tasks, such as dimensionalityreduction via autoencoding [14], learning generative classifiers [8, 9, 10, 15], generating images [16],and training regression models [17, 18].
Revised: Hence, EBMs have been applied to a wide variety of machine learning tasks, such as dimensionality reduction via autoencoding [14], learning generative classifiers [8, 9, 10, 15], generating images [16], and training regression models [17, 18].

Original: Wang et al.
Revised: Wang et al.

Original: [19] have explored connections between EBMsand InfoNCE to enhance generative performance of EBMs.
Revised: [19] have explored connections between EBMs and InfoNCE to enhance generative performance of EBMs.

Original: However, to the best of our knowledge,this paper is the first to combine EBMs with contrastive learning for representation learning.
Revised: However, to the best of our knowledge, this paper is the first to combine EBMs with contrastive learning for representation learning.

Original: Given a target distribution, an EBM can be used to estimate its density p when we can only samplefrom p .
Revised: Given a target distribution, an EBM can be used to estimate its density p when we can only sample from p .

Original: One way of achieving this is by minimizing the Kullback-Leibler (KL) divergence betweenq θ and p that maximizes the expected log-likelihood of q θ under p [20]:
Revised: One way of achieving this is by minimizing the Kullback-Leibler (KL) divergence between q θ and p that maximizes the expected log-likelihood of q θ under p [20]:

Original: [Equation]
Revised: [Equation]

Original: Stochastic gradient ascent can be used to solve (2) [20].
Revised: Stochastic gradient ascent can be used to solve (2) [20].

Original: Specifically, the gradient of the expectedlog-likelihood with respect to the parameters θ can be shown to be
Revised: Specifically, the gradient of the expected log-likelihood with respect to the parameters θ can be shown to be

Original: [Equation]
Revised: [Equation]

Original: Hence, updating θ with (3) amounts to pushing up on the energy for samples from q θ and pushingdown on the energy for samples from p .
Revised: Hence, updating θ with (3) amounts to pushing up on the energy for samples from q θ and pushing down on the energy for samples from p .

Original: This optimization method is also known as contrastivedivergence [21].
Revised: This optimization method is also known as contrastive divergence [21].

Original: While the second term in (3) can be easily calculated as we have access to samples from p , the firstterm requires sampling from q θ .
Revised: While the second term in (3) can be easily calculated as we have access to samples from p , the first term requires sampling from q θ .

Original: Previous works [16, 22, 10, 15] have used Stochastic Gradient
Revised: Previous works [16, 22, 10, 15] have used Stochastic Gradient

Original: Langevin Dynamics (SGLD)
Revised: Langevin Dynamics (SGLD)

Original: [11] to generate samples from q θ .
Revised: [11] to generate samples from q θ .

Original: Specifically, given a sample v 0 fromsome proposal distribution q 0 , the iteration
Revised: Specifically, given a sample v 0 from some proposal distribution q 0 , the iteration

Original: [Equation]
Revised: [Equation]

Original: guarantees that the sequence { v t } converges to a sample from q θ assuming { α t } decays at a polynomial rate [11].
Revised: guarantees that the sequence { v t } converges to a sample from q θ assuming { α t } decays at a polynomial rate [11].

Original: However, SGLD requires an infinite number of steps until samples from the proposal distributionconverge to samples from the target distribution.
Revised: However, SGLD requires an infinite number of steps until samples from the proposal distribution converge to samples from the target distribution.

Original: This is unfeasible, so in practice, only a finitenumber of steps along with constant step size, i.e. α t = α and constant noise variance σ t = σ 2 areused [16, 22, 10, 15].
Revised: This is unfeasible, so in practice, only a finite number of steps along with constant step size, i.e. α t = α and constant noise variance σ t = σ 2 are used [16, 22, 10, 15].

Original: Moreover, Yang and Ji [15] noted SGLD often generates samples with extremepixel values that cause EBMs to diverge during training.
Revised: Moreover, Yang and Ji [15] noted SGLD often generates samples with extreme pixel values that cause EBMs to diverge during training.

Original: Hence, they have proposed proximal SGLDwhich clamps gradient values into an interval [ − δ, δ ] for a threshold δ > 0 .
Revised: Hence, they have proposed proximal SGLD which clamps gradient values into an interval [ − δ, δ ] for a threshold δ > 0 .

Original: Then, the update equationbecomes
Revised: Then, the update equation becomes

Original: [Figure]

Original: [Equation]
Revised: [Equation]

Original: for t = 0 , . . .
Revised: for t = 0 , . . .

Original: , T − 1 , where ϵ ∼ N (0 , σ 2 ) and clamp {· , δ } clamps each element of the input vectorinto [ − δ, δ ] .
Revised: , T − 1 , where ϵ ∼ N (0 , σ 2 ) and clamp {· , δ } clamps each element of the input vector into [ − δ, δ ] .

Original: In our work, we introduce additional modifications to SGLD which accelerate theconvergence of EBCLR.
Revised: In our work, we introduce additional modifications to SGLD which accelerate the convergence of EBCLR.

Original: 𝑡 ∼ 𝒯 𝑣 𝑛 ′ 𝑣 𝑛 𝑧 𝑛′ ෤𝑣 𝑛 𝑧 𝑛 ǁ𝑧 𝑛
Revised: 𝑡 ∼ 𝒯 𝑣 𝑛 ′ 𝑣 𝑛 𝑧 𝑛′ ෤𝑣 𝑛 𝑧 𝑛 ǁ𝑧 𝑛

Original: Contrastive Loss 𝑥
Revised: [Figure] Loss 𝑥

Original: 𝑛 −𝜆 ⋅𝐸 𝜃 ( ǁ𝑧 𝑛 ;{𝑧 𝑚′ }) 𝜆 ⋅𝐸 𝜃 (𝑧 𝑛 ;{𝑧 𝑚 ′ })
Revised: 𝑛 −𝜆 ⋅𝐸 𝜃 ( ǁ𝑧 𝑛 ;{𝑧 𝑚′ }) 𝜆 ⋅𝐸 𝜃 (𝑧 𝑛 ;{𝑧 𝑚 ′ })

Original: Replay Buffer
Revised: Replay Buffer

Original: Update
Revised: Update

Original: Shared Parameters 𝑓 𝜃
Revised: Shared Parameters 𝑓 𝜃

Original: 𝑓
Revised: 𝑓

Original: 𝜃 𝑓 𝜃
Revised: 𝜃 𝑓 𝜃

Revised: Update ෤𝑣 𝑛

Original: Update ෤𝑣 𝑛 via SGLD using ∇ ෤𝑣 𝑛 𝐸 𝜃 ( ǁ𝑧 𝑛 ;{𝑧 𝑚′ })
Revised: [Figure] SGLD [Figure] ∇ ෤𝑣 𝑛 𝐸 𝜃 ( ǁ𝑧 𝑛 ;{𝑧 𝑚′ })

Original: [Figure]
Revised: [Figure]

Original:  Theory 
Revised:  Theory 3.1 Energy-Based Contrastive Learning

Original: 3.1 Energy-Based Contrastive Learning

Original: Let D be a distribution of images and T a distribution of stochastic image transformations.
Revised: Let D be a distribution of images and T a distribution of stochastic image transformations.

Original: Given x ∼ D and i.i.d.
Revised: Given x ∼ D and i.i.d.

Original: t, t ′ ∼ T , our goal is to approximate the joint distribution of the views p ( v, v ′ ) , where v = t ( x ) , v ′ =
Revised: t, t ′ ∼ T , our goal is to approximate the joint distribution of the views p ( v, v ′ ) , where v = t ( x ) , v ′ =

Original: t ′ ( x ) using the model distribution
Revised: t ′ ( x ) using the model distribution

Original: [Equation]
Revised: [Equation]

Original: 6) where Z ( θ ) is a normalization constant, τ > 0 is a temperature hyper-parameter, and z and z ′ areprojections computed by passing the views v and v ′ through the DNN f θ and then normalizing tohave unit norm.
Revised: 6) where Z ( θ ) is a normalization constant, τ > 0 is a temperature hyper-parameter, and z and z ′ are projections computed by passing the views v and v ′ through the DNN f θ and then normalizing to have unit norm.

Original: We now explain the intuitive meaning of matching q θ to p .
Revised: We now explain the intuitive meaning of matching q θ to p .

Original: Our key idea is to use p ( v, v ′ ) as a measure of semantic similarity of v and v ′ .
Revised: Our key idea is to use p ( v, v ′ ) as a measure of semantic similarity of v and v ′ .

Original: If two images v and v ′are semantically similar, they are likely to be transformations of similar images.
Revised: If two images v and v ′ are semantically similar, they are likely to be transformations of similar images.

Original: So, p ( v, v ′ ) will behigh when v and v ′ are semantically similar and low otherwise.
Revised: So, p ( v, v ′ ) will be high when v and v ′ are semantically similar and low otherwise.

Original: Suppose q θ successfully approximatesp .
Revised: Suppose q θ successfully approximates p .

Original: If we equate p ( v, v ′ ) to q θ ( v, v ′ ) in (6) and solve for ∥ z − z ′ ∥ , we see that the distance between zand z ′ will become a monotone increasing function of 1 /p ( v, v ′ ) , which is the inverse of semanticsimilarity of v and v ′ .
Revised: If we equate p ( v, v ′ ) to q θ ( v, v ′ ) in (6) and solve for ∥ z − z ′ ∥ , we see that the distance between z and z ′ will become a monotone increasing function of 1 /p ( v, v ′ ) , which is the inverse of semantic similarity of v and v ′ .

Original: So, semantically similar images will have nearby projections, and dissimilarimages will have distant projections.
Revised: So, semantically similar images will have nearby projections, and dissimilar images will have distant projections.

Original: This idea is illustrated in Figure 1.
Revised: This idea is illustrated in Figure 1.

Original: To approximate p using q θ , we train f θ to maximize the expected log-likelihood of q θ under p :
Revised: To approximate p using q θ , we train f θ to maximize the expected log-likelihood of q θ under p :

Original: [Equation]
Revised: [Equation]

Original: In order to solve this problem with stochastic gradient ascent, we could naively extend (3) to thesetting of joint distributions to obtain the following result.
Revised: In order to solve this problem with stochastic gradient ascent, we could naively extend (3) to the setting of joint distributions to obtain the following result.

Original: Proposition 1.
Revised: Proposition 1.

Original: The the joint distribution (6) can be formulated as an EBM
Revised: The the joint distribution (6) can be formulated as an EBM

Original: [Equation]
Revised: [Equation]

Original: and the gradient of the objective of (7) is given by
Revised: and the gradient of the objective of (7) is given by

Original: [Equation]
Revised: [Equation]

Original: However, computing the first expectation in (9) requires sampling pairs of views ( v, v ′ ) from q θ ( v, v ′ )via SGLD, which could be expensive.
Revised: However, computing the first expectation in (9) requires sampling pairs of views ( v, v ′ ) from q θ ( v, v ′ ) via SGLD, which could be expensive.

Original: To avert this problem, we use Bayes’ rule to decompose
Revised: To avert this problem, we use Bayes’ rule to decompose

Original: E p [log q θ ( v, v ′ )]
Revised: E p [log q θ ( v, v ′ )]

Original: =
Revised: =

Original: E p [log q θ ( v ′ | v )]
Revised: E p [log q θ ( v ′ | v )]

Original: + E p [log q θ ( v )] where q θ ( v ) = (cid:90) q θ ( v, v ′ ) dv ′ .
Revised: + E p [log q θ ( v )] where q θ ( v ) = (cid:90) q θ ( v, v ′ ) dv ′ .

Original: In the first equation of (10), the first and second terms at the RHS will be referred to as discriminativeand generative terms, respectively, throughout the paper.
Revised: In the first equation of (10), the first and second terms at the RHS will be referred to as discriminative and generative terms, respectively, throughout the paper.

Original: A similar decomposition was used by
Revised: A similar decomposition was used by

Original: Grathwohl et al.
Revised: Grathwohl et al.

Original: [10] in the setting of learning generative classifiers.
Revised: [10] in the setting of learning generative classifiers.

Original: Furthermore, we add a hyper-parameter λ to balance the strength of the discriminative term and thegenerative term.
Revised: Furthermore, we add a hyper-parameter λ to balance the strength of the discriminative term and the generative term.

Original: The advantage of this modification will be discussed in Section 4.3.
Revised: The advantage of this modification will be discussed in Section 4.3.

Original: This yields our
Revised: This yields our

Original: Energy-Based Contrastive Learning (EBCLR) objective
Revised: Energy-Based Contrastive Learning (EBCLR) objective

Original: [Equation]
Revised: [Equation]

Original: [Equation] The discriminative term can be easily differentiated since the partition function Z ( θ ) cancels outwhen q θ ( v, v ′ ) is divided by q θ ( v ) .
Revised: The discriminative term can be easily differentiated since the partition function Z ( θ ) cancels out when q θ ( v, v ′ ) is divided by q θ ( v ) .

Original: However, the generative term still contains Z ( θ ) .
Revised: However, the generative term still contains Z ( θ ) .

Original: We now presentour key result, which addresses this problem by relating the generative term to EBMs.
Revised: We now present our key result, which is used to maximize (11).

Original: The proof isdeferred to Appendix C.1.
Revised: The proof is deferred to Appendix C.1.

Original: Theorem 2.
Revised: Theorem 2.

Original: The marginal distribution in (10) can be formulated as an EBM
Revised: The marginal distribution in (10) can be formulated as an EBM

Original: [Equation]
Revised: [Equation]

Original: where Z ( θ ) is the partition function in (6) , and the gradient of the generative term is given by
Revised: where Z ( θ ) is the partition function in (6) , and the gradient of the generative term is given by

Original: [Equation]
Revised: [Equation]

Original: Thus, the gradient of the EBCLR objective is
Revised: Thus, the gradient of the EBCLR objective is

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Theorem 2 suggests that the EBM for the joint distribution can be learned by computing the gradientsof the discriminative term and the EBM for the marginal distribution.
Revised: Theorem 2 suggests that the EBM for the joint distribution can be learned by computing the gradients of the discriminative term and the EBM for the marginal distribution.

Original: Moreover, we only need tosample v from q θ ( v ) to compute the second expectation in (14).
Revised: Moreover, we only need to sample v from q θ ( v ) to compute the second expectation in (14).

Original: 3.2 Approximating the EBCLR Objective
Revised: 3.2 Approximating the EBCLR Objective

Original: To implement EBCLR, we need to approximate expectations in (11) with their empirical means.
Revised: To implement EBCLR, we need to approximate expectations in (11) with their empirical means.

Original: Suppose samples { ( v n , v ′ n ) }
Revised: Suppose samples { ( v n , v ′ n ) }

Original: Nn =1 from p ( v, v ′ ) are given, and let { ( z n , z ′ n ) }
Revised: Nn =1 from p ( v, v ′ ) are given, and let { ( z n , z ′ n ) }

Original: Nn =1 be the correspondingprojections.
Revised: Nn =1 be the corresponding projections.

Original: As the learning goal is to make q θ ( v n , v ′ n ) approximate the joint probability densityfunction p ( v n , v ′ n ) , the empirical mean (cid:98) q θ ( v n ) can be defined as:
Revised: As the learning goal is to make q θ ( v n , v ′ n ) approximate the joint probability density function p ( v n , v ′ n ) , the empirical mean (cid:98) q θ ( v n ) can be defined as:

Original: [Equation]
Revised: [Equation]

Original: where the sum is over the collection of v ′ m defined as
Revised: where the sum is over the collection of v ′ m defined as

Original: [Equation]
Revised: [Equation]

Original: 16) and N ′ :
Revised: and N ′ :

Original: = |{ v ′ m : v ′ m ̸
Revised: = |{ v ′ m : v ′ m ̸

Original: = v n }| = 2 N − 1 .
Revised: = v n }| = 2 N − 1 .

Original: One could also use a simpler form of the empirical mean:
Revised: One could also use a simpler form of the empirical mean:

Original: [Equation]
Revised: [Equation]

Original: [Equation] Similarly, q θ ( v ′ | v ) in (11), which should approximate the conditional probability density p ( v ′ | v ) , canbe represented in terms of q θ ( v n , v ′ n ) .
Revised: Similarly, q θ ( v ′ | v ) in (11), which should approximate the conditional probability density p ( v ′ | v ) , can be represented in terms of q θ ( v n , v ′ n ) .

Original: Specifically, we have
Revised: Specifically, we have

Original: [Equation]
Revised: [Equation]

Original: It is then immediately apparent that the empirical form of the discriminative term using (18) isa particular instance of the contrastive learning objective such as InfoNCE and SimCLR.
Revised: It is then immediately apparent that the empirical form of the discriminative term using (18) is a particular instance of the contrastive learning objective such as InfoNCE and SimCLR.

Original: Hence,
Revised: Hence,

Original: EBCLR can be interpreted as complementing contrastive learning with a generative term defined byan EBM.
Revised: EBCLR can be interpreted as complementing contrastive learning with a generative term defined by an EBM.

Original: We will demonstrate in Section 4.1 that the generative term offers significant advantagesover other contrastive learning methods.
Revised: We will demonstrate in Section 4.1 that the generative term offers significant advantages over other contrastive learning methods.

Original: For the second term, we use the simpler form of the empirical mean in (17):
Revised: For the second term, we use the simpler form of the empirical mean in (17):

Original: [Equation]
Revised: [Equation]

Original: We could also use (15) as the empirical mean, but either choice showed identical performance (c.f.
Revised: We could also use (15) as the empirical mean, but either choice showed identical performance (see

Original: Appendix E.3).
Revised: Appendix E.3).

Original: So, we have found (15) to be not worth the additional complexity, and have resortedto the simpler approximation (17) instead.
Revised: So, we have found (15) to be not worth the additional complexity, and have resorted to the simpler approximation (17) instead.

Original: In Appendix C.2, we theoretically justify that EBCLR willwork as intended even with the approximations (15) or (17).
Revised: In Appendix C.2, we theoretically justify that EBCLR will work as intended even with the approximations (15) or (17).

Original: If we compare (19) with (12), we cansee that this approximation of q θ ( v ) yields the energy function (after ignoring the constant log N )
Revised: If we compare (19) with (12), we can see that this approximation of q θ ( v ) yields the energy function (after ignoring the constant log N )

Original: [Equation]
Revised: [Equation]

Original: 3.3 Modifications to SGLD
Revised: 3.3 Modifications to SGLD

Original: According to Theorem 2, we need samples from the marginal q θ ( v ) to calculate the second expectationin (14).
Revised: According to Theorem 2, we need samples from the marginal q θ ( v ) to calculate the second expectation in (14).

Original: Hence, we apply proximal SGLD (5) with the energy function (20) to sample from q θ ( v ) as
Revised: Hence, we apply proximal SGLD (5) with the energy function (20) to sample from q θ ( v ) as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for t = 0 , . . .
Revised: for t = 0 , . . .

Original: , T − 1 , where ϵ ∼ N (0 , σ 2 ) .
Revised: , T − 1 , where ϵ ∼ N (0 , σ 2 ) .

Original: We make three additional modifications to proximal
Revised: We make three additional modifications to proximal

Original: SGLD to expedite the training process.
Revised: SGLD to expedite the training process.

Original: From here on, we will be referring to proximal SGLD in (5)when we say SGLD.
Revised: From here on, we will be referring to proximal SGLD in (5) when we say SGLD.

Original: First, we initialize SGLD from generated samples from previous iterations, and with probabilityρ , we reinitialize SGLD chains from samples from a proposal distribution q 0 .
Revised: First, we initialize SGLD from generated samples from previous iterations, and with probability ρ , we reinitialize SGLD chains from samples from a proposal distribution q 0 .

Original: This is achieved bykeeping a replay buffer B of SGLD samples from previous iterations.
Revised: This is achieved by keeping a replay buffer B of SGLD samples from previous iterations.

Original: This technique of maintaininga replay buffer has also been used in previous works and has proven to be crucial for stabilizing andaccelerating the convergence of EBMs [16, 10, 15].
Revised: This technique of maintaining a replay buffer has also been used in previous works and has proven to be crucial for stabilizing and accelerating the convergence of EBMs [16, 10, 15].

Original: Second, the proposal distribution q 0 is set to be the data distribution p ( v ) .
Revised: Second, the proposal distribution q 0 is set to be the data distribution p ( v ) .

Original: This choice differs fromthose of previous works [16, 10, 15] which have either used the uniform distribution or a mixture of
Revised: This choice differs from those of previous works [16, 10, 15] which have either used the uniform distribution or a mixture of

Original: Gaussians as the proposal distribution.
Revised: Gaussians as the proposal distribution.

Original: Finally, we use multi-stage SGLD (MSGLD) , which adaptively controls the magnitude of noise addedin SGLD.
Revised: Finally, we use multi-stage SGLD (MSGLD) , which adaptively controls the magnitude of noise added in SGLD.

Original: For each sample ˜ v in the replay buffer B , we keep a count κ ˜ v of number of times it hasbeen used as the initial point of SGLD.
Revised: For each sample ˜ v in the replay buffer B , we keep a count κ ˜ v of number of times it has been used as the initial point of SGLD.

Original: For samples with a low count, we use noise of high variance,and for samples with a high count, we use noise of low variance.
Revised: For samples with a low count, we use noise of high variance, and for samples with a high count, we use noise of low variance.

Original: Specifically, in (5), we set
Revised: Specifically, in (5), we set

Original: [Equation]
Revised: [Equation]

Original: where [ · ] + := max { 0 , ·} , σ 2max and σ 2min are the upper and lower bounds on the noise variance,respectively, and K controls the decay rate of noise variance.
Revised: where [ · ] + := max { 0 , ·} , σ 2max and σ 2min are the upper and lower bounds on the noise variance, respectively, and K controls the decay rate of noise variance.

Original: The purpose of this technique is tofacilitate quick exploration of the modes of q θ and still guarantee SGLD generates samples withsufficiently low energy.
Revised: The purpose of this technique is to facilitate quick exploration of the modes of q θ and still guarantee SGLD generates samples with sufficiently low energy.

Original: The pseudocodes for MSGLD and EBCLR are given in Algorithms 1 and 2,respectively, in Appendix B, and the overall learning flow of EBCLR is described in Figure 2.
Revised: The pseudocodes for MSGLD and EBCLR are given in Algorithms 1 and 2, respectively, in Appendix B, and the overall learning flow of EBCLR is described in Figure 2.

Original:  Experiments
Revised:  Experiments

Original: We now describe the experimental settings.
Revised: We now describe the experimental settings.

Original: A complete description is deferred to Appendix D.
Revised: A complete description is deferred to Appendix D.

Original: Baseline methods and datasets.
Revised: Baseline methods and datasets.

Original: The baseline methods are SimCLR, MoCo v2, SimSiam, and
Revised: The baseline methods are SimCLR, MoCo v2, SimSiam, and

Original: BYOL.
Revised: BYOL.

Original: The hyper-parameters are chosen closely following the original works [4, 12, 7, 6].
Revised: The hyper-parameters are chosen closely following the original works [4, 12, 7, 6].

Original: We usefour datasets: MNIST [23], Fashion MNIST (FMNIST) [24], CIFAR10, and CIFAR100 [25].
Revised: We use four datasets: MNIST [23], Fashion MNIST (FMNIST) [24], CIFAR10, and CIFAR100 [25].

Original: DNN architecture.
Revised: DNN architecture.

Original: We decompose f θ = π θ ◦ ϕ θ where ϕ θ is the encoder network and π θ is theprojection network.
Revised: We decompose f θ = π θ ◦ ϕ θ where ϕ θ is the encoder network and π θ is the projection network.

Original: Rather than using the output of f θ for downstream tasks, we follow previousworks [4, 5, 1, 2, 3, 6, 7] and use the output of ϕ θ instead.
Revised: Rather than using the output of f θ for downstream tasks, we follow previous works [4, 5, 1, 2, 3, 6, 7] and use the output of ϕ θ instead.

Original: In our experiments, we set ϕ θ to bea ResNet-18 [26] up to the global average pooling layer and π θ to be a 2-layer MLP with outputdimension 128 .
Revised: In our experiments, we set ϕ θ to be a ResNet-18 [26] up to the global average pooling layer and π θ to be a 2-layer MLP with output dimension 128 .

Original: However, we remove batch normalization because batch normalization hurts SGLD[16].
Revised: However, we remove batch normalization because batch normalization hurts SGLD [16].

Original: We also replace ReLU with leaky ReLU to expedite the convergence of SGLD.
Revised: We also replace ReLU with leaky ReLU to expedite the convergence of SGLD.

Original: For the baselines,we use settings proposed in the original works while keeping the backbone fixed to be ResNet-18.
Revised: For the baselines, we use settings proposed in the original works while keeping the backbone fixed to be ResNet-18.

Original: Evaluation.
Revised: Evaluation.

Original: We evaluate the representations by training a linear classifier on top of frozen ϕ θ .
Revised: We evaluate the representations by training a linear classifier on top of frozen ϕ θ .

Original: [Table]
Revised: [Table]

Original: 4.1 Comparison with Baselines
Revised: 4.1 Comparison with Baselines

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: We use batch size 128 for EBCLR andbatch size 256 for the baseline methodsfollowing Wang et.
Revised: We use batch size 128 for EBCLR and batch size 256 for the baseline methods following Wang et.

Original: al [27] and train eachmethod for 100 epochs.
Revised: al [27] and train each method for 100 epochs.

Original: Table 1 showsthe result of training each method for 100epochs.
Revised: Table 1 shows the result of training each method forepochs.

Original: Observe that EBCLR consistentlyoutperforms all baseline methods in termsof linear evaluation accuracy.
Revised: Observe that EBCLR consistently outperforms all baseline methods in terms of linear evaluation accuracy.

Original: Moreover,relative efficiency indicates EBCLR is capable of achieving the same level of performance as the baseline methods with muchfewer training epochs.
Revised: Moreover, relative efficiency indicates EBCLR is capable of achieving the same level of performance as the baseline methods with much fewer training epochs.

Original: Concretely, we observe at least × 4 acceleration in terms ofepochs compared to contrastive methods.
Revised: Concretely, we observe at least × 4 acceleration in terms of epochs compared to contrastive methods.

Original: Hence, EBCLR is a much more desirablechoice than SimCLR or MoCo v2 for learning visual representations when we have a small number of training samples.
Revised: Hence, EBCLR is a much more desirable choice than SimCLR or MoCo v2 for learning visual representations when we have a small number of training samples.

Original: We also investigate the transfer learning performance of EBCLR.
Revised: We also investigate the transfer learning performance of EBCLR.

Original: Table 2 compares the transferlearning accuracies.
Revised: Table 2 compares the transfer learning accuracies.

Original: EBCLR always outperforms the baseline methods, and the performance gap isespecially large on CIFAR10 and CIFAR100.
Revised: EBCLR always outperforms the baseline methods, and the performance gap is especially large on CIFAR10 and CIFAR100.

Original: This indicates EBCLR learns visual representationsthat generalize well across datasets.
Revised: This indicates EBCLR learns visual representations that generalize well across datasets.

Original: Repeating the above experiments with longer training or KNNclassification led to similar conclusions (c.f. Appendixes E.1 and E.4, respectively).
Revised: Repeating the above experiments with longer training or KNN classification led to similar conclusions (see Appendixes E.1 and E.4, respectively).

Original: 4.2 Effect of Reducing Negative Pairs
Revised: 4.2 Effect of Reducing Negative Pairs

Original: We compared the performances of EBCLR and SimCLR as we reduced the number of negativepairs per positive pair.
Revised: We compared the performances of EBCLR and SimCLR as we reduced the number of negative pairs per positive pair.

Original: For MoCo v2, the negative samples are provided by a queue updated by amomentum encoder.
Revised: For MoCo v2, the negative samples are provided by a queue updated by a momentum encoder.

Original: On the other hand, for EBCLR and SimCLR, negative samples come from thesame batch as the positive pair.
Revised: On the other hand, for EBCLR and SimCLR, negative samples come from the same batch as the positive pair.

Original: So, we did not have a way of fairly comparing EBCLR and SimCLRwith MoCo v2.
Revised: So, we did not have a way of fairly comparing EBCLR and SimCLR with MoCo v2.

Original: Hence, we excluded MoCo v2 from this experiment.
Revised: Hence, we excluded MoCo v2 from this experiment.

Original: We note that, according to (18), given a batch of size N , we obtain 2 N − 2 negative pairs foreach positive pair.
Revised: We note that, according to (18), given a batch of size N , we obtain 2 N − 2 negative pairs for each positive pair.

Original: SimCLR also has 2 N − 2 negative pairs for each positive pair.
Revised: SimCLR also has 2 N − 2 negative pairs for each positive pair.

Original: Hence, we canconveniently compare the sensitivity of EBCLR and SimCLR to the number of negative pairs byvarying the batch size.
Revised: Hence, we can conveniently compare the sensitivity of EBCLR and SimCLR to the number of negative pairs by varying the batch size.

Original: Table 3 shows the result of training each method for 100 epochs with batch sizes in { 16 , 64 , 128 } .
Revised: Table 3 shows the result of training each method for 100 epochs with batch sizes in { 16 , 64 , 128 } .

Original: We make three important observations.
Revised: We make three important observations.

Original: First, EBCLR consistently beats SimCLR in terms of linearevaluation accuracy for every batch size.
Revised: First, EBCLR consistently beats SimCLR in terms of linear evaluation accuracy for every batch size.

Original: Second, EBCLR is invariant to the choice of batch size.
Revised: Second, EBCLR is invariant to the choice of batch size.

Original: This contrasts with SimCLR whose performance degrades as batch size decreases.
Revised: This contrasts with SimCLR whose performance degrades as batch size decreases.

Original: Consequently,
Revised: Consequently,

Original: EBCLR with batch size 16 beats SimCLR with batch size 128 .
Revised: EBCLR with batch size 16 beats SimCLR with batch size 128 .

Original: Finally, as a byproduct of the secondobservation, the efficiency of EBCLR relative to SimCLR increases as batch size decreases.
Revised: Finally, as a byproduct of the second observation, the efficiency of EBCLR relative to SimCLR increases as batch size decreases.

Original: Theseproperties make EBCLR suitable for situations where we cannot use large batch sizes, e.g., when we
Revised: These properties make EBCLR suitable for situations where we cannot use large batch sizes, e.g., when we

Original: [Table]
Revised: [Table]

Original: have a small number of GPUs.
Revised: have a small number of GPUs.

Original: Repeating the experiments with longer training or KNN classificationagain led to similar conclusions (c.f. Appendixes E.2 and E.4, respectively).
Revised: Repeating the experiments with longer training or KNN classification again led to similar conclusions (see Appendixes E.2 and E.4, respectively).

Original: 4.3 Effect of λ and Projection Dimension
Revised: 4.3 Effect of λ and Projection Dimension

Original: [Figure]
Revised: [Figure]

Original: We explored the effect of changing the hyperparameter λ which controls the importance ofthe generative term relative to the discriminativeterm (c.f. Equation (11)).
Revised: We explored the effect of changing the hyperparameter λ which controls the importance of the generative term relative to the discriminative term (see Equation (11)).

Original: Figure 3a shows theperformance of EBCLR with various values of λas training progresses.
Revised: Figure 3a shows the performance of EBCLR with various values of λ as training progresses.

Original: We observe that naivelyusing λ = 1 . [-
-] leads to poor results.
Revised: We observe that naively using λ = 1 . [-
-]0 leads to poor results.

Original: The performance peaks at λ = 0 .
Revised: The performance peaks at λ = 0 .

Original: 1 , and then degrades aswe further decrease λ .
Revised: 1 , and then degrades as we further decrease λ .

Original: This result has two crucial implications.
Revised: This result has two crucial implications.

Original: First,the generative term plays a non-trivial role in
Revised: First, the generative term plays a non-trivial role in

Original: EBCLR.
Revised: EBCLR.

Original: Second, we need to strike a right balance between the discriminative term and thegenerative term to achieve good performance ondownstream tasks 1 .
Revised: Second, we need to strike a right balance between the discriminative term and the generative term to achieve good performance on downstream tasks 1 .

Original: We also investigated the effect of varying the output dimension of π θ .
Revised: We also investigated the effect of varying the output dimension of π θ .

Original: Figure 3b shows linearevaluation results for projection dimensions in { 128 , 256 , 512 } .
Revised: Figure 3b shows linear evaluation results for projection dimensions in { 128 , 256 , 512 } .

Original: We observe that the projectiondimension has essentially no influence on the training process.
Revised: We observe that the projection dimension has essentially no influence on the training process.

Original: In this respect, EBCLR resembles
Revised: In this respect, EBCLR resembles

Original: SimCLR which is also invariant to the output dimension (c.f. Figure 8 in the work by Chen et.
Revised: SimCLR which is also invariant to the output dimension (see Figure 8 in the work by Chen et.

Original: al [4]).
Revised: al [4]).

Original: 4.4 Effect of SGLD Modifications
Revised: 4.4 Effect of SGLD Modifications

Original: We now study the roles of the three SGLD modifications proposed in Section 3.3.
Revised: We now study the roles of the three SGLD modifications proposed in Section 3.3.

Original: Figure 4 shows theresults of varying one parameter of MSGLD while keeping the others fixed.
Revised: Figure 4 shows the results of varying one parameter of MSGLD while keeping the others fixed.

Original: Effect of reintialization frequency ρ .
Revised: Effect of reintialization frequency ρ .

Original: Figure 4a displays linear evaluation results for ρ ∈{ 0 . 0 , 0 . 2 , 1 . 0 } .
Revised: Figure 4a displays linear evaluation results for ρ ∈ { 0 . 0 , 0 . 2 , 1 . 0 } .

Original: We note that setting ρ = 1 . 0 is equivalent to removing the replay buffer.
Revised: We note that setting ρ = 1 . 0 is equivalent to removing the replay buffer.

Original: Also,setting ρ = 0 . 0 is equivalent to never reinitializing SGLD chains.
Revised: Also, setting ρ = 0 . 0 is equivalent to never reinitializing SGLD chains.

Original: Initially, ρ = 0 . 0 shows the best performance, as SGLD quickly reaches samples of lower energy.
Revised: Initially, ρ = 0 . 0 shows the best performance, as SGLD quickly reaches samples of lower energy.

Original: However, learning then slows down because of the lack of diversity of samples in the replay buffer B .
Revised: However, learning then slows down because of the lack of diversity of samples in the replay buffer B .

Original: This implies that it is necessary to set ρ > 0 in order to learn good representations.
Revised: This implies that it is necessary to set ρ > 0 in order to learn good representations.

Original: On the other hand, ρ = 1 . 0 shows slow convergence in the beginning because samples in the replaybuffer are not given enough iterations to reach low energy.
Revised: On the other hand, ρ = 1 . 0 shows slow convergence in the beginning because samples in the replay buffer are not given enough iterations to reach low energy.

Original: Although it does beat ρ = 0 . [-
-] at latterepochs, it still often performs worse than ρ = 0 .
Revised: Although it does beat ρ = 0 . [-
-]0 at latter epochs, it still often performs worse than ρ = 0 .

Original: Moreover, it is not sample-efficient compared toρ = 0 .
Revised: Moreover, it is not sample-efficient compared to ρ = 0 .

Original: 2 since we have to provide an entire batch of new samples for reinitializing SGLD chains ateach iteration.
Revised: 2 since we have to provide an entire batch of new samples for reinitializing SGLD chains at each iteration.

Original: Given the above observations, it is clear why the intermediate value 0 .
Revised: Given the above observations, it is clear why the intermediate value 0 .

Original: 2 is the best choice out ofρ ∈ { 0 . 0 , 0 . 2 , 1 . 0 } .
Revised: 2 is the best choice out of ρ ∈ { 0 . 0 , 0 . 2 , 1 . 0 } .

Original: ρ = 0 .
Revised: ρ = 0 .

Original: 2 allows enough time for samples in the replay buffer to reach low energywhile still maintaining the diversity of samples in B .
Revised: 2 allows enough time for samples in the replay buffer to reach low energy while still maintaining the diversity of samples in B .

Original: Also, it is sample-efficient compared to ρ = 1 . 0 .
Revised: Also, it is sample-efficient compared to ρ = 1 . 0 .

Original: Effect of proposal distribution q 0 .
Revised: Effect of proposal distribution q 0 .

Original: Figure 4b compares linear evaluation accuracies with q 0 as theuniform distribution and q 0 = p ( v ) .
Revised: Figure 4b compares linear evaluation accuracies with q 0 as the uniform distribution and q 0 = p ( v ) .

Original: We observe prominent acceleration in the initial epochs [-forq 
-] = p ( v ) .
Revised: We observe prominent acceleration in the initial epochs [-forq 
-]for q 0 = p ( v ) .

Original: Hence, we can conclude that this choice of proposal distribution is crucial for the highefficiency of EBCLR compared to the baseline methods in Tables 1 and 3.
Revised: Hence, we can conclude that this choice of proposal distribution is crucial for the high efficiency of EBCLR compared to the baseline methods in Tables 1 and 3.

Original: [Figure]
Revised: [Figure]

Original: We believe this acceleration effect can be explainedby the work of Hinton [21].
Revised: We believe this acceleration effect can be explained by the work of Hinton [21].

Original: Specifically, let us observe that the EBM update equation (3) pushes upthe energy on the model distribution q θ .
Revised: Specifically, let us observe that the EBM update equation (3) pushes up the energy on the model distribution q θ .

Original: In the implementation of EBCLR with q 0 = p ( v ) , however,q θ is replaced by the distribution of samples createdby a finite number of (noisy) gradient steps on realdata points (c.f. Section 3.3).
Revised: In the implementation of EBCLR with q 0 = p ( v ) , however, q θ is replaced by the distribution of samples created by a finite number of (noisy) gradient steps on real data points (see Section 3.3).

Original: Hence, the modified
Revised: Hence, the modified

Original: EMB update equation contains the curvature information of the data manifold.
Revised: EMB update equation contains the curvature information of the data manifold.

Original: This curvature informationmay expedite the training process of EBCLR.
Revised: This curvature information may expedite the training process of EBCLR.

Original: For adetailed discussion on this, we refer the readers to
Revised: For a detailed discussion on this, we refer the readers to

Original: Section 3 of the work by Hinton [21].
Revised: Section 3 of the work by Hinton [21].

Original: Comparison of SGLD and MSGLD.
Revised: Comparison of SGLD and MSGLD.

Original: Figure 4cshows results with SGLD with σ ∈ { 0 . 01 , 0 . 05 } and
Revised: Figure 4c shows results with SGLD with σ ∈ { 0 . 01 , 0 . 05 } and

Original: MSGLD with σ min = 0 .
Revised: MSGLD with σ min = 0 .

Original: and σ max = 0 . 05 .
Revised: and σ max = 0 . 05 .

Original: Wenote that setting σ min = σ max reduces MSGLD to
Revised: We note that setting σ min = σ max reduces MSGLD to

Original: SGLD.
Revised: SGLD.

Original: We observe σ = 0 .
Revised: We observe σ = 0 .

Original: 01 initially shows fast convergence but then saturates due to the lack of diversityof generated samples.
Revised: 01 initially shows fast convergence but then saturates due to the lack of diversity of generated samples.

Original: On the other hand, σ = 0 . 05initially has the worst performance but eventuallybeats σ = 0 .
Revised: On the other hand, σ = 0 .initially has the worst performance but eventually beats σ = 0 .

Original: 01 since σ = 0 .
Revised: 01 since σ = 0 .

Original: 05 quickly explores themodes of q θ .
Revised: 05 quickly explores the modes of q θ .

Original: MSGLD inherits the best of both settings.
Revised: MSGLD inherits the best of both settings.

Original: Specifically, MSGLD is as fast as σ = 0 .
Revised: Specifically, MSGLD is as fast as σ = 0 .

Original: 01 inthe beginning, and it does not suffer from the saturation problem.
Revised: 01 in the beginning, and it does not suffer from the saturation problem.

Original:  Limitations and Societal Impacts
Revised:  Limitations and Societal Impacts

Original: Limitations.
Revised: Limitations.

Original: The main limitation of our work is of scale.
Revised: The main limitation of our work is of scale.

Original: While EBCLR demonstrates superiorsample efficiency, it requires inner SGLD iterations (which cannot be parallelized) and a replay buffer
Revised: While EBCLR demonstrates superior sample efficiency, it requires inner SGLD iterations (which cannot be parallelized) and a replay buffer

Original: B .
Revised: B .

Original: These two components increase the computational burden of EBCLR.
Revised: These two components increase the computational burden of EBCLR.

Original: So, we found it difficult toapply EBCLR to large-scale data such as ImageNet.
Revised: So, we found it difficult to apply EBCLR to large-scale data such as ImageNet.

Original: However, we note that inner SGLD iterationsand the replay buffer are not particular limitations of EBCLR, but limitations of EBMs in general.
Revised: However, we note that inner SGLD iterations and the replay buffer are not particular limitations of EBCLR, but limitations of EBMs in general.

Original: Given the increasing efforts to overcome these limitations such as Proximal-YOPO-SGLD (for morediscussion, see Appendix F), we believe EBCLR will eventually be applicable to larger data.
Revised: Given the increasing efforts to overcome these limitations such as Proximal-YOPO-SGLD (for more discussion, see Appendix F), we believe EBCLR will eventually be applicable to larger data.

Original: Social Impacts.
Revised: Social Impacts.

Original: We generally expect positive outcomes from this research.
Revised: We generally expect positive outcomes from this research.

Original: Further developmentof EBCLR can mitigate the need for large amount of data and large batch sizes to learn goodrepresentations and ultimately lead to a reduction in resource consumption.
Revised: Further development of EBCLR can mitigate the need for large amount of data and large batch sizes to learn good representations and ultimately lead to a reduction in resource consumption.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we proposed EBCLR which combines contrastive learning with EBMs.
Revised: In this work, we proposed EBCLR which combines contrastive learning with EBMs.

Original: This amalgamation of ideas has led to both theoretical and practical contributions.
Revised: This amalgamation of ideas has led to both theoretical and practical contributions.

Original: Theoretically, EBCLRassociates distance in the projection space with the density of positive samples.
Revised: Theoretically, EBCLR associates distance in the projection space with the density of positive samples.

Original: Since the distributionof positive samples reflects the semantic similarity of images, EBCLR is capable of learning goodvisual representations.
Revised: Since the distribution of positive samples reflects the semantic similarity of images, EBCLR is capable of learning good visual representations.

Original: Practically, EBCLR is several times more sample-efficient than conventionalcontrastive and non-contrastive learning approaches and is robust to small numbers of negative pairs.
Revised: Practically, EBCLR is several times more sample-efficient than conventional contrastive and non-contrastive learning approaches and is robust to small numbers of negative pairs.

Original: Hence, EBCLR is applicable even in scenarios with limited data or devices.
Revised: Hence, EBCLR is applicable even in scenarios with limited data or devices.

Original: We believe that EBCLRmakes representation learning available to a wider range of machine learning practitioners.
Revised: We believe that EBCLR makes representation learning available to a wider range of machine learning practitioners.

Revised: Acknowledgments and Disclosure of Funding

Revised: This work was supported by the National Research Foundation of Korea under Grant NRF2020R1A2B5B03001980, KAIST Key Research Institute (Interdisciplinary Research Group) Project, and Field-oriented Technology Development Project for Customs Administration through National

Revised: Research Foundation of Korea(NRF) funded by the Ministry of Science & ICT and Korea Customs

Revised: Service(**NRF-2021M3I1A1097938**).

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Appendix C.3.
Revised: [Yes] See Appendix C. 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimentalresults (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experimentsmultiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] Each EBM was trained once due to computation costs.
Revised: [No] Each EBM was trained once due to computation costs.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix D. 
Revised: [Yes] See Appendix D. 4.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A](d) Did you discuss whether
Revised: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re us- ing/curating?
Revised: and how consent was obtained from people whose data you’re us- ing/curating?

Original: [N/A]  Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content?

Original: [N/A] 
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent onparticipant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: A Additional Background 
Revised: A Additional Background A.1 Contrastive Divergence and Contrastive Learning

Original: A.1 Contrastive Divergence and Contrastive Learning

Original: Contrastive divergence, initially proposed by Hinton et al.
Revised: Contrastive divergence, initially proposed by Hinton et al.

Original: [21], is a particular method for solving the maximum-454 likelihood estimation problem for EBMs.
Revised: [21], is a particular method for solving the maximum- likelihood estimation problem for EBMs.

Original: Intuitively, it proceeds by minimizing the energy function on dataand maximizing the energy function on (possibly short-run) MCMC samples of the current EBM distribution456 through gradient ascent.
Revised: Intuitively, it proceeds by minimizing the energy function on data and maximizing the energy function on (possibly short-run) MCMC samples of the current EBM distribution through gradient ascent.

Original: In fact, we use an instance of contrastive divergence to optimize the generative term inour EBCLR objective (11).
Revised: In fact, we use an instance of contrastive divergence to optimize the generative term in our EBCLR objective (11).

Original: We also disambiguate contrastive learning from contrastive divergence.
Revised: We also disambiguate contrastive learning from contrastive divergence.

Original: They are similar in the sense that they459 contrast certain pairs of data.
Revised: They are similar in the sense that they contrast certain pairs of data.

Original: Contrastive learning contrasts positive pairs and negative pairs.
Revised: Contrastive learning contrasts positive pairs and negative pairs.

Original: Contrastivedivergence contrasts data and MCMC samples.
Revised: Contrastive divergence contrasts data and MCMC samples.

Original: However, they differ in the aspect that contrastive learning461 is inherently a discriminative learning method (in contrastive learning, the model “predicts” the positive pairamong negative pairs) and contrastive divergence is a generative learning method.
Revised: However, they differ in the aspect that contrastive learning is inherently a discriminative learning method (in contrastive learning, the model “predicts” the positive pair among negative pairs) and contrastive divergence is a generative learning method.

Original: In the context of unsupervised visual representation learning, there are several works which connect contrastive464 learning with generative learning [28, 29, 30].
Revised: In the context of unsupervised visual representation learning, there are several works which connect contrastive learning with generative learning [28, 29, 30].

Original: For instance, Chen et al.
Revised: For instance, Chen et al.

Original: [30] combine contrastive learning with
Revised: [30] combine contrastive learning with

Original: GANs to improve person re-identification.
Revised: GANs to improve person re-identification.

Original: Zimmermann et al.
Revised: Zimmermann et al.

Original: [29] and Wang et al.
Revised: [29] and Wang et al.

Original: [28] study the distributional466 properties of embeddings found by contrastive learning.
Revised: [28] study the distributional properties of embeddings found by contrastive learning.

Original: Our work, for the first time, explores the synergybetween contrastive learning (the discriminative term in (11)) and EBMs / contrastive divergence (the generative468 term in (11)) for learning visual representations.
Revised: Our work, for the first time, explores the synergy between contrastive learning (the discriminative term in (11)) and EBMs / contrastive divergence (the generative term in (11)) for learning visual representations.

Original: A.2 Relation to Contrastive Energy-Based Models (CEMs) [19]
Revised: A.2 Relation to Contrastive Energy-Based Models (CEMs) [19]

Original: According to the terms introduced in the work of Wang et al.
Revised: According to the terms introduced in the work of Wang et al.

Original: [19], our EBCLR model distribution (6) with471 τ = 1 is identical to the model distribution of Non-Parametric CEM ((4) in [19]).
Revised: [19], our EBCLR model distribution (6) with τ = 1 is identical to the model distribution of Non-Parametric CEM ((4) in [19]).

Original: This is because the squaredℓ 2 norm between two unit-norm vectors z and z ′ is equivalent to the inner product between z and z ′ up to anadditive constant.
Revised: This is because the squared ℓ 2 norm between two unit-norm vectors z and z ′ is equivalent to the inner product between z and z ′ up to an additive constant.

Original: However, our work differs from the work of Wang et al.
Revised: However, our work differs from the work of Wang et al.

Original: [19] in three aspects.
Revised: [19] in three aspects.

Original: Objective function.
Revised: Objective function.

Original: Wang et al.
Revised: Wang et al.

Original: [19] optimizes the log-likelihood of the joint model distribution q θ ( v, v ′ )
Revised: [19] optimizes the log-likelihood of the joint model distribution q θ ( v, v ′ )

Original: directly whereas we use Bayes’ rule to decompose the objective into two terms, the discriminative term and the476 generative term.
Revised: directly whereas we use Bayes’ rule to decompose the objective into two terms, the discriminative term and the generative term.

Original: This allows us to reweight the generative term to learn better visual representations (as exploredin Section 4.3).
Revised: This allows us to reweight the generative term to learn better visual representations (as explored in Section 4.3).

Original: Optimization method.
Revised: Optimization method.

Original: Wang et al.
Revised: Wang et al.

Original: [19] maximizes the log-likelihood of the joint model distribution directly479 via adversarial training.
Revised: [19] maximizes the log-likelihood of the joint model distribution directly via adversarial training.

Original: On the other hand, we use gradient ascent to optimize the discriminative term andcontrastive divergence to optimize the generative term.
Revised: On the other hand, we use gradient ascent to optimize the discriminative term and contrastive divergence to optimize the generative term.

Original: Computation costs are similar, as adversarial training481 also requires inner iterations to generate adversarial examples at every model update step.
Revised: Computation costs are similar, as adversarial training also requires inner iterations to generate adversarial examples at every model update step.

Original: Application of interest.
Revised: Application of interest.

Original: Wang et al.
Revised: Wang et al.

Original: [19] focuses on enhancing image generation, while we focus on learning483 visual representations useful for downstream tasks.
Revised: [19] focuses on enhancing image generation, while we focus on learning visual representations useful for downstream tasks.

Original: Wang et al.
Revised: Wang et al.

Original: [19] also establishes a connection between contrastive divergence and InfoNCE (Section 5.1 in485 [19]).
Revised: [19] also establishes a connection between contrastive divergence and InfoNCE (Section 5.1 in [19]).

Original: Specifically, Wang et al.
Revised: Specifically, Wang et al.

Original: [19] show that when we approximate the model marginal q θ ( v ) with the datadistribution p ( v ) , MLE gradient for q θ ( v, v ′ ) becomes the InfoNCE loss gradient.
Revised: [19] show that when we approximate the model marginal q θ ( v ) with the data distribution p ( v ) , MLE gradient for q θ ( v, v ′ ) becomes the InfoNCE loss gradient.

Original: Surprisingly, we find thisholds in our framework as well.
Revised: Surprisingly, we find this holds in our framework as well.

Original: If we set q θ ( v ) = p ( v ) in (11), the generative term in (11) becomes independent488 of θ , so we end up optimizing only the discriminative term.
Revised: If we set q θ ( v ) = p ( v ) in (11), the generative term in (11) becomes independent of θ , so we end up optimizing only the discriminative term.

Original: We have shown that the discriminative term can beapproximated by a contrastive learning objective, so we recover the result of Wang et al.
Revised: We have shown that the discriminative term can be approximated by a contrastive learning objective, so we recover the result of Wang et al.

Original: In this sense, the490 derivations in Section 3 of our paper generalizes the connection discovered in the work of Wang et al.
Revised: In this sense, the derivations in Section 3 of our paper generalizes the connection discovered in the work of Wang et al.

Original: [19], aswe work with weaker assumptions (we do not set q θ ( v ) =
Revised: [19], as we work with weaker assumptions (we do not set q θ ( v ) =

Original: p ( v ) in (11)).
Revised: p ( v ) in (11)).

Original: A.3 EBMs for Learning Useful Representations
Revised: A.3 EBMs for Learning Useful Representations

Original:  et al.
Revised: Ranzato et al.

Original: [ proposes to train an EBM with an autoencoder structure by minimizing the  of an encoding loss and a decoding loss to learn an energy surface.
Revised: [14] proposes to train an EBM with an autoencoder structure by minimizing the sum of an encoding loss and a decoding loss to learn an energy surface.

Original: A byproduct is that the EBM learns a compressed representation of the data, which can be used for, e g., denoising.
Revised: A byproduct is that the EBM learns a compressed representation of the data, which can be used for, e.g., denoising.

Original: Our work is similar in the aspect that we use a particular EBM architecture (a triplet network) and training the EBM with our proposed method results in a compressed representation of the data which is useful for downstream tasks, e g., classification and transfer learning
Revised: Our work is similar in the aspect that we use a particular EBM architecture (a triplet network) and training the EBM with our proposed method results in a compressed representation of the data which is useful for downstream tasks, e.g., classification and transfer learning.

Original: In sense roughly 
Revised: In that sense, our work is also roughly related to Denoising Autoencoders (DAEs) [31] and Restricted

Original: Denoising Autoencoders ( Machines 
Revised: Boltzmann Machines (RBMs) as well.

Original:  EBMs  goal is to extract useful representations from 
Revised: Both are EBMs whose goal is to extract useful representations from data.

Revised: However, in relating contrastive learning to EBMs, they about contrasting positive pairs pairs.

Original: A.4 Sample Efficiency of Generative Models
Revised: A.4 Sample Efficiency of Generative Models

Original: We motivate the sample efficiency of EBCLR through the lens of discriminative and generative learning.
Revised: We motivate the sample efficiency of EBCLR through the lens of discriminative and generative learning.

Original: We504 note that standard contrastive learning (e.g., SimCLR) optimizes a discriminative objective (the model mustpredict positive pairs) and EBCLR optimizes a generative objective (given a pair of images, the model must506 output how likely it is to be a positive pair).
Revised: We note that standard contrastive learning (e.g., SimCLR) optimizes a discriminative objective (the model must predict positive pairs) and EBCLR optimizes a generative objective (given a pair of images, the model must output how likely it is to be a positive pair).

Original: So, we can compare the performance of discriminative models andgenerative models.
Revised: So, we can compare the performance of discriminative models and generative models.

Original: There is a long line of works which show (theoretically and / or empirically) that generative models can509 outperform discriminative ones.
Revised: There is a long line of works which show (theoretically and / or empirically) that generative models can outperform discriminative ones.

Original: For instance, Efron [32] has theoretically shown that Normal Discriminant
Revised: For instance, Efron [32] has theoretically shown that Normal Discriminant

Original: Analysis can be more efficient than logistic regression.
Revised: Analysis can be more efficient than logistic regression.

Original: Ng et al.
Revised: Ng et al.

Original: [33] has also theoretically shown that in the511 small sample regime, naïve Bayes can be more resistant to overfitting than logistic regression (i.e., find bettersolutions when given a small number of training samples).
Revised: [33] has also theoretically shown that in the small sample regime, naïve Bayes can be more resistant to overfitting than logistic regression (i.e., find better solutions when given a small number of training samples).

Original: There are also works [8, 9, 10] that consider optimizing a weighted combination of a discriminative loss and a514 generative loss, just like our EBCLR objective (11).
Revised: There are also works [8, 9, 10] that consider optimizing a weighted combination of a discriminative loss and a generative loss, just like our EBCLR objective (11).

Original: They find that with an appropriate weight, the model canoutperform discriminative models.
Revised: They find that with an appropriate weight, the model can outperform discriminative models.

Original: In particular, Larochelle et al.
Revised: In particular, Larochelle et al.

Original: [9] finds this hybrid approach to be beneficial516 in the limited training data setting.
Revised: [9] finds this hybrid approach to be beneficial in the limited training data setting.

Original: Grathwohl et al.
Revised: Grathwohl et al.

Original: [10] uses this hybrid approach to train a classifier and findthat the classifier exhibits excellent calibration, unlike other calibration methods that require additional training518 data.
Revised: [10] uses this hybrid approach to train a classifier and find that the classifier exhibits excellent calibration, unlike other calibration methods that require additional training data.

Original: We can also address the efficiency of generative models from the perspective of generative pre-training.
Revised: We can also address the efficiency of generative models from the perspective of generative pre-training.

Original: Bengio520 et al.
Revised: Bengio et al.

Original: [34] finds that pre-training a network with a generative loss brings the network weights towards a goodlocal minimum, thus bringing better generalization.
Revised: [34] finds that pre-training a network with a generative loss brings the network weights towards a good local minimum, thus bringing better generalization.

Original: Erhan et al.
Revised: Erhan et al.

Original: [35] finds that generative pre-training acts as a522 regularizer and that this regularization effect can be beneficial when we have a small training set.
Revised: [35] finds that generative pre-training acts as a regularizer and that this regularization effect can be beneficial when we have a small training set.

Original: Another explanation specific to EBCLR is that the generative loss has an effect orthogonal to the effect of the524 discriminative loss.
Revised: Another explanation specific to EBCLR is that the generative loss has an effect orthogonal to the effect of the discriminative loss.

Original: To minimize the generative loss, the model needs to place low energy on data and highenergy elsewhere.
Revised: To minimize the generative loss, the model needs to place low energy on data and high energy elsewhere.

Original: This can be achieved only when the model has features that accurately capture the manifold526 of the data.
Revised: This can be achieved only when the model has features that accurately capture the manifold of the data.

Original: When we have a small training set, the model may easily solve the discriminative objective andoverfit.
Revised: When we have a small training set, the model may easily solve the discriminative objective and overfit.

Original: By providing an auxiliary generative task, the overfitting can be mitigated.
Revised: By providing an auxiliary generative task, the overfitting can be mitigated.

Original: An analogous effect with528 Joint Energy-Based Models is demonstrated by Zhao et al.
Revised: An analogous effect with Joint Energy-Based Models is demonstrated by Zhao et al.

Original: B Psuedocodes
Revised: B Psuedocodes

Original:  1: Input: Energy E θ , initialization point ˜ v , count κ ˜ v , step size α , number of iterations T , proximity parameter δ , decay parameter K , variance bounds σ 2min , σ 2max 2: σ = σ min + ( σ max − σ min ) · [1 − κ ˜ v /K ]
Revised: Algorithm 1: Input: Energy E θ , initialization point ˜ v , count κ ˜ v , step size α , number of iterations T , proximity parameter δ , decay parameter K , variance bounds σ 2min , σ 2max 2: σ = σ min + ( σ max − σ min ) · [1 − κ ˜ v /K ]

Original: + 3: for t 
Revised: + 3: for t = 0 , 1 , 2 , . . .

Original: T − 1 do 4:
Revised: T − 1 do 4:

Original: Sample ϵ ∼ N (0 , σ 2 ) 5: ˜ v ← ˜ v − α · clamp {∇˜ v E θ (˜ v ) , δ } + ϵ 6: end for 7: κ ˜ v ← κ ˜ v +
Revised: Sample ϵ ∼ N (0 , σ 2 ) 5: ˜ v ← ˜ v − α · clamp {∇ v E θ (˜ v ) , δ } + ϵ 6: end for 7: κ ˜ v ← κ ˜ v +

Original: 1: Input: DNN f θ , batch size N , generative term weight λ , replay buffer B , reinit. frequency ρ 2: while not converged do 3:
Revised: Calculate

Original: Sample{ ( v n , v ′ n ) } Nn =1 from p ( v, v ′ ) 4:
Revised: Sample

Original: Calculate the disc. term with Eq. Calculate gradient of the disc. term 6:
Revised: Calculate E v

Original: Sample {˜ vn } Nn =1 from B with prob. 1 − ρ and from p ( v ) with prob. ρ 7: Calculate E θ ( v ; { v ′ m } Nm =1 ) with Eq.
Revised: { v

Original: Update {˜ v(2) n } N n =1 with MSGLDin Algo.9:
Revised: Update { v MSGLD

Original: Calculategradient of the gen. term with Eq.
Revised: Calculate

Original: Update θvia gradient ascent 11:
Revised: Update θ gradient ascent 11:

Original: UpdateB with MSGLDsamples {
Revised: Update with MSGLD

Original: 12: end while
Revised: 12: end while

Original: C Missing Proofs 
Revised: C Missing Proofs C.1 Proof of Theorem

Original: C.1 Proof of Theorem

Original: Proof.
Revised: Proof.

Original: We first observe that
Revised: We first observe that

Original: [Equation]
Revised: [Equation]

Original: which proves the relation (12).
Revised: which proves the relation (12).

Original: We also have 
Revised: We also have log q θ (

Original: [Equation]
Revised: [Equation]

Original: and so
Revised: and so

Original: [Equation]
Revised: [Equation]

Original: Taking expectation w.r.t.
Revised: Taking expectation w.r.t.

Original: p ( v ) establishes the relation (13).
Revised: p ( v ) establishes the relation (13).

Original: C.2 Another Theoretical Justification for EBCLR
Revised: C.2 Another Theoretical Justification for EBCLR

Original: Here we provide another theoretical justification on how solving the EBCLR objective (11) with λ = 1 causes539 q θ ( v, v ′ ) to approximate p ( v, v ′ ) .
Revised: Here we provide another theoretical justification on how solving the EBCLR objective (11) with λ = 1 causes q θ ( v, v ′ ) to approximate p ( v, v ′ ) .

Original: It is known that optimizing the first term of (11) with (18) in place of q θ ( v ′ | v )will cause (18) to approximate p ( v ′ | v ) [37].
Revised: It is known that optimizing the first term of (11) with (18) in place of q θ ( v ′ | v ) will cause (18) to approximate p ( v ′ | v ) [37].

Original: Also, optimizing the second term of (11) with (15) or (17) in placeof q θ ( v ) will cause (15) or (17) to be proportional to p ( v ) (indeed, in Table 10, we see SGLD samples of the542 EBCLR marginal q θ ( v ) approximated by (17) achieve a non-trivial FID score).
Revised: Also, optimizing the second term of (11) with (15) or (17) in place of q θ ( v ) will cause (15) or (17) to be proportional to p ( v ) (indeed, in Table 10, we see SGLD samples of the EBCLR marginal q θ ( v ) approximated by (17) achieve a non-trivial FID score).

Original: So, the product of (15) or (17)543 with (18) will be proportional to p ( v, v ′ ) .
Revised: So, the product of (15) or (17) with (18) will be proportional to p ( v, v ′ ) .

Original: Moreover, by construction, the product of (15) or (17) with (18) is(approximately) q θ ( v, v ′ ) .
Revised: Moreover, by construction, the product of (15) or (17) with (18) is (approximately) q θ ( v, v ′ ) .

Original: Hence, optimizing the EBCLR objective (11) will cause q θ ( v, v ′ ) to model p ( v, v ′ ) .
Revised: Hence, optimizing the EBCLR objective (11) will cause q θ ( v, v ′ ) to model p ( v, v ′ ) .

Original: D Complete Training and Evaluation Details
Revised: D Complete Training and Evaluation Details

Original: Baseline methods and datasets.
Revised: Baseline methods and datasets.

Original: The baseline methods are SimCLR, MoCo v2, SimSiam, and BYOL.
Revised: The baseline methods are SimCLR, MoCo v2, SimSiam, and BYOL.

Original: The548 hyper-parameters are chosen closely following the original works [4, 12, 7, 6].
Revised: The hyper-parameters are chosen closely following the original works [4, 12, 7, 6].

Original: We use four datasets: MNIST[23], Fashion MNIST (FMNIST)
Revised: We use four datasets: MNIST [23], Fashion MNIST (FMNIST)

Original: [24], CIFAR10, and CIFAR100 [25].
Revised: [24], CIFAR10, and CIFAR100 [25].

Original: Data transformation.
Revised: Data transformation.

Original: For fair comparison, we use the stochastic data transformation proposed by Chen et.
Revised: For fair comparison, we use the stochastic data transformation proposed by Chen et.

Original: al [4]551 for all methods.
Revised: al [4] for all methods.

Original: In the case of grayscale images, we only use the random cropping augmentation.
Revised: In the case of grayscale images, we only use the random cropping augmentation.

Original: For EBCLR,we also add small Gaussian noise N (0 , 0 .
Revised: For EBCLR, we also add small Gaussian noise N (0 , 0 .

Original: 03 2 ) to stabilize training [16, 10, 15].
Revised: 03 2 ) to stabilize training [16, 10, 15].

Original: We remark that Gaussian noiseof standard deviation 0 .
Revised: We remark that Gaussian noise of standard deviation 0 .

Original: 03 is nearly invisible to the human eye.
Revised: 03 is nearly invisible to the human eye.

Original: DNN architecture.
Revised: DNN architecture.

Original: We decompose f θ = π θ ◦ ϕ θ where ϕ θ is the encoder network and π θ is the projection555 network.
Revised: We decompose f θ = π θ ◦ ϕ θ where ϕ θ is the encoder network and π θ is the projection network.

Original: Rather than directly using the output of f θ for downstream tasks, we follow previous works [4, 5, 1, 2,3, 6, 7] and use the output of ϕ θ instead.
Revised: Rather than directly using the output of f θ for downstream tasks, we follow previous works [4, 5, 1, 2, 3, 6, 7] and use the output of ϕ θ instead.

Original: In our experiments, we set ϕ θ to be a ResNet-18 [26] up to the global average pooling layer.
Revised: In our experiments, we set ϕ θ to be a ResNet-18 [26] up to the global average pooling layer.

Original: The architecture of558 π θ and π θ to be a 2-layer MLP with output dimension 128 .
Revised: The architecture of π θ and π θ to be a 2-layer MLP with output convergence of SGLD.

Original: However, we remove batch normalization because 559 batch normalization hurts SGLD [16].

Original: We also replace ReLU activations with leaky ReLU to expedite theconvergence of SGLD.

Original: For the baseline methods, we use the settings proposed in the original works while561 keeping the backbone fixed to be ResNet-18.
Revised: For the baseline methods, we use the settings proposed in the original works while keeping the backbone fixed to be ResNet-18.

Original: SGLD settings.
Revised: SGLD settings.

Original: The default parameters for MSGLD are |B| =
Revised: The default parameters for MSGLD are |B| =

Original: 50 k , ρ = 0 .
Revised: 50 k , ρ = 0 .

Original: 2 , T = 10 , α = 0 .
Revised: 2 , T = 10 , α = 0 .

Original: 05 , δ = 1 . 0 ,563 K = 3 , σ min = 0 .
Revised: 05 , δ = 1 . 0 , K = 3 , σ min = 0 .

Original: 01 , and σ max = 0 .
Revised: 01 , and σ max = 0 .

Original: We use ρ = 0 .
Revised: We use ρ = 0 .

Original: 4 for MNIST, ρ = 0 .
Revised: 4 for MNIST, ρ = 0 .

Original: 6 for FMNIST, and ρ = 0 . 2 for564 CIFAR10 and CIFAR100.
Revised: 6 for FMNIST, and ρ = 0 . 2 for CIFAR10 and CIFAR100.

Original: At the beginning of training, we fill the buffer B entirely with proposal distributionsamples.
Revised: At the beginning of training, we fill the buffer B entirely with proposal distribution samples.

Original: Training.
Revised: Training.

Original: The temperature is τ = 0 .
Revised: The temperature is τ = 0 .

Original: 1 for EBCLR.
Revised: 1 for EBCLR.

Original: For EBCLR, as we cannot use batch normalization, it567 is difficult to apply SGD with momentum.
Revised: For EBCLR, as we cannot use batch normalization, it is difficult to apply SGD with momentum.

Original: Hence, we use the Adam optimizer [38].
Revised: Hence, we use the Adam optimizer [38].

Original: The learning rate is0 .
Revised: The learning rate is 0 .

Original: 0002 if the batch size is 128 and 0 .
Revised: if the batch size is 128 and 0 .

Original: if the batch size is smaller than 128 .
Revised: if the batch size is smaller than 128 .

Original: We also weakly regularize569 the squared ℓ 2 -norm of the outputs of f θ with weight 0 .
Revised: We also weakly regularize the squared ℓ 2 -norm of the outputs of f θ with weight 0 .

Original: For the baseline methods, we closely follow the570 settings of previous works [4, 12, 27].
Revised: For the baseline methods, we closely follow the settings of previous works [4, 12, 27].

Original: Specifically, we use SGD with momentum 0.9 and learning rate of0 . {+
+}
Revised: Specifically, we use SGD with momentum 0.9 and learning rate of 0 . {+
+}

Original: 03 · ( batch size / 256) and weight decay 0 . 0001 .
Revised: · ( batch size / 256) and weight decay 0 . 0001 .

Original: Each model is trained with a single GeForce RTX 3090.
Revised: Each model is trained with a single GeForce RTX 3090.

Original: Evaluation.
Revised: Evaluation.

Original: We assess the quality of the representations by training a linear classifier on top of frozen ϕ θ .
Revised: We assess the quality of the representations by training a linear classifier on top of frozen ϕ θ .

Original: The573 linear classifier is trained with Adam for 200 epochs with batch size 512 .
Revised: The linear classifier is trained with Adam for 200 epochs with batch size 512 .

Original: The learning rate η is found by gridsearch for log 10 η in [ − 4 , − 1] .
Revised: The learning rate η is found by grid search for log 10 η in [ − 4 , − 1] .

Original: E Additional ExperimentsE.1 Section 4.1 with Longer Training
Revised: E Additional Experiments E.1 Section 4.1 with Longer Training

Original: [Table]

Original: We repeat the experiments in Section 4.1 by training the models for 200 epochs.
Revised: We repeat the experiments in Section 4.1 by training the models for 200 epochs.

Original: For EBCLR, the number of578 SGLD iterations T is increased by 5 when SGLD is unable to generate samples with sufficiently high energy.
Revised: For EBCLR, the number of SGLD iterations T is increased by 5 when SGLD is unable to generate samples with sufficiently high energy.

Original: Tables 4 and 5 describe the results.
Revised: Tables 4 and 5 describe the results.

Original: We again note that EBCLR consistently beats all baseline methods and580 demonstrates high efficiency.
Revised: We again note that EBCLR consistently beats all baseline methods and demonstrates high efficiency.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: E.2 Section 4.2 with Longer Training
Revised: E.2 Section 4.2 with Longer Training

Original: We repeat the experiments in Section 4.2 by training the models for 200 epochs.
Revised: We repeat the experiments in Section 4.2 by training the models for 200 epochs.

Original: For EBCLR, the number of583 SGLD iterations T is increased by 5 when SGLD is unable to generate samples with sufficiently high energy.
Revised: For EBCLR, the number of SGLD iterations T is increased by 5 when SGLD is unable to generate samples with sufficiently high energy.

Original: Table 6 describes the results.
Revised: Table 6 describes the results.

Original: We observe similar patterns as in the case of training for 100 epochs.
Revised: We observe similar patterns as in the case of training for 100 epochs.

Original: In the585 case of CIFAR10, SimCLR seems to have a pattern agnostic to the batch size.
Revised: In the case of CIFAR10, SimCLR seems to have a pattern agnostic to the batch size.

Original: However, we note that there isa significant gap between the performance of SimCLR with batch size 256 (c.f. Table 4) and batch sizes in587 { 16 , 64 , 128 } .
Revised: However, we note that there is a significant gap between the performance of SimCLR with batch size 256 (see Table 4) and batch sizes in { 16 , 64 , 128 } .

Original: Interestingly, for SimCLR with small batch sizes, we observe that training longer with small batch sizes leads to589 worse performance than training only for 100 epochs.
Revised: Interestingly, for SimCLR with small batch sizes, we observe that training longer with small batch sizes leads to worse performance than training only for 100 epochs.

Original: This is particularly prominent on CIFAR10 and CIFAR100(compare Tables 3 and 6).
Revised: This is particularly prominent on CIFAR10 and CIFAR (compare Tables 3 and 6).

Original: We speculate that this is because, with small batches, SimCLR is unable to provide591 hard negatives, and this leads to overfitting of the DNN.
Revised: We speculate that this is because, with small batches, SimCLR is unable to provide hard negatives, and this leads to overfitting of the DNN.

Original: On the other hand, EBCLR does not suffer from thisphenomenon.
Revised: On the other hand, EBCLR does not suffer from this phenomenon.

Original: This again demonstrates the importance of the generative term.
Revised: This again demonstrates the importance of the generative term.

Original: [Table]
Revised: [Table]

Original:  EBCLR is calculated by the following formula: (number of epochs used by EBCLR with the same batch size to reach the final accuracy of SimCLR) /
Revised: Efficiency of SimCLR relative to EBCLR is calculated by the following formula: (number of epochs used by EBCLR with the same batch size to reach the final accuracy of SimCLR) /

Original: (total number of training epochs).
Revised: (total number of training epochs).

Original: E.3 Comparison of Approximations
Revised: E.3 Comparison of Approximations

Original: We can either use Equation (15) or Equation (17) when approximating the generative term.
Revised: We can either use Equation (15) or Equation (17) when approximating the generative term.

Original: Figure 5 shows595 results for both approximations on CIFAR10.
Revised: Figure 5 shows results for both approximations on CIFAR10.

Original: We observe identical performance for both approximations, whichimplies that the generative term is robust to the number of samples used in the calculation of the empirical mean.
Revised: We observe identical performance for both approximations, which implies that the generative term is robust to the number of samples used in the calculation of the empirical mean.

Original: This justifies our choice of using the simpler Equation (17) to approximate the marginal distribution q ( v ) .
Revised: This justifies our choice of using the simpler Equation (17) to approximate the marginal distribution q ( v ) .

Original: E.4 Evaluation with KNN Classifiers
Revised: E.4 Evaluation with KNN Classifiers

Original: We also reproduce Tables 1,
Revised: We also reproduce Tables 1,

Original: [Table]
Revised: [Table]

Original: and 3 using K-nearest neighbour (KNN) classifiers.
Revised: and 3 using K-nearest neighbour (KNN) classifiers.

Original: We followed the evaluation600 protocol of Caron et al.
Revised: We followed the evaluation protocol of Caron et al.

Original: [39] and used a weighted KNN classifier with K = 20 .
Revised: [39] and used a weighted KNN classifier with K = 20 .

Original: The results are given below.
Revised: The results are given below.

Original: In
Revised: In

Original: Table 7, except for the case of BYOL on MNIST, we see that EBCLR beats all baselines again by a non-trivial602 margin.
Revised: Table 7, except for the case of BYOL on MNIST, we see that EBCLR beats all baselines again by a non-trivial margin.

Original: EBCLR is also shown to be sample efficient.
Revised: EBCLR is also shown to be sample efficient.

Original: In Table 8, we see EBCLR outperforms all baselineson the task of transfer learning.
Revised: In Table 8, we see EBCLR outperforms all baselines on the task of transfer learning.

Original: In Table 9, we see EBCLR is robust to small batch sizes.
Revised: In Table 9, we see EBCLR is robust to small batch sizes.

Original: This leads to the604 lower efficiency of SimCLR compared to EBCLR.
Revised: This leads to the lower efficiency of SimCLR compared to EBCLR.

Original: On CIFAR10 and CIFAR100, relative efficiency does notdecrease below 0.05 since we saved checkpoints every 5 epochs.
Revised: On CIFAR10 and CIFAR100, relative efficiency does not decrease below 0.05 since we saved checkpoints every 5 epochs.

Original: We can reasonably expect SimCLR to have606 lower relative efficiency at smaller batch sizes if we had saved checkpoints every epoch.
Revised: We can reasonably expect SimCLR to have lower relative efficiency at smaller batch sizes if we had saved checkpoints every epoch.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: E.5 Generative Performance of EBCLR
Revised: E.5 Generative Performance of EBCLR

Original: In this section, we explore the effect of the discriminative loss on the generative performance of EBCLR.
Revised: In this section, we explore the effect of the discriminative loss on the generative performance of EBCLR.

Original: To this609 end, we solved
Revised: To this end, we solved

Original: [Equation]
Revised: [Equation]

Original: (23) for various values of γ on FMNIST.
Revised: (23) for various values of γ on FMNIST.

Original: Recall that in the original EBCLR objective, weight λ comes in front of611 the generative term.
Revised: Recall that in the original EBCLR objective, weight λ comes in front of the generative term.

Original: We then measured the Fréchet Inception Distance (FID, lower is better) between samplesfrom the true marginal p ( v ) and SGLD samples from the EBM marginal q θ ( v ) approximated by (19) / (17).
Revised: We then measured the Fréchet Inception Distance (FID, lower is better) between samples from the true marginal p ( v ) and SGLD samples from the EBM marginal q θ ( v ) approximated by (19) / (17).

Original: The613 results are shown in Table 10.
Revised: The results are shown in Table 10.

Original: Interestingly, we found using an appropriate γ > 0 led to improvements in FID over γ = 0 .
Revised: Interestingly, we found using an appropriate γ > 0 led to improvements in FID over γ = 0 .

Original: In other words,615 contrastive learning did help improve the generative performance of the model.
Revised: In other words, contrastive learning did help improve the generative performance of the model.

Original: However, using an excessivelylarge γ led to a deterioration of the performance.
Revised: However, using an excessively large γ led to a deterioration of the performance.

Original: This is analogous to the result of Section 4.3 where we observed617 using an appropriate weight on the generative term led to improvements over using only the discriminativeterm (contrastive loss).
Revised: This is analogous to the result of Section 4.3 where we observed using an appropriate weight on the generative term led to improvements over using only the discriminative term (contrastive loss).

Original: The difference is that to achieve optimal generative performance, the generative term619 weight needs to be larger (i.e., γ < 1 ) whereas to achieve optimal discriminative performance, the discriminativeterm weight needs to be larger (i.e., λ < 1 in (11)).
Revised: The difference is that to achieve optimal generative performance, the generative term weight needs to be larger (i.e., γ < 1 ) whereas to achieve optimal discriminative performance, the discriminative term weight needs to be larger (i.e., λ < 1 in (11)).

Original: We speculate that using a larger model could mitigate this621 trade-off, but this topic is beyond the scope of this work.
Revised: We speculate that using a larger model could mitigate this trade-off, but this topic is beyond the scope of this work.

Original: [Table]
Revised: [Table]

Original: F A Discussion on Making EBCLR Scalable
Revised: F A Discussion on Making EBCLR Scalable

Original: We could consider three alternative losses / methods for training EBCLR: noise contrastive estimation (NCE)624 [40], VERA [41], and score matching [42].
Revised: We could consider three alternative losses / methods for training EBCLR: noise contrastive estimation (NCE) [40], VERA [41], and score matching [42].

Original: Although the NCE loss does not depend on MCMC, it requiresspecification of a noise distribution whose normalized density can be easily evaluated.
Revised: Although the NCE loss does not depend on MCMC, it requires specification of a noise distribution whose normalized density can be easily evaluated.

Original: In most cases, such noise626 distribution samples are easily distinguishable from natural image samples and lead to the density chasm problem[43], making optimization difficult.
Revised: In most cases, such noise distribution samples are easily distinguishable from natural image samples and lead to the density chasm problem [43], making optimization difficult.

Original: Telescoping density-ratio estimation [43] attempts to mitigate this problem,[-
-] but it does not seem to scale well to large data.
Revised: Telescoping density-ratio estimation [43] attempts to mitigate this problem,[-
-] but it does not seem to scale well to large data.

Original: Flow contrastive estimation [44] uses a flow model to define thenoise distribution and trains a flow model along with the EBM.
Revised: Flow contrastive estimation [44] uses a flow model to define the noise distribution and trains a flow model along with the EBM.

Original: Though flow contrastive estimation requires an630 auxiliary flow model, we believe this is the most promising alternative to MCMC.
Revised: Though flow contrastive estimation requires an auxiliary flow model, we believe this is the most promising alternative to MCMC.

Original: In a similar sense, VERA,which uses a generator to sample from the EBM, seems promising as well.
Revised: In a similar sense, VERA, which uses a generator to sample from the EBM, seems promising as well.

Original: Yet, flow contrastive estimation and632 VERA also did not demonstrate scalability to large-scale datasets such as ImageNet.
Revised: Yet, flow contrastive estimation and VERA also did not demonstrate scalability to large-scale datasets such as ImageNet.

Original: Score matching matchesthe gradient of model log-density to the gradient of data log-density, so it does not rely on MCMC.
Revised: Score matching matches the gradient of model log-density to the gradient of data log-density, so it does not rely on MCMC.

Original: However, it634 requires calculation of the Hessian, so in the deep learning setting, this approach is also quite expensive.
Revised: However, it requires calculation of the Hessian, so in the deep learning setting, this approach is also quite expensive.