Original: Energy-Based Contrastive Learning of Visual Representations
Revised: Energy-Based Contrastive Learning of Visual Representations

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress
Revised: AffiliationAddress

Original: Abstract
Revised: Abstract

Original: Contrastive learning is a method of learning visual representations by training
Revised: Contrastive learning is a method of learning visual representations by training

Original: Deep Neural Networks (DNNs) to increase the similarity between representationsof positive pairs (transformations of the same image) and reduce the similaritybetween representations of negative pairs (transformations of different images).
Revised: Deep Neural Networks (DNNs) to increase the similarity between representationsof positive pairs (transformations of the same image) and reduce the similaritybetween representations of negative pairs (transformations of different images).

Original: Here we explore Energy-Based Contrastive Learning (EBCLR) that leveragesthe power of generative learning by combining contrastive learning with EnergyBased Models (EBMs).
Revised: Here we explore Energy-Based Contrastive Learning (EBCLR) that leveragesthe power of generative learning by combining contrastive learning with EnergyBased Models (EBMs).

Original: EBCLR can be theoretically interpreted as learning thejoint distribution of positive pairs, and it shows promising results on small andmedium-scale datasets such as MNIST, Fashion-MNIST, CIFAR-10, and CIFAR100.
Revised: EBCLR can be theoretically interpreted as learning thejoint distribution of positive pairs, and it shows promising results on small andmedium-scale datasets such as MNIST, Fashion-MNIST, CIFAR-10, and CIFAR100.

Original: Specifically, we find EBCLR demonstrates from × 4 up to × 20 accelerationcompared to SimCLR and MoCo v2 in terms of training epochs.
Revised: Specifically, we find EBCLR demonstrates from × 4 up to × 20 accelerationcompared to SimCLR and MoCo v2 in terms of training epochs.

Original: Furthermore, incontrast to SimCLR, we observe EBCLR achieves nearly the same performancewith 254 negative pairs (batch size 128 ) and 30 negative pairs (batch size 16 ) perpositive pair, demonstrating the robustness of EBCLR to small numbers of negativepairs.
Revised: Furthermore, incontrast to SimCLR, we observe EBCLR achieves nearly the same performancewith 254 negative pairs (batch size 128 ) and 30 negative pairs (batch size 16 ) perpositive pair, demonstrating the robustness of EBCLR to small numbers of negativepairs.

Original: Hence, EBCLR provides a novel avenue for improving contrastive learningmethods that usually require large datasets with a significant number of negativepairs per iteration to achieve reasonable performance on downstream tasks.
Revised: Hence, EBCLR provides a novel avenue for improving contrastive learningmethods that usually require large datasets with a significant number of negativepairs per iteration to achieve reasonable performance on downstream tasks.

Original:  Introduction
Revised:  Introduction

Original: In computer vision supervised learning requires a large-scale human-annotated dataset of images totrain accurate deep neural networks (DNNs).
Revised: In computer vision supervised learning requires a large-scale human-annotated dataset of images totrain accurate deep neural networks (DNNs).

Original: However, acquiring labels for millions of images can bedifficult or impossible in practice.
Revised: However, acquiring labels for millions of images can bedifficult or impossible in practice.

Original: This has led to the rise of self-supervised learning, which learnsuseful visual representations by forcing DNNs to be invariant or equivariant to image transformations.
Revised: This has led to the rise of self-supervised learning, which learnsuseful visual representations by forcing DNNs to be invariant or equivariant to image transformations.

Original: Among self-supervised learning algorithms, contrastive methods are rapidly gaining popularity fortheir superb performance.
Revised: Among self-supervised learning algorithms, contrastive methods are rapidly gaining popularity fortheir superb performance.

Original: Specifically, contrastive learning methods [1, 2, 3, 4, 5] train DNNs by increasing the similaritybetween representations of positive pairs (transformations of the same image) and decreasing thesimilarity between representations of negative pairs (transformations of different images).
Revised: Specifically, contrastive learning methods [1, 2, 3, 4, 5] train DNNs by increasing the similaritybetween representations of positive pairs (transformations of the same image) and decreasing thesimilarity between representations of negative pairs (transformations of different images).

Original: Thenegative pairs prevent DNNs from collapsing to the trivial solution, i.e., the constant function.
Revised: Thenegative pairs prevent DNNs from collapsing to the trivial solution, i.e., the constant function.

Original: Thereare numerous variants of contrastive learning methods, such as SimCLR [4], Momentum Contrast(MoCo) [5], etc.
Revised: Thereare numerous variants of contrastive learning methods, such as SimCLR [4], Momentum Contrast(MoCo) [5], etc.

Original: Despite this flurry of research in contrastive learning, contrastive methods require large datasets anda large number of negative pairs per positive pair to achieve reasonable performance on downstreamtasks.
Revised: Despite this flurry of research in contrastive learning, contrastive methods require large datasets anda large number of negative pairs per positive pair to achieve reasonable performance on downstreamtasks.

Original: Although there are recently proposed non-contrastive methods such as BYOL [6] and SimSiam[7] that do not rely on negative pairs
Revised: Although there are recently proposed non-contrastive methods such as BYOL [6] and SimSiam[7] that do not rely on negative pairs

Original: , they require heuristic techniques such as stop-gradient to avertcollapsing to the trivial solution.
Revised: , they require heuristic techniques such as stop-gradient to avertcollapsing to the trivial solution.

Original: There has been an effort to explain the dynamics of non-contrastivemethods with linear neural networks [3], but it is unclear how the analyses generalize to DNNs.
Revised: There has been an effort to explain the dynamics of non-contrastivemethods with linear neural networks [3], but it is unclear how the analyses generalize to DNNs.

Original: 𝑣 + 𝑣 −
Revised: 𝑣 + 𝑣 −

Original: Shared Params 𝑣 ∝1/𝑝(𝑣,𝑣 + ) 𝑓 𝜃 𝑓
Revised: Shared Params 𝑣 ∝1/𝑝(𝑣,𝑣 + ) 𝑓 𝜃 𝑓

Original: 𝜃 𝑓 𝜃 𝑧 𝑧 + 𝑧 −
Revised: 𝜃 𝑓 𝜃 𝑧 𝑧 + 𝑧 −

Original: Projection Space 𝑧
Revised: Projection Space 𝑧

Original: + 𝑧 𝑧 − 𝑝 𝑣,𝑣 + ≫ 𝑝(𝑣,𝑣 − )
Revised: + 𝑧 𝑧 − 𝑝 𝑣,𝑣 + ≫ 𝑝(𝑣,𝑣 − )

Original: Projections ∝1/𝑝(𝑣,𝑣 − )
Revised: Projections ∝1/𝑝(𝑣,𝑣 − )

Original: In this paper, we explore a novel avenue in visual representation learning: Energy-Based Contrastive
Revised: In this paper, we explore a novel avenue in visual representation learning: Energy-Based Contrastive

Original: Learning (EBCLR) which leverages the power of generative learning [8, 9, 10] by combiningcontrastive learning with energy-based models (EBMs).
Revised: Learning (EBCLR) which leverages the power of generative learning [8, 9, 10] by combiningcontrastive learning with energy-based models (EBMs).

Original: EBCLR complements the contrastivelearning loss with a generative loss, and it can be interpreted as learning the joint distribution ofpositive pairs.
Revised: EBCLR complements the contrastivelearning loss with a generative loss, and it can be interpreted as learning the joint distribution ofpositive pairs.

Original: In fact, we demonstrate that the existing contrastive loss is a special case of the EBCLRloss if the generative term is not used.
Revised: In fact, we demonstrate that the existing contrastive loss is a special case of the EBCLRloss if the generative term is not used.

Original: Although EBMs are notorious for being difficult to traindue to their reliance on Stochastic Gradient Langevin Dynamics (SGLD) [11], another importantcontribution of this work is that we overcome this by appropriate modifications to SGLD.
Revised: Although EBMs are notorious for being difficult to traindue to their reliance on Stochastic Gradient Langevin Dynamics (SGLD) [11], another importantcontribution of this work is that we overcome this by appropriate modifications to SGLD.

Original: Extensive experiments on a variety of small and medium-scale datasets demonstrate that EBCLR isrobust to small numbers of negative pairs, and it outperforms SimCLR and MoCo v2 [12] in terms ofsample efficiency and linear evaluation accuracy.
Revised: Extensive experiments on a variety of small and medium-scale datasets demonstrate that EBCLR isrobust to small numbers of negative pairs, and it outperforms SimCLR and MoCo v2 [12] in terms ofsample efficiency and linear evaluation accuracy.

Original: Hence, EBCLR opens up a new research directionfor alleviating the dependence of contrastive methods on large datasets and large batches.
Revised: Hence, EBCLR opens up a new research directionfor alleviating the dependence of contrastive methods on large datasets and large batches.

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: • We propose a novel contrastive learning method called EBCLR which learns the joint
Revised: • We propose a novel contrastive learning method called EBCLR which learns the joint

Original: distribution of positive pairs.
Revised: distribution of positive pairs.

Original: We show that EBCLR loss is equivalent to a combination of acontrastive term and a generative term (Section 3).
Revised: We show that EBCLR loss is equivalent to a combination of acontrastive term and a generative term (Section 3).

Original: To the best of our knowledge, this is the
Revised: To the best of our knowledge, this is the

Original: first work to apply EBMs to contrastive learning of visual representations.
Revised: first work to apply EBMs to contrastive learning of visual representations.

Original: • We show that EBCLR offers two advantages over conventional contrastive learning methods:
Revised: • We show that EBCLR offers two advantages over conventional contrastive learning methods:

Original: EBCLR is several times more sample efficient (Section 4.1) and robust to small batch sizes
Revised: EBCLR is several times more sample efficient (Section 4.1) and robust to small batch sizes

Original: (Section 4.2).
Revised: (Section 4.2).

Original: These factors lead to a non-trivial performance gain for EBCLR.
Revised: These factors lead to a non-trivial performance gain for EBCLR.

Original: • We perform thorough ablation studies of the components of EBCLR: effect of changing the
Revised: • We perform thorough ablation studies of the components of EBCLR: effect of changing the

Original: weight of the generative term (Section 4.3), effect of projection space dimension (Section4.3), and the effect of the proposed SGLD modifications (Section 4.4).
Revised: weight of the generative term (Section 4.3), effect of projection space dimension (Section4.3), and the effect of the proposed SGLD modifications (Section 4.4).

Original:  Related Works
Revised:  Related Works

Original: In this section, we go over related works necessary for understanding EBCLR.
Revised: In this section, we go over related works necessary for understanding EBCLR.

Original: In Appendix A, wegive a more extensive review of relevant works for those not familiar with EBMs, contrastive learning,or generative models.
Revised: In Appendix A, wegive a more extensive review of relevant works for those not familiar with EBMs, contrastive learning,or generative models.

Original: 2.1 Contrastive Learning
Revised: 2.1 Contrastive Learning

Original: For a given batch of images { x n } Nn =1 and two image transformations t , t ′ , contrastive learningmethods first create two views v n = t ( x n ) , v ′ n = t ′ ( x n ) of each instance x n .
Revised: For a given batch of images { x n } Nn =1 and two image transformations t , t ′ , contrastive learningmethods first create two views v n = t ( x n ) , v ′ n = t ′ ( x n ) of each instance x n .

Original: Here, the pair ( v n , v ′ m ) is called a positive pair if n = m and a negative pair if n ̸ = m .
Revised: Here, the pair ( v n , v ′ m ) is called a positive pair if n = m and a negative pair if n ̸ = m .

Original: Given a DNN f θ , the views are thenembedded into the projection space by passing the views through f θ and normalizing.
Revised: Given a DNN f θ , the views are thenembedded into the projection space by passing the views through f θ and normalizing.

Original: Contrastive methods train f θ to increase agreement between projections of positive pairs and decreaseagreement between projections of negative pairs.
Revised: Contrastive methods train f θ to increase agreement between projections of positive pairs and decreaseagreement between projections of negative pairs.

Original: Specifically, f θ is trained to maximize the InfoNCEobjective [1].
Revised: Specifically, f θ is trained to maximize the InfoNCEobjective [1].

Original: After training, outputs from the final layer or an intermediate layer of f θ are used fordownstream tasks.
Revised: After training, outputs from the final layer or an intermediate layer of f θ are used fordownstream tasks.

Original: There are numerous variants of contrastive methods.
Revised: There are numerous variants of contrastive methods.

Original: For instance, SimCLR [4] uses a compositionof random cropping, random flipping, color jittering, color dropping, and blurring as the imagetransformation.
Revised: For instance, SimCLR [4] uses a compositionof random cropping, random flipping, color jittering, color dropping, and blurring as the imagetransformation.

Original: Negative pairs are created by transforming different images within a batch.
Revised: Negative pairs are created by transforming different images within a batch.

Original: On theother hand, MoCo [12] maintains a queue of negative samples, so negative samples are not limited toviews of images from the same batch.
Revised: On theother hand, MoCo [12] maintains a queue of negative samples, so negative samples are not limited toviews of images from the same batch.

Original: 2.2 Energy-Based Models
Revised: 2.2 Energy-Based Models

Original: Given a scalar-valued energy function E θ ( v ) with parameter θ , an energy-based model (EBM) [13]defines a distribution by the formula
Revised: Given a scalar-valued energy function E θ ( v ) with parameter θ , an energy-based model (EBM) [13]defines a distribution by the formula

Original: [Equation]
Revised: [Equation]

Original: 1) where Z ( θ ) is the partition function which guarantees q θ integrates to 1 .
Revised: 1) where Z ( θ ) is the partition function which guarantees q θ integrates to 1 .

Original: Since there are essentially norestrictions on the choice of the energy function, EBMs have great flexibility in modeling distributions.
Revised: Since there are essentially norestrictions on the choice of the energy function, EBMs have great flexibility in modeling distributions.

Original: Hence, EBMs have been applied to a wide variety of machine learning tasks, such as dimensionalityreduction via autoencoding [27], learning generative classifiers [8, 9, 10, 14], generating images [15],and training regression models [16, 17].
Revised: Hence, EBMs have been applied to a wide variety of machine learning tasks, such as dimensionalityreduction via autoencoding [14], learning generative classifiers [8, 9, 10, 15], generating images [16],and training regression models [17, 18].

Original: Wang et al.
Revised: Wang et al.

Original: [26] have explored connections between EBMsand InfoNCE to enhance generative performance of EBMs.
Revised: [19] have explored connections between EBMsand InfoNCE to enhance generative performance of EBMs.

Original: However, to the best of our knowledge,this paper is the first to combine EBMs with contrastive learning for representation learning.
Revised: However, to the best of our knowledge,this paper is the first to combine EBMs with contrastive learning for representation learning.

Original: Given a target distribution, an EBM can be used to estimate its density p when we can only samplefrom p .
Revised: Given a target distribution, an EBM can be used to estimate its density p when we can only samplefrom p .

Original: One way of achieving this is by minimizing the Kullback-Leibler (KL) divergence betweenq θ and p that maximizes the expected log-likelihood of q θ under p [18]:
Revised: One way of achieving this is by minimizing the Kullback-Leibler (KL) divergence betweenq θ and p that maximizes the expected log-likelihood of q θ under p [20]:

Original: [Equation]
Revised: [Equation]

Original: Stochastic gradient ascent can be used to solve (2) [18].
Revised: Stochastic gradient ascent can be used to solve (2) [20].

Original: Specifically, the gradient of the expectedlog-likelihood with respect to the parameters θ can be shown to be
Revised: Specifically, the gradient of the expectedlog-likelihood with respect to the parameters θ can be shown to be

Original: [Equation]
Revised: [Equation]

Original: Hence, updating θ with (3) amounts to pushing up on the energy for samples from q θ and pushingdown on the energy for samples from p .
Revised: Hence, updating θ with (3) amounts to pushing up on the energy for samples from q θ and pushingdown on the energy for samples from p .

Original: This optimization method is also known as contrastivedivergence [19].
Revised: This optimization method is also known as contrastivedivergence [21].

Original: While the second term in (3) can be easily calculated as we have access to samples from p , the firstterm requires sampling from q θ .
Revised: While the second term in (3) can be easily calculated as we have access to samples from p , the firstterm requires sampling from q θ .

Original: Previous works [15, 20, 10, 14] have used Stochastic Gradient
Revised: Previous works [16, 22, 10, 15] have used Stochastic Gradient

Original: Langevin Dynamics (SGLD)
Revised: Langevin Dynamics (SGLD)

Original: [11] to generate samples from q θ .
Revised: [11] to generate samples from q θ .

Original: Specifically, given a sample v 0 fromsome proposal distribution q 0 , the iteration
Revised: Specifically, given a sample v 0 fromsome proposal distribution q 0 , the iteration

Original: [Equation]
Revised: [Equation]

Original: guarantees that the sequence { v t } converges to a sample from q θ assuming { α t } decays at a polynomial rate [11].
Revised: guarantees that the sequence { v t } converges to a sample from q θ assuming { α t } decays at a polynomial rate [11].

Original: However, SGLD requires an infinite number of steps until samples from the proposal distributionconverge to samples from the target distribution.
Revised: However, SGLD requires an infinite number of steps until samples from the proposal distributionconverge to samples from the target distribution.

Original: This is unfeasible, so in practice, only a finitenumber of steps along with constant step size, i.e. α t = α and constant noise variance σ t =
Revised: This is unfeasible, so in practice, only a finitenumber of steps along with constant step size, i.e. α t = α and constant noise variance σ t =

Original: σ 2 areused [15, 20, 10, 14].
Revised: σ 2 areused [16, 22, 10, 15].

Original: Moreover, Yang and Ji [14] noted SGLD often generates samples with extremepixel values that cause EBMs to diverge during training.
Revised: Moreover, Yang and Ji [15] noted SGLD often generates samples with extremepixel values that cause EBMs to diverge during training.

Original: Hence, they have proposed proximal SGLDwhich clamps gradient values into an interval [ − δ, δ ] for a threshold δ > 0 .
Revised: Hence, they have proposed proximal SGLDwhich clamps gradient values into an interval [ − δ, δ ] for a threshold δ > 0 .

Original: Then, the update equationbecomes
Revised: Then, the update equationbecomes

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: for t = 0 , . . .
Revised: for t = 0 , . . .

Original: , T − 1 , where ϵ ∼ N (0 , σ 2 ) and clamp {· , δ } clamps each element of the input vectorinto [ − δ, δ ] .
Revised: , T − 1 , where ϵ ∼ N (0 , σ 2 ) and clamp {· , δ } clamps each element of the input vectorinto [ − δ, δ ] .

Original: In our work, we introduce additional modifications to SGLD which accelerate theconvergence of EBCLR.
Revised: In our work, we introduce additional modifications to SGLD which accelerate theconvergence of EBCLR.

Original: 𝑡 ∼ 𝒯 𝑣 𝑛 ′ 𝑣 𝑛 𝑧 𝑛′ ෤𝑣 𝑛 𝑧 𝑛 ǁ𝑧 𝑛
Revised: 𝑡 ∼ 𝒯 𝑣 𝑛 ′ 𝑣 𝑛 𝑧 𝑛′ ෤𝑣 𝑛 𝑧 𝑛 ǁ𝑧 𝑛

Original: Contrastive Loss 𝑥
Revised: Contrastive Loss 𝑥

Original: 𝑛 −𝜆 ⋅𝐸 𝜃 ( ǁ𝑧 𝑛 ;{𝑧 𝑚′ }) 𝜆 ⋅𝐸 𝜃 (𝑧 𝑛 ;{𝑧 𝑚 ′ })
Revised: 𝑛 −𝜆 ⋅𝐸 𝜃 ( ǁ𝑧 𝑛 ;{𝑧 𝑚′ }) 𝜆 ⋅𝐸 𝜃 (𝑧 𝑛 ;{𝑧 𝑚 ′ })

Original: Replay Buffer
Revised: Replay Buffer

Original: Update
Revised: Update

Original: Shared Parameters 𝑓 𝜃
Revised: Shared Parameters 𝑓 𝜃

Original: 𝑓
Revised: 𝑓

Original: 𝜃 𝑓 𝜃
Revised: 𝜃 𝑓 𝜃

Original: Update ෤𝑣 𝑛 via SGLD using ∇ ෤𝑣 𝑛 𝐸 𝜃 ( ǁ𝑧 𝑛 ;{𝑧 𝑚′ })
Revised: Update ෤𝑣 𝑛 via SGLD using ∇ ෤𝑣 𝑛 𝐸 𝜃 ( ǁ𝑧 𝑛 ;{𝑧 𝑚′ })

Original: [Figure]
Revised: [Figure]

Original:  Theory
Revised:  Theory

Original: 3.1 Energy-Based Contrastive Learning
Revised: 3.1 Energy-Based Contrastive Learning

Original: Let D be a distribution of images and T a distribution of stochastic image transformations.
Revised: Let D be a distribution of images and T a distribution of stochastic image transformations.

Original: Given x ∼ D and i.i.d.
Revised: Given x ∼ D and i.i.d.

Original: t, t ′ ∼ T , our goal is to approximate the joint distribution of the views p ( v, v ′ ) , where v = t ( x ) , v ′ =
Revised: t, t ′ ∼ T , our goal is to approximate the joint distribution of the views p ( v, v ′ ) , where v = t ( x ) , v ′ =

Original: t ′ ( x ) using the model distribution
Revised: t ′ ( x ) using the model distribution

Original: [Equation]
Revised: [Equation]

Original: 6) where Z ( θ ) is a normalization constant, τ > 0 is a temperature hyper-parameter, and z and z ′ areprojections computed by passing the views v and v ′ through the DNN f θ and then normalizing tohave unit norm.
Revised: 6) where Z ( θ ) is a normalization constant, τ > 0 is a temperature hyper-parameter, and z and z ′ areprojections computed by passing the views v and v ′ through the DNN f θ and then normalizing tohave unit norm.

Original: We now explain the intuitive meaning of matching q θ to p .
Revised: We now explain the intuitive meaning of matching q θ to p .

Original: Our key idea is to use p ( v, v ′ ) as a measure of semantic similarity of v and v ′ .
Revised: Our key idea is to use p ( v, v ′ ) as a measure of semantic similarity of v and v ′ .

Original: If two images v and v ′are semantically similar, they are likely to be transformations of similar images.
Revised: If two images v and v ′are semantically similar, they are likely to be transformations of similar images.

Original: So, p ( v, v ′ ) will behigh when v and v ′ are semantically similar and low otherwise.
Revised: So, p ( v, v ′ ) will behigh when v and v ′ are semantically similar and low otherwise.

Original: Suppose q θ successfully approximatesp .
Revised: Suppose q θ successfully approximatesp .

Original: If we equate p ( v, v ′ ) to q θ ( v, v ′ ) in (6) and solve for ∥ z − z ′ ∥ , we see that the distance between zand z ′ will become a monotone increasing function of 1 /p ( v, v ′ ) , which is the inverse of semanticsimilarity of v and v ′ .
Revised: If we equate p ( v, v ′ ) to q θ ( v, v ′ ) in (6) and solve for ∥ z − z ′ ∥ , we see that the distance between zand z ′ will become a monotone increasing function of 1 /p ( v, v ′ ) , which is the inverse of semanticsimilarity of v and v ′ .

Original: So, semantically similar images will have nearby projections, and dissimilarimages will have distant projections.
Revised: So, semantically similar images will have nearby projections, and dissimilarimages will have distant projections.

Original: This idea is illustrated in Figure 1.
Revised: This idea is illustrated in Figure 1.

Original: To approximate p using q θ , we train f θ to maximize the expected log-likelihood of q θ under p :
Revised: To approximate p using q θ , we train f θ to maximize the expected log-likelihood of q θ under p :

Original: [Equation]
Revised: [Equation]

Original: In order to solve this problem with stochastic gradient ascent, we could naively extend (3) to thesetting of joint distributions to obtain the following result.
Revised: In order to solve this problem with stochastic gradient ascent, we could naively extend (3) to thesetting of joint distributions to obtain the following result.

Original: Proposition 1.
Revised: Proposition 1.

Original: The the joint distribution (6) can be formulated as an EBM
Revised: The the joint distribution (6) can be formulated as an EBM

Original: [Equation]
Revised: [Equation]

Original: and the gradient of the objective of (7) is given by
Revised: and the gradient of the objective of (7) is given by

Original: [Equation]
Revised: [Equation]

Original: However, computing the first expectation in (9) requires sampling pairs of views ( v, v ′ ) from q θ ( v, v ′ )via SGLD, which could be expensive.
Revised: However, computing the first expectation in (9) requires sampling pairs of views ( v, v ′ ) from q θ ( v, v ′ )via SGLD, which could be expensive.

Original: To avert this problem, we use Bayes’ rule to decompose
Revised: To avert this problem, we use Bayes’ rule to decompose

Original: E p [log q θ ( v, v ′ )]
Revised: E p [log q θ ( v, v ′ )]

Original: =
Revised: =

Original: E p [log q θ ( v ′ | v )]
Revised: E p [log q θ ( v ′ | v )]

Original: + E p [log q θ ( v )] where q θ ( v ) = (cid:90) q θ ( v, v ′ ) dv ′ .
Revised: + E p [log q θ ( v )] where q θ ( v ) = (cid:90) q θ ( v, v ′ ) dv ′ .

Original: In the first equation of (10), the first and second terms at the RHS will be referred to as discriminativeand generative terms, respectively, throughout the paper.
Revised: In the first equation of (10), the first and second terms at the RHS will be referred to as discriminativeand generative terms, respectively, throughout the paper.

Original: A similar decomposition was used by
Revised: A similar decomposition was used by

Original: Grathwohl et al.
Revised: Grathwohl et al.

Original: [10] in the setting of learning generative classifiers.
Revised: [10] in the setting of learning generative classifiers.

Original: Furthermore, we add a hyper-parameter λ to balance the strength of the discriminative term and thegenerative term.
Revised: Furthermore, we add a hyper-parameter λ to balance the strength of the discriminative term and thegenerative term.

Original: The advantage of this modification will be discussed in Section 4.3.
Revised: The advantage of this modification will be discussed in Section 4.3.

Original: This yields our
Revised: This yields our

Original: Energy-Based Contrastive Learning (EBCLR) objective
Revised: Energy-Based Contrastive Learning (EBCLR) objective

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The discriminative term can be easily differentiated since the partition function Z ( θ ) cancels outwhen
Revised: The discriminative term can be easily differentiated since the partition function Z ( θ ) cancels outwhen

Original: q θ ( v, v ′ ) is divided by q θ ( v ) .
Revised: q θ ( v, v ′ ) is divided by q θ ( v ) .

Original: However, the generative term still contains Z ( θ ) .
Revised: However, the generative term still contains Z ( θ ) .

Original: We now presentour key result, which addresses this problem by relating the generative term to EBMs.
Revised: We now presentour key result, which addresses this problem by relating the generative term to EBMs.

Original: The proof isdeferred to Appendix C.1.
Revised: The proof isdeferred to Appendix C.1.

Original: Theorem 2.
Revised: Theorem 2.

Original: The marginal distribution in (10) can be formulated as an EBM
Revised: The marginal distribution in (10) can be formulated as an EBM

Original: [Equation]
Revised: [Equation]

Original: where Z ( θ ) is the partition function in (6) , and the gradient of the generative term is given by
Revised: where Z ( θ ) is the partition function in (6) , and the gradient of the generative term is given by

Original: [Equation]
Revised: [Equation]

Original: Thus, the gradient of the EBCLR objective is
Revised: Thus, the gradient of the EBCLR objective is

Original: [Equation]
Revised: [Equation]

Original: Theorem 2 suggests that the EBM for the joint distribution can be learned by computing the gradientsof the discriminative term and the EBM for the marginal distribution.
Revised: Theorem 2 suggests that the EBM for the joint distribution can be learned by computing the gradientsof the discriminative term and the EBM for the marginal distribution.

Original: Moreover, we only need tosample v from q θ ( v ) to compute the second expectation in (14).
Revised: Moreover, we only need tosample v from q θ ( v ) to compute the second expectation in (14).

Original: 3.2 Approximating the EBCLR Objective
Revised: 3.2 Approximating the EBCLR Objective

Original: To implement EBCLR, we need to approximate expectations in (11) with their empirical means.
Revised: To implement EBCLR, we need to approximate expectations in (11) with their empirical means.

Original: Suppose samples { ( v n , v ′ n ) }
Revised: Suppose samples { ( v n , v ′ n ) }

Original: Nn =1 from p ( v, v ′ ) are given, and let { ( z n , z ′ n ) }
Revised: Nn =1 from p ( v, v ′ ) are given, and let { ( z n , z ′ n ) }

Original: Nn =1 be the correspondingprojections.
Revised: Nn =1 be the correspondingprojections.

Original: As the learning goal is to make q θ ( v n , v ′ n ) approximate the joint probability densityfunction p ( v n , v ′ n ) , the empirical mean (cid:98) q θ ( v n ) can be defined as:
Revised: As the learning goal is to make q θ ( v n , v ′ n ) approximate the joint probability densityfunction p ( v n , v ′ n ) , the empirical mean (cid:98) q θ ( v n ) can be defined as:

Original: [Equation]
Revised: [Equation]

Original: where the sum is over the collection of v ′ m defined as
Revised: where the sum is over the collection of v ′ m defined as

Original: [Equation]
Revised: [Equation]

Original: 16) and N ′ :
Revised: 16) and N ′ :

Original: = |{ v ′ m : v ′ m ̸
Revised: = |{ v ′ m : v ′ m ̸

Original: = v n }| = 2 N − 1 .
Revised: = v n }| = 2 N − 1 .

Original: One could also use a simpler form of the empirical mean:
Revised: One could also use a simpler form of the empirical mean:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Similarly, q θ ( v ′ | v ) in (11), which should approximate the conditional probability density p ( v ′ | v ) , canbe represented in terms of q θ ( v n , v ′ n ) .
Revised: Similarly, q θ ( v ′ | v ) in (11), which should approximate the conditional probability density p ( v ′ | v ) , canbe represented in terms of q θ ( v n , v ′ n ) .

Original: Specifically, we have
Revised: Specifically, we have

Original: [Equation]
Revised: [Equation]

Original: It is then immediately apparent that the empirical form of the discriminative term using (18) isa particular instance of the contrastive learning objective such as InfoNCE and SimCLR.
Revised: It is then immediately apparent that the empirical form of the discriminative term using (18) isa particular instance of the contrastive learning objective such as InfoNCE and SimCLR.

Original: Hence,
Revised: Hence,

Original: EBCLR can be interpreted as complementing contrastive learning with a generative term defined byan EBM.
Revised: EBCLR can be interpreted as complementing contrastive learning with a generative term defined byan EBM.

Original: We will demonstrate in Section 4.1 that the generative term offers significant advantagesover other contrastive learning methods.
Revised: We will demonstrate in Section 4.1 that the generative term offers significant advantagesover other contrastive learning methods.

Original: For the second term, we use the simpler form of the empirical mean in (17):
Revised: For the second term, we use the simpler form of the empirical mean in (17):

Original: [Equation]
Revised: [Equation]

Original: We could also use (15) as the empirical mean, but either choice showed identical performance (c.f.
Revised: We could also use (15) as the empirical mean, but either choice showed identical performance (c.f.

Original: Appendix E.3).
Revised: Appendix E.3).

Original: So, we have found (15) to be not worth the additional complexity, and have resortedto the simpler approximation (17) instead.
Revised: So, we have found (15) to be not worth the additional complexity, and have resortedto the simpler approximation (17) instead.

Original: In Appendix C.2, we theoretically justify that EBCLR willwork as intended even with the approximations (15) or (17).
Revised: In Appendix C.2, we theoretically justify that EBCLR willwork as intended even with the approximations (15) or (17).

Original: If we compare (19) with (12), we cansee that this approximation of q θ ( v ) yields the energy function (after ignoring the constant log N )
Revised: If we compare (19) with (12), we cansee that this approximation of q θ ( v ) yields the energy function (after ignoring the constant log N )

Original: [Equation]
Revised: [Equation]

Original: 3.3 Modifications to SGLD
Revised: 3.3 Modifications to SGLD

Original: According to Theorem 2, we need samples from the marginal q θ ( v ) to calculate the second expectationin (14).
Revised: According to Theorem 2, we need samples from the marginal q θ ( v ) to calculate the second expectationin (14).

Original: Hence, we apply proximal SGLD (5) with the energy function (20) to sample from q θ ( v ) as
Revised: Hence, we apply proximal SGLD (5) with the energy function (20) to sample from q θ ( v ) as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for t = 0 , . . .
Revised: for t = 0 , . . .

Original: , T − 1 , where ϵ ∼ N (0 , σ 2 ) .
Revised: , T − 1 , where ϵ ∼ N (0 , σ 2 ) .

Original: We make three additional modifications to proximal
Revised: We make three additional modifications to proximal

Original: SGLD to expedite the training process.
Revised: SGLD to expedite the training process.

Original: From here on, we will be referring to proximal SGLD in (5)when we say SGLD.
Revised: From here on, we will be referring to proximal SGLD in (5)when we say SGLD.

Original: First, we initialize SGLD from generated samples from previous iterations, and with probabilityρ , we reinitialize SGLD chains from samples from a proposal distribution q 0 .
Revised: First, we initialize SGLD from generated samples from previous iterations, and with probabilityρ , we reinitialize SGLD chains from samples from a proposal distribution q 0 .

Original: This is achieved bykeeping a replay buffer B of SGLD samples from previous iterations.
Revised: This is achieved bykeeping a replay buffer B of SGLD samples from previous iterations.

Original: This technique of maintaininga replay buffer has also been used in previous works and has proven to be crucial for stabilizing andaccelerating the convergence of EBMs [15, 10, 14].
Revised: This technique of maintaininga replay buffer has also been used in previous works and has proven to be crucial for stabilizing andaccelerating the convergence of EBMs [16, 10, 15].

Original: Second, the proposal distribution q 0 is set to be the data distribution p ( v ) .
Revised: Second, the proposal distribution q 0 is set to be the data distribution p ( v ) .

Original: This choice differs fromthose of previous works [15, 10, 14] which have either used the uniform distribution or a mixture of
Revised: This choice differs fromthose of previous works [16, 10, 15] which have either used the uniform distribution or a mixture of

Original: Gaussians as the proposal distribution.
Revised: Gaussians as the proposal distribution.

Original: Finally, we use multi-stage SGLD (MSGLD) , which adaptively controls the magnitude of noise addedin SGLD.
Revised: Finally, we use multi-stage SGLD (MSGLD) , which adaptively controls the magnitude of noise addedin SGLD.

Original: For each sample ˜ v in the replay buffer B , we keep a count κ ˜ v of number of times it hasbeen used as the initial point of SGLD.
Revised: For each sample ˜ v in the replay buffer B , we keep a count κ ˜ v of number of times it hasbeen used as the initial point of SGLD.

Original: For samples with a low count, we use noise of high variance,and for samples with a high count, we use noise of low variance.
Revised: For samples with a low count, we use noise of high variance,and for samples with a high count, we use noise of low variance.

Original: Specifically, in (5), we set
Revised: Specifically, in (5), we set

Original: [Equation]
Revised: [Equation]

Original: where [ · ] + := max { 0 , ·} , σ 2max and σ 2min are the upper and lower bounds on the noise variance,respectively, and K controls the decay rate of noise variance.
Revised: where [ · ] + := max { 0 , ·} , σ 2max and σ 2min are the upper and lower bounds on the noise variance,respectively, and K controls the decay rate of noise variance.

Original: The purpose of this technique is tofacilitate quick exploration of the modes of q θ and still guarantee SGLD generates samples withsufficiently low energy.
Revised: The purpose of this technique is tofacilitate quick exploration of the modes of q θ and still guarantee SGLD generates samples withsufficiently low energy.

Original: The pseudocodes for MSGLD and EBCLR are given in Algorithms 1 and 2,respectively, in Appendix B, and the overall learning flow of EBCLR is described in Figure 2.
Revised: The pseudocodes for MSGLD and EBCLR are given in Algorithms 1 and 2,respectively, in Appendix B, and the overall learning flow of EBCLR is described in Figure 2.

Original:  Experiments
Revised:  Experiments

Original: We now describe the experimental settings.
Revised: We now describe the experimental settings.

Original: A complete description is deferred to Appendix D.
Revised: A complete description is deferred to Appendix D.

Original: Baseline methods and datasets.
Revised: Baseline methods and datasets.

Original: The baseline methods are SimCLR, MoCo v2, SimSiam, and
Revised: The baseline methods are SimCLR, MoCo v2, SimSiam, and

Original: BYOL.
Revised: BYOL.

Original: The hyper-parameters are chosen closely following the original works [4, 12, 7, 6].
Revised: The hyper-parameters are chosen closely following the original works [4, 12, 7, 6].

Original: We usefour datasets: MNIST [21], Fashion MNIST (FMNIST) [22], CIFAR10, and CIFAR100 [23].
Revised: We usefour datasets: MNIST [23], Fashion MNIST (FMNIST) [24], CIFAR10, and CIFAR100 [25].

Original: DNN architecture.
Revised: DNN architecture.

Original: We decompose f θ = π θ ◦ ϕ θ where ϕ θ is the encoder network and π θ is theprojection network.
Revised: We decompose f θ = π θ ◦ ϕ θ where ϕ θ is the encoder network and π θ is theprojection network.

Original: Rather than using the output of f θ for downstream tasks, we follow previousworks [4, 5, 1, 2, 3, 6, 7] and use the output of ϕ θ instead.
Revised: Rather than using the output of f θ for downstream tasks, we follow previousworks [4, 5, 1, 2, 3, 6, 7] and use the output of ϕ θ instead.

Original: In our experiments, we set ϕ θ to bea ResNet-18 [24] up to the global average pooling layer and π θ to be a 2-layer MLP with outputdimension 128 .
Revised: In our experiments, we set ϕ θ to bea ResNet-18 [26] up to the global average pooling layer and π θ to be a 2-layer MLP with outputdimension 128 .

Original: However, we remove batch normalization because batch normalization hurts SGLD[15].
Revised: However, we remove batch normalization because batch normalization hurts SGLD[16].

Original: We also replace ReLU with leaky ReLU to expedite the convergence of SGLD.
Revised: We also replace ReLU with leaky ReLU to expedite the convergence of SGLD.

Original: For the baselines,we use settings proposed in the original works while keeping the backbone fixed to be ResNet-18.
Revised: For the baselines,we use settings proposed in the original works while keeping the backbone fixed to be ResNet-18.

Original: Evaluation.
Revised: Evaluation.

Original: We evaluate the representations by training a linear classifier on top of frozen ϕ θ .
Revised: We evaluate the representations by training a linear classifier on top of frozen ϕ θ .

Original: [Table]
Revised: [Table]

Original: 4.1 Comparison with Baselines
Revised: 4.1 Comparison with Baselines

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: We use batch size 128 for EBCLR andbatch size 256 for the baseline methodsfollowing Wang et.
Revised: We use batch size 128 for EBCLR andbatch size 256 for the baseline methodsfollowing Wang et.

Original: al [25] and train eachmethod for 100 epochs.
Revised: al [27] and train eachmethod for 100 epochs.

Original: Table 1 showsthe result of training each method for 100epochs.
Revised: Table 1 showsthe result of training each method for 100epochs.

Original: Observe that EBCLR consistentlyoutperforms all baseline methods in termsof linear evaluation accuracy.
Revised: Observe that EBCLR consistentlyoutperforms all baseline methods in termsof linear evaluation accuracy.

Original: Moreover,relative efficiency indicates EBCLR is capable of achieving the same level of performance as the baseline methods with muchfewer training epochs.
Revised: Moreover,relative efficiency indicates EBCLR is capable of achieving the same level of performance as the baseline methods with muchfewer training epochs.

Original: Concretely, we observe at least × 4 acceleration in terms ofepochs compared to contrastive methods.
Revised: Concretely, we observe at least × 4 acceleration in terms ofepochs compared to contrastive methods.

Original: Hence, EBCLR is a much more desirablechoice than SimCLR or MoCo v2 for learning visual representations when we have a small number of training samples.
Revised: Hence, EBCLR is a much more desirablechoice than SimCLR or MoCo v2 for learning visual representations when we have a small number of training samples.

Original: We also investigate the transfer learning performance of EBCLR.
Revised: We also investigate the transfer learning performance of EBCLR.

Original: Table 2 compares the transferlearning accuracies.
Revised: Table 2 compares the transferlearning accuracies.

Original: EBCLR always outperforms the baseline methods, and the performance gap isespecially large on CIFAR10 and CIFAR100.
Revised: EBCLR always outperforms the baseline methods, and the performance gap isespecially large on CIFAR10 and CIFAR100.

Original: This indicates EBCLR learns visual representationsthat generalize well across datasets.
Revised: This indicates EBCLR learns visual representationsthat generalize well across datasets.

Original: Repeating the above experiments with longer training or KNNclassification led to similar conclusions (c.f.
Revised: Repeating the above experiments with longer training or KNNclassification led to similar conclusions (c.f.

Original: Appendixes E.1 and E.4, respectively).
Revised: Appendixes E.1 and E.4, respectively).

Original: 4.2 Effect of Reducing Negative Pairs
Revised: 4.2 Effect of Reducing Negative Pairs

Original: We compared the performances of EBCLR and SimCLR as we reduced the number of negativepairs per positive pair.
Revised: We compared the performances of EBCLR and SimCLR as we reduced the number of negativepairs per positive pair.

Original: For MoCo v2, the negative samples are provided by a queue updated by amomentum encoder.
Revised: For MoCo v2, the negative samples are provided by a queue updated by amomentum encoder.

Original: On the other hand, for EBCLR and SimCLR, negative samples come from thesame batch as the positive pair.
Revised: On the other hand, for EBCLR and SimCLR, negative samples come from thesame batch as the positive pair.

Original: So, we did not have a way of fairly comparing EBCLR and SimCLRwith MoCo v2.
Revised: So, we did not have a way of fairly comparing EBCLR and SimCLRwith MoCo v2.

Original: Hence, we excluded MoCo v2 from this experiment.
Revised: Hence, we excluded MoCo v2 from this experiment.

Original: We note that, according to (18), given a batch of size N , we obtain 2 N − 2 negative pairs foreach positive pair.
Revised: We note that, according to (18), given a batch of size N , we obtain 2 N − 2 negative pairs foreach positive pair.

Original: SimCLR also has 2 N − 2 negative pairs for each positive pair.
Revised: SimCLR also has 2 N − 2 negative pairs for each positive pair.

Original: Hence, we canconveniently compare the sensitivity of EBCLR and SimCLR to the number of negative pairs byvarying the batch size.
Revised: Hence, we canconveniently compare the sensitivity of EBCLR and SimCLR to the number of negative pairs byvarying the batch size.

Original: Table 3 shows the result of training each method for 100 epochs with batch sizes in { 16 , 64 , 128 } .
Revised: Table 3 shows the result of training each method for 100 epochs with batch sizes in { 16 , 64 , 128 } .

Original: We make three important observations.
Revised: We make three important observations.

Original: First, EBCLR consistently beats SimCLR in terms of linearevaluation accuracy for every batch size.
Revised: First, EBCLR consistently beats SimCLR in terms of linearevaluation accuracy for every batch size.

Original: Second, EBCLR is invariant to the choice of batch size.
Revised: Second, EBCLR is invariant to the choice of batch size.

Original: This contrasts with SimCLR whose performance degrades as batch size decreases.
Revised: This contrasts with SimCLR whose performance degrades as batch size decreases.

Original: Consequently,
Revised: Consequently,

Original: EBCLR with batch size 16 beats SimCLR with batch size 128 .
Revised: EBCLR with batch size 16 beats SimCLR with batch size 128 .

Original: Finally, as a byproduct of the secondobservation, the efficiency of EBCLR relative to SimCLR increases as batch size decreases.
Revised: Finally, as a byproduct of the secondobservation, the efficiency of EBCLR relative to SimCLR increases as batch size decreases.

Original: Theseproperties make EBCLR suitable for situations where we cannot use large batch sizes, e.g., when we
Revised: Theseproperties make EBCLR suitable for situations where we cannot use large batch sizes, e.g., when we

Original: [Table]
Revised: [Table]

Original: have a small number of GPUs.
Revised: have a small number of GPUs.

Original: Repeating the experiments with longer training or KNN classificationagain led to similar conclusions (c.f.
Revised: Repeating the experiments with longer training or KNN classificationagain led to similar conclusions (c.f.

Original: Appendixes E.2 and E.4, respectively).
Revised: Appendixes E.2 and E.4, respectively).

Original: 4.3 Effect of λ and Projection Dimension
Revised: 4.3 Effect of λ and Projection Dimension

Original: [Figure]
Revised: [Figure]

Original: We explored the effect of changing the hyperparameter λ which controls the importance ofthe generative term relative to the discriminativeterm (c.f. Equation (11)).
Revised: We explored the effect of changing the hyperparameter λ which controls the importance ofthe generative term relative to the discriminativeterm (c.f. Equation (11)).

Original: Figure 3a shows theperformance of EBCLR with various values of λas training progresses.
Revised: Figure 3a shows theperformance of EBCLR with various values of λas training progresses.

Original: We observe that naivelyusing λ = 1 . 

Revised: We observe that naivelyusing λ = 1 .

Original: leads to poor results.
Revised: leads to poor results.

Original: The performance peaks at λ = 0 .
Revised: The performance peaks at λ = 0 .

Original: 1 , and then degrades aswe further decrease λ .
Revised: 1 , and then degrades aswe further decrease λ .

Original: This result has two crucial implications.
Revised: This result has two crucial implications.

Original: First,the generative term plays a non-trivial role in
Revised: First,the generative term plays a non-trivial role in

Original: EBCLR.
Revised: EBCLR.

Original: Second, we need to strike a right balance between the discriminative term and thegenerative term to achieve good performance ondownstream tasks 1 .
Revised: Second, we need to strike a right balance between the discriminative term and thegenerative term to achieve good performance ondownstream tasks 1 .

Original: We also investigated the effect of varying the output dimension of π θ .
Revised: We also investigated the effect of varying the output dimension of π θ .

Original: Figure 3b shows linearevaluation results for projection dimensions in { 128 , 256 , 512 } .
Revised: Figure 3b shows linearevaluation results for projection dimensions in { 128 , 256 , 512 } .

Original: We observe that the projectiondimension has essentially no influence on the training process.
Revised: We observe that the projectiondimension has essentially no influence on the training process.

Original: In this respect, EBCLR resembles
Revised: In this respect, EBCLR resembles

Original: SimCLR which is also invariant to the output dimension (c.f. Figure 8 in the work by Chen et.
Revised: SimCLR which is also invariant to the output dimension (c.f. Figure 8 in the work by Chen et.

Original: al [4]).
Revised: al [4]).

Original: 4.4 Effect of SGLD Modifications
Revised: 4.4 Effect of SGLD Modifications

Original: We now study the roles of the three SGLD modifications proposed in Section 3.3.
Revised: We now study the roles of the three SGLD modifications proposed in Section 3.3.

Original: Figure 4 shows theresults of varying one parameter of MSGLD while keeping the others fixed.
Revised: Figure 4 shows theresults of varying one parameter of MSGLD while keeping the others fixed.

Original: Effect of reintialization frequency ρ .
Revised: Effect of reintialization frequency ρ .

Original: Figure 4a displays linear evaluation results for ρ ∈{ 0 . 0 , 0 . 2 , 1 . 0 } .
Revised: Figure 4a displays linear evaluation results for ρ ∈{ 0 . 0 , 0 . 2 , 1 . 0 } .

Original: We note that setting ρ = 1 . 0 is equivalent to removing the replay buffer.
Revised: We note that setting ρ = 1 . 0 is equivalent to removing the replay buffer.

Original: Also,setting ρ = 0 . 0 is equivalent to never reinitializing SGLD chains.
Revised: Also,setting ρ = 0 . 0 is equivalent to never reinitializing SGLD chains.

Original: Initially, ρ = 0 . 0 shows the best performance, as SGLD quickly reaches samples of lower energy.
Revised: Initially, ρ = 0 . 0 shows the best performance, as SGLD quickly reaches samples of lower energy.

Original: However, learning then slows down because of the lack of diversity of samples in the replay buffer B .
Revised: However, learning then slows down because of the lack of diversity of samples in the replay buffer B .

Original: This implies that it is necessary to set ρ > 0 in order to learn good representations.
Revised: This implies that it is necessary to set ρ > 0 in order to learn good representations.

Original: On the other hand, ρ = 1 . 0 shows slow convergence in the beginning because samples in the replaybuffer are not given enough iterations to reach low energy.
Revised: On the other hand, ρ = 1 . 0 shows slow convergence in the beginning because samples in the replaybuffer are not given enough iterations to reach low energy.

Original: Although it does beat ρ = 0 . 

Revised: Although it does beat ρ = 0 .

Original: at latterepochs, it still often performs worse than ρ = 0 .
Revised: at latterepochs, it still often performs worse than ρ = 0 .

Original: Moreover, it is not sample-efficient compared toρ = 0 .
Revised: Moreover, it is not sample-efficient compared toρ = 0 .

Original: 2 since we have to provide an entire batch of new samples for reinitializing SGLD chains ateach iteration.
Revised: 2 since we have to provide an entire batch of new samples for reinitializing SGLD chains ateach iteration.

Original: Given the above observations, it is clear why the intermediate value 0 .
Revised: Given the above observations, it is clear why the intermediate value 0 .

Original: 2 is the best choice out ofρ ∈ { 0 . 0 , 0 . 2 , 1 . 0 } .
Revised: 2 is the best choice out ofρ ∈ { 0 . 0 , 0 . 2 , 1 . 0 } .

Original: ρ = 0 .
Revised: ρ = 0 .

Original: 2 allows enough time for samples in the replay buffer to reach low energywhile still maintaining the diversity of samples in B .
Revised: 2 allows enough time for samples in the replay buffer to reach low energywhile still maintaining the diversity of samples in B .

Original: Also, it is sample-efficient compared to ρ = 1 . 0 .
Revised: Also, it is sample-efficient compared to ρ = 1 . 0 .

Original: Effect of proposal distribution q 0 .
Revised: Effect of proposal distribution q 0 .

Original: Figure 4b compares linear evaluation accuracies with q 0 as theuniform distribution and q 0 = p ( v ) .
Revised: Figure 4b compares linear evaluation accuracies with q 0 as theuniform distribution and q 0 = p ( v ) .

Original: We observe prominent acceleration in the initial epochs forq 

Revised: We observe prominent acceleration in the initial epochs forq

Original: = p ( v ) .
Revised: = p ( v ) .

Original: Hence, we can conclude that this choice of proposal distribution is crucial for the highefficiency of EBCLR compared to the baseline methods in Tables 1 and 3.
Revised: Hence, we can conclude that this choice of proposal distribution is crucial for the highefficiency of EBCLR compared to the baseline methods in Tables 1 and 3.

Original: [Figure]
Revised: [Figure]

Original: We believe this acceleration effect can be explainedby the work of Hinton [19].
Revised: We believe this acceleration effect can be explainedby the work of Hinton [21].

Original: Specifically, let us observe that the EBM update equation (3) pushes upthe energy on the model distribution q θ .
Revised: Specifically, let us observe that the EBM update equation (3) pushes upthe energy on the model distribution q θ .

Original: In the implementation of EBCLR with q 0 = p ( v ) , however,q θ is replaced by the distribution of samples createdby a finite number of (noisy) gradient steps on realdata points (c.f. Section 3.3).
Revised: In the implementation of EBCLR with q 0 = p ( v ) , however,q θ is replaced by the distribution of samples createdby a finite number of (noisy) gradient steps on realdata points (c.f. Section 3.3).

Original: Hence, the modified
Revised: Hence, the modified

Original: EMB update equation contains the curvature information of the data manifold.
Revised: EMB update equation contains the curvature information of the data manifold.

Original: This curvature informationmay expedite the training process of EBCLR.
Revised: This curvature informationmay expedite the training process of EBCLR.

Original: For adetailed discussion on this, we refer the readers to
Revised: For adetailed discussion on this, we refer the readers to

Original: Section 3 of the work by Hinton [19].
Revised: Section 3 of the work by Hinton [21].

Original: Comparison of SGLD and MSGLD.
Revised: Comparison of SGLD and MSGLD.

Original: Figure 4cshows results with SGLD with σ ∈ { 0 . 01 , 0 . 05 } and
Revised: Figure 4cshows results with SGLD with σ ∈ { 0 . 01 , 0 . 05 } and

Original: MSGLD with σ min = 0 .
Revised: MSGLD with σ min = 0 .

Original: and σ max = 0 . 05 .
Revised: and σ max = 0 . 05 .

Original: Wenote that setting σ min = σ max reduces MSGLD to
Revised: Wenote that setting σ min = σ max reduces MSGLD to

Original: SGLD.
Revised: SGLD.

Original: We observe σ = 0 .
Revised: We observe σ = 0 .

Original: 01 initially shows fast convergence but then saturates due to the lack of diversityof generated samples.
Revised: 01 initially shows fast convergence but then saturates due to the lack of diversityof generated samples.

Original: On the other hand, σ = 0 .
Revised: On the other hand, σ = 0 .

Original: 05initially has the worst performance but eventuallybeats σ = 0 .
Revised: 05initially has the worst performance but eventuallybeats σ = 0 .

Original: 01 since σ = 0 .
Revised: 01 since σ = 0 .

Original: 05 quickly explores themodes of q θ .
Revised: 05 quickly explores themodes of q θ .

Original: MSGLD inherits the best of both settings.
Revised: MSGLD inherits the best of both settings.

Original: Specifically, MSGLD is as fast as σ = 0 .
Revised: Specifically, MSGLD is as fast as σ = 0 .

Original: 01 inthe beginning, and it does not suffer from the saturation problem.
Revised: 01 inthe beginning, and it does not suffer from the saturation problem.

Original:  Limitations and Societal Impacts
Revised:  Limitations and Societal Impacts

Original: Limitations.
Revised: Limitations.

Original: The main limitation of our work is of scale.
Revised: The main limitation of our work is of scale.

Original: While EBCLR demonstrates superiorsample efficiency, it requires inner SGLD iterations (which cannot be parallelized) and a replay buffer
Revised: While EBCLR demonstrates superiorsample efficiency, it requires inner SGLD iterations (which cannot be parallelized) and a replay buffer

Original: B .
Revised: B .

Original: These two components increase the computational burden of EBCLR.
Revised: These two components increase the computational burden of EBCLR.

Original: So, we found it difficult toapply EBCLR to large-scale data such as ImageNet.
Revised: So, we found it difficult toapply EBCLR to large-scale data such as ImageNet.

Original: However, we note that inner SGLD iterationsand the replay buffer are not particular limitations of EBCLR, but limitations of EBMs in general.
Revised: However, we note that inner SGLD iterationsand the replay buffer are not particular limitations of EBCLR, but limitations of EBMs in general.

Original: Given the increasing efforts to overcome these limitations such as Proximal-YOPO-SGLD (for morediscussion, see Appendix F), we believe EBCLR will eventually be applicable to larger data.
Revised: Given the increasing efforts to overcome these limitations such as Proximal-YOPO-SGLD (for morediscussion, see Appendix F), we believe EBCLR will eventually be applicable to larger data.

Original: Social Impacts.
Revised: Social Impacts.

Original: We generally expect positive outcomes from this research.
Revised: We generally expect positive outcomes from this research.

Original: Further developmentof EBCLR can mitigate the need for large amount of data and large batch sizes to learn goodrepresentations and ultimately lead to a reduction in resource consumption.
Revised: Further developmentof EBCLR can mitigate the need for large amount of data and large batch sizes to learn goodrepresentations and ultimately lead to a reduction in resource consumption.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we proposed EBCLR which combines contrastive learning with EBMs.
Revised: In this work, we proposed EBCLR which combines contrastive learning with EBMs.

Original: This amalgamation of ideas has led to both theoretical and practical contributions.
Revised: This amalgamation of ideas has led to both theoretical and practical contributions.

Original: Theoretically, EBCLRassociates distance in the projection space with the density of positive samples.
Revised: Theoretically, EBCLRassociates distance in the projection space with the density of positive samples.

Original: Since the distributionof positive samples reflects the semantic similarity of images, EBCLR is capable of learning goodvisual representations.
Revised: Since the distributionof positive samples reflects the semantic similarity of images, EBCLR is capable of learning goodvisual representations.

Original: Practically, EBCLR is several times more sample-efficient than conventionalcontrastive and non-contrastive learning approaches and is robust to small numbers of negative pairs.
Revised: Practically, EBCLR is several times more sample-efficient than conventionalcontrastive and non-contrastive learning approaches and is robust to small numbers of negative pairs.

Original: Hence, EBCLR is applicable even in scenarios with limited data or devices.
Revised: Hence, EBCLR is applicable even in scenarios with limited data or devices.

Original: We believe that EBCLRmakes representation learning available to a wider range of machine learning practitioners.
Revised: We believe that EBCLRmakes representation learning available to a wider range of machine learning practitioners.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Appendix C.
Revised: [Yes] See Appendix C.3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as 
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimentalresults (either in the supplemental material or as a URL)? [Yes]

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?

Original: [Table]
Revised: [Yes]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments [Table] multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experimentsmultiple times)?

Original: [No] Each EBM was trained once due to computation costs.
Revised: [No] Each EBM was trained once due to computation costs.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,internal cluster, or cloud provider)?

Original: [Yes] See Appendix D.
Revised: [Yes] See Appendix D.

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: a) If your work uses existing assets, did you cite the creators
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: b) you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: [Table]

Original: c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A](d)

Original: d) Did you discuss whether and how consent was obtained from people whose data you’re us-
Revised: Did you discuss whether and how consent was obtained from people whose data you’re us-

Original: ing/curating?
Revised: ing/curating?

Original: [N/A]
Revised: [N/A]

Original: e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: Did you discuss whether the data you are using/curating contains personally identifiable

Original: informa-
Revised: informa-

Original: [Table]

Original: tion or offensive content?
Revised: tion or offensive content?

Original: [N/A]
Revised: [N/A]

Original:  applicable?[Table]
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: N/A]
Revised: [N/A]

Original:  you describe any potential participant , with  to Institutional Review Board (IRB)[Table]
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)

Original: approvals, if applicable?
Revised: approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: c) Did you include the estimated hourly wage paid to participants and the total amount spent on [Table] participant compensation?
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount spent onparticipant compensation?

Original: [N/A]
Revised: [N/A]

Original: A Additional BackgroundA.1 Contrastive Divergence and Contrastive Learning
Revised: A Additional Background

Revised: A.1 Contrastive Divergence and Contrastive Learning

Original: Contrastive divergence, initially proposed by Hinton et al.
Revised: Contrastive divergence, initially proposed by Hinton et al.

Original: [19], is a particular method for solving the maximum- 448 likelihood estimation problem for EBMs.
Revised: [21], is a particular method for solving the maximum- 454 likelihood estimation problem for EBMs.

Original: Intuitively, it proceeds by minimizing the energy function on dataand maximizing the energy function on (possibly short-run) MCMC samples of the current EBM distribution 450 through gradient ascent.
Revised: Intuitively, it proceeds by minimizing the energy function on dataand maximizing the energy function on (possibly short-run) MCMC samples of the current EBM distribution 456 through gradient ascent.

Original: In fact, we use an instance of contrastive divergence to optimize the generative term inour EBCLR objective (11).
Revised: In fact, we use an instance of contrastive divergence to optimize the generative term inour EBCLR objective (11).

Original: We also disambiguate contrastive learning from contrastive divergence.
Revised: We also disambiguate contrastive learning from contrastive divergence.

Original: They are similar in the sense that they 453 contrast certain pairs of data.
Revised: They are similar in the sense that they 459 contrast certain pairs of data.

Original: Contrastive learning contrasts positive pairs and negative pairs.
Revised: Contrastive learning contrasts positive pairs and negative pairs.

Original: Contrastivedivergence contrasts data and MCMC samples.
Revised: Contrastivedivergence contrasts data and MCMC samples.

Original: However, they differ in the aspect that contrastive learning 455 is inherently a discriminative learning method (in contrastive learning, the model “predicts” the positive pairamong negative pairs) and contrastive divergence is a generative learning method.
Revised: However, they differ in the aspect that contrastive learning 461 is inherently a discriminative learning method (in contrastive learning, the model “predicts” the positive pairamong negative pairs) and contrastive divergence is a generative learning method.

Original: In the context of unsupervised457 visual representation learning,to the best of our knowledge, there are no works which use discriminative andgenerative losses simultaneously.
Revised: In the context of unsupervised visual representation learning, there are several works which connect contrastive 464 learning with generative learning [28, 29, 30]. For instance, Chen et al. [30] combine contrastive learning with GANs to improve person re-identification.

Original:  Our work, for the first time, explores the synergy between contrastive learning459 (the discriminative term in (11)) and  contrastive divergence (the generative  term in (11)) for learning visualrepresentations.
Revised: Zimmermann et al. [29] and Wang et al. [28] study the distributional 466 properties of embeddings found by contrastive learning. Our work, for the first time, explores the synergybetween contrastive learning (the discriminative term in (11)) and EBMs / contrastive divergence (the generative 468 term in (11)) for learning visual representations.

Original: A.2 Relation to Contrastive Energy-Based Models (CEMs) [26]
Revised: A.2 Relation to Contrastive Energy-Based Models (CEMs) [19]

Original: According to the terms introduced in the work of Wang et al.
Revised: According to the terms introduced in the work of Wang et al.

Original: [26], our EBCLR model distribution (6) with 463 τ = 1 is identical to the model distribution of Non-Parametric CEM ((4) in [26]).
Revised: [19], our EBCLR model distribution (6) with 471 τ = 1 is identical to the model distribution of Non-Parametric CEM ((4) in [19]).

Original: This is because the squaredℓ 2 norm between two unit-norm vectors z and z ′ is equivalent to the inner product between z and z ′ up to anadditive constant.
Revised: This is because the squaredℓ 2 norm between two unit-norm vectors z and z ′ is equivalent to the inner product between z and z ′ up to anadditive constant.

Original: However, our work differs from the work of Wang et al.
Revised: However, our work differs from the work of Wang et al.

Original: [26] in three aspects.
Revised: [19] in three aspects.

Original: Objective function.
Revised: Objective function.

Original: Wang et al.
Revised: Wang et al.

Original: [26] optimizes the log-likelihood of the joint model distribution q θ ( v, v ′ )
Revised: [19] optimizes the log-likelihood of the joint model distribution q θ ( v, v ′ )

Original: directly whereas we use Bayes’ rule to decompose the objective into two terms, the discriminative term and the 468 generative term.
Revised: directly whereas we use Bayes’ rule to decompose the objective into two terms, the discriminative term and the 476 generative term.

Original: This allows us to reweight the generative term to learn better visual representations (as exploredin Section 4.3).
Revised: This allows us to reweight the generative term to learn better visual representations (as exploredin Section 4.3).

Original: Optimization method.
Revised: Optimization method.

Original: Wang et al.
Revised: Wang et al.

Original: [26] maximizes the log-likelihood of the joint model distribution directly 471 via adversarial training.
Revised: [19] maximizes the log-likelihood of the joint model distribution directly 479 via adversarial training.

Original: On the other hand, we use gradient ascent to optimize the discriminative term andcontrastive divergence to optimize the generative term.
Revised: On the other hand, we use gradient ascent to optimize the discriminative term andcontrastive divergence to optimize the generative term.

Original: Computation costs are similar, as adversarial training 473 also requires inner iterations to generate adversarial examples at every model update step.
Revised: Computation costs are similar, as adversarial training 481 also requires inner iterations to generate adversarial examples at every model update step.

Original: Application of interest.
Revised: Application of interest.

Original: Wang et al.
Revised: Wang et al.

Original: [26] focuses on enhancing image generation, while we focus on learning 475 visual representations useful for downstream tasks.
Revised: [19] focuses on enhancing image generation, while we focus on learning 483 visual representations useful for downstream tasks.

Original: Wang et al.
Revised: Wang et al.

Original: [26] also establishes a connection between contrastive divergence and InfoNCE (Section 5.1 in 477 [26]).
Revised: [19] also establishes a connection between contrastive divergence and InfoNCE (Section 5.1 in 485 [19]).

Original: Specifically, Wang et al.
Revised: Specifically, Wang et al.

Original: [26] show that when we approximate the model marginal q θ ( v ) with the datadistribution p ( v ) , MLE gradient for q θ ( v, v ′ ) becomes the InfoNCE loss gradient.
Revised: [19] show that when we approximate the model marginal q θ ( v ) with the datadistribution p ( v ) , MLE gradient for q θ ( v, v ′ ) becomes the InfoNCE loss gradient.

Original: Surprisingly, we find thisholds in our framework as well.
Revised: Surprisingly, we find thisholds in our framework as well.

Original: If we set q θ ( v ) = p ( v ) in (11), the generative term in (11) becomes independent 480 of θ , so we end up optimizing only the discriminative term.
Revised: If we set q θ ( v ) = p ( v ) in (11), the generative term in (11) becomes independent 488 of θ , so we end up optimizing only the discriminative term.

Original: We have shown that the discriminative term can beapproximated by a contrastive learning objective, so we recover the result of Wang et al.
Revised: We have shown that the discriminative term can beapproximated by a contrastive learning objective, so we recover the result of Wang et al.

Original: In this sense, the 482 derivations in Section 3 of our paper generalizes the connection discovered in the work of Wang et al.
Revised: In this sense, the 490 derivations in Section 3 of our paper generalizes the connection discovered in the work of Wang et al.

Original: [26], aswe work with weaker assumptions (we do not set q θ ( v ) =
Revised: [19], aswe work with weaker assumptions (we do not set q θ ( v ) =

Original: p ( v ) in (11)).
Revised: p ( v ) in (11)).

Original: A.3 EBMs for Learning Useful Representations
Revised: A.3 EBMs for Learning Useful Representations

Original: Ranzato et al.
Revised: et al.

Original: [27] proposes to train an EBM with an autoencoder structure by minimizing thesum of an486 encoding loss and a decoding loss to learn an energy surface.
Revised: [ proposes to train an EBM with an autoencoder structure by minimizing the of an encoding loss and a decoding loss to learn an energy surface.

Original: A byproduct is that the EBM learns a compressedrepresentation of the data, which can be used for, e.g., denoising.
Revised: A byproduct is that the EBM learns a compressed representation of the data, which can be used for, e g., denoising.

Original: Our work is similar in the aspect that we use488 a particular EBM architecture (a triplet network) and training the EBM with our proposed method results ina compressed representation of the data which is useful for downstream tasks, e.g., classification and transfer 490 learning.
Revised: Our work is similar in the aspect that we use a particular EBM architecture (a triplet network) and training the EBM with our proposed method results in a compressed representation of the data which is useful for downstream tasks, e g., classification and transfer learning

Original: In that sense, our work is also roughlyrelated to Denoising Autoencoders (DAEs) [28] and Restricted
Revised: In sense roughly

Original: Boltzmann Machines(RBMs) as well.
Revised: Denoising Autoencoders ( Machines

Original: Both are EBMswhose goal is to extract useful representations fromdata.
Revised: EBMs goal is to extract useful representations from

Original: However, we cannot say Ranzato et al.

Original: [2], RBMs, or DAEs precede our work in relating contrastive learning to

Original: EBMs, because they mention anything about contrasting positive pairs against negative pairs.

Original: A.4 Sample Efficiency of Generative Models
Revised: A.4 Sample Efficiency of Generative Models

Original: We motivate the sample efficiency of EBCLR through the lens of discriminative and generative learning.
Revised: We motivate the sample efficiency of EBCLR through the lens of discriminative and generative learning.

Original: We 496 note that standard contrastive learning (e.g., SimCLR) optimizes a discriminative objective (the model mustpredict positive pairs) and EBCLR optimizes a generative objective (given a pair of images, the model must  output how likely it is to be a positive pair).
Revised: We 504 note that standard contrastive learning (e.g., SimCLR) optimizes a discriminative objective (the model mustpredict positive pairs) and EBCLR optimizes a generative objective (given a pair of images, the model must 506 output how likely it is to be a positive pair).

Original: So, we can compare the performance of discriminative models and 499 generative models.
Revised: So, we can compare the performance of discriminative models andgenerative models.

Original: There is a long line of works which show (theoretically and / or empirically) that generative models can 501 outperform discriminative ones.
Revised: There is a long line of works which show (theoretically and / or empirically) that generative models can 509 outperform discriminative ones.

Original: For instance, Efron [29] has theoretically shown that Normal Discriminant
Revised: For instance, Efron [32] has theoretically shown that Normal Discriminant

Original: Analysis can be more efficient than logistic regression.
Revised: Analysis can be more efficient than logistic regression.

Original: Ng et al.
Revised: Ng et al.

Original: [30] has also theoretically shown that in the 503 small sample regime, naïve Bayes can be more resistant to overfitting than logistic regression (i.e., find bettersolutions when given a small number of training samples).
Revised: [33] has also theoretically shown that in the 511 small sample regime, naïve Bayes can be more resistant to overfitting than logistic regression (i.e., find bettersolutions when given a small number of training samples).

Original: There are also works [8, 9, 10] that consider optimizing a weighted combination of a discriminative loss and a 506 generative loss, just like our EBCLR objective (11).
Revised: There are also works [8, 9, 10] that consider optimizing a weighted combination of a discriminative loss and a 514 generative loss, just like our EBCLR objective (11).

Original: They find that with an appropriate weight, the model canoutperform discriminative models.
Revised: They find that with an appropriate weight, the model canoutperform discriminative models.

Original: In particular, Larochelle et al.
Revised: In particular, Larochelle et al.

Original: [9] finds this hybrid approach to be beneficial 508 in the limited training data setting.
Revised: [9] finds this hybrid approach to be beneficial 516 in the limited training data setting.

Original: Grathwohl et al.
Revised: Grathwohl et al.

Original: [10] uses this hybrid approach to train a classifier and findthat the classifier exhibits excellent calibration, unlike other calibration methods that require additional training 510 data.
Revised: [10] uses this hybrid approach to train a classifier and findthat the classifier exhibits excellent calibration, unlike other calibration methods that require additional training 518 data.

Original: We can also address the efficiency of generative models from the perspective of generative pre-training.
Revised: We can also address the efficiency of generative models from the perspective of generative pre-training.

Original: Bengio [-
-] et al.
Revised: Bengio [-
-]520 et al.

Original: [31] finds that pre-training a network with a generative loss brings the network weights towards a goodlocal minimum, thus bringing better generalization.
Revised: [34] finds that pre-training a network with a generative loss brings the network weights towards a goodlocal minimum, thus bringing better generalization.

Original: Erhan et al.
Revised: Erhan et al.

Original: [32] finds that generative pre-training acts as a 514 regularizer and that this regularization effect can be beneficial when we have a small training set.
Revised: [35] finds that generative pre-training acts as a 522 regularizer and that this regularization effect can be beneficial when we have a small training set.

Original: Another explanation specific to EBCLR is that the generative loss has an effect orthogonal to the effect of the 516 discriminative loss.
Revised: Another explanation specific to EBCLR is that the generative loss has an effect orthogonal to the effect of the 524 discriminative loss.

Original: To minimize the generative loss, the model needs to place low energy on data and highenergy elsewhere.
Revised: To minimize the generative loss, the model needs to place low energy on data and highenergy elsewhere.

Original: This can be achieved only when the model has features that accurately capture the manifold 518 of the data.
Revised: This can be achieved only when the model has features that accurately capture the manifold 526 of the data.

Original: When we have a small training set, the model may easily solve the discriminative objective andoverfit.
Revised: When we have a small training set, the model may easily solve the discriminative objective andoverfit.

Original: By providing an auxiliary generative task, the overfitting can be mitigated.
Revised: By providing an auxiliary generative task, the overfitting can be mitigated.

Original: An analogous effect with 520 Joint Energy-Based Models is demonstrated by Zhao et al.
Revised: An analogous effect with 528 Joint Energy-Based Models is demonstrated by Zhao et al.

Original: B Psuedocodes
Revised: B Psuedocodes

Revised: 1: Input: Energy E θ , initialization point ˜ v , count κ ˜ v , step size α , number of iterations T , proximity parameter δ , decay parameter K , variance bounds σ 2min , σ 2max 2: σ = σ min + ( σ max − σ min ) · [1 − κ ˜ v /K ]

Revised: + 3: for t

Revised: T − 1 do 4:

Original: C Missing Proofs
Revised: Sample ϵ ∼ N (0 , σ 2 ) 5: ˜ v ← ˜ v − α · clamp {∇ ˜ v E θ (˜ v ) , δ } + ϵ 6: end for 7: κ ˜ v ← κ ˜ v + 1: Input: DNN f θ , batch size N , generative term weight λ , replay buffer B , reinit. frequency ρ 2: while not converged do 3:

Revised: Sample { ( v n , v ′ n ) } Nn =1 from p ( v, v ′ ) 4:

Revised: Calculate the disc.

Revised: term with Eq.

Revised: Calculate gradient of the disc.

Revised: term 6:

Revised: Sample { ˜ v n } Nn =1 from B with prob.

Revised: 1 − ρ and from p ( v ) with prob.

Revised: ρ 7:

Revised: Calculate E θ ( v ; { v ′ m } Nm =1 ) with Eq.

Revised: Update { ˜ v (2) n } N n =1 with MSGLD in Algo.9:

Revised: Calculate gradient of the gen. term with Eq.

Revised: Update θ via gradient ascent 11:

Revised: Update B with MSGLD samples {

Revised: 12: end while

Revised: C Missing Proofs

Original: C.1 Proof of Theorem2
Revised: C.1 Proof of Theorem

Original: Proof.
Revised: Proof.

Original: We first observe that
Revised: We first observe that

Original: [Equation]
Revised: [Equation]

Original: which proves the relation (12).
Revised: which proves the relation (12).

Original: We also have
Revised: We also have

Original: [Equation]
Revised: [Equation]

Original: and so
Revised: and so

Original: [Equation]
Revised: [Equation]

Original: Taking expectation w.r.t.
Revised: Taking expectation w.r.t.

Original: p ( v ) establishes the relation (13).
Revised: p ( v ) establishes the relation (13).

Original: C.2 Another Theoretical Justification for EBCLR
Revised: C.2 Another Theoretical Justification for EBCLR

Original: Here we provide another theoretical justification on how solving the EBCLR objective (11) with λ = 1 causes 531 q θ ( v, v ′ ) to approximate p ( v, v ′ ) .
Revised: Here we provide another theoretical justification on how solving the EBCLR objective (11) with λ = 1 causes 539 q θ ( v, v ′ ) to approximate p ( v, v ′ ) .

Original: It is known that optimizing the first term of (11) with (18) in place of q θ ( v ′ | v )will cause (18) to approximate p ( v ′ | v ) [34].
Revised: It is known that optimizing the first term of (11) with (18) in place of q θ ( v ′ | v )will cause (18) to approximate p ( v ′ | v ) [37].

Original: Also, optimizing the second term of (11) with (15) or (17) in placeof q θ ( v ) will cause (15) or (17) to be proportional to p ( v ) (indeed, in Table 10, we see SGLD samples of the 534 EBCLR marginal q θ ( v ) approximated by (17) achieve a non-trivial FID score).
Revised: Also, optimizing the second term of (11) with (15) or (17) in placeof q θ ( v ) will cause (15) or (17) to be proportional to p ( v ) (indeed, in Table 10, we see SGLD samples of the 542 EBCLR marginal q θ ( v ) approximated by (17) achieve a non-trivial FID score).

Original: So, the product of (15) or (17) 535 with (18) will be proportional to p ( v, v ′ ) .
Revised: So, the product of (15) or (17) 543 with (18) will be proportional to p ( v, v ′ ) .

Original: Moreover, by construction, the product of (15) or (17) with (18) is(approximately) q θ ( v, v ′ ) .
Revised: Moreover, by construction, the product of (15) or (17) with (18) is(approximately) q θ ( v, v ′ ) .

Original: Hence, optimizing the EBCLR objective (11) will cause q θ ( v, v ′ ) to model p ( v, v ′ ) .
Revised: Hence, optimizing the EBCLR objective (11) will cause q θ ( v, v ′ ) to model p ( v, v ′ ) .

Original: D Complete Training and Evaluation Details
Revised: D Complete Training and Evaluation Details

Original: Baseline methods and datasets.
Revised: Baseline methods and datasets.

Original: The baseline methods are SimCLR, MoCo v2, SimSiam, and BYOL.
Revised: The baseline methods are SimCLR, MoCo v2, SimSiam, and BYOL.

Original: The 540 hyper-parameters are chosen closely following the original works [4, 12, 7, 6].
Revised: The 548 hyper-parameters are chosen closely following the original works [4, 12, 7, 6].

Original: We use four datasets: MNIST[21], Fashion MNIST (FMNIST) [22], CIFAR10, and CIFAR100 [23].
Revised: We use four datasets: MNIST[23], Fashion MNIST (FMNIST) [24], CIFAR10, and CIFAR100 [25].

Original: Data transformation.
Revised: Data transformation.

Original: For fair comparison, we use the stochastic data transformation proposed by Chen et.
Revised: For fair comparison, we use the stochastic data transformation proposed by Chen et.

Original: al [4] 543 for all methods.
Revised: al [4] 551 for all methods.

Original: In the case of grayscale images, we only use the random cropping augmentation.
Revised: In the case of grayscale images, we only use the random cropping augmentation.

Original: For EBCLR,we also add small Gaussian noise N (0 , 0 .
Revised: For EBCLR,we also add small Gaussian noise N (0 , 0 .

Original: 03 2 ) to stabilize training [15, 10, 14].
Revised: 03 2 ) to stabilize training [16, 10, 15].

Original: We remark that Gaussian noiseof standard deviation 0 .
Revised: We remark that Gaussian noiseof standard deviation 0 .

Original: 03 is nearly invisible to the human eye.
Revised: 03 is nearly invisible to the human eye.

Original: DNN architecture.
Revised: DNN architecture.

Original: We decompose f θ = π θ ◦ ϕ θ where ϕ θ is the encoder network and π θ is the projection 547 network.
Revised: We decompose f θ = π θ ◦ ϕ θ where ϕ θ is the encoder network and π θ is the projection 555 network.

Original: Rather than directly using the output of f θ for downstream tasks, we follow previous works [4, 5, 1, 2,3, 6, 7] and use the output of ϕ θ instead.
Revised: Rather than directly using the output of f θ for downstream tasks, we follow previous works [4, 5, 1, 2,3, 6, 7] and use the output of ϕ θ instead.

Original: In our experiments, we set ϕ θ to be a ResNet-18 [24] up to the global average pooling layer.
Revised: In our experiments, we set ϕ θ to be a ResNet-18 [26] up to the global average pooling layer.

Original: The architecture of 550 π θ and π θ to be a 2-layer MLP with output dimension 128 .
Revised: The architecture of 558 π θ and π θ to be a 2-layer MLP with output dimension 128 .

Original: However, we remove batch normalization because 551 batch normalization hurts SGLD [15].
Revised: However, we remove batch normalization because 559 batch normalization hurts SGLD [16].

Original: We also replace ReLU activations with leaky ReLU to expedite theconvergence of SGLD.
Revised: We also replace ReLU activations with leaky ReLU to expedite theconvergence of SGLD.

Original: For the baseline methods, we use the settings proposed in the original works while 553 keeping the backbone fixed to be ResNet-18.
Revised: For the baseline methods, we use the settings proposed in the original works while 561 keeping the backbone fixed to be ResNet-18.

Original: SGLD settings.
Revised: SGLD settings.

Original: The default parameters for MSGLD are |B| =
Revised: The default parameters for MSGLD are |B| =

Original: 50 k , ρ = 0 .
Revised: 50 k , ρ = 0 .

Original: 2 , T = 10 , α = 0 .
Revised: 2 , T = 10 , α = 0 .

Original: 05 , δ = 1 . 0 , 555 K = 3 , σ min = 0 .
Revised: 05 , δ = 1 . 0 , 563 K = 3 , σ min = 0 .

Original: 01 , and σ max = 0 .
Revised: 01 , and σ max = 0 .

Original: We use ρ = 0 .
Revised: We use ρ = 0 .

Original: 4 for MNIST, ρ = 0 .
Revised: 4 for MNIST, ρ = 0 .

Original: 6 for FMNIST, and ρ = 0 . 2 for 556 CIFAR10 and CIFAR100.
Revised: 6 for FMNIST, and ρ = 0 . 2 for 564 CIFAR10 and CIFAR100.

Original: At the beginning of training, we fill the buffer B entirely with proposal distributionsamples.
Revised: At the beginning of training, we fill the buffer B entirely with proposal distributionsamples.

Original: Training.
Revised: Training.

Original: The temperature is τ = 0 .
Revised: The temperature is τ = 0 .

Original: 1 for EBCLR.
Revised: 1 for EBCLR.

Original: For EBCLR, as we cannot use batch normalization, it 559 is difficult to apply SGD with momentum.
Revised: For EBCLR, as we cannot use batch normalization, it 567 is difficult to apply SGD with momentum.

Original: Hence, we use the Adam optimizer [35].
Revised: Hence, we use the Adam optimizer [38].

Original: The learning rate is0 .
Revised: The learning rate is0 .

Original: 0002 if the batch size is 128 and 0 .
Revised: 0002 if the batch size is 128 and 0 .

Original: if the batch size is smaller than 128 .
Revised: if the batch size is smaller than 128 .

Original: We also weakly regularize 561 the squared ℓ 2 -norm of the outputs of f θ with weight 0 .
Revised: We also weakly regularize 569 the squared ℓ 2 -norm of the outputs of f θ with weight 0 .

Original: For the baseline methods, we closely follow the 562 settings of previous works [4, 12, 25].
Revised: For the baseline methods, we closely follow the 570 settings of previous works [4, 12, 27].

Original: Specifically, we use SGD with momentum 0.9 and learning rate of0 .
Revised: Specifically, we use SGD with momentum 0.9 and learning rate of0 .

Original: 03 · ( batch size / 256) and weight decay 0 . 0001 .
Revised: 03 · ( batch size / 256) and weight decay 0 . 0001 .

Original: Each model is trained with a single GeForce RTX 3090.
Revised: Each model is trained with a single GeForce RTX 3090.

Original: Evaluation.
Revised: Evaluation.

Original: We assess the quality of the representations by training a linear classifier on top of frozen ϕ θ .
Revised: We assess the quality of the representations by training a linear classifier on top of frozen ϕ θ .

Original: The 565 linear classifier is trained with Adam for 200 epochs with batch size 512 .
Revised: The 573 linear classifier is trained with Adam for 200 epochs with batch size 512 .

Original: The learning rate η is found by gridsearch for log 10 η in [ − 4 , − 1] .
Revised: The learning rate η is found by gridsearch for log 10 η in [ − 4 , − 1] .

Original: E Additional ExperimentsE.1 Section 4.1 with Longer Training
Revised: E Additional ExperimentsE.1 Section 4.1 with Longer Training

Revised: [Table]

Original: We repeat the experiments in Section 4.1 by training the models for 200 epochs.
Revised: We repeat the experiments in Section 4.1 by training the models for 200 epochs.

Original: For EBCLR, the number of 570 SGLD iterations T is increased by 5 when SGLD is unable to generate samples with sufficiently high energy.
Revised: For EBCLR, the number of 578 SGLD iterations T is increased by 5 when SGLD is unable to generate samples with sufficiently high energy.

Original: Tables 4 and 5 describe the results.
Revised: Tables 4 and 5 describe the results.

Original: We again note that EBCLR consistently beats all baseline methods and 572 demonstrates high efficiency.
Revised: We again note that EBCLR consistently beats all baseline methods and 580 demonstrates high efficiency.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: E.2 Section 4.2 with Longer Training
Revised: E.2 Section 4.2 with Longer Training

Original: We repeat the experiments in Section 4.2 by training the models for 200 epochs.
Revised: We repeat the experiments in Section 4.2 by training the models for 200 epochs.

Original: For EBCLR, the number of 575 SGLD iterations T is increased by 5 when SGLD is unable to generate samples with sufficiently high energy.
Revised: For EBCLR, the number of 583 SGLD iterations T is increased by 5 when SGLD is unable to generate samples with sufficiently high energy.

Original: Table 6 describes the results.
Revised: Table 6 describes the results.

Original: We observe similar patterns as in the case of training for 100 epochs.
Revised: We observe similar patterns as in the case of training for 100 epochs.

Original: In the 577 case of CIFAR10, SimCLR seems to have a pattern agnostic to the batch size.
Revised: In the 585 case of CIFAR10, SimCLR seems to have a pattern agnostic to the batch size.

Original: However, we note that there isa significant gap between the performance of SimCLR with batch size 256 (c.f.
Revised: However, we note that there isa significant gap between the performance of SimCLR with batch size 256 (c.f.

Original: Table 4) and batch sizes in 579 { 16 , 64 , 128 } .
Revised: Table 4) and batch sizes in 587 { 16 , 64 , 128 } .

Original: Interestingly, for SimCLR with small batch sizes, we observe that training longer with small batch sizes leads to 581 worse performance than training only for 100 epochs.
Revised: Interestingly, for SimCLR with small batch sizes, we observe that training longer with small batch sizes leads to 589 worse performance than training only for 100 epochs.

Original: This is particularly prominent on CIFAR10 and CIFAR100(compare Tables 3 and 6).
Revised: This is particularly prominent on CIFAR10 and CIFAR100(compare Tables 3 and 6).

Original: We speculate that this is because, with small batches, SimCLR is unable to provide 583 hard negatives, and this leads to overfitting of the DNN.
Revised: We speculate that this is because, with small batches, SimCLR is unable to provide 591 hard negatives, and this leads to overfitting of the DNN.

Original: On the other hand, EBCLR does not suffer from thisphenomenon.
Revised: On the other hand, EBCLR does not suffer from thisphenomenon.

Original: This again demonstrates the importance of the generative term.
Revised: This again demonstrates the importance of the generative term.

Original: [Table]
Revised: [Table]

Original: EBCLR is calculated by the following formula: (number of epochs used by EBCLR with the same batch size to reach the final accuracy of SimCLR) /
Revised: EBCLR is calculated by the following formula: (number of epochs used by EBCLR with the same batch size to reach the final accuracy of SimCLR) /

Original: (total number of training epochs).
Revised: (total number of training epochs).

Original: E.3 Comparison of Approximations
Revised: E.3 Comparison of Approximations

Original: We can either use Equation (15) or Equation (17) when approximating the generative term.
Revised: We can either use Equation (15) or Equation (17) when approximating the generative term.

Original: Figure 5 shows 587 results for both approximations on CIFAR10.
Revised: Figure 5 shows 595 results for both approximations on CIFAR10.

Original: We observe identical performance for both approximations, whichimplies that the generative term is robust to the number of samples used in the calculation of the empirical mean.
Revised: We observe identical performance for both approximations, whichimplies that the generative term is robust to the number of samples used in the calculation of the empirical mean.

Original: This justifies our choice of using the simpler Equation (17) to approximate the marginal distribution q ( v ) .
Revised: This justifies our choice of using the simpler Equation (17) to approximate the marginal distribution q ( v ) .

Original: E.4 Evaluation with KNN Classifiers
Revised: E.4 Evaluation with KNN Classifiers

Original: We also reproduce Tables 1,
Revised: We also reproduce Tables 1,

Original: [Table]
Revised: [Table]

Original: and 3 using K-nearest neighbour (KNN) classifiers.
Revised: and 3 using K-nearest neighbour (KNN) classifiers.

Original: We followed the evaluation 592 protocol of Caron et al.
Revised: We followed the evaluation 600 protocol of Caron et al.

Original: [36] and used a weighted KNN classifier with K = 20 .
Revised: [39] and used a weighted KNN classifier with K = 20 .

Original: The results are given below.
Revised: The results are given below.

Original: In
Revised: In

Original: Table 7, except for the case of BYOL on MNIST, we see that EBCLR beats all baselines again by a non-trivial 594 margin.
Revised: Table 7, except for the case of BYOL on MNIST, we see that EBCLR beats all baselines again by a non-trivial 602 margin.

Original: EBCLR is also shown to be sample efficient.
Revised: EBCLR is also shown to be sample efficient.

Original: In Table 8, we see EBCLR outperforms all baselineson the task of transfer learning.
Revised: In Table 8, we see EBCLR outperforms all baselineson the task of transfer learning.

Original: In Table 9, we see EBCLR is robust to small batch sizes.
Revised: In Table 9, we see EBCLR is robust to small batch sizes.

Original: This leads to the 596 lower efficiency of SimCLR compared to EBCLR.
Revised: This leads to the 604 lower efficiency of SimCLR compared to EBCLR.

Original: On CIFAR10 and CIFAR100, relative efficiency does notdecrease below 0.05 since we saved checkpoints every 5 epochs.
Revised: On CIFAR10 and CIFAR100, relative efficiency does notdecrease below 0.05 since we saved checkpoints every 5 epochs.

Original: We can reasonably expect SimCLR to have 598 lower relative efficiency at smaller batch sizes if we had saved checkpoints every epoch.
Revised: We can reasonably expect SimCLR to have 606 lower relative efficiency at smaller batch sizes if we had saved checkpoints every epoch.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: E.5 Generative Performance of EBCLR
Revised: E.5 Generative Performance of EBCLR

Original: In this section, we explore the effect of the discriminative loss on the generative performance of EBCLR.
Revised: In this section, we explore the effect of the discriminative loss on the generative performance of EBCLR.

Original: To this 601 end, we solved
Revised: To this 609 end, we solved

Original: [Equation]
Revised: [Equation]

Original: (23) for various values of γ on FMNIST.
Revised: (23) for various values of γ on FMNIST.

Original: Recall that in the original EBCLR objective, weight λ comes in front of 603 the generative term.
Revised: Recall that in the original EBCLR objective, weight λ comes in front of 611 the generative term.

Original: We then measured the Fréchet Inception Distance (FID, lower is better) between samplesfrom the true marginal p ( v ) and SGLD samples from the EBM marginal q θ ( v ) approximated by (19) / (17).
Revised: We then measured the Fréchet Inception Distance (FID, lower is better) between samplesfrom the true marginal p ( v ) and SGLD samples from the EBM marginal q θ ( v ) approximated by (19) / (17).

Original: The 605 results are shown in Table 10.
Revised: The 613 results are shown in Table 10.

Original: Interestingly, we found using an appropriate γ > 0 led to improvements in FID over γ = 0 .
Revised: Interestingly, we found using an appropriate γ > 0 led to improvements in FID over γ = 0 .

Original: In other words, 607 contrastive learning did help improve the generative performance of the model.
Revised: In other words, 615 contrastive learning did help improve the generative performance of the model.

Original: However, using an excessivelylarge γ led to a deterioration of the performance.
Revised: However, using an excessivelylarge γ led to a deterioration of the performance.

Original: This is analogous to the result of Section 4.3 where we observed 609 using an appropriate weight on the generative term led to improvements over using only the discriminativeterm (contrastive loss).
Revised: This is analogous to the result of Section 4.3 where we observed 617 using an appropriate weight on the generative term led to improvements over using only the discriminativeterm (contrastive loss).

Original: The difference is that to achieve optimal generative performance, the generative term 611 weight needs to be larger (i.e., γ < 1 ) whereas to achieve optimal discriminative performance, the discriminativeterm weight needs to be larger (i.e., λ < 1 in (11)).
Revised: The difference is that to achieve optimal generative performance, the generative term 619 weight needs to be larger (i.e., γ < 1 ) whereas to achieve optimal discriminative performance, the discriminativeterm weight needs to be larger (i.e., λ < 1 in (11)).

Original: We speculate that using a larger model could mitigate this 613 trade-off, but this topic is beyond the scope of this work.
Revised: We speculate that using a larger model could mitigate this 621 trade-off, but this topic is beyond the scope of this work.

Original: [Table]
Revised: [Table]

Original: F A Discussion on Making EBCLR Scalable
Revised: F A Discussion on Making EBCLR Scalable

Original: We could consider three alternative losses / methods for training EBCLR: noise contrastive estimation (NCE) 616 [37], VERA [38], and score matching [39].
Revised: We could consider three alternative losses / methods for training EBCLR: noise contrastive estimation (NCE) 624 [40], VERA [41], and score matching [42].

Original: Although the NCE loss does not depend on MCMC, it requiresspecification of a noise distribution whose normalized density can be easily evaluated.
Revised: Although the NCE loss does not depend on MCMC, it requiresspecification of a noise distribution whose normalized density can be easily evaluated.

Original: In most cases, such noise 618 distribution samples are easily distinguishable from natural image samples and lead to the density chasm problem[40], making optimization difficult.
Revised: In most cases, such noise 626 distribution samples are easily distinguishable from natural image samples and lead to the density chasm problem[43], making optimization difficult.

Original: Telescoping density-ratio estimation [40] attempts to mitigate this problem, 

Revised: Telescoping density-ratio estimation [43] attempts to mitigate this problem,

Original: but it does not seem to scale well to large data.
Revised: but it does not seem to scale well to large data.

Original: Flow contrastive estimation [41] uses a flow model to define thenoise distribution and trains a flow model along with the EBM.
Revised: Flow contrastive estimation [44] uses a flow model to define thenoise distribution and trains a flow model along with the EBM.

Original: Though flow contrastive estimation requires an 622 auxiliary flow model, we believe this is the most promising alternative to MCMC.
Revised: Though flow contrastive estimation requires an 630 auxiliary flow model, we believe this is the most promising alternative to MCMC.

Original: In a similar sense, VERA,which uses a generator to sample from the EBM, seems promising as well.
Revised: In a similar sense, VERA,which uses a generator to sample from the EBM, seems promising as well.

Original: Yet, flow contrastive estimation and 624 VERA also did not demonstrate scalability to large-scale datasets such as ImageNet.
Revised: Yet, flow contrastive estimation and 632 VERA also did not demonstrate scalability to large-scale datasets such as ImageNet.

Original: Score matching matchesthe gradient of model log-density to the gradient of data log-density, so it does not rely on MCMC.
Revised: Score matching matchesthe gradient of model log-density to the gradient of data log-density, so it does not rely on MCMC.

Original: However, it 626 requires calculation of the Hessian, so in the deep learning setting, this approach is also quite expensive.
Revised: However, it 634 requires calculation of the Hessian, so in the deep learning setting, this approach is also quite expensive.